888826 - (regression) panda android no longer setting shutdown.stamp for graceful shutdown of buildbot

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Justin Wood (:Callek)]

So we recently changed the android jobs to touch shutdown.stamp in order to graceful themselves between jobs rather than forcibly kill themselves when we see the error.flg.

It seems Panda Android on Mozharness has regressed this by no longer touching the shutdown.stamp file.

It was added in http://hg.mozilla.org/build/buildbotcustom/rev/ac664092d9d8 back in May. Kim is this something you think can be fixed easily?

At the moment there is nothing that will stop a panda job from running even when it won't ping due to this issue.

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: patch

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 770190 [details] [diff] [review]
patch

The shutdown_dir could be specified in query_abs_dirs(), but this works too.

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: patch

thanks aki for the suggestion, that's a cleaner way to do this

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 770318 [details] [diff] [review]
patch

>+                shutdown_file = os.path.join(dirs['shutdown_dir'],'shutdown.stamp')

Nit: You'll probably get a pep8 warning unless you put a space after the comma.

>+                open(shutdown_file, 'w').close()

A couple things:
a) I wish this were logged (open() is silent by default)
b) we may at some point need to bypass this for developer non-buildbot test runs.

Sorry I didn't catch those two things the first time around.
I'm ok with this landing as-is, but we may have to address them in the future.

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: patch

added logging + fixed all the pep8 errors in that script

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 770380 [details] [diff] [review]
patch

Thanks Kim!

Comment 7

[Jonathan Griffin (:jgriffin)]

Comment on attachment 770380 [details] [diff] [review]
patch

in production

Comment 8

[Kim Moir [:kmoir] ET]

Verified that this is working on a recent build on tbpl.