719511 - scriptfactory doesn't reboot slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[cmtalbert]

This is causing us issues with peptest in try.  Essentially something doesn't get cleaned up due to an error then because the slave never rebooted when the slave goes on to run another job, it is in an unexpected state causing other tests to fail.

See bug 719504 for the resulting mayhem.

Comment 1

[[e-mail] (:bhearsum)]

Aki, ISTR that you were looking at a patch for this. Throw this bug if that's wrong or you can't work on it.

Comment 2

[Aki Sasaki (not active)]

ScriptFactory is a buildbot factory, and is not part of mozharness.  Fixing summary.
However, this still needs to be fixed for ScriptFactory to run viable mozharness-based tests.

Comment 3

[Aki Sasaki (not active)]

Attachment: add an optional reboot_command to ScriptFactory

I'm not hardcoding the path or the options because ScriptFactory currently runs both mozharness and build/tools scripts.

I suppose I could get around that by checking in count_and_reboot.py into mozharness; this seems like a decent alternative.

Comment 4

[Aki Sasaki (not active)]

Attachment: call reboot() =P

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 594827 [details] [diff] [review]
call reboot() =P

http://hg.mozilla.org/build/buildbotcustom/rev/a14af12e9545

Now I need to add a build/tools checkout to peptest, and add the reboot_command to the builders.

Comment 6

[Armen [:armenzg]]

This is live now as of 1300 PST.

Comment 7

[Armen [:armenzg]]

Backed out from http://hg.mozilla.org/build/buildbotcustom/rev/6b81b41b58b3

Can you please back it out from default or land a fix?

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 594827 [details] [diff] [review]
call reboot() =P

I failed to set self.reboot_command.
I might as well test the whole thing before the next r?

Comment 9

[Aki Sasaki (not active)]

Attachment: (custom) add optional reboot command to ScriptFactory

I was able to get this, combined with the following two patches, to reboot a Fedora box after running Peptest.

Comment 10

[Aki Sasaki (not active)]

Attachment: (configs) reenable peptest; add count_and_reboot.py reboot_command to peptest

Comment 11

[Aki Sasaki (not active)]

Attachment: (mozharness) peptest default_actions and repos changes

This patch does two things:

1) changes the default actions for peptest to be more developer friendly (calls get-latest-tinderbox by default).  It then changes the default_actions for the production configs to switch out get-latest-tinderbox for read-buildbot-config.

2) adds the build/tools repo to all production configs, and checks out that repo if 'repos' is defined in self.config.  This is so we have count_and_reboot.py on disk.

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 596713 [details] [diff] [review]
(configs) reenable peptest; add count_and_reboot.py reboot_command to peptest

http://hg.mozilla.org/build/buildbot-configs/rev/8107a795bf38

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 596712 [details] [diff] [review]
(custom) add optional reboot command to ScriptFactory

http://hg.mozilla.org/build/buildbotcustom/rev/cc74a82b64a3

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 596714 [details] [diff] [review]
(mozharness) peptest default_actions and repos changes

http://hg.mozilla.org/build/mozharness/rev/18f1775c7c59

Peptest should be reenabled, with ScriptFactory rebooting, on the next reconfig (Thursday? I may take this tomorrow during 11.0b3 as well.)

Comment 15

[Mark Côté [:mcote]]

Thursday would be good. I have another patch to land to ensure that peptest doesn't try to use the network. Also there are some more command-line options to disable network access--do you need to modify try/mozharness for these?

Comment 16

[Aki Sasaki (not active)]

(In reply to Mark Côté ( :mcote ) from comment #15)
> Thursday would be good. I have another patch to land to ensure that peptest
> doesn't try to use the network.

I think that peptest hitting the network makes peptest orange; not rebooting made other tests orange.
We should be able to reenable and hide before you're ready, but sounds like I can leave the reconfig for this for Thursday without blocking you guys.

> Also there are some more command-line
> options to disable network access--do you need to modify try/mozharness for
> these?

I think we'll need to update the command line generation logic here:
http://hg.mozilla.org/build/mozharness/file/18f1775c7c59/scripts/peptest.py#l314

Comment 17

[Mark Côté [:mcote]]

Yup, you will need to add these options:

--proxy with the location to the server-locations.txt file (peptest/tests/firefox/server-locations.txt)
--proxy-host-dirs
--server-path with the location of the root server dir (tests/peptest/tests/firefox/server)

That will proxy network requests to local files. Let me know if you have any questions.

Comment 18

[Aki Sasaki (not active)]

Oops, reopening for comment 17.

Comment 19

[Aki Sasaki (not active)]

Attachment: add proxy options to peptest

With this patch, we get a command line of

12:09:43     INFO - Running command: ['/home/cltbld/talos-slave/test/build/venv/bin/python', '-u', '/home/cltbld/talos-slave/test/build/tests/peptest/peptest/runpeptests.py', '--app', 'firefox', '--binary', '/home/cltbld/talos-slave/test/build/application/firefox/firefox', '--test-path', '/home/cltbld/talos-slave/test/build/tests/peptest/tests/firefox/firefox_all.ini', '--timeout', '60', '--tracer-threshold', '50', '--tracer-interval', '10', '--proxy', '/home/cltbld/talos-slave/test/build/tests/peptest/tests/firefox/server-locations.txt', '--proxy-host-dirs', '--server-path', '/home/cltbld/talos-slave/test/build/tests/peptest/tests/firefox/server', '--log-level', 'INFO']

Comment 20

[Mark Côté [:mcote]]

Comment on attachment 599308 [details] [diff] [review]
add proxy options to peptest

Review of attachment 599308 [details] [diff] [review]:
-----------------------------------------------------------------

New command line looks good.

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 599308 [details] [diff] [review]
add proxy options to peptest

http://hg.mozilla.org/build/mozharness/rev/f230e358cedc

I think all that's left is outside the scope of this bug: adjust peptests to go green, add new peptests.