818569 - When installing new gecko, we should stop B2G first

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Jonathan Griffin (:jgriffin)]

According to cjones, some of the b2g process crashes in our logs occur because we are copying a new gecko before shutting B2G down.  We should try changing this and see what happens.

The problem is that when any test fails for any reason, it looks in the log like it's caused by the B2G process crash, even though that crash happened during installing of a new gecko and has nothing to do with the test run.  E.g., https://tbpl.mozilla.org/php/getParsedLog.php?id=17628314&tree=Mozilla-Inbound, which got starred as bug 814551, even though it's a completely separate failure.

One problem with shutting down B2G before copying the files is that it may make our intermittent 'hang while adb pushing' bug more frequent; we'll have to see if that turns out to be a problem.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: try: -b o -p ics_armv7a_gecko -t none -u all --post-to-bugzilla

Pushing this to try to see what happens: https://tbpl.mozilla.org/?tree=Try&rev=8f433ac1fe19

Comment 2

[Jonathan Griffin (:jgriffin)]

On the only job that's finished (R2), this did indeed trigger the adb-push hang.  Our code recovered from it, but if this happens too often we'll get buildbot timeouts.  On the positive side, there aren't any crashes in the log.

I'll try a tweaked version of this.

Comment 3

[Jonathan Griffin (:jgriffin)]

take 2: https://tbpl.mozilla.org/?tree=Try&rev=522bfaeb19ec

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> take 2: https://tbpl.mozilla.org/?tree=Try&rev=522bfaeb19ec

also https://tbpl.mozilla.org/?tree=Cedar&noignore=1&rev=edfd3cedd1a6, since I think cedar has shorter turnaround times than try

Comment 5

[Andrew Halberstadt [:ahal]]

I backed this out of cedar since it seems to consistently break the Mn tests and I was worried it might also be affecting the xpcshell tests:
https://hg.mozilla.org/projects/cedar/rev/4f5570fee38

Comment 6

[Jonathan Griffin (:jgriffin)]

That's really weird.  Unfortunately, I cannot reproduce this locally.

Comment 7

[Jonathan Griffin (:jgriffin)]

I still can't reproduce this.  I'm going to file a bug for an emulator refresh to see if that fixes things.

Comment 8

[Jonathan Griffin (:jgriffin)]

Pushed to try again since we have a new emulator: https://tbpl.mozilla.org/?tree=Try&rev=4e4301a88d29

Comment 9

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #8)
> Pushed to try again since we have a new emulator:
> https://tbpl.mozilla.org/?tree=Try&rev=4e4301a88d29

This was all green except for xpcshell; I've pushed to try again to see if it's consistent:

https://tbpl.mozilla.org/?tree=Try&rev=fe7f6ca531c2

Comment 10

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> (In reply to Jonathan Griffin (:jgriffin) from comment #8)
> > Pushed to try again since we have a new emulator:
> > https://tbpl.mozilla.org/?tree=Try&rev=4e4301a88d29
> 
> This was all green except for xpcshell; I've pushed to try again to see if
> it's consistent:
> 
> https://tbpl.mozilla.org/?tree=Try&rev=fe7f6ca531c2

Same result; this appears to cause xpcshell update tests to fail, which makes no sense to me.  Any ideas why this might happen, Andrew?

Comment 11

[Andrew Halberstadt [:ahal]]

Those tests require a special binary that gets built and added when --enable-updater is added to the mozconfig. Is it possible that this binary exists, then we rmDir /system/b2g and fail to copy it back?

Additionally the subset of tests that are included in xpcshell_updater.ini also require a TestAUSHelper binary (see bug 821343)

Comment 12

[Jonathan Griffin (:jgriffin)]

Attachment: Stop B2G before updating gecko

WIP.  Ugh, I had a stupid mistake that was causing us not to copy any files at all.  I'm glad xpcshell complained

Comment 13

[Jonathan Griffin (:jgriffin)]

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=33db0aa82f14

Comment 14

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #13)
> pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=33db0aa82f14

xpcshell tests are green now. The others haven't shown up yet, so still waiting...

Comment 15

[Jonathan Griffin (:jgriffin)]

Comment on attachment 693627 [details] [diff] [review]
Stop B2G before updating gecko

Review of attachment 693627 [details] [diff] [review]:
-----------------------------------------------------------------

Try run was green this time.

Comment 16

[Andrew Halberstadt [:ahal]]

Comment on attachment 693627 [details] [diff] [review]
Stop B2G before updating gecko

Review of attachment 693627 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM

Comment 17

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/21c0b3fbad22

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/21c0b3fbad22

Comment 19

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/0448e0709981
https://hg.mozilla.org/releases/mozilla-b2g18/rev/7c0800e148d6