982225 - Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[Phil Ringnalda (:philor)]

Given that we know absolutely for sure that 3-chunk behaves differently than 1-chunk, "on m-i and then other branches" needs to be "and then a few hours later" not even "a few days later."

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

we have tested 3 chunk on cedar, changing this delicate balance will cause fires to burn, earthquakes to strike and RyanVM to lose the brown in his hair.  We need to move to ec2 machines instead of hardware for linux bc tests, and on debug the only realistic way to do that is via chunking.

Comment 3

[Phil Ringnalda (:philor)]

I'm told that the actual plan is to run 3-chunk hidden on m-i while still running 1-chunk visible, and then once 3-chunk shows it's stable and green, switch over to 3-chunk on all trunk trees, which is a fine and sensible plan.

Comment 4

[Armen [:armenzg]]

Attachment: run 3 chunks for debug mochitest-browser-chrome on EC2 machines (side-by-side with Fedora 1-chunk)

Comment 5

[Jordan Lund (:jlund)]

hey there,

I'm not sure if this is it but I think our test masters will fail if we merge to default.

I see this land: http://hg.mozilla.org/build/buildbot-configs/rev/4da229fa0b07

but this differs from https://bugzilla.mozilla.org/attachment.cgi?id=8389378&action=edit

in that it adds 'linux-debug'. checkconfig complains about that key.

Jenkiins should have caught this but it got up and quit on march 6th... :(

Disclaimer: I am probably wrong about something here :)

Comment 6

[Jordan Lund (:jlund)]

I backed it out: http://hg.mozilla.org/build/buildbot-configs/rev/4da229fa0b07

checkconfig works again. apologies if I was wrong in doing so.

Comment 7

[Jordan Lund (:jlund)]

err, sorry I backed it out here: https://hg.mozilla.org/build/buildbot-configs/rev/3a3d88f97064

Comment 8

[John Hopkins (:jhopkins)]

in production

Comment 9

[Armen [:armenzg]]

:(

m-i is perma orange (hidden):
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Ubuntu.*mochitest-browser-chrome-&showall=1
vs:
https://tbpl.mozilla.org/?tree=Cedar&jobname=Ubuntu.*mochitest-browser-chrome-&showall=1

Comment 10

[Armen [:armenzg]]

Attachment: bump bc2 10 more minutes

bc2 & bc3 are looking green on m-i.
bc2 sometimes goes red when it goes over 4200 seconds (successful runs go around 4000 seconds).

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8390659 [details] [diff] [review]
bump bc2 10 more minutes

Review of attachment 8390659 [details] [diff] [review]:
-----------------------------------------------------------------

this looks great.

Comment 12

[Armen [:armenzg]]

Live.

Comment 13

[Phil Ringnalda (:philor)]

Attachment: and another 20 minutes

4800 wasn't enough, we still exceed it. 6000 has the amusing property of being half of what I increased the unchunked maxtime to, since bc2 is really (along with parts of bc3) devtools-chrome, and that's where the slow is.

Comment 14

[Armen [:armenzg]]

Comment on attachment 8392452 [details] [diff] [review]
and another 20 minutes

Review of attachment 8392452 [details] [diff] [review]:
-----------------------------------------------------------------

Thank you philor.

Comment 15

[Phil Ringnalda (:philor)]

Comment on attachment 8392452 [details] [diff] [review]
and another 20 minutes

https://hg.mozilla.org/build/buildbot-configs/rev/029b6eee4fbc

Comment 16

[Armen [:armenzg]]

Live in production.

Comment 17

[Armen [:armenzg]]

Attachment: run 3 chunks for debug mochitest-browser-chrome on EC2 machines (side-by-side with Fedora 1-chunk) on up to Gecko30 branches

Comment 18

[Armen [:armenzg]]

Attachment: builders being added by previous patch

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8395917 [details] [diff] [review]
run 3 chunks for debug mochitest-browser-chrome on EC2 machines (side-by-side with Fedora 1-chunk) on up to Gecko30 branches

Review of attachment 8395917 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1992,5 @@
>  
> +# Enable debug mochitest-browser-chrome on EC2
> +# For branches newer than Gecko 30 (including)
> +for name, branch in items_at_least(BRANCHES, 'gecko_version', 30):
> +    if branch['platforms'].has_key('linux'):

A nit. Can you use the following instead of has_key (PEP8 suggest to avoid using has_key):
if 'linux' in branch['platforms']:
  ....

@@ +1994,5 @@
> +# For branches newer than Gecko 30 (including)
> +for name, branch in items_at_least(BRANCHES, 'gecko_version', 30):
> +    if branch['platforms'].has_key('linux'):
> +        branch['platforms']['linux']['ubuntu32_vm']['debug_unittest_suites'] += MOCHITEST_BC_3[:]
> +    if branch['platforms'].has_key('linux'):

the same here.

Comment 20

[Armen [:armenzg]]

Comment on attachment 8395917 [details] [diff] [review]
run 3 chunks for debug mochitest-browser-chrome on EC2 machines (side-by-side with Fedora 1-chunk) on up to Gecko30 branches

https://hg.mozilla.org/build/buildbot-configs/rev/8fc0876e3c1c

Comment 21

[Armen [:armenzg]]

Running across all trunk and aurora trees:
https://tbpl.mozilla.org/?tree=Fx-Team&rev=f162ae54353d&jobname=debug.*mochitest-browser-chrome

Hiding it on Aurora:
https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=debug.*mochitest-browser-chrome

T-FAIL CRASH (filed as bug 987892):
https://tbpl.mozilla.org/php/getParsedLog.php?id=36674042&tree=Mozilla-Aurora&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=36673963&tree=Mozilla-Aurora&full=1

Comment 22

[[e-mail] (:bhearsum)]

something is in production

Comment 23

[Armen [:armenzg]]

Attachment: enable b2g reftests on EC2 across 1.2, 1.3 & 1.3t

I want to see on what state we're on those branches.

Comment 24

[John Hopkins (:jhopkins)]

in production

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Hidden on b2g28 and b2g26 until they're greened up.

Comment 26

[Armen [:armenzg]]

Attachment: Disable b2g reftests on minis up to m-a

Comment 27

[Armen [:armenzg]]

Attachment: Enable debug mochitest-browser chrome for Ubuntu {x64} for the release branches

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8401447 [details] [diff] [review]
Enable debug mochitest-browser chrome for Ubuntu {x64} for the release branches

Wooooo!

Comment 29

[Armen [:armenzg]]

Comment on attachment 8401447 [details] [diff] [review]
Enable debug mochitest-browser chrome for Ubuntu {x64} for the release branches

http://hg.mozilla.org/build/buildbot-configs/rev/575ccb9b4185

Comment 30

[Armen [:armenzg]]

Live in production.

Comment 31

[Armen [:armenzg]]

Attachment: Disable b2g reftests on the minis

We need to wait until one last uplift happens and some good communications, however, it is not bad to have the patch reviewed already.

Once we're completely done with the minis, a clean up patch will be required to remove the platform completely.

Comment 32

[Armen [:armenzg]]

Attachment: Disabled builders with attachment 8402669

Comment 33

[Armen [:armenzg]]

I'm going crazy with so many bugs.
I will do any follow up patches in bug 864866 (the tracking bug).