837268 - Enable Ubuntu tests slaves on cedar b2g

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Comment 1

[Andrew Halberstadt [:ahal]]

Reftests (and I think crashtests?) will need to stay on physical machines as they need hardware acceleration, everything else can be moved over.

Comment 2

[Rail Aliiev [:rail]]

Attachment: puppet-manifests

Comment 3

[Rail Aliiev [:rail]]

Attachment: configs

It passes test_masters.sh and generates a sane diff. Still need to look at the dump_master.py output.

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 709403 [details] [diff] [review]
configs

Review of attachment 709403 [details] [diff] [review]:
-----------------------------------------------------------------

dump_masters.sh shows a sane output as well.

::: mozilla-tests/b2g_config.py
@@ +382,5 @@
>          },
> +        'ubuntu32-b2g': {
> +            'opt_unittest_suites': UNITTEST_SUITES['opt_unittest_suites'][:],
> +            'debug_unittest_suites': UNITTEST_SUITES['debug_unittest_suites'][:],
> +            'suite_config': {

This block is a copy of fedora-b2g block above

@@ +929,5 @@
> +# explicitly set slave platforms per branch
> +for branch in BRANCHES.keys():
> +    for platform in BRANCHES[branch]['platforms']:
> +        if 'slave_platforms' not in BRANCHES[branch]['platforms'][platform]:
> +            BRANCHES[branch]['platforms'][platform]['slave_platforms'] = PLATFORMS[platform]['slave_platforms']

If per branch slave_platform is not set, misc.py tries to use platform specific slave_platforms.

@@ +954,2 @@
>  
> +    for k, v in sorted(items.iteritems()):

copy/paste from config.py

Comment 5

[Rail Aliiev [:rail]]

Attachment: configs (no ubuntu32-b2g)

intediff is here: https://gist.github.com/rail/4715081
basically s/ubuntu32-b2g/ubuntu32/g

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 710205 [details] [diff] [review]
configs (no ubuntu32-b2g)

http://hg.mozilla.org/build/buildbot-configs/rev/91c4a7367f38

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 710205 [details] [diff] [review]
configs (no ubuntu32-b2g)

http://hg.mozilla.org/build/buildbot-configs/rev/e3241dfffca5

Hmm, for some reason it didn't pass test-masters.sh...

Comment 8

[Rail Aliiev [:rail]]

It turns out that "-b2g" suffix is needed to generate unique builddirs:

ValueError: builder b2g_ics_armv7a_gecko_emulator_ubuntu32 cedar opt test mochitest-1 reuses builddir cedar_ubuntu32_test-mochitest-1

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 709401 [details] [diff] [review]
puppet-manifests

Reincarnation in progress...

Comment 10

[Chris AtLee [:catlee]]

Comment on attachment 709403 [details] [diff] [review]
configs

Review of attachment 709403 [details] [diff] [review]:
-----------------------------------------------------------------

booooo

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 709401 [details] [diff] [review]
puppet-manifests

http://hg.mozilla.org/build/puppet-manifests/rev/fac2a07117d5

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 709403 [details] [diff] [review]
configs

http://hg.mozilla.org/build/buildbot-configs/rev/c8d9abc885ab

Comment 13

[Rail Aliiev [:rail]]

in production

Comment 14

[Rail Aliiev [:rail]]

Attachment: Rename builders

It turns out that "ubuntu" substring goes to Ubuntu platform on tbpl. It would be better to have more distinguishable name not containing the slave platform. The next patch will add another row to tbpl as well (B2G Arm (VM)).

Comment 15

[Rail Aliiev [:rail]]

Attachment: TBPL changes

Comment 16

[Ed Morley [:emorley]]

Comment on attachment 710675 [details] [diff] [review]
TBPL changes

Presume at some point in the future when we're happy the VM runs are reliable and have switched most things over (other than those in comment 1), we can remove this special-case and just have everything on one row? 

(Similar to how both normal and AWS builds appear on the same style rows on TBPL currently - with the slave name still allowing one to distinguish if needed)

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 710674 [details] [diff] [review]
Rename builders

http://hg.mozilla.org/build/buildbot-configs/rev/4f0e15059ddd

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 710675 [details] [diff] [review]
TBPL changes

http://hg.mozilla.org/webtools/tbpl/rev/c256cae3f5ca

Comment 19

[Justin Wood (:Callek)]

In production

Comment 20

[Andrew Halberstadt [:ahal]]

I noticed that reftests are being run, these shouldn't be and are expected to fail. I don't know why everything else is failing.

We can either:
A) Try running them on the 64 bit pool, this is what I tested and it worked (need to install ia32-libs though)
B) Get a loaner 32 bit VM and try to debug the problems

Unfortunately I have no idea how hard either approach will be.

Comment 21

[Rail Aliiev [:rail]]

I'm a little bit worried about A) approach because it may affect other tests, since we're going to change system libraries (well, add some).

Can we try B), maybe the fix is trivial? We can also try adding more CPU/memory to see if that's the problem.

Comment 22

[Andrew Halberstadt [:ahal]]

Filed bug 840042 for access to a loaner VM

Comment 23

[Andrew Halberstadt [:ahal]]

Attachment: don't run reftests on ubuntu32-b2g in cedar

Due to graphics requirements we shouldn't be running reftests on a VM. This should disable them again on cedar.

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 712925 [details] [diff] [review]
don't run reftests on ubuntu32-b2g in cedar

Sorry for the delay with review. Looks good to me. Thanks!

Comment 25

[Andrew Halberstadt [:ahal]]

Comment on attachment 712925 [details] [diff] [review]
don't run reftests on ubuntu32-b2g in cedar

http://hg.mozilla.org/build/buildbot-configs/rev/86e7ec905b68

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 712925 [details] [diff] [review]
don't run reftests on ubuntu32-b2g in cedar

This patch is in production.

Comment 27

[Rail Aliiev [:rail]]

Attachment: [puppet] install ia32-libs

... if bug 837268 goes smoothly.

Comment 28

[Rail Aliiev [:rail]]

Attachment: puppet-manifests

ubuntu32-b2g will be removed once we switch.

Comment 29

[Rail Aliiev [:rail]]

Attachment: configs

Comment 30

[Chris AtLee [:catlee]]

Comment on attachment 716127 [details] [diff] [review]
configs

Review of attachment 716127 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/BuildSlaves.py.template
@@ +17,5 @@
>      "fedora-b2g": "pass",
>      "panda_android": "pass",
>      "b2g_panda": "pass",
>      "ubuntu32": "pass",
> +    "ubuntu64-b2g": "pass",

I was just thinking that a comment here or somewhere more appropriate about why we need these -b2g variants of the platforms would be great. I keep forgetting why we need them :)

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 716124 [details] [diff] [review]
[puppet] install ia32-libs

Can you add a comment as to why this is needed?  It's not clear to me from reading this bug.

Comment 32

[Rail Aliiev [:rail]]

Comment on attachment 716125 [details] [diff] [review]
puppet-manifests

http://hg.mozilla.org/build/puppet-manifests/rev/99d975bc7697

Comment 33

[Rail Aliiev [:rail]]

Comment on attachment 716124 [details] [diff] [review]
[puppet] install ia32-libs

http://hg.mozilla.org/build/puppet/rev/63a5b3005adb

Comment 34

[Rail Aliiev [:rail]]

Attachment: kill ubuntu32-b2g

Comment 35

[Rail Aliiev [:rail]]

Comment on attachment 716127 [details] [diff] [review]
configs

This was http://hg.mozilla.org/build/buildbot-configs/rev/f0c04d1e7c28

Comment 36

[Rail Aliiev [:rail]]

Comment on attachment 717168 [details] [diff] [review]
kill ubuntu32-b2g

http://hg.mozilla.org/build/puppet-manifests/rev/edbd9b502812