820235 - Perform verification of new Linux64 and Linux32 test reference platform on iX node

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris Cooper [:coop] (he/him)]

Like the other new test reference platforms, we need to get the evaluation node hooked up to a dev buildbot master, run the full suite of tests against it, and document the test failure delta from the current ref platform.

Comment 1

[Carsten Book [:Tomcat]]

(In reply to Chris Cooper [:coop] from comment #0)
> Like the other new test reference platforms, we need to get the evaluation
> node hooked up to a dev buildbot master, run the full suite of tests against
> it, and document the test failure delta from the current ref platform.

Hey Coop, do we have that test failure delta data from the current ref platform somewhere?

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Carsten Book [:Tomcat] from comment #1) 
> Hey Coop, do we have that test failure delta data from the current ref
> platform somewhere?

Once the new test machine is ready for verification, it's as simple as pulling down the log from a current test run on mozilla-central and making sure you test the same packaged build on the new machine, i.e. uploading that build to dev-stage so the new machine can grab it. Alternatively, you could run the same build against the both the old ref platform and the new ref platform in staging. That would eliminate any production vs. staging differences.

Not really something we can do in advance unless you preserve the logs & build...best to wait until the new machine is setup to minimize result drift.

Comment 3

[Carsten Book [:Tomcat]]

(In reply to Chris Cooper [:coop] from comment #2)

> Not really something we can do in advance unless you preserve the logs &
> build...best to wait until the new machine is setup to minimize result drift.

thanks! maschine is up and working on the puppet stuff, will update the bug as i get news/results

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Tomcat, are you working on getting this to use X11 on the external graphics card?  I can take a whack at it if you're not.  That should probably be in a sub-bug anyway, since this bug is about evaluating the result.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

So bug 838351 is for the graphics and audio, which I and/or someone else from IT will take care of.  Tomcat, do you know of anything else that will need to be done before we put these into production?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Tomcat, I'm going to grab ix-mn-linux64-001 for this purpose.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug820235.patch

As I had mentioned in irc, the disableservices line is disabling services which aren't even installed in the Base group.  Puppet fails to stop a service it doesn't know about, so instead this just uninstalls them where they're installed.

This also adds support for ipmitools on Ubuntu, since it's required on this hardware.

Graphics work will be in a patch on bug 838351.

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 710451 [details] [diff] [review]
bug820235.patch

s/modem-manager/modemmanager/
Otherwise looks good.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks for checking - I fixed that but I think I forgot to push the diff update.

Comment 10

[Carsten Book [:Tomcat]]

Comment on attachment 710451 [details] [diff] [review]
bug820235.patch

r+ also see the notes from rail :)

Comment 11

[Carsten Book [:Tomcat]]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> So bug 838351 is for the graphics and audio, which I and/or someone else
> from IT will take care of.  Tomcat, do you know of anything else that will
> need to be done before we put these into production?

no so far i don't know anything else, thanks for checking, will update when i have somehting

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 710451 [details] [diff] [review]
bug820235.patch

Per IRC conversations with Rail, this needs a bit more (installing ubuntu-desktop) and some changes to disableservices (install, but disable, anacron, since it's required for ubuntu-desktop)

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug820235-r2.patch

This should do the trick..

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 710788 [details] [diff] [review]
bug820235-r2.patch

Review of attachment 710788 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Close!  I landed this to avoid including linux_desktop on Darwin:

diff --git a/modules/toplevel/manifests/slave/test.pp b/modules/toplevel/manifests/slave/test.pp
index 4416dbd..ab1e9aa 100644
--- a/modules/toplevel/manifests/slave/test.pp
+++ b/modules/toplevel/manifests/slave/test.pp
@@ -6,11 +6,15 @@ class toplevel::slave::test inherits toplevel::slave {
     # so we get the GUI for free and just need to ensure VNC is enabled.
     include vnc
     include screenresolution::talos
-    include packages::linux_desktop
     include users::builder::autologin
     include talos
     include ntp::atboot
     include packages::fonts
     include tweaks::fonts
     include tweaks::cleanup
+
+    # this will get fixed in a subsequent patch for bug 838351
+    if ($::operatingsystem == 'Ubuntu') {
+        include packages::linux_desktop
+    }
 }

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

I think this is ready to go, and in fact may already be done.  Rail?

Comment 17

[Rail Aliiev [:rail]]

We still need to hookup these machines to one of the non-staging branches and run them in parallel with fedora slaves. This would require some changes in builbot-configs (evil loops, of course) and probably a person from a-team to look at the possible failures.

We attached some of the machines to my staging master, replacing existing fedora slaves, but evaluating results without TBPL is hard.

Comment 18

[Chris Cooper [:coop] (he/him)]

(In reply to Rail Aliiev [:rail] from comment #17)
> We attached some of the machines to my staging master, replacing existing
> fedora slaves, but evaluating results without TBPL is hard.

Is cedar being specifically used for Win8 or could we also hook up the Fedora slaves there and just have the Windows guys ignore the Linux results and vice versa? Seems like the best use of existing resources to me.

Comment 19

[Rail Aliiev [:rail]]

Yeah, Cedar is my favorite too. :)

Comment 20

[Rail Aliiev [:rail]]

Attachment: sekrets

Comment 21

[Rail Aliiev [:rail]]

Attachment: buildapi

Comment 22

[Rail Aliiev [:rail]]

Attachment: buildbotcustom

* use talos_slave_platforms by default and slave_platforms as fallback

Comment 23

[Rail Aliiev [:rail]]

Attachment: configs

It generates sane diffs:

config.py: https://gist.github.com/rail/5240897
builders: https://gist.github.com/rail/5240891

Comment 24

[Rail Aliiev [:rail]]

Attachment: configs

the only difference is number of slaves (100 vs 50) in production_config.py

Comment 25

[Chris AtLee [:catlee]]

Comment on attachment 729232 [details] [diff] [review]
configs

Review of attachment 729232 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/BuildSlaves.py.template
@@ +25,5 @@
>      "ubuntu64_vm-b2g": "pass",
>      "ubuntu64_vm": "pass",
> +    "ubuntu32_hw": "pass",
> +    "ubuntu64_hw-b2g": "pass",
> +    "ubuntu64_hw": "pass",

nit: can you sort these platforms? maybe group all the ubuntu32 variants together, and then all the ubuntu64 variants together?

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 729232 [details] [diff] [review]
configs

http://hg.mozilla.org/build/puppet-manifests/rev/b4887372734d
with platforms sorted

Comment 27

[Rail Aliiev [:rail]]

Comment on attachment 729207 [details] [diff] [review]
buildapi

http://hg.mozilla.org/build/buildapi/rev/d6c6f0fe65b1

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 729211 [details] [diff] [review]
buildbotcustom

http://hg.mozilla.org/build/buildbotcustom/rev/1b90a741fdd9

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 729232 [details] [diff] [review]
configs

http://hg.mozilla.org/build/buildbot-configs/rev/fa82bb34dd25 actually

Comment 30

[Rail Aliiev [:rail]]

Back out: http://hg.mozilla.org/build/buildbot-configs/rev/c5631ad322a0


INFO  - created  "bm18-tests1-linux" master, running checkconfig
INFO  - starting to print log file '/builds/buildbot/preproduction/slave/test-masters/buildbot-configs/test-output/bm18-tests1-linux-jp7wI4-checkconfig.log'
INFO  - /builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
INFO  -   import MimeWriter, tempfile, rfc822
INFO  - Traceback (most recent call last):
INFO  -   File "/builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
INFO  -     ConfigLoader(configFileName=configFileName)
INFO  -   File "/builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
INFO  -   File "/builds/buildbot/preproduction/slave/test-masters/sandbox/lib/python2.6/site-packages/buildbot-0.8.2_hg_41fc8a9db7a0_production_0.8-py2.6.egg/buildbot/master.py", line 808, in loadConfig
INFO  -     % (b['name'], n))
INFO  - ValueError: builder Ubuntu HW 12.04 x64 cedar talos svgr uses undefined slave talos-linux64-ix-001
INFO  - finished printing log file '/builds/buildbot/preproduction/slave/test-masters/buildbot-configs/test-output/bm18-tests1-linux-jp7wI4-checkconfig.log'
ERROR - TEST-FAIL bm18-tests1-linux failed to run checkconfig


Hmmm. It worked fine on my <s>laptop</s> dev-master...

Comment 31

[Rail Aliiev [:rail]]

Attachment: configs v2

Hmmm, it turns out that running dump_master.py and builder_list.py doesn't check the configs...

A trivial interdiff: https://gist.github.com/rail/5248201

It passes test-master.sh now and has the same diff of list of builders.

Comment 32

[Rail Aliiev [:rail]]

Comment on attachment 729700 [details] [diff] [review]
configs v2

http://hg.mozilla.org/build/buildbot-configs/rev/dfb9ba1d7083

Comment 33

[Chris AtLee [:catlee]]

random aside - can we make the hostnames consistent between HW and VMs? We currently have

tst-linux32-ec2-XXX
talos-linux32-ix-YYY

Does it still make sense to have 'talos' in the hostname?

Comment 34

[Rail Aliiev [:rail]]

b2g emulator tests are enabled on cedar in parallel with the old ones

Comment 35

[Rail Aliiev [:rail]]

(In reply to Chris AtLee [:catlee] from comment #33)
> random aside - can we make the hostnames consistent between HW and VMs? We
> currently have
> 
> tst-linux32-ec2-XXX
> talos-linux32-ix-YYY
> 
> Does it still make sense to have 'talos' in the hostname?

Yeah, I think this will be better.

Ami, how long it may take to rename the slaves?

From our side this will require changes in puppet patterns and buildbot configs.

Comment 36

[Amy Rich [:arr] [:arich]]

So this is a significant couple of days worth of work to reimage all of the linux hosts we already have.  We would also need to change dns, inventory, and nagios.  We also would need to change all of inventory and dns for the windows xp and w7 hosts as well since the hardware is on ordered and all of these things have already been pre-populated.

I thought that talos was still a useful designator, because the whole reason we have physical hardware (vs the AWS machines) was because we *needed* it for talos (and graphics) tests (eg AWS can't do talos).  So to me it makes sense to still differentiate.

Comment 37

[Armen [:armenzg]]

(In reply to Amy Rich [:arich] [:arr] from comment #36)
> So this is a significant couple of days worth of work to reimage all of the
> linux hosts we already have.  We would also need to change dns, inventory,
> and nagios.  We also would need to change all of inventory and dns for the
> windows xp and w7 hosts as well since the hardware is on ordered and all of
> these things have already been pre-populated.
> 
> I thought that talos was still a useful designator, because the whole reason
> we have physical hardware (vs the AWS machines) was because we *needed* it
> for talos (and graphics) tests (eg AWS can't do talos).  So to me it makes
> sense to still differentiate.

This probably got clarified somewhere else but in-house test machines will be doing mainly two types of jobs:
* unit tests that need graphic cards support
* talos jobs

We also established a hostname naming convention that is going through review by Relops and DCops.

Comment 38

[Rail Aliiev [:rail]]

ATM this platform can be used for Talos without any problems. B2G emulator failures are tracked by bug 850105.

Comment 39

[Rail Aliiev [:rail]]

BTW, (almost) everything here applies to 32-bit platform as well. Updating the summary accordingly.