892691 - Add HD Panda chassis to android production pandas

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jake Watkins [:dividehex]]

https://mana.mozilla.org/wiki/display/IT/HD+Panda+Chassis+System

The HD Panda Prototype Chassis is alive and kicking in SCL1.  We would like to add the pandas within to the test pool.  (no better proving ground than production imo)
I've run selftests and put the android image on all of them via mozpool.

The pandas are:
panda-{0887-0909}.p10.releng.scl1.mozilla.com

and the associated relay board is panda-relay-080.p10.releng.scl1.mozilla.com

Comment 1

[Jake Watkins [:dividehex]]

This is on hold until after Moz Summit.  Relops will be demoing the Chassis and Mozpool at the Santa Clara fair.

Comment 2

[Jake Watkins [:dividehex]]

I'm removing the HD Chassis for the mozsummit demo.  All pandas in this chassis have been locked out in mozpool.

Comment 3

[Jake Watkins [:dividehex]]

HD Panda Chassis is back and racked in SCL1. (according to bug927068)

Comment 4

[Kim Moir [:kmoir] ET]

I talked to dividehex and these pandas are already in inventory and exist in mozpool

Comment 5

[Pete Moore [:pmoore][:pete]]

Attachment: Puppet patch to add buildbot-master89.srv.releng.scl3.mozilla.com to manifests/moco-nodes.pp

Tested on releng-puppet2.srv.releng.scl3.mozilla.com (environment=pmoore).

Comment 6

[Pete Moore [:pmoore][:pete]]

Attachment: Patch for production-masters.json (added buildbot master 89)

Untested ;)

Comment 7

[Pete Moore [:pmoore][:pete]]

Attachment: bug892691_slavealloc-inserts.sql

The attached SQL should import the new pandas (panda-0887 -> panda-0909) and buildbot-master89.srv.releng.scl3.mozilla.com into the production MySQL database, with pandas in a disabled (not enabled) state.

The slavealloc dbimport mechanism imports a CSV file, and converts it to SQL insert statements, denormalising the normalised fields using the following mapping:

http://mxr.mozilla.org/build/source/tools/lib/python/slavealloc/scripts/dbimport.py?mark=123-137#123

This shows that the non-specified slave fields (locked_masterid, current_masterid, notes, custom_tplid) are not included in the generated insert statements. A look at the live database schema shows that NULLs are allowed for the non-specified fields. Therefore the attached SQL import statements are equivalent to using the dbimport mechanism, since they set these unspecified fields to NULL also.

I'm including this attachment for informational purposes, I will now also attach a CSV file which can be used by slavealloc dbimport tool. However this attachment may be useful for validating the data in the database after the dbimport has run.

Please note the pandas are *not* associated to the new buildbot-master89 at this point (i.e. "locked master" and "current master" are not set on the new pandas).

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: bug892691_data-for-slavealloc-dbimport-tool.csv

Hi Kim,

Here is the data I would like to import into *production* slavealloc database. Can you review my change?

I will import it using [e-mail] by running /data/releng/www/slavealloc/slavealloc dbimport -D <db url including password> --slave-data <this file>

I believe, since I don't specify --enable on the command line, they will be initially disabled.

Will also attach a patch now for adding the new master.

Many thanks!
Pete

Comment 9

[Pete Moore [:pmoore][:pete]]

Attachment: bug892691_data-for-slavealloc-dbimport-tool-master.csv

This is the entry for adding buildbot-master89 to the *production* slavealloc database.

I will import it using [e-mail] by running /data/releng/www/slavealloc/slavealloc dbimport -D <db url including password> --master-data <this file>

Again, without the --enabled, the master should be initially disabled.

Please note I can't set the "notes" field with the initial dbimport import. I can do this afterwards in slavealloc (I would add this bug number into the notes field).

Thanks for your review Kim!

Pete

Comment 10

[Kim Moir [:kmoir] ET]

Comment on attachment 821762 [details]
bug892691_data-for-slavealloc-dbimport-tool.csv

Looks good.  As an aside, I noticed that  pandas don't have cnames yet, it would be good to open a bug with IT to set these up.  

https://wiki.mozilla.org/ReferencePlatforms/HowToSetupNewPlatform#Slavealloc_changes_and_cnames_for_slaves

Comment 11

[Pete Moore [:pmoore][:pete]]

Slavealloc production database updated to include masters and slaves (as per attachments https://bugzilla.mozilla.org/attachment.cgi?id=821762 and https://bugzilla.mozilla.org/attachment.cgi?id=821772).

Comment 12

[Pete Moore [:pmoore][:pete]]

Puppet change applied: https://hg.mozilla.org/build/puppet/rev/d37313946ebe
Tools change applied: https://hg.mozilla.org/build/tools/rev/effb9660a59d

Comment 13

[Pete Moore [:pmoore][:pete]]

Attachment: bug892691_buildbot-configs.patch

Added pandas to buildbot-configs/mozilla-tests/production_config.py

Comment 14

[Pete Moore [:pmoore][:pete]]

Buildbot-configs change applied: https://hg.mozilla.org/build/buildbot-configs/rev/d68aa2da1617

(not merged into production branch)

Comment 15

[John Hopkins (:jhopkins)]

in production

Comment 16

[Pete Moore [:pmoore][:pete]]

Attachment: New panda relay and foopy data in devices.json

A patch for devices.json with all the necessary data about foopies and relays (taken from https://inventory.mozilla.org/en-US/#inventory).

Comment 17

[Pete Moore [:pmoore][:pete]]

I have created all the necessary directories under /builds for each panda, on each of the associated foopies.

Comment 18

[Pete Moore [:pmoore][:pete]]

The distribution of new pandas to foopies is as follows:

Foopy name | Number of pandas   | New pandas added
===========+====================+=========================
foopy102   | 13 (1 new panda)   | panda-0901
foopy103   | 13 (1 new panda)   | panda-0902
foopy104   | 13 (1 new panda)   | panda-0903
foopy105   | 13                 | 
foopy106   | 19                 | 
foopy107   | 14 (6 new pandas)  | panda-0904 -> panda-0909
foopy108   | 14 (14 new pandas) | panda-0887 -> panda-0900

Comment 19

[Justin Wood (:Callek)]

Comment on attachment 827989 [details] [diff] [review]
New panda relay and foopy data in devices.json

Review of attachment 827989 [details] [diff] [review]:
-----------------------------------------------------------------

r-, but rubber-stamp+ once fixed:

We sort devices.json by device name not foopy. So please place them in the list accordingly.

Comment 20

[Pete Moore [:pmoore][:pete]]

Hey Callek,

In that case, I'll supply two patches - the first to tidy up devices.json (to sort by device name, since currently this convention is not strictly followed) and a second patch to add the new pandas.

Pete

Comment 21

[Pete Moore [:pmoore][:pete]]

Attachment: Sort entries in devices.json (cleanup)

This is just a cleanup of the initial devices.json file, before I'll add my changes to it.

Comment 22

[Pete Moore [:pmoore][:pete]]

In case you see a lot of change in that patch, and are scared, I wanted to offer some comforting words to you.

Please note that this diff is considerably smaller if you filter out space changes, and apply a more aggressive diff, e.g. using the "diff" utility (rather than an hg diff).

To demonstrate, I left the old version as devices.json, and put the new version as devices2.json locally on my computer, and then ran:

pmoore@Peters-MacBook-Pro-2:~/hg/tools $ diff -d -b buildfarm/mobile/devices.json buildfarm/mobile/devices2.json
1318a1319
>         "_comment": "decommissioned in bug 786366",
1321,1322c1322
<         "pduid": ".AA2",
<         "_comment": "decommissioned in bug 786366"
---
>         "pduid": ".AA2"
1615,1616c1615,1616
<         "foopy": "None",
<         "_comment": "decommissioned in bug 900507"
---
>         "_comment": "decommissioned in bug 900507",
>         "foopy": "None"
1677a1678,1682
>     "panda-0033": {
>         "foopy": "foopy52",
>         "relayhost": "panda-relay-022.p2.releng.scl1.mozilla.com",
>         "relayid": "1:4"
>     },
1732a1738,1742
>     "panda-0045": {
>         "foopy": "foopy69",
>         "relayhost": "panda-relay-042.p5.releng.scl1.mozilla.com",
>         "relayid": "1:2"
>     },
1787a1798,1802
>     "panda-0057": {
>         "foopy": "foopy77",
>         "relayhost": "panda-relay-051.p6.releng.scl1.mozilla.com",
>         "relayid": "2:1"
>     },
1842a1858,1862
>     "panda-0069": {
>         "foopy": "foopy78",
>         "relayhost": "panda-relay-051.p6.releng.scl1.mozilla.com",
>         "relayid": "2:5"
>     },
1897a1918,1922
>     "panda-0081": {
>         "foopy": "foopy85",
>         "relayhost": "panda-relay-059.p7.releng.scl1.mozilla.com",
>         "relayid": "2:5"
>     },
2738,2742d2762
<     "panda-0033": {
<         "foopy": "foopy52",
<         "relayhost": "panda-relay-022.p2.releng.scl1.mozilla.com",
<         "relayid": "1:4"
<     }, 
3828,3832d3847
<     "panda-0045": {
<         "foopy": "foopy69",
<         "relayhost": "panda-relay-042.p5.releng.scl1.mozilla.com",
<         "relayid": "1:2"
<     }, 
4338,4342d4352
<     "panda-0057": {
<         "foopy": "foopy77",
<         "relayhost": "panda-relay-051.p6.releng.scl1.mozilla.com",
<         "relayid": "2:1"
<     },
4358,4362d4367
<     "panda-0069": {
<         "foopy": "foopy78",
<         "relayhost": "panda-relay-051.p6.releng.scl1.mozilla.com",
<         "relayid": "2:5"
<     },
4793,4797d4797
<     "panda-0081": {
<         "foopy": "foopy85",
<         "relayhost": "panda-relay-059.p7.releng.scl1.mozilla.com",
<         "relayid": "2:5"
<     },
5388,5392d5387
<     "panda-0885": {
<         "foopy": "foopy94",
<         "relayhost": "panda-relay-070.p8.releng.scl1.mozilla.com",
<         "relayid": "2:3"
<     },
5448,5452d5442
<     "panda-0886": {
<         "foopy": "foopy95",
<         "relayhost": "panda-relay-071.p9.releng.scl1.mozilla.com",
<         "relayid": "2:4"
<     },
5906a5897,5906
>     },
>     "panda-0885": {
>         "foopy": "foopy94",
>         "relayhost": "panda-relay-070.p8.releng.scl1.mozilla.com",
>         "relayid": "2:3"
>     },
>     "panda-0886": {
>         "foopy": "foopy95",
>         "relayhost": "panda-relay-071.p9.releng.scl1.mozilla.com",
>         "relayid": "2:4"
5908c5908
<  }
---
> }

So you see, a lot of the actual patch is just about tidying up whitespace etc (e.g. trailing spaces in lines).

Please note, I generated this patch by simply sorting the existing devices.json file using this (very simple) python code:

import simplejson as json
devices = json.load(open('/Users/pmoore/hg/tools/buildfarm/mobile/devices.json', 'r'))
with open('/Users/pmoore/hg/tools/buildfarm/mobile/devices2.json', 'w') as devices2:
    devices2.write(json.dumps(devices, sort_keys=True, indent='    '))

Comment 23

[Pete Moore [:pmoore][:pete]]

Attachment: New panda relay and foopy data in devices.json

As noted above, this patch is to be applied after first sorting/tidying devices.json with previous attachment https://bugzilla.mozilla.org/attachment.cgi?id=828810

Comment 24

[Pete Moore [:pmoore][:pete]]

Both changes committed to devices.json:

https://hg.mozilla.org/build/tools/rev/5c00ae0da6d6
https://hg.mozilla.org/build/tools/rev/5e9a55306338

Comment 25

[Pete Moore [:pmoore][:pete]]

Looks like I never got round to closing this bug, but all parts seem to be completed, so closing it now.