942299 - Disable Mac OS X 10.7 testing automation and repurpose it as 10.6

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Armen [:armenzg]]

I hope to have the 10.7 testing infrastructure disabled by the end of Tuesday.
I will file a Relops bug on Monday/Tuesday with all the details.

Comment 1

[Amy Rich [:arr] [:arich]]

Ccing uberj so he sees the dep bugs as they roll in, too.

Comment 2

[Armen [:armenzg]]

Attachment: remove 10.7 testing across the board

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 8338564 [details] [diff] [review]
remove 10.7 testing across the board

Review of attachment 8338564 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. Might be good to sanity check with a dump-master or builder_list run.

Comment 4

[Armen [:armenzg]]

Comment on attachment 8338564 [details] [diff] [review]
remove 10.7 testing across the board

https://hg.mozilla.org/build/buildbot-configs/rev/1f56e6dc01e1

Comment 5

[Armen [:armenzg]]

Attachment: add the 10.7 machines as 10.6 machines

Comment 6

[Armen [:armenzg]]

Attachment: remove talos-r4-lion machines from slavealloc

Comment 7

[Armen [:armenzg]]

Attachment: add 10.6 machines to slavealloc

This should be included with something like this:
/data/releng/www/slavealloc/slavealloc dbimport -D <db url including password> --slave-data <the csv file you just created containing slaves>

Comment 8

[Armen [:armenzg]]

Attachment: remove 10.7 from trychooser

Comment 9

[Armen [:armenzg]]

Attachment: remove 10.7 from buildfaster report

Comment 10

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8338775 [details] [diff] [review]
remove 10.7 from buildfaster report

Review of attachment 8338775 [details] [diff] [review]:
-----------------------------------------------------------------

We may want to wait a day or two before landing this so that 10.7 test jobs from today get parsed correctly.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8338753 [details] [diff] [review]
remove talos-r4-lion machines from slavealloc

This DB is case sensitive for table names, it seems. The where clause is fine but make sure you adjust the table name before you run it:
mysql> select name FROM SLAVES WHERE NAME LIKE 'talos-r4-lion-%';
ERROR 1146 (42S02): Table 'buildslaves.SLAVES' doesn't exist
mysql> select name FROM slaves WHERE NAME LIKE 'talos-r4-lion-%';
+-------------------+
| name              |
+-------------------+
| talos-r4-lion-001 |

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 8338744 [details] [diff] [review]
add the 10.7 machines as 10.6 machines

Per IRC, this should only go up to 173. r=me with that changed.

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 8338765 [details]
add 10.6 machines to slavealloc

Per IRC, this should only go up to 173. r=me with that changed.

Comment 14

[Armen [:armenzg]]

Comment on attachment 8338744 [details] [diff] [review]
add the 10.7 machines as 10.6 machines

Landed as:
-    'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(1,85) \
+    'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(1,174) \

Comment 15

[Armen [:armenzg]]

Comment on attachment 8338753 [details] [diff] [review]
remove talos-r4-lion machines from slavealloc

Deployed.

mysql> DELETE FROM slaves WHERE NAME LIKE 'talos-r4-lion-%';
Query OK, 89 rows affected (0.02 sec)

Comment 16

[Armen [:armenzg]]

Attachment: add the 10.6 machines to the graph server

Here are the values that are used:

mysql> select * from machines where name like 'talos-r4-snow-%' limit 1;
+------+-------+---------------+-----------+-------------------+-----------+------------+
| id   | os_id | is_throttling | cpu_speed | name              | is_active | date_added |
+------+-------+---------------+-----------+-------------------+-----------+------------+
| 1455 |    21 |             0 | 2.4       | talos-r4-snow-001 |         1 | 1317830270 |
+------+-------+---------------+-----------+-------------------+-----------+------------+
1 row in set (0.00 sec)

Comment 17

[Armen [:armenzg]]

Comment on attachment 8338765 [details]
add 10.6 machines to slavealloc

Deployed without 174 and 175.

Comment 18

[Armen [:armenzg]]

Comment on attachment 8339311 [details] [diff] [review]
add the 10.6 machines to the graph server

Too my surprise, somebody had already these entries in 2011 (10/05/11 @ 3:57:50pm UTC) up to machine 183.
Both staging and production graphs are up-to-date.

Comment 19

[Armen [:armenzg]]

I've marked all slaves to be added to my staging master:
update slaves set enabled=1, envid=3, poolid=16, locked_masterid=133 where enabled=0 and notes is NULL and name like 'talos-r4-snow-%';

mysql> select count(*) from slaves where locked_masterid=133 and name like 'talos-r4-snow-%';
+----------+
| count(*) |
+----------+
|       89 |
+----------+
1 row in set (0.00 sec)

Comment 20

[John Hopkins (:jhopkins)]

in production

Comment 21

[Armen [:armenzg]]

I added all of the slaves that have had green runs.

I've left the following on staging:
* talos-r4-snow-130 - running red on staging
* talos-r4-snow-102 - ping down; I triggered a slaveapi reboot
* talos-r4-snow-103 - ping down; I triggered a slaveapi reboot
* talos-r4-snow-104 - ping down; I triggered a slaveapi reboot
* talos-r4-snow-144 - it is odd; the ping does not timeout; slaveapi reboot takes long to respond
* talos-r4-snow-166 - I can ping it but I can't ssh; slaveapi reboot
* talos-r4-snow-170 - it has not taken a job; uptime 13hours; sudo reboot

Comment 22

[Armen [:armenzg]]

Attachment: final adjustment

Comment 23

[Marco Castelluccio [:marco]]

The 10.7 machines were the only machines that allowed me to test apps installations/updates (because on 10.6 and 10.8 machines we don't have enough privileges to write to /Applications).
Could you prioritize bug 942445 now that 10.7 machines are gone?

Comment 24

[Armen [:armenzg]]

(In reply to Marco Castelluccio [:marco] from comment #23)
> The 10.7 machines were the only machines that allowed me to test apps
> installations/updates (because on 10.6 and 10.8 machines we don't have
> enough privileges to write to /Applications).
> Could you prioritize bug 942445 now that 10.7 machines are gone?

Let me follow up on it.

Comment 25

[Armen [:armenzg]]

Machines put in production:
* talos-r4-snow-102
* talos-r4-snow-104
* talos-r4-snow-130

Machines left behind:
* talos-r4-snow-103
* talos-r4-snow-166
* talos-r4-snow-170

Being dealt with on problem tracking bugs:
* talos-r4-snow-144

Comment 26

[Armen [:armenzg]]

I've filed the following:
https://bugzilla.mozilla.org/show_bug.cgi?id=talos-r4-snow-103
https://bugzilla.mozilla.org/show_bug.cgi?id=talos-r4-snow-144
https://bugzilla.mozilla.org/show_bug.cgi?id=talos-r4-snow-166
https://bugzilla.mozilla.org/show_bug.cgi?id=talos-r4-snow-170

Comment 27

[Armen [:armenzg]]

coop, I'm not going to land this.
I believe it has value to leave it around in case someone wants to look at historic data in gofaster.

WFY?

(In reply to Chris Cooper [:coop] from comment #10)
> Comment on attachment 8338775 [details] [diff] [review]
> remove 10.7 from buildfaster report
> 
> Review of attachment 8338775 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> We may want to wait a day or two before landing this so that 10.7 test jobs
> from today get parsed correctly.

Comment 28

[[e-mail] (:bhearsum)]

something here is in production

Comment 29

[Armen [:armenzg]]

coop, if you disagree with what I say please let me know and re-open the bug.

Comment 30

[Chris Cooper [:coop] (he/him)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #27)
> coop, I'm not going to land this.
> I believe it has value to leave it around in case someone wants to look at
> historic data in gofaster.
> 
> WFY?

Sure.