Closed Bug 914302 Opened 11 years ago Closed 11 years ago

Upgrade production tegras and pandas to Watcher 1.16

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: pmoore, Assigned: pmoore)

References

Details

There is a new version of Watcher (1.16) which should be applied to all production tegras. This has already been applied successfully to all staging devices (see https://bugzilla.mozilla.org/show_bug.cgi?id=876715).

The plan is to deploy this to all production tegras and pandas on Friday 13 (!!) September, in the following period (stated in multiple timezones, for convenience):

    00:30 - 04:30 PDT
    03:30 - 07:30 EDT
    09:30 - 13:30 CEST
    19:30 - 23:30 NZST

If any major issues are discovered, pandas can be reimaged via Mozpool, and tegras can be reimaged using the imaging notebook.

In addition, we will be providing a new image for tegras which contains the updated Watcher (1.16) and the new SUT Agent (1.19). Assuming no major issues are found after go-live, these new images can replace the existing images we have, so that future tegra-reimaging will use the new versions of the tools.
Some requests:

Sheriffs: please can you confirm that you are ok with this deployment window?

All: My current strategy is to have a complete outage during this time for mobile testing - however, if required, I can deploy in 2 batches of masters, so that half the tegras and pandas are available while the other half are being updated, and vice versa. Please let me know if this is required. This would then not require a complete outage, but we would have a longer period of low capacity.

Callek/kmoir: can you review my description and check you agree with everything I have said.

Callek: can you please confirm how the rollback procedure should be triggered for tegras, and let me know if I need to make special arrangements that the notebook is available, powered on, and connected to the network, at this time, in case nobody is physically present at the time the rollback needs to take place?

Callek, could you also provide the bug # of the ticket you raised for getting a new panda image containing the new watcher and sut agent?

Thanks all!
Flags: needinfo?(ryanvm)
Flags: needinfo?(philringnalda)
Flags: needinfo?(kwierso)
Flags: needinfo?(kmoir)
Flags: needinfo?(emorley)
Flags: needinfo?(cbook)
Flags: needinfo?(bugspam.Callek)
I'll defer the sheriff part of this to Ed/Tomcat since the window falls during the bulk of their work day.
Flags: needinfo?(ryanvm)
I'm even further from Ed/Tomcat's timezone than Ryan.
Flags: needinfo?(kwierso)
lgtm

As an aside, the mobile test jobs will just queue and then be served when the buildbot masters are up again.  Just to clarify that the the nuance behind the word outage :-)
Flags: needinfo?(kmoir)
(In reply to Pete Moore [:pete][:pmoore] from comment #1)
> Callek/kmoir: can you review my description and check you agree with
> everything I have said.

I don't see it articulated, but part of the process should be to have someone (relops, or you/us if they tell us how) update the image in mozpool that is "android panda" so that requests from buildbot get the new image, for any that get reimaged after a self-test, etc. (so we don't accidentally downgrade). This should be easy once we test the image update (which is still pending)

> Callek: can you please confirm how the rollback procedure should be
> triggered for tegras, and let me know if I need to make special arrangements
> that the notebook is available, powered on, and connected to the network, at
> this time, in case nobody is physically present at the time the rollback
> needs to take place?

For tegras, I would say the rollback, would be to:
(a) Manually Deploy old watcher apk to all reachable devices. (risky, but better than 0 capacity)
(b) ask DCOps to reimage ALL tegras with "known good" image in 2 or 3 batches, where we have reduced capacity for each batch.

I don't expect rollback to be easy here, but I also don't forsee a reasonable way to mitigate that risk/pain.

For pandas rollback would be to:
(a) graceful panda masters
(b) swap default "android panda" image back to old image
(c) manually reimage all android pandas
(d) re-enable panda masters

> Callek, could you also provide the bug # of the ticket you raised for
> getting a new panda image containing the new watcher and sut agent?

Bug 914472
Flags: needinfo?(bugspam.Callek)
If you mean "we're going to close all trees that do mobile, except maybe Try," then that sounds fine to me. Comment 4 could be interpreted as meaning "since jobs will just queue up, no need to close trees," which would only be sort of vaguely true if we didn't coalesce jobs. It may also result in an interesting situation on Try, since we typically carry an 8-10 hour backlog of Tegra jobs into that time of day, and catch up then, so if we don't catch up then... interesting.
Flags: needinfo?(philringnalda)
Yeah this will need a tree closure, unless we do them in 2-3 batches at a quiet time, IMO.
Flags: needinfo?(emorley)
Depends on: 914841
Depends on: 876715
Depends on: 914849
All, this requires that Relops be involved and a CAB review and a communication out to everyon@mozilla.com if this has the risk of taking a tree down.  Please have those planned out for the next week CAB changes before this is implemented.
(In reply to SylvieV from comment #8)

Just to make sure the right group gets invited to the table, this requires dcops for the worst case recovery scenario, not relops.
per meeting w/pmoore, coop, callek, joduinn:

1) We're cancelling tomorrow's planned upgrade work, because this time would be during the Oslo-based b2g workweek. We expect early next week to be much lower load, as everyone travels back from the b2g workweek. 

2) The nature of this rollout can be done in chunks, which avoids the need for any tree closure / downtime. If this could only be done as one-big-bang-treeclosure, it should have gone through CAB review, like all other tree closures. However, doing in chunks like this will take physically longer, but is worth doing a) avoids tree closures and b) is safer in case of any need to rollback.

3) Here is the revised plan:
* On each of Monday / Tues / Wed / Thurs, during 09:30-13:30 CET, pmoore will be taking down one of the 4 tegra-masters per day, and upgrading the foopies+tegras attached to that master.
* Based on earlier discussions with sheriffs, we believe doing this upgrade in 4 chunks, during early-morning low load times means we do *not* need a tree closure for this. Edmorley/ryanvm, please confirm?
* Doing the upgrade on Pandas will be different, because of MozPool, and does not need to be 
at same time as the above tegra work. More details once we see how the tegra rollout goes.
Flags: needinfo?(ryanvm)
Flags: needinfo?(emorley)
Given the times being discussed, I'll defer back to comment 2 :)
Flags: needinfo?(ryanvm)
wfm
Flags: needinfo?(emorley)
ryamvm, ed: glad that proposal works for you both, and avoids a tree closure!

dmoore: I know there were various in-person meetings about this yesterday, and I believe this is works for you... but I've I just noticed you're not cc'd, so adding you.
Latest plan can be seen here: http://petemoore.tumblr.com/post/61010673496/runbook-for-tegra-watcher-rollout-16-19-sep-2013

Here is a copy for convenience:


Schedule

Mon 16 Sep 2013
http://buildbot-master10.build.mtv1.mozilla.com:8201/

   * foopy28 - all devices
   * foopy29 - all devices *except* tegra-342
   * foopy30 - all devices *except* tegra-349
   * foopy31 - all devices
   * foopy32 - all devices


Tue 17 Sep 2013
http://buildbot-master19.build.mtv1.mozilla.com:8201/

Wed 18 Sep 2013
http://buildbot-master20.build.mtv1.mozilla.com:8201/

Thu 19 Sep 2013
http://buildbot-master22.build.mtv1.mozilla.com:8201/


Steps

   * Inform sheriffs via #releng that buildbot master will be shutdown
   * Send email to confirm that buildbot master will be shutdown (release@mozilla.com)
   * Update bug 914302 to say deployment is starting for this master
   * Click on “Clean Shutdown”
   * Wait until graceful shutdown is completed
   * Upgrade all tegras
   * Start master using fabric, e.g. inside virtual env containing fabric:
     python <full path to>/manage_masters.py -f <full path to>/production-masters.json -H bm10-tests1-tegra start
   * Monitor results
   * Advise sheriffs that buildbot master is back again, and ask to be notified in case of problems
   * Email release@mozilla.com to say all is ok again
   * Update bug 914302 to say master is completed
Dear all,
* Starting phase 1 now (tegras on buildbot-master10)
* No outage - but buildbot-master10 will be down while upgrading
Thanks,
Pete
buildbot-master10 down, update of tegras starting
All tegras are now currently being updated in parallel on foopy28, foopy29, foopy30, foopy31, foopy32 with the following exceptions:

    * tegra-342 on foopy29 is configured to connect to buildbot-master20.build.mtv1.mozilla.com (not sure why - will investigate)
    * tegra-349 on foopy30 is disabled in slavealloc

I will post an update with the results when completed.
Tegras upgraded, now collecting results to check for any failures, etc.
These upgrades were all successful:

SUCCESS: tegra-286 (foopy30)
SUCCESS: tegra-308 (foopy30)
SUCCESS: tegra-309 (foopy28)
SUCCESS: tegra-313 (foopy28)
SUCCESS: tegra-315 (foopy30)
SUCCESS: tegra-316 (foopy28)
SUCCESS: tegra-317 (foopy28)
SUCCESS: tegra-318 (foopy28)
SUCCESS: tegra-321 (foopy28)
SUCCESS: tegra-322 (foopy31)
SUCCESS: tegra-324 (foopy29)
SUCCESS: tegra-326 (foopy29)
SUCCESS: tegra-327 (foopy31)
SUCCESS: tegra-329 (foopy31)
SUCCESS: tegra-330 (foopy29)
SUCCESS: tegra-331 (foopy29)
SUCCESS: tegra-332 (foopy29)
SUCCESS: tegra-334 (foopy29)
SUCCESS: tegra-335 (foopy31)
SUCCESS: tegra-338 (foopy29)
SUCCESS: tegra-340 (foopy29)
SUCCESS: tegra-343 (foopy29)
SUCCESS: tegra-344 (foopy31)
SUCCESS: tegra-346 (foopy29)
SUCCESS: tegra-347 (foopy30)
SUCCESS: tegra-348 (foopy31)
SUCCESS: tegra-351 (foopy30)
SUCCESS: tegra-352 (foopy32)
SUCCESS: tegra-353 (foopy32)
SUCCESS: tegra-354 (foopy32)
SUCCESS: tegra-355 (foopy32)
SUCCESS: tegra-357 (foopy32)
SUCCESS: tegra-358 (foopy32)
SUCCESS: tegra-359 (foopy30)
SUCCESS: tegra-362 (foopy30)
SUCCESS: tegra-363 (foopy32)
SUCCESS: tegra-364 (foopy30)
SUCCESS: tegra-367 (foopy30)
SUCCESS: tegra-369 (foopy32)

The following tegra upgrades need to be investigated, since they did not log a successful upgrade:
INVESTIGATE: tegra-307 (foopy28)
INVESTIGATE: tegra-310 (foopy28)
INVESTIGATE: tegra-311 (foopy28)
INVESTIGATE: tegra-312 (foopy28)
INVESTIGATE: tegra-314 (foopy28)
INVESTIGATE: tegra-319 (foopy28)
INVESTIGATE: tegra-320 (foopy31)
INVESTIGATE: tegra-323 (foopy28)
INVESTIGATE: tegra-325 (foopy29)
INVESTIGATE: tegra-328 (foopy31)
INVESTIGATE: tegra-333 (foopy31)
INVESTIGATE: tegra-336 (foopy31)
INVESTIGATE: tegra-337 (foopy31)
INVESTIGATE: tegra-339 (foopy29)
INVESTIGATE: tegra-341 (foopy31)
INVESTIGATE: tegra-345 (foopy31)
INVESTIGATE: tegra-350 (foopy32)
INVESTIGATE: tegra-356 (foopy30)
INVESTIGATE: tegra-360 (foopy32)
INVESTIGATE: tegra-361 (foopy30)
INVESTIGATE: tegra-365 (foopy30)
Foopy 28 first investigation results
====================================

INVESTIGATE: tegra-307 (foopy28)
Could not connect
Buildbot slave not running
No error.flg (strange)

INVESTIGATE: tegra-310 (foopy28)
Upgraded, but timed out waiting for device to come back up
Buildbot slave not running
error.flg says "Remote Device Error: unable to write to sdcard"

INVESTIGATE: tegra-311 (foopy28)
Upgraded, but timed out waiting for device to come back up
Buildbot slave now running again (device passed verify.py)

INVESTIGATE: tegra-312 (foopy28)
Could not connect
Buildbot slave now running again (device passed verify.py)

INVESTIGATE: tegra-314 (foopy28)
Upgraded, but timed out waiting for device to come back up
Buildbot slave not running
error.flg says "Remote Device Error: unable to write to sdcard"

INVESTIGATE: tegra-319 (foopy28)
Upgraded, but timed out waiting for device to come back up
Buildbot slave now running again (device passed verify.py)

INVESTIGATE: tegra-323 (foopy28)
Upgraded, but timed out waiting for device to come back up
Buildbot slave now running again (device passed verify.py)
Foopy 28 further results
========================

INVESTIGATE: tegra-307 (foopy28)
Could not connect
Buildbot slave not running
error.flg: Unable to ping device after 5 attempts

INVESTIGATE: tegra-310 (foopy28)
Upgrade ok
Buildbot not running
error.flg: Remote Device Error: unable to write to sdcard

INVESTIGATE: tegra-311 (foopy28)
Intermittent connection problems, but device upgraded ok
Buildbot currently running

INVESTIGATE: tegra-312 (foopy28)
**Not upgraded**
Buildbot running

INVESTIGATE: tegra-314 (foopy28)
Upgrade ok
Buildbot not running
error.flg: Remote Device Error: unable to write to sdcard

INVESTIGATE: tegra-319 (foopy28)
Now ok, upgrade worked, buildbot running

INVESTIGATE: tegra-323 (foopy28)
Now ok, upgrade worked, buildbot running
Complete results:



Foopy 28 investigation results
==============================

INVESTIGATE: tegra-307 (foopy28)
Could not connect
error.flg: Unable to ping device after 5 attempts

INVESTIGATE: tegra-310 (foopy28)
Upgrade ok
error.flg: Remote Device Error: unable to write to sdcard

INVESTIGATE: tegra-311 (foopy28)
Intermittent connection problems, but device upgraded ok
Buildbot currently running

INVESTIGATE: tegra-312 (foopy28)
**Not upgraded**
Buildbot running

INVESTIGATE: tegra-314 (foopy28)
Upgrade ok
error.flg: Remote Device Error: unable to write to sdcard



Foopy 29 investigation results
==============================

INVESTIGATE: tegra-324 (foopy29)
Watcher upgraded ok
Remote Device Error: unable to write to sdcard
Buildbot not running, no error.flg yet

INVESTIGATE: tegra-325 (foopy29)
Automation Error: Unable to connect to device after 5 attempts

INVESTIGATE: tegra-339 (foopy29)
Watcher upgraded ok
Remote Device Error: unable to write to sdcard



Foopy 30 investigation results
==============================

INVESTIGATE: tegra-356 (foopy30)
Automation Error: Unable to connect to device after 5 attempts

INVESTIGATE: tegra-361 (foopy30)
Watcher upgraded ok
Remote Device Error: unable to write to sdcard

INVESTIGATE: tegra-365 (foopy30)
Watcher upgraded ok
Remote Device Error: unable to write to sdcard



Foopy 31 investigation results
==============================

INVESTIGATE: tegra-320 (foopy31)
Not upgraded

INVESTIGATE: tegra-328 (foopy31)
Cannot connect
No error.flg at the moment

INVESTIGATE: tegra-333 (foopy31) - all ok again
INVESTIGATE: tegra-336 (foopy31) - all ok again

INVESTIGATE: tegra-337 (foopy31)
Watcher upgrade ok
Remote Device Error: unable to write to sdcard

INVESTIGATE: tegra-341 (foopy31)
Watcher upgrade ok
Remote Device Error: unable to write to sdcard

INVESTIGATE: tegra-345 (foopy31)
Cannot connect
No error.flg at the moment



Foopy 32 investigation results
==============================

INVESTIGATE: tegra-350 (foopy32)
Watcher upgrade ok
Remote Device Error: unable to write to sdcard

***NEW***: tegra-358 (foopy32)
Watcher upgrade ok
Remote Device Error: unable to write to sdcard

INVESTIGATE: tegra-360 (foopy32)
Cannot connect
No error.flg at the moment
I don't consider any of the above issues to be blocking, so will restart buildbot master 10.
http://buildbot-master10.build.mtv1.mozilla.com:8201/ up and running again, notified #releng.

Work completed for day 1/4, with the exception of the individual tegra issues described in connect 22 above, which will be handed as BAU.

Tomorrow - Tue Sep 17 (Day 2/4) will be the upgrade of tegras attached to the next tegra buildbot master (http://buildbot-master19.build.mtv1.mozilla.com:8201/) - starting at 9:30 CEST...
REGARDING ***PANDA*** ROLLOUT PLANS
===================================

Regarding panda roll out - I have discussed this with kmoir and jmaher, and bug 829211 should be landing in the next week or so, so I would propose we roll out to pandas after bug 829211 lands.

The biggest motivator for this, is that once all pandas are mozpool-managed, we simply have to update the image name in mozharness to the new mozpool panda image that contains the new watcher, and this should take care that all the pandas then run with the correct image.

It is also trivial to rollback the change, since it would be a case of reverting the change.
Depends on: 829211
Dear all,
* Starting phase 2 now (tegras on buildbot-master19)
* No outage - but buildbot-master19 will be down while upgrading
Thanks,
Pete
All tegras on buildbot-master19 patched, checking results...
Everything it taking a little longer today - hope to be all done in the next hour.
Upgrade problems on foopy27
===========================

tegra-250: Cannot connect to device



Upgrade problems on foopy111
============================

No upgrade problems found



Upgrade problems on foopy115
============================

No upgrade problems found



Upgrade problems on foopy117
============================

tegra-108: Cannot connect to device
tegra-113: Cannot connect to device
tegra-115: Cannot connect to device



Upgrade problems on foopy121
============================

No upgrade problems found



Upgrade problems on foopy123...
============================

tegra-223: Cannot connect to device
tegra-229: Cannot connect to device



Other notes
===========

tegra-228 (foopy123)
=========
Not upgraded - intentional - not a problem

tegra-261 (foopy123)
=========
Not upgraded - intentional - not a problem
After starting up buildbot master and puppet on foopies, and watch_devices.sh on foopies, I'm now additionally having issues connecting to the following three tegras:

tegra-087 on foopy115
tegra-177 and tegra-189 on foopy121

I could connect to these devices before the upgrade. I'm not sure at the moment what the problem is.
Correction to comment above in comment 31: I mean, I could connect to them after the upgrade, before starting up buildbot master and slaves...
Strangely, we seem to have made a nett win today. See results below:


Tegras that had no error.flg before or after the watcher upgrade on buildbot-master19 today:

tegra-031
tegra-032
tegra-035
tegra-036
tegra-037
tegra-038
tegra-039
tegra-078
tegra-080
tegra-081
tegra-082
tegra-083
tegra-085
tegra-086
tegra-088
tegra-089
tegra-105
tegra-107
tegra-109
tegra-110
tegra-111
tegra-112
tegra-117
tegra-146
tegra-178
tegra-179
tegra-181
tegra-182
tegra-183
tegra-188
tegra-190
tegra-194
tegra-199
tegra-200
tegra-225
tegra-227
tegra-231
tegra-233
tegra-248
tegra-249
tegra-251
tegra-252
tegra-253
tegra-254
tegra-255
tegra-256
tegra-257
tegra-258
tegra-259
tegra-260

Tegras that previously had an error.flg file, before the upgrade, and still have one, afterwards:

tegra-064
tegra-087
tegra-108
tegra-113
tegra-114
tegra-177
tegra-189
tegra-220
tegra-221
tegra-223
tegra-229
tegra-232

Tegras that previously had error.flg files, that were cleared after the Watcher upgrade (!!!!)

tegra-079
tegra-084
tegra-106
tegra-116
tegra-173
tegra-174
tegra-187
tegra-196

Tegras that previously did not have an error.flg file, that now, after the upgrade, have one:

tegra-115
tegra-250
There are 6 tegras that don't have an error.flg file, but that are in any case not connected when I look at http://buildbot-master19.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1 - so I will investigate these now:

tegra-032
tegra-111
tegra-116
tegra-117
tegra-233
tegra-248
So of these 6 non-connected tegras, it turns out 4 were rebooting, and are connected again. The other two now have error.flg files. See below:

tegra-032: was rebooting, now is connected again.

tegra-111: was rebooting, now is connected again.

tegra-116:
Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/writetest 11865'; err='Push failed!'

error.flg: Remote Device Error: unable to write to sdcard

tegra-117:
09/17/2013 08:15:08: INFO: INFO: Got expected SUTAgent version '1.19'
09/17/2013 08:15:08: INFO: Uninstalling org.mozilla.fennec...
09/17/2013 08:15:08: INFO: Waiting for device to come back...
09/17/2013 08:17:08: INFO: Try 1
09/17/2013 08:18:04: INFO: devroot None
2013-09-17 08:20:02 -- *** ERROR *** failed to aquire lockfile

error.flg: Remote Device Error: Unhandled exception in cleanupDevice

tegra-233: was rebooting, now is connected again.

tegra-248: was rebooting, now is connected again.
I'm calling a day today, will continue tomorrow. :)
Dear all,
    * Starting phase 3 of https://bugzilla.mozilla.org/show_bug.cgi?id=914302
    * Updating all buildbot-master20 tegras to new Watcher 1.16 (production)
    * No outage - but buildbot-master20 will be down during this upgrade.
Thanks,
Pete
Summary report for day 3/4 (buildbot-master20)
==========================

    * only one device failed to upgrade:
          tegra-076

mozdevice.devicemanager.DMError: Did not successfully run command ['dd', 'if=/mnt/sdcard/tests/Watcher.1.16.apk', 'of=/data/local/watcher.apk'] (output: '', retval: 'None')

    * two devices upgraded, but did not reboot successfully (so are currently unreachable):
          tegra-219
          tegra-300

    * nine devices could not be upgraded, because they could not be reached:
          tegra-050
          tegra-066
          tegra-073
          tegra-074
          tegra-090
          tegra-150
          tegra-166
          tegra-171
          tegra-279


Will now restart puppet, watch_devices.sh, and buildbot master 20...
buildbot-master20 is back up and running, I'm monitoring the tegras here: http://buildbot-master20.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1

I will file bugs for any tegras that need to be reimaged.

Thanks,
Pete
Dear all,
* Phase 4 is underway (tegras on buildbot-master22)
* No tree outage - but buildbot-master22 is down while I am upgrading the tegras
Thanks,
Pete
Summary report for day 4/4 (buildbot-master22)
==========================

    * two devices failed to upgrade:
          tegra-264@foopy126
          tegra-270@foopy126

mozdevice.devicemanager.DMError: Did not successfully run command ['dd', 'if=/mnt/sdcard/tests/Watcher.1.16.apk', 'of=/data/local/watcher.apk'] (output: '', retval: 'None')


    * four devices could not be upgraded, because they could not be reached:
          tegra-130@foopy119
          tegra-218@foopy122
          tegra-242@foopy124
          tegra-269@foopy126
Started up buildbot-master22 again, and am monitoring http://buildbot-master22.build.mtv1.mozilla.com:8201/buildslaves?no_builders=1

This bug is now complete.

I will follow up with a separate bug for tegras that still require action, such as a new image.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Reopening, because this ticket was also to encompass panda upgrades (whoops) .... Thanks jmaher for reminding me!
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Just generating a final list of tegras that had issues, and will be adding them to bug 918162...
Tegras that we cannot connect to, to see if the Watcher upgraded ok
===================================================================

foopy109: tegra-184: Cannot connect to device
foopy109: tegra-185: Cannot connect to device
foopy109: tegra-224: Cannot connect to device
foopy110: tegra-020: Cannot connect to device
foopy112: tegra-050: Cannot connect to device
foopy112: tegra-090: Cannot connect to device
foopy114: tegra-066: Cannot connect to device
foopy114: tegra-073: Cannot connect to device
foopy114: tegra-074: Cannot connect to device
foopy117: tegra-108: Cannot connect to device
foopy117: tegra-113: Cannot connect to device
foopy118: tegra-150: Cannot connect to device
foopy118: tegra-152: Cannot connect to device
foopy119: tegra-130: Cannot connect to device
foopy120: tegra-166: Cannot connect to device
foopy120: tegra-171: Cannot connect to device
foopy122: tegra-218: Cannot connect to device
foopy123: tegra-223: Cannot connect to device
foopy123: tegra-228: Cannot connect to device
foopy123: tegra-229: Cannot connect to device
foopy124: tegra-239: Cannot connect to device
foopy125: tegra-293: Cannot connect to device
foopy126: tegra-269: Cannot connect to device
foopy126: tegra-270: Cannot connect to device
foopy128: tegra-279: Cannot connect to device
foopy28: tegra-307: Cannot connect to device
foopy28: tegra-309: Cannot connect to device
foopy29: tegra-338: Cannot connect to device
foopy29: tegra-339: Cannot connect to device
foopy30: tegra-349: Cannot connect to device
foopy30: tegra-356: Cannot connect to device
foopy30: tegra-364: Cannot connect to device
foopy31: tegra-328: Cannot connect to device
foopy31: tegra-341: Cannot connect to device
foopy31: tegra-345: Cannot connect to device
foopy32: tegra-360: Cannot connect to device
foopy34: panda-0046: Cannot connect to device
foopy34: panda-0047: Cannot connect to device
foopy34: panda-0048: Cannot connect to device
foopy34: panda-0049: Cannot connect to device
foopy34: panda-0050: Cannot connect to device
foopy34: panda-0051: Cannot connect to device
foopy34: panda-0052: Cannot connect to device
foopy34: panda-0053: Cannot connect to device
foopy34: panda-0054: Cannot connect to device
foopy34: panda-0055: Cannot connect to device
foopy34: panda-0056: Cannot connect to device
foopy34: panda-0057: Cannot connect to device
foopy35: panda-0058: Cannot connect to device
foopy35: panda-0059: Cannot connect to device
foopy35: panda-0060: Cannot connect to device
foopy35: panda-0061: Cannot connect to device
foopy35: panda-0062: Cannot connect to device
foopy35: panda-0063: Cannot connect to device
foopy35: panda-0064: Cannot connect to device
foopy35: panda-0065: Cannot connect to device
foopy35: panda-0066: Cannot connect to device
foopy35: panda-0067: Cannot connect to device
foopy35: panda-0068: Cannot connect to device
foopy35: panda-0069: Cannot connect to device



Devices that do not have the upgraded Watcher
=============================================

foopy109: tegra-010: Not upgraded
foopy109: tegra-011: Not upgraded
foopy109: tegra-012: Not upgraded
foopy109: tegra-013: Not upgraded
foopy110: tegra-014: Not upgraded
foopy110: tegra-015: Not upgraded
foopy110: tegra-016: Not upgraded
foopy110: tegra-017: Not upgraded
foopy110: tegra-018: Not upgraded
foopy110: tegra-019: Not upgraded
foopy110: tegra-021: Not upgraded
foopy110: tegra-022: Not upgraded
foopy110: tegra-023: Not upgraded
foopy110: tegra-025: Not upgraded
foopy110: tegra-026: Not upgraded
foopy110: tegra-027: Not upgraded
foopy110: tegra-028: Not upgraded
foopy110: tegra-029: Not upgraded
foopy112: tegra-042: Not upgraded
foopy112: tegra-051: Not upgraded
foopy113: tegra-053: Not upgraded
foopy113: tegra-060: Not upgraded
foopy114: tegra-076: Not upgraded
foopy118: tegra-155: Not upgraded
foopy119: tegra-142: Not upgraded
foopy124: tegra-242: Not upgraded
foopy124: tegra-246: Not upgraded
foopy126: tegra-264: Not upgraded
foopy28: tegra-312: Not upgraded
foopy29: tegra-325: Not upgraded
foopy31: tegra-320: Not upgraded


Devices that were successfully upgraded
=======================================

foopy109: tegra-288: Watcher upgraded
foopy111: tegra-031: Watcher upgraded
foopy111: tegra-032: Watcher upgraded
foopy111: tegra-035: Watcher upgraded
foopy111: tegra-036: Watcher upgraded
foopy111: tegra-037: Watcher upgraded
foopy111: tegra-038: Watcher upgraded
foopy111: tegra-039: Watcher upgraded
foopy111: tegra-064: Watcher upgraded
foopy111: tegra-078: Watcher upgraded
foopy111: tegra-199: Watcher upgraded
foopy111: tegra-200: Watcher upgraded
foopy112: tegra-040: Watcher upgraded
foopy112: tegra-041: Watcher upgraded
foopy112: tegra-045: Watcher upgraded
foopy112: tegra-046: Watcher upgraded
foopy112: tegra-047: Watcher upgraded
foopy112: tegra-048: Watcher upgraded
foopy112: tegra-052: Watcher upgraded
foopy113: tegra-054: Watcher upgraded
foopy113: tegra-055: Watcher upgraded
foopy113: tegra-056: Watcher upgraded
foopy113: tegra-057: Watcher upgraded
foopy113: tegra-058: Watcher upgraded
foopy113: tegra-059: Watcher upgraded
foopy113: tegra-061: Watcher upgraded
foopy113: tegra-062: Watcher upgraded
foopy113: tegra-063: Watcher upgraded
foopy114: tegra-067: Watcher upgraded
foopy114: tegra-068: Watcher upgraded
foopy114: tegra-070: Watcher upgraded
foopy114: tegra-071: Watcher upgraded
foopy114: tegra-072: Watcher upgraded
foopy114: tegra-091: Watcher upgraded
foopy114: tegra-219: Watcher upgraded
foopy115: tegra-079: Watcher upgraded
foopy115: tegra-080: Watcher upgraded
foopy115: tegra-081: Watcher upgraded
foopy115: tegra-082: Watcher upgraded
foopy115: tegra-083: Watcher upgraded
foopy115: tegra-084: Watcher upgraded
foopy115: tegra-085: Watcher upgraded
foopy115: tegra-086: Watcher upgraded
foopy115: tegra-087: Watcher upgraded
foopy115: tegra-088: Watcher upgraded
foopy115: tegra-089: Watcher upgraded
foopy116: tegra-092: Watcher upgraded
foopy116: tegra-093: Watcher upgraded
foopy116: tegra-094: Watcher upgraded
foopy116: tegra-095: Watcher upgraded
foopy116: tegra-096: Watcher upgraded
foopy116: tegra-097: Watcher upgraded
foopy116: tegra-098: Watcher upgraded
foopy116: tegra-099: Watcher upgraded
foopy116: tegra-100: Watcher upgraded
foopy116: tegra-101: Watcher upgraded
foopy116: tegra-102: Watcher upgraded
foopy116: tegra-103: Watcher upgraded
foopy116: tegra-104: Watcher upgraded
foopy117: tegra-105: Watcher upgraded
foopy117: tegra-106: Watcher upgraded
foopy117: tegra-107: Watcher upgraded
foopy117: tegra-109: Watcher upgraded
foopy117: tegra-110: Watcher upgraded
foopy117: tegra-111: Watcher upgraded
foopy117: tegra-112: Watcher upgraded
foopy117: tegra-114: Watcher upgraded
foopy117: tegra-115: Watcher upgraded
foopy117: tegra-116: Watcher upgraded
foopy117: tegra-117: Watcher upgraded
foopy118: tegra-118: Watcher upgraded
foopy118: tegra-119: Watcher upgraded
foopy118: tegra-120: Watcher upgraded
foopy118: tegra-121: Watcher upgraded
foopy118: tegra-145: Watcher upgraded
foopy118: tegra-148: Watcher upgraded
foopy118: tegra-149: Watcher upgraded
foopy118: tegra-151: Watcher upgraded
foopy118: tegra-154: Watcher upgraded
foopy118: tegra-157: Watcher upgraded
foopy119: tegra-129: Watcher upgraded
foopy119: tegra-132: Watcher upgraded
foopy119: tegra-133: Watcher upgraded
foopy119: tegra-134: Watcher upgraded
foopy119: tegra-135: Watcher upgraded
foopy119: tegra-136: Watcher upgraded
foopy119: tegra-138: Watcher upgraded
foopy119: tegra-139: Watcher upgraded
foopy119: tegra-140: Watcher upgraded
foopy119: tegra-141: Watcher upgraded
foopy119: tegra-144: Watcher upgraded
foopy120: tegra-158: Watcher upgraded
foopy120: tegra-159: Watcher upgraded
foopy120: tegra-160: Watcher upgraded
foopy120: tegra-162: Watcher upgraded
foopy120: tegra-163: Watcher upgraded
foopy120: tegra-164: Watcher upgraded
foopy120: tegra-165: Watcher upgraded
foopy120: tegra-167: Watcher upgraded
foopy120: tegra-168: Watcher upgraded
foopy120: tegra-169: Watcher upgraded
foopy120: tegra-170: Watcher upgraded
foopy121: tegra-173: Watcher upgraded
foopy121: tegra-174: Watcher upgraded
foopy121: tegra-177: Watcher upgraded
foopy121: tegra-178: Watcher upgraded
foopy121: tegra-179: Watcher upgraded
foopy121: tegra-181: Watcher upgraded
foopy121: tegra-182: Watcher upgraded
foopy121: tegra-183: Watcher upgraded
foopy121: tegra-187: Watcher upgraded
foopy121: tegra-188: Watcher upgraded
foopy121: tegra-189: Watcher upgraded
foopy121: tegra-190: Watcher upgraded
foopy121: tegra-196: Watcher upgraded
foopy122: tegra-206: Watcher upgraded
foopy122: tegra-207: Watcher upgraded
foopy122: tegra-208: Watcher upgraded
foopy122: tegra-209: Watcher upgraded
foopy122: tegra-210: Watcher upgraded
foopy122: tegra-211: Watcher upgraded
foopy122: tegra-212: Watcher upgraded
foopy122: tegra-213: Watcher upgraded
foopy122: tegra-214: Watcher upgraded
foopy122: tegra-215: Watcher upgraded
foopy122: tegra-216: Watcher upgraded
foopy122: tegra-217: Watcher upgraded
foopy123: tegra-146: Watcher upgraded
foopy123: tegra-194: Watcher upgraded
foopy123: tegra-220: Watcher upgraded
foopy123: tegra-221: Watcher upgraded
foopy123: tegra-225: Watcher upgraded
foopy123: tegra-227: Watcher upgraded
foopy123: tegra-231: Watcher upgraded
foopy123: tegra-232: Watcher upgraded
foopy123: tegra-233: Watcher upgraded
foopy123: tegra-261: Watcher upgraded
foopy124: tegra-234: Watcher upgraded
foopy124: tegra-235: Watcher upgraded
foopy124: tegra-236: Watcher upgraded
foopy124: tegra-237: Watcher upgraded
foopy124: tegra-238: Watcher upgraded
foopy124: tegra-240: Watcher upgraded
foopy124: tegra-243: Watcher upgraded
foopy124: tegra-244: Watcher upgraded
foopy124: tegra-245: Watcher upgraded
foopy124: tegra-247: Watcher upgraded
foopy125: tegra-290: Watcher upgraded
foopy125: tegra-294: Watcher upgraded
foopy125: tegra-295: Watcher upgraded
foopy125: tegra-296: Watcher upgraded
foopy125: tegra-297: Watcher upgraded
foopy125: tegra-298: Watcher upgraded
foopy125: tegra-299: Watcher upgraded
foopy125: tegra-300: Watcher upgraded
foopy125: tegra-302: Watcher upgraded
foopy125: tegra-304: Watcher upgraded
foopy125: tegra-305: Watcher upgraded
foopy125: tegra-306: Watcher upgraded
foopy126: tegra-195: Watcher upgraded
foopy126: tegra-262: Watcher upgraded
foopy126: tegra-263: Watcher upgraded
foopy126: tegra-265: Watcher upgraded
foopy126: tegra-266: Watcher upgraded
foopy126: tegra-267: Watcher upgraded
foopy126: tegra-271: Watcher upgraded
foopy126: tegra-272: Watcher upgraded
foopy126: tegra-273: Watcher upgraded
foopy126: tegra-274: Watcher upgraded
foopy128: tegra-191: Watcher upgraded
foopy128: tegra-192: Watcher upgraded
foopy128: tegra-201: Watcher upgraded
foopy128: tegra-276: Watcher upgraded
foopy128: tegra-277: Watcher upgraded
foopy128: tegra-278: Watcher upgraded
foopy128: tegra-280: Watcher upgraded
foopy128: tegra-281: Watcher upgraded
foopy128: tegra-282: Watcher upgraded
foopy128: tegra-283: Watcher upgraded
foopy128: tegra-284: Watcher upgraded
foopy128: tegra-285: Watcher upgraded
foopy27: tegra-248: Watcher upgraded
foopy27: tegra-249: Watcher upgraded
foopy27: tegra-250: Watcher upgraded
foopy27: tegra-251: Watcher upgraded
foopy27: tegra-252: Watcher upgraded
foopy27: tegra-253: Watcher upgraded
foopy27: tegra-254: Watcher upgraded
foopy27: tegra-255: Watcher upgraded
foopy27: tegra-256: Watcher upgraded
foopy27: tegra-257: Watcher upgraded
foopy27: tegra-258: Watcher upgraded
foopy27: tegra-259: Watcher upgraded
foopy27: tegra-260: Watcher upgraded
foopy28: tegra-310: Watcher upgraded
foopy28: tegra-311: Watcher upgraded
foopy28: tegra-313: Watcher upgraded
foopy28: tegra-314: Watcher upgraded
foopy28: tegra-316: Watcher upgraded
foopy28: tegra-317: Watcher upgraded
foopy28: tegra-318: Watcher upgraded
foopy28: tegra-319: Watcher upgraded
foopy28: tegra-321: Watcher upgraded
foopy28: tegra-323: Watcher upgraded
foopy29: tegra-324: Watcher upgraded
foopy29: tegra-326: Watcher upgraded
foopy29: tegra-330: Watcher upgraded
foopy29: tegra-331: Watcher upgraded
foopy29: tegra-332: Watcher upgraded
foopy29: tegra-334: Watcher upgraded
foopy29: tegra-340: Watcher upgraded
foopy29: tegra-342: Watcher upgraded
foopy29: tegra-343: Watcher upgraded
foopy29: tegra-346: Watcher upgraded
foopy30: tegra-286: Watcher upgraded
foopy30: tegra-308: Watcher upgraded
foopy30: tegra-315: Watcher upgraded
foopy30: tegra-347: Watcher upgraded
foopy30: tegra-351: Watcher upgraded
foopy30: tegra-359: Watcher upgraded
foopy30: tegra-361: Watcher upgraded
foopy30: tegra-362: Watcher upgraded
foopy30: tegra-365: Watcher upgraded
foopy30: tegra-367: Watcher upgraded
foopy31: tegra-322: Watcher upgraded
foopy31: tegra-327: Watcher upgraded
foopy31: tegra-329: Watcher upgraded
foopy31: tegra-333: Watcher upgraded
foopy31: tegra-335: Watcher upgraded
foopy31: tegra-336: Watcher upgraded
foopy31: tegra-337: Watcher upgraded
foopy31: tegra-344: Watcher upgraded
foopy31: tegra-348: Watcher upgraded
foopy32: tegra-350: Watcher upgraded
foopy32: tegra-352: Watcher upgraded
foopy32: tegra-353: Watcher upgraded
foopy32: tegra-354: Watcher upgraded
foopy32: tegra-355: Watcher upgraded
foopy32: tegra-357: Watcher upgraded
foopy32: tegra-358: Watcher upgraded
foopy32: tegra-363: Watcher upgraded
foopy32: tegra-369: Watcher upgraded
Flags: needinfo?(cbook)
Depends on: 925753
This work is now completed. Closing bug.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.