1147480 - Autophone - Regressions from retrigger/cancel jobs

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect, P1)

Description

[Bob Clary [:bc] (inactive)]

bug 1133580 caused several regressions.

* The test status reported to Treeherder is not set properly upon failure. Some tests which fail are being reported as Green even though job details shows an error.

* The  PhoneTest.job_details properly is not reinitialized when a test begins running. Previously the job_details was initialized in AutophoneTreeherder.submit_pending, but now that submit_pending is called from autophone.py when the build is first detected, and submit_running/submit_complete are called on a different when the test runs/completes, the job_details is not initialized.

Other issues which are not really regressions:

* #moc has implemented rate limiting on ftp.m.o and the utils.get_remote_text can fail due to 503 errors especially during trigger_runs.py from outside the moco network. They suggest retrying after random waits.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: pr 16 - v1

Due to the severe nature of the bugs, I have been running this in production since about 11 AM PT. You can see the example results at:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=autophone&exclusion_profile=false

https://treeherder.mozilla.org/#/jobs?repo=fx-team&filter-searchStr=autophone&exclusion_profile=false

Note the mochitest exceptions do not appear to be related to the patch but instead appear to be related to sutagent issues. I'm reaching out to stephend for help investigating the devices in the lab.

Comment 2

[Bob Clary [:bc] (inactive)]

For an example of the mochitests running without sutagent issues see a result from my local testing on staging:

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=6465d9a4d0dc&exclusion_profile=false&filter-searchStr=autophone

Comment 3

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/57123d38251d60ad42a5ac0787b37492150920c7
https://github.com/mozilla/autophone/commit/9e37b83d5bbb0f6f0e9e445044d3d8b7c89f2c36
https://github.com/mozilla/autophone/commit/57123d38251d60ad42a5ac0787b37492150920c7