Autophone - Regressions from retrigger/cancel jobs

RESOLVED FIXED

Status

P1
critical
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: bc, Assigned: bc)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

44 bytes, text/x-github-pull-request
mcote
: review+
Details | Review | Splinter Review
(Assignee)

Description

4 years ago
bug 1133580 caused several regressions.

* The test status reported to Treeherder is not set properly upon failure. Some tests which fail are being reported as Green even though job details shows an error.

* The  PhoneTest.job_details properly is not reinitialized when a test begins running. Previously the job_details was initialized in AutophoneTreeherder.submit_pending, but now that submit_pending is called from autophone.py when the build is first detected, and submit_running/submit_complete are called on a different when the test runs/completes, the job_details is not initialized.

Other issues which are not really regressions:

* #moc has implemented rate limiting on ftp.m.o and the utils.get_remote_text can fail due to 503 errors especially during trigger_runs.py from outside the moco network. They suggest retrying after random waits.
(Assignee)

Comment 1

4 years ago
Created attachment 8583271 [details] [review]
pr 16 - v1

Due to the severe nature of the bugs, I have been running this in production since about 11 AM PT. You can see the example results at:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=autophone&exclusion_profile=false

https://treeherder.mozilla.org/#/jobs?repo=fx-team&filter-searchStr=autophone&exclusion_profile=false

Note the mochitest exceptions do not appear to be related to the patch but instead appear to be related to sutagent issues. I'm reaching out to stephend for help investigating the devices in the lab.
Attachment #8583271 - Flags: review?(mcote)
(Assignee)

Comment 2

4 years ago
For an example of the mochitests running without sutagent issues see a result from my local testing on staging:

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=6465d9a4d0dc&exclusion_profile=false&filter-searchStr=autophone

Updated

4 years ago
Attachment #8583271 - Flags: review?(mcote) → review+
You need to log in before you can comment on or make changes to this bug.