1248172 - Autophone - treeherder job collections created with invalid job_guid

Status: RESOLVED WONTFIX

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: autophone 2 log

In bug 1216578 we started queuing treeherder submissions to the treeherder table in the jobs database.

Overnight, it appear several entries were created with null job_guid. This stalled the submission of other pending treeherder jobs completely. 

I also did not receive any email notifications that there was a problem.

The error in the log was:

Traceback (most recent call last):
  File "/mozilla/autophone/autophone/autophonetreeherder.py", line 91, in post_request
    client.post_collection(project, job_collection)
  File "/mozilla/autophone/venv/local/lib/python2.7/site-packages/thclient/client.py", line 923, in post_collection
    collection_inst.validate()
  File "/mozilla/autophone/venv/local/lib/python2.7/site-packages/thclient/client.py", line 529, in validate
    d.validate()
  File "/mozilla/autophone/venv/local/lib/python2.7/site-packages/thclient/client.py", line 62, in validate
    cb(prop.split('.'), required_properties[prop], prop)
  File "/mozilla/autophone/venv/local/lib/python2.7/site-packages/thclient/client.py", line 117, in validate_existence
    raise TreeherderClientError(msg, [])
TreeherderClientError: TreeherderJob structure validation errors detected for property:job.job_guid
        Value not defined for job.job_guid


I am not sure of the root cause, but there were apparently network issues where downloads were failing due to incomplete downloads.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: autophone 2 bad_jobs.txt

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: autophone 3 log

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: autophone 3 bad_jobs.txt

Comment 4

[Bob Clary [:bc] (inactive)]

Apart from not creating invalid jobs, when we see structural fatal errors that are not due to transient network or treeherder server issues, we should make sure to email notification of the issues and not block on the bad jobs. We can either delete the structurally bad jobs from the database or at least skip over them so the other jobs are submitted in a timely fashion.

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Bob Clary [:bc:] from comment #0)
> I also did not receive any email notifications that there was a problem.

Neither did I. That's odd, since there is code to send email here, and we've seen that type of email recently:

Phone: nexus-6p-2
TreeherderClientError: HTTPSConnectionPool(host='treeherder.mozilla.org', port=443): Max retries exceeded with url: /api/project/fx-team/jobs/ (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1310187d90>, 'Connection to treeherder.mozilla.org timed out. (connect timeout=120)'))
Last attempt: None
Response: 2016-02-09T14:49:28.689539

(I note as an aside that the Response and Last attempt are reversed.)

Comment 6

[Bob Clary [:bc] (inactive)]

Heh. I haven't looked closer into what happened but I wonder if the failures related to the downloading and unzipping the build caused us to hit a dead lock for some reason. I mostly wanted to file this so we wouldn't lose the datum and could think about the potential causes at our leisure. heh. ;-)

Comment 7

[Geoff Brown [:gbrown]]

Attachment: improve mail notification

I'm not sure what conditions we want to use to decide when to discard a job - let's discuss.

In the mean time, I noticed these 3 issues related to email notification.

Comment 8

[Bob Clary [:bc] (inactive)]

Comment on attachment 8720027 [details] [diff] [review]
improve mail notification

Review of attachment 8720027 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 9

[Geoff Brown [:gbrown]]

https://github.com/mozilla/autophone/commit/d6823b825c7cd0db532f24efb61cabff0b1baf7a

There more to do here - figure out when to retry and when to discard a failed submission.

Comment 10

[Bob Clary [:bc] (inactive)]

Autophone is going away. Resolving these to wontfix.