1270167 - Error 500 when submitting to treeherder for version-control-tools repo

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Robert Wood [:rwood]]

11:51:26 Getting revision hash from: https://treeherder.allizom.org/api/project/version-control-tools/resultset/?revision=72bc5a490bb3
11:51:26 Sending results to Treeherder: [{"project": "version-control-tools", "job": {"end_timestamp": 0, "submit_timestamp": 1462377086141, "start_timestamp": 1462377086141, "build_platform": {"platform": "linux64", "os_name": "linux", "architecture": "x86_64"}, "name": "reviewboard", "option_collection": {"opt": true}, "artifacts": [], "machine_platform": {"platform": "linux64", "os_name": "linux", "architecture": "x86_64"}, "who": "", "group_symbol": "MozReview", "reason": "", "group_name": "MozReview", "machine": "ip-10-1-0-177", "state": "running", "result": "", "log_references": [], "job_symbol": "RB", "job_guid": "af1061ba-df75-45c5-b219-a2e72e49e76a", "product_name": "mozreview", "desc": ""}, "revision_hash": "08a18c4f821c62ad0c680ef0cc866f839173a3d7", "coalesced": []}]
11:51:26 Traceback (most recent call last):
11:51:26   File "./post-to-treeherder.py", line 362, in <module>
11:51:26     th.submit_running_job(job)
11:51:26   File "./post-to-treeherder.py", line 246, in submit_running_job
11:51:26     self.submit(job)
11:51:26   File "./post-to-treeherder.py", line 238, in submit
11:51:26     client.post_collection(self.repository, job_collection)
11:51:26   File "/var/lib/jenkins/.local/lib/python2.7/site-packages/thclient/client.py", line 926, in post_collection
11:51:26     collection_inst.get_collection_data(), timeout)
11:51:26   File "/var/lib/jenkins/.local/lib/python2.7/site-packages/thclient/client.py", line 710, in _post_json
11:51:26     resp.raise_for_status()
11:51:26   File "/var/lib/jenkins/.local/lib/python2.7/site-packages/requests/models.py", line 840, in raise_for_status
11:51:26     raise HTTPError(http_error_msg, response=self)
11:51:26 requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://treeherder.allizom.org/api/project/version-control-tools/jobs/

Comment 1

[Cameron Dawson [:camd]]

Hey Rob--  Are you using the most recent version of the treeherder client?  We now prefer using the revision itself rather than the revision_hash.  It SHOULD still work with the revision_hash.  But I just wanted to mention that going forward we should move to using the revision.

I'm still looking into this error, regardless..  :)

Comment 2

[Robert Wood [:rwood]]

Thanks Cam - I just upgraded from treeherder-client 2.0.1 to the latest 2.1.0 (thanks) and changed the code to add.revision to the job and submit with that instead of revision_hash, unfortunately still seeing the same issue. Btw here are the dependencies I'm using:

boto==2.38.0
mozinfo==0.8
treeherder-client==2.1.0

# Required by treeherder-client
mohawk==0.3.2.1
requests==2.9.1
requests-hawk==1.0.0
six==1.10.0

Comment 3

[Cameron Dawson [:camd]]

I was only able to reproduce this error in a specific circumstance.  If the project in the data for the job had "project": "version-control-tools" but when I posted the job the first time, I posted it with:

    client.post_collection('mozilla-inbound', myjobcollection)

It went through without an error, but didn't show up because I posted it to the wrong project.  I then realized my mistake and did:

    client.post_collection('version-control-tools', myjobcollection)

In the second call, it errors out because we have a table that tracks where job guids belong.  It already had an entry for that guid in mozilla-central and so it bombed out.

This is a treeherder bug, to be sure.  But certainly an odd case.  Working on a fix now.

Comment 4

[Cameron Dawson [:camd]]

I just noticed that we never use the ``project`` param of a job for anything.  We ONLY determine the project for a submission based on the URL.  So it's unnecessary to have code to add a project to a job.  We should take that out of the client...

Nevertheless, I'm not certain that this was caused by the same error.  I find no record of job_guid af1061ba-df75-45c5-b219-a2e72e49e76a in the treeherder-stage.job table.  So there couldn't have been the same conflict I hit locally.

I think I need to do some log grepin'

Comment 5

[Cameron Dawson [:camd]]

Rob: ahh, ok.  I found the exception in the log.  Your timestamps are in milliseconds, but we need them in seconds (field is INT(10)).  Sure would be great if we returned a more useful error message.  Or detect that they're too long and truncate them, even.  Or have the client validate that they're the right size.

Would you try again with the shorter timestamps?

Comment 6

[Cameron Dawson [:camd]]

Attachment: server stack trace

Comment 7

[Robert Wood [:rwood]]

Changed timestamps to seconds and now it works \o/ thank you :camd!

Comment 8

[Ed Morley [:emorley]]

Let's leave this open to improve the error handling on the Treeherder side - a 500 is not very user friendly :-)

Comment 9

[Ed Morley [:emorley]]

Submitting jobs/pushes via the API is now discouraged in favour of using Pulse (bug 1349182), so we're wontfixing API-specific issues.