1128474 - "HTTPError: 400 Client Error: no such method: pushes" from json-pushes

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Ed Morley [:emorley]]

Treeherder's staging instance has seen a lot of "HTTPError: 400 Client Error: no such method: pushes" from json-pushes today.

Sadly our logs our not yet adequate enough to work out which repo's json-pushes call is to blame (the output is interleaved below), but it will be one of the /json-pushes/?full=1&fromchange=FOO requests. (Note: version 1 of the json-pushes API).

Is this more fallout from bug 1127676? (Staging still had the cached 'bad' rev aiui, which we've cleared in the last few mins, but IMO we still shouldn't HTTP 400)

gps, any ideas? :-)



[2015-02-02 04:14:00,437: INFO/MainProcess] Received task: fetch-hg-push-logs[c974da23-a2d6-4ec7-a515-6038011a6a6a]
[2015-02-02 04:14:00,469: INFO/MainProcess] Extracted last push for 'larch', 'feb54976e5b90a08846f136defb7dd0cd04660d3', from cache, attempting to get changes only from that point
[2015-02-02 04:14:00,471: INFO/MainProcess] Starting new HTTPS connection (1): hg.mozilla.org
[2015-02-02 04:14:00,508: INFO/MainProcess] Task fetch-hg-push-logs[9a219393-8f90-4f64-a902-b81bda417018] succeeded in 0.201298613043s: None
[2015-02-02 04:14:00,508: INFO/MainProcess] Received task: fetch-hg-push-logs[de7ce17c-764a-4776-8583-debe33635c6e]
[2015-02-02 04:14:00,520: ERROR/MainProcess] Task fetch-hg-push-logs[87ff9879-f3f4-4416-8e4d-551e24c0609d] raised unexpected: HTTPError('400 Client Error: no such method: pushes',)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/newrelic-2.40.0.34-py2.6-linux-x86_64.egg/newrelic/hooks/application_celery.py", line 66, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/tasks/buildapi_tasks.py", line 69, in fetch_hg_push_log
    process.run(repo_url + '/json-pushes/?full=1', repo_name)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/pushlog.py", line 103, in run
    raise e
HTTPError: 400 Client Error: no such method: pushes
[2015-02-02 04:14:00,530: INFO/MainProcess] Received task: fetch-hg-push-logs[2f0d6181-dce8-4f8d-9522-3bc82f18b624]
[2015-02-02 04:14:00,540: INFO/MainProcess] Extracted last push for 'comm-central', 'f0194350ebf3c31805e745e851280093d685f2a3', from cache, attempting to get changes only from that point
[2015-02-02 04:14:00,541: INFO/MainProcess] Starting new HTTPS connection (1): hg.mozilla.org
[2015-02-02 04:14:00,542: INFO/MainProcess] Extracted last push for 'oak', 'a9b91a13402a7aaed28be9bfc1d5fa72ffaf355a', from cache, attempting to get changes only from that point
[2015-02-02 04:14:00,543: INFO/MainProcess] Starting new HTTPS connection (1): hg.mozilla.org
[2015-02-02 04:14:00,574: INFO/MainProcess] Task fetch-hg-push-logs[b5e16bf2-dcc4-4af7-ae1d-2f66573fac20] succeeded in 0.136495200975s: None
[2015-02-02 04:14:00,585: INFO/MainProcess] Task fetch-hg-push-logs[2efd7f31-ecd8-4383-87f5-2b6605b74d49] succeeded in 0.265664344945s: None
[2015-02-02 04:14:00,604: INFO/MainProcess] Received task: fetch-hg-push-logs[c92ebf67-a31f-4fee-a8c3-da7a3d1ef87b]
[2015-02-02 04:14:00,610: INFO/MainProcess] Received task: fetch-hg-push-logs[57df9672-e1db-46e5-b8ad-443a863309c6]
[2015-02-02 04:14:00,631: INFO/MainProcess] Task fetch-hg-push-logs[fcda1a4e-6b23-4f6f-8fb8-9d273e695bd2] succeeded in 0.330557483016s: None
[2015-02-02 04:14:00,642: INFO/MainProcess] Received task: fetch-hg-push-logs[6f403eda-1745-4b26-9e78-7ca43c0555f8]
[2015-02-02 04:14:00,648: INFO/MainProcess] Extracted last push for 'comm-aurora', 'e8b1aa4336226be76486cba40a5900d083a45455', from cache, attempting to get changes only from that point
[2015-02-02 04:14:00,650: INFO/MainProcess] Starting new HTTPS connection (1): hg.mozilla.org
[2015-02-02 04:14:00,651: INFO/MainProcess] Extracted last push for 'gaia-try', '40de3f6b8b6fae95c6edd1fb8819a07fae9898f9', from cache, attempting to get changes only from that point
[2015-02-02 04:14:00,653: INFO/MainProcess] Starting new HTTPS connection (1): hg.mozilla.org
[2015-02-02 04:14:00,654: INFO/MainProcess] Extracted last push for 'mozilla-b2g32_v2_0', 'b961877cacba81e9077519b1876ee4a4d05a70b0', from cache, attempting to get changes only from that point
[2015-02-02 04:14:00,656: INFO/MainProcess] Starting new HTTPS connection (1): hg.mozilla.org
[2015-02-02 04:14:00,684: INFO/MainProcess] Task fetch-hg-push-logs[de7ce17c-764a-4776-8583-debe33635c6e] succeeded in 0.153503638983s: None
[2015-02-02 04:14:00,695: INFO/MainProcess] Received task: fetch-hg-push-logs[0cd3b9f9-7067-4d69-9392-64fed86619c3]
[2015-02-02 04:14:00,703: INFO/MainProcess] Extracted last push for 'mozilla-esr31', '212f6ca519de824f43245c45f75e50aaef8f682c', from cache, attempting to get changes only from that point
[2015-02-02 04:14:00,705: INFO/MainProcess] Starting new HTTPS connection (1): hg.mozilla.org
[2015-02-02 04:14:00,780: ERROR/MainProcess] Task fetch-hg-push-logs[c974da23-a2d6-4ec7-a515-6038011a6a6a] raised unexpected: HTTPError('400 Client Error: no such method: pushes',)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/newrelic-2.40.0.34-py2.6-linux-x86_64.egg/newrelic/hooks/application_celery.py", line 66, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/tasks/buildapi_tasks.py", line 69, in fetch_hg_push_log
    process.run(repo_url + '/json-pushes/?full=1', repo_name)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/pushlog.py", line 103, in run
    raise e
HTTPError: 400 Client Error: no such method: pushes

Comment 1

[Ed Morley [:emorley]]

Actually, we're seeing this on prod too:
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/3017852935

Comment 3

[Gregory Szorc [:gps]]

There was exactly one host generating HTTP 400 errors: hgweb2.dmz.scl3.

I poked around the host and could find nothing that varied between it and other hosts which were working.

I deleted the .pyc files for pushlog-related web extensions and json-pushes on this host magically started working again. This is a real WTF. I'm not sure what happened. The deployment for all hosts is identical. Most weird.

Hopefully this gets fixed when we eventually refactor the Mercurial environment on production to be a proper virtualenv.

Comment 4

[Ed Morley [:emorley]]

That is weird! Many thanks :-)