1453249 - [esr60] Cannot GET /json-automationrelevance/$HASH: Internal Server Error

Status: RESOLVED WORKSFORME

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Johan Lorenzo [:jlorenzo]]

STR:
* GET https://hg.mozilla.org/releases/mozilla-esr60/json-automationrelevance/2c000486eac466da6623e4d7f7f1fd4e318f60e8

Results: 500 Internal Server Error

This prevents Taskcluster's taskgraph to generate optimized graphs. I think it blocks bug 1452958.

I'm not if bug 1441340 comment 5 will solve this or not. If so, please feel free to mark it as a dupe.

Comment 1

[Johan Lorenzo [:jlorenzo]]

Hi :gps! Just to confirm, does bug 1441340 comment 5 solve this bug? Should I mark it as a dupe?

Comment 2

[Phil Ringnalda (:philor)]

json-automationrelevance uses the pushlog to determine what files were touched by the push which includes the revision it was passed, and the pushlog is empty. So either some other not-written comment in that bug fixes it if we clone the pushlog when we clone a repo, which I don't think we do, or, it'll be fixed by having an actual push to mozilla-esr60.

Comment 3

[Gregory Szorc [:gps]]

This was due to the hgweb replicas being out of sync for an extended period of time.

As explained at https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/notifications.html#common-properties-of-notifications, there is a race condition on every push where each hgweb read-only mirror obtains the new data at a different moment in time. We don't guarantee that https://hg.mozilla.org/ exposes a consistent snapshot of a repository until the Pulse or SNS message is sent. Systems that react to events on hg.mo should be subscribed to one of these feeds instead of polling hg.mo (e.g. via the pushlog).

Even if consumers are polling for changes, the window between replication inconsistency is often very small - <5s. So by the time a Taskcluster task hits hg.mo, the data is completely replicated and the window is long over. But we had an event that was causing long inconsistency windows and this caused chaos for downstream consumers.