Make searchfox index both mozilla-central nightlies in order to allow having reliable coverage data without adding undue latency
Categories
(Webtools :: Searchfox, enhancement)
Tracking
(firefox86 fixed)
| Tracking | Status | |
|---|---|---|
| firefox86 | --- | fixed |
People
(Reporter: asuth, Assigned: asuth)
References
Details
Attachments
(1 file)
As part of our attempts to ensure that we have revision-consistent coverage data for mozilla-central at every indexing run in bug 1677903 and generally improve the searchfox experience for users in Europe, we should start running searchfox indexing jobs on taskcluster for both of the mozilla-central: [{hour: 10, minute: 0}, {hour: 22, minute: 0}] nightlies; right now we only run at hour: 10. This entails changing https://searchfox.org/mozilla-central/source/.cron.yml to run at the second job as well (but only for mozilla-central, which is supported by use of by-project) and making the AWS lambda job for config1.json run twice a day.
The general motivation here is two-fold:
- In order to ensure coverage data is available, we need to delay the searchfox AWS lambda job (currently 12 UTC/GMT and 8a) further, which increases staleness of the index (which is based on the nightly's initiation at 10 UTC). Currently the AWS job completes around ~9:30am Eastern Time, 6:30am Pacific Time. Ensuring we have coverage could move this to somewhere between 12:30pm-2:30pm Eastern or 9:30am-12:30pm Pacific depending on how the coverage cron job stabilizes. By adding a second job we can help mitigate the impact of this new latency.
- The above has always been Americas-centric. 9:30am Eastern is 3:30pm in Berlin and 11:30pm in Tokyo. Compensating for coverage could move Berlin to 6:30pm-8:30pm and Tokyo to 2:30am-4:30am. By adding a second job we can greatly improve the effective latency experienced by those at least in Europe and Africa.
The expected timeline for this new nightly would be (using current effective time zones and place names in an ad hoc way):
- 22:00 UTC, 5pm Eastern, 11pm Berlin, 7am Tokyo: Taskcluster triggers nightly and searchfox jobs
- 00:00 UTC, 7pm Eastern, 1am Berlin, 9am Tokyo: The searchfox taskcluster jobs are done. This is when we would trigger the AWS job if we weren't waiting for coverage.
- 03:00-05:00 UTC, 10pm-12am Eastern, 4-6am Berlin, 12-2pm Tokyo: We can trigger the searchfox AWS job.
- 04:30-06:30 UTC, 11:30pm-1:30am Eastern, 5:30-7:30am Berlin, 1:30-3:30pm Tokyo: Searchfox AWS job completes and new web server is active.
|
Assignee | |
Comment 1•3 years ago
|
||
Pushed by bugmail@asutherland.org: https://hg.mozilla.org/integration/autoland/rev/d740fe0f03ba Make searchfox index both mozilla-central nightlies. r=kats,releng-reviewers,aki DONTBUILD
|
||
Comment 3•3 years ago
|
||
| bugherder | ||
|
|
Assignee | |
Comment 4•3 years ago
|
||
The .cron.yml changes worked for the 22utc build tonight and the changes to the AWS cron jobs I described in https://bugzilla.mozilla.org/show_bug.cgi?id=1677903#c15 likewise worked. This does, however, introduce a change in behavior that violates the assumptions that trigger-web-server.py has about which web servers to leave running since it uses a time-based heuristic; spinning off a new bug on that.
Description
•