1503547 - Add windows indexing to searchfox

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Kartikaya Gupta (email:[e-mail])]

After bug 1418415 lands we should have windows indexing jobs running on taskcluster. This allows us to download and merge the windows index data into the final searchfox index.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozsearch/mozsearch/pull/162 - a couple of fixes needed for the merging to work properly

Comment 2

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozsearch/mozsearch-mozilla/pull/24 - refactoring and enabling of the windows stuff

Comment 3

[Kartikaya Gupta (email:[e-mail])]

(In case you hadn't guessed, current dev instance has this deployed)

Comment 4

[Kartikaya Gupta (email:[e-mail])]

We're also going to need to fiddle with the timing of the cron jobs. The windows indexing job on taskcluster today took 199 minutes. The jobs get triggered at 6:30 AM EDT and so the windows job would finish at ~9:50 AM EDT, while the lambda task starts at 9 AM EDT. This means the windows artifacts won't be ready in time.

I definitely want to move the taskcluster cron from 6:30 AM EDT to 6:00 AM EDT because (a) the merges are done usually between 5:30 AM EDT and 6:00 AM EDT, and (b) the nightly builds are kicked off cron at 6:00 AM EDT and it makes sense to build searchfox artifacts on the same push. So that will buy us an extra half-hour, but I'll probably still need to move the lambda task down by a half hour and start at 9:30 AM EDT to ensure success.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1503547 - Move searchfox indexing jobs a half hour earlier to match nightly build times. r?mccr8

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/522fe7ae8d3a
Move searchfox indexing jobs a half hour earlier to match nightly build times. r=mccr8

Comment 7

[Kartikaya Gupta (email:[e-mail])]

leave-open until the mozsearch-mozilla PR is merged

Comment 8

[Kartikaya Gupta (email:[e-mail])]

I also changed the lambda task to start at 9:30 AM EDT

Comment 9

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/522fe7ae8d3a

Comment 10

[Kartikaya Gupta (email:[e-mail])]

Next indexing run should have Windows.