Closed Bug 1646118 Opened 4 years ago Closed 4 years ago

Add mozilla-esr78 to searchfox

Categories

(Webtools :: Searchfox, task)

Product:
Webtools
Component:
Searchfox
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Callek, Assigned: kats)

References

Details

(Keywords: leave-open)

We need to add esr78 to the list of branches indexed by searchfox

Assignee: nobody → kats

Looks like esr78 was branched off beta, so I added a esr78 branch on the gecko-blame git tarball using git branch esr78 beta and reuploaded that. That should be the only manual tarball change needed, the patch to update the mozsearch-mozilla configs will take care of the rest. There's no searchfox taskcluster jobs on the ESR78 branch, presumably because the .cron.yml change only got merged last night and so the first taskcluster jobs should get scheduled in an hour or so. Once that's done I can test my PR.

https://github.com/mozsearch/mozsearch-mozilla/pull/103

I tried the PR a couple of days ago but it hit the indexer timeout, probably because we now have 4 repos (beta, release, two ESRs) that are all doing the equivalent of m-c and it takes a long time. Options are to (a) increase the timeout, (b) split the indexer into two channels, (c) move ESR68 to the mozilla-archived channel (which I think is undesirable because it's still getting code updates) or (d) do more performance optimization work to bring down the indexing time.

Obviously (d) is the most work but also the biggest payoff. I'd prefer (a) to (b), and I don't think (c) is a good option but just mentioned it for completeness.

(a) increase the timeout seems like the best short-term fix.

I think there's also option (e) which is to move primary indexing to be a taskcluster job that basically produces a tarball that's the indexer output (perhaps pre-codesearch ingestion). This would likely entail some efforts to compress anything potentially compressible like to pre-gzip HTML that can be served by nginx compression, and to compress the JSON payloads in the cross-reference database (which is not entirely trivial thanks to it being a text file that probably could be a LMDB database or SQLite db using a btree that could compete with binary search).

mkindex.sh on mozilla-beta (which is what esr78 was forked from) took 1:35:05 to run on my attempt that timed out. Add about 20 minutes for the indexer-setup phase (normally takes ~5 minutes but esr78 will need to build a bunch of blame on the first run) and round up to 2 hours. So that's what I need to add to the crontab.

Sigh. Even 8 hours wasn't enough, it died while the web server was getting set up.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.