1493527 - Make merge-analyses.py faster

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Kartikaya Gupta (email:[e-mail])]

It's pretty slow, like a couple of hours to merge Linux and macOS m-c analysis files. I'm sure it can be optimized further by taking the common case into account but we can also rewrite it in rust which allows us to reuse the read_analysis code. That might be preferable.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

For the record: it currently takes ~2h15m to merge the m-c analysis files. From local experiments it seems like a good chunk of time is spent in the `sort` command that I pump the output of merge-analyses.py into. The purpose of the sort|uniq combo is to deduplicate the target lines, but we can do that using a hashset inside merge-analyses.py itself. And I don't think we need the output analysis sorted for any other reason except aesthetics. I'll see how much that helps.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Yeah that didn't help, ~2h34m with that change.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

After a rewrite in rust it's down to ~22m or so. Still working out the bugs but that's much more promising.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

A few tweaks later it's down to ~8 minutes! Granted, it's not exactly an apples-to-apples comparison since the output isn't exactly the same as the python version, but it's semantically equivalent.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozsearch/mozsearch/pull/154
https://github.com/mozsearch/mozsearch-mozilla/pull/21

Comment 6

[Kartikaya Gupta (email:[e-mail])]

PRs merged