846425 - Compare builds for moz.build switchover regressions

Status: NEW

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

As part of switching to moz.build files, there is the possibility there were errors in the largely mechanical conversion of Makefile.in data to moz.build data.

We should compare a before and after run on TBPL and look for any obvious discrepancies.

Before builds:

https://tbpl.mozilla.org/?rev=1f56495eaa56
https://tbpl.mozilla.org/?rev=b0e08db3bc2a

After build:

https://tbpl.mozilla.org/?rev=c65d59d33aa8

I think a good proxy for "did the build change" is to look at the number of tests run. If there is a difference, we might have accidentally dropped or added a directory.

Over time, I'd like a better way to perform this analysis automatically. I'm not sure if we can leverage the log parsers used to display the numbers on TBPL or what. Longer term, we need tools to emit machine readable files and TBPL should present high-level diffs between builds (test X added, etc).

Comment 1

[Phil Ringnalda (:philor)]

That would only have been reasonably possible if it had gone m-i -> m-c -> b-s -> m-c. Since the merge from b-s merged m-i to m-c, your after includes things which were added on m-i.

For reftests (by which I mean things run by the reftest harness), "number of tests" means "number of lines in the manifest" so before linux32 opt did 8872/0/305 and after linux32 opt did 8873/0/305 and that has nothing to do with you, because either the files in the manifest are there or the suite fails. So m-i added one reftest to what was on your m-c before.

But the unit for mochitests, where you could have disappeared some test files, is not number of test files, it's number of calls to SimpleTest's testing functions, so a before of 223556/0/748 and an after of 223501/0/751 doesn't mean anything, just like it doesn't mean anything that the first winxp opt M1 on your after did 225285/0/711 and when I retriggered it it did 225334/0/711.

So what you need is for someone to write a log parser which will extract unique test filenames from a mochitest log, and to then do a three-way compare between the m-i rev that went to b-s and m-c before and m-c after.

Not it.