Open Bug 846425 Opened 12 years ago Updated 2 years ago

Compare builds for switchover regressions


(Firefox Build System :: General, defect)



(Not tracked)


(Reporter: gps, Unassigned)



As part of switching to files, there is the possibility there were errors in the largely mechanical conversion of data to data.

We should compare a before and after run on TBPL and look for any obvious discrepancies.

Before builds:

After build:

I think a good proxy for "did the build change" is to look at the number of tests run. If there is a difference, we might have accidentally dropped or added a directory.

Over time, I'd like a better way to perform this analysis automatically. I'm not sure if we can leverage the log parsers used to display the numbers on TBPL or what. Longer term, we need tools to emit machine readable files and TBPL should present high-level diffs between builds (test X added, etc).
That would only have been reasonably possible if it had gone m-i -> m-c -> b-s -> m-c. Since the merge from b-s merged m-i to m-c, your after includes things which were added on m-i.

For reftests (by which I mean things run by the reftest harness), "number of tests" means "number of lines in the manifest" so before linux32 opt did 8872/0/305 and after linux32 opt did 8873/0/305 and that has nothing to do with you, because either the files in the manifest are there or the suite fails. So m-i added one reftest to what was on your m-c before.

But the unit for mochitests, where you could have disappeared some test files, is not number of test files, it's number of calls to SimpleTest's testing functions, so a before of 223556/0/748 and an after of 223501/0/751 doesn't mean anything, just like it doesn't mean anything that the first winxp opt M1 on your after did 225285/0/711 and when I retriggered it it did 225334/0/711.

So what you need is for someone to write a log parser which will extract unique test filenames from a mochitest log, and to then do a three-way compare between the m-i rev that went to b-s and m-c before and m-c after.

Not it.
Depends on: 847369
Depends on: 851975
Product: Core → Firefox Build System
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.