962156 - Investigate why mochitest-bc runs slowly on all platforms, especially debug

Status: RESOLVED INCOMPLETE

(Core :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

We have seen many intermittent failures in the mochitest-bc suite that appear to be related to intermittent spikes in test run time.

For example, browser_webconsole_console_logging_api.js on Linux32 debug typically runs in ~4s, but intermittent would spike to 9+s, leading to bugs like bug 946066. I've seen other instances of tests hitting the 30s timeout that normally run in a fraction of that time.

Overall, debug mochitest-bc has a runtime of over 100 minutes on all platforms now. Linux32/64 and OSX 10.6 are generally 140-150min (and we've hit the 9000s timeout even then on occasion).

Since mochitest-bc chunking seems to be constantly held up by various factors, it would be good if we could investigate what is going on here and whether there are things we can be doing better now until that work is finished.

Comment 1

[Andrew McCreight [:mccr8]]

I think Taras was looking at profiling mochitest runs at the end of last year.  I don't know if he got any useful results out of that or not.

Comment 2

[Mihai Sucan [:msucan]]

Would it be possible to populate a database with test files and runtimes, for every machine and build, every run? We need data to learn about what's going slower and slower.

Comment 3

[Mihai Sucan [:msucan]]

I have a new round of intermittent test failures in the webconsole. Again it seems that the #1 culprit is slower and slower test runners...

Comment 4

[Nathan Froyd [:froydnj]]

(In reply to Mihai Sucan [:msucan] from comment #3)
> I have a new round of intermittent test failures in the webconsole. Again it
> seems that the #1 culprit is slower and slower test runners...

For avoidance of doubt: when you say "test runners", you mean the machines that the tests run on, correct?

Comment 5

[Mihai Sucan [:msucan]]

(In reply to Nathan Froyd (:froydnj) from comment #4)
> (In reply to Mihai Sucan [:msucan] from comment #3)
> > I have a new round of intermittent test failures in the webconsole. Again it
> > seems that the #1 culprit is slower and slower test runners...
> 
> For avoidance of doubt: when you say "test runners", you mean the machines
> that the tests run on, correct?

Yes, that is correct. However, it might as well be slower and slower builds. We have no measurements to tell us why tests run slower.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

This isn't actionable.