1405338 - Test browser_source_map-reload.js fails when run with --repeat 2

Status: RESOLVED FIXED

(DevTools :: Debugger, enhancement, P3)

Description

[Julian Descottes [:jdescottes]]

STRs:
- run ./mach test devtools/client/framework/test/browser_source_map-reload.js --repeat 2
-> Second iteration fails and test times out

This makes the test harder to debug in case it becomes intermittent, and also triggers a false positive if it happens to be run in a TV (test-verify) suite on try.

Comment 1

[Julian Descottes [:jdescottes]]

Geoff: a bit off-topic, but this kind of tests are wrongly flagged as intermittent by the TV test suite. Maybe it could be interesting to run the TV test suite once for all tests in order to identify all the tests that fail TV even though they were never logged as intermittents before?

Comment 2

[Tom Tromey :tromey]

Attachment: Bug 1405338 - change source-map reloading test to work with --repeat;

Review commit: https://reviewboard.mozilla.org/r/186048/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/186048/

Comment 3

[Julian Descottes [:jdescottes]]

Comment on attachment 8914770 [details]
Bug 1405338 - change source-map reloading test to work with --repeat;

https://reviewboard.mozilla.org/r/186048/#review191060

Fixes the issue, thanks Tom!

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/89ffef35daad
change source-map reloading test to work with --repeat; r=jdescottes

Comment 5

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to Julian Descottes [:jdescottes] from comment #1)
> Geoff: a bit off-topic, but this kind of tests are wrongly flagged as
> intermittent by the TV test suite. Maybe it could be interesting to run the
> TV test suite once for all tests in order to identify all the tests that
> fail TV even though they were never logged as intermittents before?

Julian, thanks for the ping, and for following up on this (you too Tom!). 

There are so many tests that trying to verify all of them is a big undertaking, both in machine time and in human follow-up/investigation of failures. I have run 100 or so tests through verify on try, and I have been watching TV as a tier-3 task for a couple of weeks; most tests pass and most failures seem to correspond to existing intermittent bugs, so I am hopeful that  issues like this one are infrequent...but that hope is tempered by the fact that this bug was found within hours of going to tier 2! I have filed bug 1405428.

I am also going to update the test-verify doc to note that:
 - test-verify failures sometimes confirm pre-existing conditions (rather than finding new problems)
 - test-verify failures sometimes indicate a test's vulnerability to --repeat or other conditions not found in "normal" automation runs

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/89ffef35daad