1205739 - Investigate test_devtools_extensions.html test collisions on Linux Debug

Status: NEW

(DevTools :: General, defect)

Description

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

After the file migration in bug 912121, test_devtools_extensions.html starts colliding with other tests.

When test_devtools_extensions.html is enabled on Linux debug, the following tests then fail:

* toolkit/components/osfile/tests/mochi/test_osfile_comms.xul
* toolkit/components/osfile/tests/mochi/test_osfile_front.xul
* xpfe/appshell/test/test_hiddenPrivateWindow.xul

This is likely a test ordering issue: after the file move, this DevTools tests runs much earlier in the suite.

In bug 912121, I disabled test_devtools_extensions.html for Linux debug, as I couldn't find an obvious fix at the time.

As a related note, bug 1110982 tracks work to use run by dir for the chrome test job, so that would likely eliminate this issue.

Comment 1

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

:zer0, any guess why test_devtools_extensions.html would cause later tests to fail like this?

Here's an example try run with the failures before I disabled test_devtools_extensions.html:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0538032d629

Comment 2

[Matteo Ferretti [:zer0] [:matteo]]

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #1)
> :zer0, any guess why test_devtools_extensions.html would cause later tests
> to fail like this?
> 
> Here's an example try run with the failures before I disabled
> test_devtools_extensions.html:
> 
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0538032d629

I don't see anything obvious; but I can try to improve a bit the way test_devtools_extensions.html, especially how the tabs are handled, to mitigate side effects.

What could be a good way to check a possible patch? Push on try with "try: -b do -p linux64 -u mochitest-o -t none"?

Comment 3

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Matteo Ferretti [:matteo] [:zer0] from comment #2)
> (In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #1)
> > :zer0, any guess why test_devtools_extensions.html would cause later tests
> > to fail like this?
> > 
> > Here's an example try run with the failures before I disabled
> > test_devtools_extensions.html:
> > 
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0538032d629
> 
> I don't see anything obvious; but I can try to improve a bit the way
> test_devtools_extensions.html, especially how the tabs are handled, to
> mitigate side effects.
> 
> What could be a good way to check a possible patch? Push on try with "try:
> -b do -p linux64 -u mochitest-o -t none"?

Well, you'd need to re-enable the test first by removing the skip-if line[1] (perhaps that was obvious...) since I disabled it to land bug 912121.  And then, yes, that try syntax should work.  The failures were only on debug, so you could change to "-b d" instead of "-b do" if you wanted.

[1]: https://dxr.mozilla.org/mozilla-central/source/devtools/shared/tests/mochitest/chrome.ini#7