Closed Bug 704010 Opened 12 years ago Closed 11 years ago

Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1

Categories

(Core :: DOM: Core & HTML, defect)

Product:
Core
Component:
DOM: Core & HTML
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: crash, Whiteboard: [buildslaves])

Attachments

(1 file)

dumpchk.exe output
13.78 KB, text/plain
Details 
We discussed this once before, about one of the brutal CSS tests, and came to the inevitable conclusion that it's the fault of the test for doing something brutal to the slave, but the test author can't be at fault because he has no way of knowing what brutal thing he is doing, and releng can't say because it happened in the past, and why don't you just ignore purple anyway?

https://tbpl.mozilla.org/php/getParsedLog.php?id=7498672&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7496770&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7495812&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7495914&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7495292&tree=Mozilla-Inbound

... WinXP mochitest-1, both debug and PGO, disconnecting in the midst of test_fullscreen-api.html.

https://tbpl.mozilla.org/php/getParsedLog.php?id=7491088&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7489527&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7482717&tree=Firefox

... Win7 mochitest-4, both debug and PGO, disconnecting in the midst of test_value_cloning.html (one or more of those, or some other recent rash, might have been in one of the other exhaustive CSS tests, not sure and since opening mochitest-4 full logs is cash-out-of-pocket on a capped connection, not looking again to see).
https://tbpl.mozilla.org/php/getParsedLog.php?id=7485935&tree=Mozilla-Aurora - Win7 debug mochitest-4, in test_value_storage.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=7501103&tree=Firefox
Rev3 WINNT 5.1 mozilla-central opt test mochitests-1/5
test_fullscreen-api.html

https://tbpl.mozilla.org/php/getParsedLog.php?id=7501160&tree=Firefox
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5
test_fullscreen-api.html
Are you sure it's not that the disconnect happens to be in the middle of one of the longer-running tests because that test is longer-running (and therefore takes a nontrivial fraction of the total time) rather than because it does something to cause the disconnect?
There is a heartbeat mechanism between slave and master which continues regardless of test execution. But if a test/firefox pushes the slave deep into swap it's possible that buildbot doesn't get any CPU time to maintain that, and the job gets interrupted.

(In reply to Ed Morley [:edmorley] from comment #2)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=7501160&tree=Firefox
> Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5
> test_fullscreen-api.html

Here's some logging from the master, which I'm reading as the master losing contact with the slave:

2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] BuildSlave.detached(talos-r3-xp-013)
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] <Build Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5>.lostRemote
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121]  stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2aaaf9c5bb50>
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] addCompleteLog(interrupt)
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=firefox/firefox.exe', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-pat
h=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '1', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.
error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
        ]
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] RemoteCommand.disconnect: lost slave
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2aaaf9c5bb50>): []
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121]  step 'mochitest-plain-1' complete: exception
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121]  <Build Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5>: build finished
In https://tbpl.mozilla.org/php/getParsedLog.php?id=7501113&tree=Firefox&full=1, the purportedly-green log I was downloading while Verizon was sending me a text saying I was at 75% of my bandwidth cap, test_value_cloning.html took 17742ms of a 421568ms run, which doesn't seem like it would give it overwhelming odds of being the innocent bystander, though it's certainly possible that 4% of the time plus the location within the run does.

Amusingly, I wouldn't call that a green run, because it hits "Output exceeded 52428800 bytes, remaining output has been truncated" during shutdown, a couple hundred domwindows before the end, and I have a vague memory of that coming up the last time we were playing the blame game about this failure pattern.
So bug 660398 was intended to speed up test_value_cloning.html a *lot* -- but it did so at the cost of more memory usage, I think.  Maybe it's too much for some of these slaves?
Maybe, but it's not unreasonable to bet on the sheer size of the log being the problem. There's variation between platforms, and opt/PGO/debug, but going by the gzipped sizes, nothing but mochitest-other even comes close to the size of mochitest-4 logs, and generally they are three to six times bigger than all the other logs. Apparently buildbot felt some need to cut off logs at 50MB, since even that green M4 did get cut off there - perhaps that's because more than that (and sometimes a bit less than that) will hang the buildbot slave.

Assuming we really do need the logorrhea (see what I did there?) of those tests, is there some non-ugly way we can split them up? m-4, m-value_cloning, m-value_storage? :)
I tried rerunning the second job in comment #2 on the same slave, looked away at the wrong moment, and found the machine was rebooted and saying 'The system has recovered from a serious error". So that looks more like an OS crash. I have the dmp file if anyone wants to take a look.
https://tbpl.mozilla.org/php/getParsedLog.php?id=7501103&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7501966&tree=Firefox

tbpl's HTMLized versions of those logs M1 are only around 14.7MB, which doesn't do much for my 50MB log theory.
https://tbpl.mozilla.org/php/getParsedLog.php?id=7502048&tree=Mozilla-Inbound - an M1 in test_bug389797.html, a mere 11.7MB in.
(In reply to Nick Thomas [:nthomas] from comment #8)
Hmm, might be a red-herring as it'd only got to webgl tests and might just be from having VNC active while the test was running. 

Second time through I got right through test_fullscreen-api.html without issue. There were extra windows open which were not fullscreen for > 30s, but the task manager was responsive and Firefox memory didn't go above 300M. The slave has 1.7G memory; pretty sure that's 2G for real but munged by boot camp/windows.
And if you're being very careful to not muck up the focus, and limit the set of tests
 python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --test-path=content/html/content/test
then it's hard to see anything wrong with memory or CPU usage. This is all outside of buildbot of course.
Blocks: 703996
https://tbpl.mozilla.org/php/getParsedLog.php?id=7507393&tree=Mozilla-Inbound - M1 test_fullscreen-api.html (though a touch earlier in it than usual)
https://tbpl.mozilla.org/php/getParsedLog.php?id=7511213&tree=Mozilla-Inbound - M1 test_bug430351.html (starting to doubt my "same place in particular tests" diagnosis)
https://tbpl.mozilla.org/php/getParsedLog.php?id=7514099&tree=Firefox - M1 test_textarea_attributes_reflection.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=7559952&tree=Mozilla-Inbound
slave: talos-r3-xp-022
73264 INFO TEST-PASS | /tests/content/html/content/test/test_fullscreen-api.html | Parent should be in full-screen mode
https://tbpl.mozilla.org/php/getParsedLog.php?id=7572351&tree=Firefox
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 on 2011-11-24 11:01:51 PST for push 84117219ded0
slave: talos-r3-xp-059
content/html/content/test/test_fullscreen-api.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=7572966&tree=Firefox
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 on 2011-11-24 11:49:51 PST for push 84117219ded0
slave: talos-r3-xp-020
content/html/content/test/test_fullscreen-api.html
https://tbpl.mozilla.org/?tree=Fx-Team&rev=0a6f4e952a3c
talos-r3-xp-058
test_fullscreen-api.html
Severity: normal → blocker
https://tbpl.mozilla.org/php/getParsedLog.php?id=7625584&tree=Mozilla-Inbound
Summary: Rashes of Windows slaves disconnecting at the same place in particular tests → Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1 and mochitest-4 runs
I made a patch to shrink mochitest-4 log size as a part of fix for bug 627885.
Is it possible to land only that part first? It's already r+'ed.
Yes, and thank you!
(In reply to Nick Thomas [:nthomas] from comment #8)
> I tried rerunning the second job in comment #2 on the same slave, looked
> away at the wrong moment, and found the machine was rebooted and saying 'The
> system has recovered from a serious error". So that looks more like an OS
> crash. I have the dmp file if anyone wants to take a look.

nthomas: can you please post the dmp file somewhere (people.m.o? not sure how big it would be) so we can rope someone into taking a look?
Assignee: nobody → nrthomas
Priority: -- → P1
Whiteboard: [crash][buildslaves]
(In reply to Chris Cooper [:coop] from comment #136)
> nthomas: can you please post the dmp file somewhere (people.m.o? not sure
> how big it would be) so we can rope someone into taking a look?

It's small, http://people.mozilla.com/~nthomas/crash.zip

As I said in comment #11 it may be an artifact of how I was testing, rather than the actual issue. Sorry, but I don't have time to work on this right now.
Assignee: nrthomas → nobody
https://tbpl.mozilla.org/php/getParsedLog.php?id=7691156&tree=Firefox
70825 INFO TEST-PASS | /tests/content/html/content/test/test_bug674558.html | Value unchanged - 5 should equal 5
I've been trying to reproduce this in a staging environment, running on buildbot. I took a changeset that hit this (9ecb0201e044) and pulled a production slave that hit this (talos-r3-xp-016) and the first run was green, the second hit the disconnect at test_fullscreen-api.html and so then I tried to _watch_ the test suite run with RDC which resulted in a few runs that were orange (complete but with some fails due to resolution perhaps?).  Went back to letting buildbot do its thing and have had several green runs.  Not much to go on here, and unsure what else to try.
Attached file dumpchk.exe output — 
Since I knew absolutely nothing about analyzing Windows minidumps before I started Googling, dunno how good the results are from running it on a different machine which lacks the offending dll, but the results of

C:\Program Files\Debugging Tools for Windows>dumpchk.exe C:\crash\Mini112011-02.
dmp -y SRV*c:\symbols*http://msdl.microsoft.com/download/symbols > c:\dmpchk.txt

is "dude, your video driver sucks." Well, literally, "Probably caused by : nv4_disp.dll ( nv4_disp+75be2 )"
(In reply to Phil Ringnalda (:philor) from comment #144)
> Created attachment 578470 [details]
> dumpchk.exe output
> 
> Since I knew absolutely nothing about analyzing Windows minidumps before I
> started Googling, dunno how good the results are from running it on a
> different machine which lacks the offending dll, but the results of
> 
> C:\Program Files\Debugging Tools for Windows>dumpchk.exe
> C:\crash\Mini112011-02.
> dmp -y SRV*c:\symbols*http://msdl.microsoft.com/download/symbols >
> c:\dmpchk.txt
> 
> is "dude, your video driver sucks." Well, literally, "Probably caused by :
> nv4_disp.dll ( nv4_disp+75be2 )"

Not sure if it would help but perhaps we have to change the graphic driver to a less crashy version.
We're re-evaluating which version to upgrade/downgrade the Win7 slaves to bug 702504.

[1] https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Upgrade_NVIDIA_driver
See Also: → 702504
My theory of the day is that there were two separate things, M4's huge log, and M1's whatever. Suggestive of the M1 problem:

with test_fullscreen_api disabled: https://tbpl.mozilla.org/?tree=Try&rev=572683838224

with test_fullscreen_api enabled: https://tbpl.mozilla.org/?tree=Try&rev=daff5a8af30e
Summary: Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1 and mochitest-4 runs → Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1
https://tbpl.mozilla.org/php/getParsedLog.php?id=7745097&tree=Firefox

I should have enabled PGO on try, that fails like crazy.
https://tbpl.mozilla.org/php/getParsedLog.php?id=7754768&tree=Firefox
73266 INFO TEST-PASS | /tests/content/html/content/test/test_fullscreen-api.html | Parent should be in full-screen mode
PGO with test_fullscreen_api disabled: https://tbpl.mozilla.org/?tree=Try&rev=53f088b4c903
PGO with test_fullscreen_api enabled: https://tbpl.mozilla.org/?tree=Try&rev=ec281df69367

I'm a little concerned by how frequently we hit bug 608634 with it disabled, but a simple little test failure is a few thousand times better than a test suite abort, slave OS crash, and a Talos run failure if the slave does Talos next.

The hypothetical person who will hypothetically try to debug the problem will have absolutely no problem triggering it with PGO builds on Try, so I say we stop running it on WinXP until that hypothetical person becomes actual. Particularly, I very strongly suggest we stop running it before the next merge to Aurora so we don't have to endure it for 12 weeks afterward no matter what we do on the trunk.
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=e90ea5280cfc&jobname=winnt%205.1 - pretty sure I'll get the opt M1 to complete before long, though.
Wups, comment 210 caught me not looking, that's not actually this.
https://tbpl.mozilla.org/php/getParsedLog.php?id=7774505&tree=Mozilla-Inbound

Starting to wonder if maybe bug 676349 exacerbated this: it took me 6 runs on that push to get a green, and only 1 of 12 runs on the next push was green.
(In reply to Phil Ringnalda (:philor) from comment #240)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=7774505&tree=Mozilla-Inbound
> 
> Starting to wonder if maybe bug 676349 exacerbated this: it took me 6 runs
> on that push to get a green, and only 1 of 12 runs on the next push was
> green.

You could try pushing a backout of bug 676349 to Try to test that theory.
In about 8 hours I'll be where I have a tree, and I can. Or, maybe I can pin the inbound Win PGO build bustage on it, and get it backed out, quicker than that :)
I'm nothing but a bundle of suspicions, but I'm suspicious of how rather than continuing to be awful as long as bug 676349 was in the tree, the frequency tapered off pretty much in parallel with the tapering off of phx's network troubles.
https://tbpl.mozilla.org/php/getParsedLog.php?id=7828157&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7828585&tree=Firefox

I tried disabling subtests, but wasn't able to find just one that we could disable rather than disabling the whole set. I realized afterward that I did it the wrong way, removing another one each try when instead I should have done eight pushes each running just a single subtest, but we're getting such a backlog of WinXP test jobs because we have to keep retriggering M1 because of this, and we have to keep retriggering Talos when a slave affected by this hits a Talos job on its next run, that I think we need to disable the whole thing, get ourselves unbroken, and then test that when we're in less desperate conditions.
Wups, comment 345 was a misstar.
Assignee: nobody → catlee
https://hg.mozilla.org/integration/mozilla-inbound/rev/1cdb116ae0b9

Disabled test_fullscreen-api on WinXP. When this bug is fixed, we should backout this changeset to re-enable the test on WinXP. We do want to be running this test on WinXP, I'm just sick of bugspam emails.
Even if someone can somehow find a less crashy driver that still meets all our other needs from a driver (whatever they may be), someone still checked something in to inbound between say November 18th and 20th, which turned this test which had been around long before that into something which crashes a driver which has been around since last February.
Assignee: catlee → nobody
Severity: blocker → critical
Component: Release Engineering → DOM: Core & HTML
Priority: P1 → --
Product: mozilla.org → Core
QA Contact: release → general
Version: other → Trunk
(In reply to Chris Pearce (:cpearce) (Mozilla Corporation) from comment #348)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/1cdb116ae0b9
> 
> Disabled test_fullscreen-api on WinXP. When this bug is fixed, we should
> backout this changeset to re-enable the test on WinXP. We do want to be
> running this test on WinXP, I'm just sick of bugspam emails.

https://hg.mozilla.org/mozilla-central/rev/1cdb116ae0b9
I've spent some time bisecting this on try, and it looks like https://hg.mozilla.org/integration/mozilla-inbound/rev/03fcc784d866 is where the failure was introduced.
That's a merge changeset, which is not super useful. :-/ There are definitely a bunch of GL-related changes encompassed in that merge, though.
Disconnect in a different test; I don't know if this was a crash or a genuine infra error.

https://tbpl.mozilla.org/php/getParsedLog.php?id=7903392&tree=Firefox
Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5 on 2011-12-13 06:41:45 PST for push 00e184077825
90773 INFO TEST-PASS | /tests/layout/style/test/test_value_computation.html | should not get initial value for 'content:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAAKElEQVR42u3NQQ0AAAgEoNP+nTWFDzcoQE1udQQCgUAgEAgEAsGTYAGjxAE/G/Q2tQAAAABJRU5ErkJggg==)' on elementn. - url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAAKElEQVR42u3NQQ0AAAgEoNP+nTWFDzcoQE1udQQCgUAgEAgEAsGTYAGjxAE/G/Q2tQAAAABJRU5ErkJggg==") should not equal none

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]
[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]
Blocks: 710942
Keywords: crash
Whiteboard: [crash][buildslaves] → [buildslaves]
https://tbpl.mozilla.org/php/getParsedLog.php?id=9306269&tree=Firefox

The new flavor would be:

47270 INFO TEST-PASS | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Successful test page (URL: conformance/more/conformance/quickCheckAPI-B1.html)
WebGL test page successful: conformance/more/conformance/quickCheckAPI-B1.html
WebGL mochitest: starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9323545&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B3.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9332815&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9338617&tree=Firefox - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9341188&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9352318&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9359355&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9368866&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9405490&tree=Firefox - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9388468&tree=Fx-Team - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9469753&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B3.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9481769&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9486858&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9491416&tree=Firefox - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9498859&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9512272&tree=Profiling - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9529632&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9544421&tree=Profiling - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9550700&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9553331&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9578850&tree=Maple - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9615819&tree=Mozilla-Inbound - starting page conformance/more/functions/bufferSubDataBadArgs.html (and getting part way through it)
https://tbpl.mozilla.org/php/getParsedLog.php?id=9617078&tree=Maple - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9624969&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9643901&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9656686&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9662707&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9696295&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9711069&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9682882&tree=Fx-Team - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9713144&tree=Fx-Team - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9726765&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9734945&tree=Mozilla-Inbound - starting page conformance/more/functions/copyTexImage2DBadArgs.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9744579&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=9748252&tree=Mozilla-Inbound - quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9749986&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9752157&tree=Mozilla-Inbound
conformance/more/conformance/quickCheckAPI-C.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9773100&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9766551&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9792180&tree=Mozilla-Inbound - conformance/more/functions/copyTexImage2D.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9801928&tree=Mozilla-Inbound - conformance/more/functions/copyTexImage2D.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9804679&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9804679&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9807864&tree=Firefox - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9813312&tree=Firefox - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9816248&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
 - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9843596&tree=Mozilla-Inbound - conformance/more/functions/deleteBufferBadArgs.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9869936&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9858303&tree=Mozilla-Inbound - conformance/more/functions/drawArrays.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9885306&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9893147&tree=Mozilla-Inbound - conformance/more/functions/copyTexSubImage2DBadArgs.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9895526&tree=Mozilla-Inbound - conformance/more/functions/copyTexSubImage2DBadArgs.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9899812&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9899694&tree=Mozilla-Inbound - conformance/more/functions/deleteBufferBadArgs.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9961160&tree=Mozilla-Inbound - conformance/more/functions/deleteBufferBadArgs.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9965204&tree=Firefox - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9976476&tree=Mozilla-Inbound
conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9976627&tree=Services-Central - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9961915&tree=Fx-Team - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest)
https://tbpl.mozilla.org/php/getParsedLog.php?id=9934518&tree=Fx-Team - conformance/more/functions/drawArrays.html] (WebGL mochitest)
https://tbpl.mozilla.org/php/getParsedLog.php?id=10003352&tree=Maple - conformance/more/functions/drawArrays.html] (WebGL mochitest)
This seems to have stopped around March 10. Let's call it WFM.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.