Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1

RESOLVED WORKSFORME

Status

()

--
critical
RESOLVED WORKSFORME
7 years ago
7 years ago

People

(Reporter: philor, Unassigned)

Tracking

({crash})

Trunk
x86
Windows XP
crash
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildslaves])

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
We discussed this once before, about one of the brutal CSS tests, and came to the inevitable conclusion that it's the fault of the test for doing something brutal to the slave, but the test author can't be at fault because he has no way of knowing what brutal thing he is doing, and releng can't say because it happened in the past, and why don't you just ignore purple anyway?

https://tbpl.mozilla.org/php/getParsedLog.php?id=7498672&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7496770&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7495812&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7495914&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=7495292&tree=Mozilla-Inbound

... WinXP mochitest-1, both debug and PGO, disconnecting in the midst of test_fullscreen-api.html.

https://tbpl.mozilla.org/php/getParsedLog.php?id=7491088&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7489527&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7482717&tree=Firefox

... Win7 mochitest-4, both debug and PGO, disconnecting in the midst of test_value_cloning.html (one or more of those, or some other recent rash, might have been in one of the other exhaustive CSS tests, not sure and since opening mochitest-4 full logs is cash-out-of-pocket on a capped connection, not looking again to see).
(Reporter)

Comment 1

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7485935&tree=Mozilla-Aurora - Win7 debug mochitest-4, in test_value_storage.html

Comment 2

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7501103&tree=Firefox
Rev3 WINNT 5.1 mozilla-central opt test mochitests-1/5
test_fullscreen-api.html

https://tbpl.mozilla.org/php/getParsedLog.php?id=7501160&tree=Firefox
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5
test_fullscreen-api.html
Are you sure it's not that the disconnect happens to be in the middle of one of the longer-running tests because that test is longer-running (and therefore takes a nontrivial fraction of the total time) rather than because it does something to cause the disconnect?
There is a heartbeat mechanism between slave and master which continues regardless of test execution. But if a test/firefox pushes the slave deep into swap it's possible that buildbot doesn't get any CPU time to maintain that, and the job gets interrupted.

(In reply to Ed Morley [:edmorley] from comment #2)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=7501160&tree=Firefox
> Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5
> test_fullscreen-api.html

Here's some logging from the master, which I'm reading as the master losing contact with the slave:

2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] BuildSlave.detached(talos-r3-xp-013)
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] <Build Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5>.lostRemote
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121]  stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2aaaf9c5bb50>
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] addCompleteLog(interrupt)
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=firefox/firefox.exe', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-pat
h=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '1', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.
error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
        ]
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] RemoteCommand.disconnect: lost slave
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2aaaf9c5bb50>): []
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121]  step 'mochitest-plain-1' complete: exception
2011-11-20 15:17:09-0800 [Broker,64988,10.12.50.121]  <Build Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5>: build finished
(Reporter)

Comment 5

7 years ago
In https://tbpl.mozilla.org/php/getParsedLog.php?id=7501113&tree=Firefox&full=1, the purportedly-green log I was downloading while Verizon was sending me a text saying I was at 75% of my bandwidth cap, test_value_cloning.html took 17742ms of a 421568ms run, which doesn't seem like it would give it overwhelming odds of being the innocent bystander, though it's certainly possible that 4% of the time plus the location within the run does.

Amusingly, I wouldn't call that a green run, because it hits "Output exceeded 52428800 bytes, remaining output has been truncated" during shutdown, a couple hundred domwindows before the end, and I have a vague memory of that coming up the last time we were playing the blame game about this failure pattern.
So bug 660398 was intended to speed up test_value_cloning.html a *lot* -- but it did so at the cost of more memory usage, I think.  Maybe it's too much for some of these slaves?
(Reporter)

Comment 7

7 years ago
Maybe, but it's not unreasonable to bet on the sheer size of the log being the problem. There's variation between platforms, and opt/PGO/debug, but going by the gzipped sizes, nothing but mochitest-other even comes close to the size of mochitest-4 logs, and generally they are three to six times bigger than all the other logs. Apparently buildbot felt some need to cut off logs at 50MB, since even that green M4 did get cut off there - perhaps that's because more than that (and sometimes a bit less than that) will hang the buildbot slave.

Assuming we really do need the logorrhea (see what I did there?) of those tests, is there some non-ugly way we can split them up? m-4, m-value_cloning, m-value_storage? :)
I tried rerunning the second job in comment #2 on the same slave, looked away at the wrong moment, and found the machine was rebooted and saying 'The system has recovered from a serious error". So that looks more like an OS crash. I have the dmp file if anyone wants to take a look.
(Reporter)

Comment 9

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7501103&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7501966&tree=Firefox

tbpl's HTMLized versions of those logs M1 are only around 14.7MB, which doesn't do much for my 50MB log theory.
(Reporter)

Comment 10

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7502048&tree=Mozilla-Inbound - an M1 in test_bug389797.html, a mere 11.7MB in.
(In reply to Nick Thomas [:nthomas] from comment #8)
Hmm, might be a red-herring as it'd only got to webgl tests and might just be from having VNC active while the test was running. 

Second time through I got right through test_fullscreen-api.html without issue. There were extra windows open which were not fullscreen for > 30s, but the task manager was responsive and Firefox memory didn't go above 300M. The slave has 1.7G memory; pretty sure that's 2G for real but munged by boot camp/windows.
And if you're being very careful to not muck up the focus, and limit the set of tests
 python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --test-path=content/html/content/test
then it's hard to see anything wrong with memory or CPU usage. This is all outside of buildbot of course.
(Reporter)

Updated

7 years ago
Blocks: 703996
(Reporter)

Comment 18

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7507393&tree=Mozilla-Inbound - M1 test_fullscreen-api.html (though a touch earlier in it than usual)
(Reporter)

Comment 19

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7511213&tree=Mozilla-Inbound - M1 test_bug430351.html (starting to doubt my "same place in particular tests" diagnosis)
(Reporter)

Comment 25

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7514099&tree=Firefox - M1 test_textarea_attributes_reflection.html
(Reporter)

Comment 36

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7517708&tree=Fx-Team
talos-r3-xp-024
test_fullscreen-api.html
(Reporter)

Comment 37

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7518813&tree=Fx-Team
talos-r3-xp-055
test_fullscreen-api.html
(Reporter)

Comment 40

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7534644&tree=Firefox
talos-r3-xp-033
test_fullscreen-api.html
(Reporter)

Comment 46

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7553414&tree=Firefox
talos-r3-w7-006
test_value_cloning.html
(Reporter)

Comment 47

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7553436&tree=Firefox
talos-r3-xp-019
test_fullscreen-api.html
(Reporter)

Comment 50

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7558462&tree=Firefox
talos-r3-xp-053
test_fullscreen-api.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=7559952&tree=Mozilla-Inbound
slave: talos-r3-xp-022
73264 INFO TEST-PASS | /tests/content/html/content/test/test_fullscreen-api.html | Parent should be in full-screen mode
https://tbpl.mozilla.org/php/getParsedLog.php?id=7572351&tree=Firefox
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 on 2011-11-24 11:01:51 PST for push 84117219ded0
slave: talos-r3-xp-059
content/html/content/test/test_fullscreen-api.html
https://tbpl.mozilla.org/php/getParsedLog.php?id=7572966&tree=Firefox
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 on 2011-11-24 11:49:51 PST for push 84117219ded0
slave: talos-r3-xp-020
content/html/content/test/test_fullscreen-api.html
(Reporter)

Comment 61

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7578401&tree=Firefox
talos-r3-w7-005
test_value_cloning.html
(Reporter)

Comment 62

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7582688&tree=Firefox
talos-r3-w7-047
test_value_cloning.html
(Reporter)

Comment 68

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7591340&tree=Firefox
talos-r3-xp-059
test_fullscreen-api.html
(Reporter)

Comment 78

7 years ago
https://tbpl.mozilla.org/?tree=Fx-Team&rev=0a6f4e952a3c
talos-r3-xp-058
test_fullscreen-api.html
(Reporter)

Updated

7 years ago
Severity: normal → blocker
(Reporter)

Comment 86

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7625584&tree=Mozilla-Inbound
Summary: Rashes of Windows slaves disconnecting at the same place in particular tests → Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1 and mochitest-4 runs
I made a patch to shrink mochitest-4 log size as a part of fix for bug 627885.
Is it possible to land only that part first? It's already r+'ed.
(Reporter)

Comment 88

7 years ago
Yes, and thank you!
(In reply to Nick Thomas [:nthomas] from comment #8)
> I tried rerunning the second job in comment #2 on the same slave, looked
> away at the wrong moment, and found the machine was rebooted and saying 'The
> system has recovered from a serious error". So that looks more like an OS
> crash. I have the dmp file if anyone wants to take a look.

nthomas: can you please post the dmp file somewhere (people.m.o? not sure how big it would be) so we can rope someone into taking a look?
Assignee: nobody → nrthomas
Priority: -- → P1
Whiteboard: [crash][buildslaves]
(In reply to Chris Cooper [:coop] from comment #136)
> nthomas: can you please post the dmp file somewhere (people.m.o? not sure
> how big it would be) so we can rope someone into taking a look?

It's small, http://people.mozilla.com/~nthomas/crash.zip

As I said in comment #11 it may be an artifact of how I was testing, rather than the actual issue. Sorry, but I don't have time to work on this right now.
Assignee: nrthomas → nobody
https://tbpl.mozilla.org/php/getParsedLog.php?id=7691156&tree=Firefox
70825 INFO TEST-PASS | /tests/content/html/content/test/test_bug674558.html | Value unchanged - 5 should equal 5
I've been trying to reproduce this in a staging environment, running on buildbot. I took a changeset that hit this (9ecb0201e044) and pulled a production slave that hit this (talos-r3-xp-016) and the first run was green, the second hit the disconnect at test_fullscreen-api.html and so then I tried to _watch_ the test suite run with RDC which resulted in a few runs that were orange (complete but with some fails due to resolution perhaps?).  Went back to letting buildbot do its thing and have had several green runs.  Not much to go on here, and unsure what else to try.
(Reporter)

Comment 144

7 years ago
Created attachment 578470 [details]
dumpchk.exe output

Since I knew absolutely nothing about analyzing Windows minidumps before I started Googling, dunno how good the results are from running it on a different machine which lacks the offending dll, but the results of

C:\Program Files\Debugging Tools for Windows>dumpchk.exe C:\crash\Mini112011-02.
dmp -y SRV*c:\symbols*http://msdl.microsoft.com/download/symbols > c:\dmpchk.txt

is "dude, your video driver sucks." Well, literally, "Probably caused by : nv4_disp.dll ( nv4_disp+75be2 )"

Comment 150

7 years ago
(In reply to Phil Ringnalda (:philor) from comment #144)
> Created attachment 578470 [details]
> dumpchk.exe output
> 
> Since I knew absolutely nothing about analyzing Windows minidumps before I
> started Googling, dunno how good the results are from running it on a
> different machine which lacks the offending dll, but the results of
> 
> C:\Program Files\Debugging Tools for Windows>dumpchk.exe
> C:\crash\Mini112011-02.
> dmp -y SRV*c:\symbols*http://msdl.microsoft.com/download/symbols >
> c:\dmpchk.txt
> 
> is "dude, your video driver sucks." Well, literally, "Probably caused by :
> nv4_disp.dll ( nv4_disp+75be2 )"

Not sure if it would help but perhaps we have to change the graphic driver to a less crashy version.
We're re-evaluating which version to upgrade/downgrade the Win7 slaves to bug 702504.

[1] https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Upgrade_NVIDIA_driver
See Also: → bug 702504
(Reporter)

Comment 183

7 years ago
My theory of the day is that there were two separate things, M4's huge log, and M1's whatever. Suggestive of the M1 problem:

with test_fullscreen_api disabled: https://tbpl.mozilla.org/?tree=Try&rev=572683838224

with test_fullscreen_api enabled: https://tbpl.mozilla.org/?tree=Try&rev=daff5a8af30e
Summary: Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1 and mochitest-4 runs → Windows slaves frequently crash (looks like a disconnect in the log) during mochitest-1
(Reporter)

Comment 188

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7745097&tree=Firefox

I should have enabled PGO on try, that fails like crazy.
https://tbpl.mozilla.org/php/getParsedLog.php?id=7754768&tree=Firefox
73266 INFO TEST-PASS | /tests/content/html/content/test/test_fullscreen-api.html | Parent should be in full-screen mode
(Reporter)

Comment 205

7 years ago
PGO with test_fullscreen_api disabled: https://tbpl.mozilla.org/?tree=Try&rev=53f088b4c903
PGO with test_fullscreen_api enabled: https://tbpl.mozilla.org/?tree=Try&rev=ec281df69367

I'm a little concerned by how frequently we hit bug 608634 with it disabled, but a simple little test failure is a few thousand times better than a test suite abort, slave OS crash, and a Talos run failure if the slave does Talos next.

The hypothetical person who will hypothetically try to debug the problem will have absolutely no problem triggering it with PGO builds on Try, so I say we stop running it on WinXP until that hypothetical person becomes actual. Particularly, I very strongly suggest we stop running it before the next merge to Aurora so we don't have to endure it for 12 weeks afterward no matter what we do on the trunk.
(Reporter)

Comment 208

7 years ago
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=e90ea5280cfc&jobname=winnt%205.1 - pretty sure I'll get the opt M1 to complete before long, though.
(Reporter)

Comment 211

7 years ago
Wups, comment 210 caught me not looking, that's not actually this.
(Reporter)

Comment 240

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7774505&tree=Mozilla-Inbound

Starting to wonder if maybe bug 676349 exacerbated this: it took me 6 runs on that push to get a green, and only 1 of 12 runs on the next push was green.
(In reply to Phil Ringnalda (:philor) from comment #240)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=7774505&tree=Mozilla-Inbound
> 
> Starting to wonder if maybe bug 676349 exacerbated this: it took me 6 runs
> on that push to get a green, and only 1 of 12 runs on the next push was
> green.

You could try pushing a backout of bug 676349 to Try to test that theory.
(Reporter)

Comment 246

7 years ago
In about 8 hours I'll be where I have a tree, and I can. Or, maybe I can pin the inbound Win PGO build bustage on it, and get it backed out, quicker than that :)
(Reporter)

Comment 261

7 years ago
I'm nothing but a bundle of suspicions, but I'm suspicious of how rather than continuing to be awful as long as bug 676349 was in the tree, the frequency tapered off pretty much in parallel with the tapering off of phx's network troubles.
(Reporter)

Comment 329

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=7828157&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=7828585&tree=Firefox

I tried disabling subtests, but wasn't able to find just one that we could disable rather than disabling the whole set. I realized afterward that I did it the wrong way, removing another one each try when instead I should have done eight pushes each running just a single subtest, but we're getting such a backlog of WinXP test jobs because we have to keep retriggering M1 because of this, and we have to keep retriggering Talos when a slave affected by this hits a Talos job on its next run, that I think we need to disable the whole thing, get ourselves unbroken, and then test that when we're in less desperate conditions.
(Reporter)

Comment 346

7 years ago
Wups, comment 345 was a misstar.

Updated

7 years ago
Assignee: nobody → catlee
https://hg.mozilla.org/integration/mozilla-inbound/rev/1cdb116ae0b9

Disabled test_fullscreen-api on WinXP. When this bug is fixed, we should backout this changeset to re-enable the test on WinXP. We do want to be running this test on WinXP, I'm just sick of bugspam emails.
(Reporter)

Comment 349

7 years ago
Even if someone can somehow find a less crashy driver that still meets all our other needs from a driver (whatever they may be), someone still checked something in to inbound between say November 18th and 20th, which turned this test which had been around long before that into something which crashes a driver which has been around since last February.
Assignee: catlee → nobody
Severity: blocker → critical
Component: Release Engineering → DOM: Core & HTML
Priority: P1 → --
Product: mozilla.org → Core
QA Contact: release → general
Version: other → Trunk
(In reply to Chris Pearce (:cpearce) (Mozilla Corporation) from comment #348)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/1cdb116ae0b9
> 
> Disabled test_fullscreen-api on WinXP. When this bug is fixed, we should
> backout this changeset to re-enable the test on WinXP. We do want to be
> running this test on WinXP, I'm just sick of bugspam emails.

https://hg.mozilla.org/mozilla-central/rev/1cdb116ae0b9
I've spent some time bisecting this on try, and it looks like https://hg.mozilla.org/integration/mozilla-inbound/rev/03fcc784d866 is where the failure was introduced.
That's a merge changeset, which is not super useful. :-/ There are definitely a bunch of GL-related changes encompassed in that merge, though.
Disconnect in a different test; I don't know if this was a crash or a genuine infra error.

https://tbpl.mozilla.org/php/getParsedLog.php?id=7903392&tree=Firefox
Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5 on 2011-12-13 06:41:45 PST for push 00e184077825
90773 INFO TEST-PASS | /tests/layout/style/test/test_value_computation.html | should not get initial value for 'content:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAAKElEQVR42u3NQQ0AAAgEoNP+nTWFDzcoQE1udQQCgUAgEAgEAsGTYAGjxAE/G/Q2tQAAAABJRU5ErkJggg==)' on elementn. - url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAAKElEQVR42u3NQQ0AAAgEoNP+nTWFDzcoQE1udQQCgUAgEAgEAsGTYAGjxAE/G/Q2tQAAAABJRU5ErkJggg==") should not equal none

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]
[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]
Blocks: 710942

Updated

7 years ago
Keywords: crash
Whiteboard: [crash][buildslaves] → [buildslaves]
(Reporter)

Comment 356

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9306269&tree=Firefox

The new flavor would be:

47270 INFO TEST-PASS | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Successful test page (URL: conformance/more/conformance/quickCheckAPI-B1.html)
WebGL test page successful: conformance/more/conformance/quickCheckAPI-B1.html
WebGL mochitest: starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 357

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9323545&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B3.html
(Reporter)

Comment 358

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9332815&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 359

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9338617&tree=Firefox - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 360

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9341188&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 361

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9352318&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 362

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9359355&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 363

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9368866&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 364

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9405490&tree=Firefox - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 365

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9388468&tree=Fx-Team - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 366

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9469753&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B3.html
(Reporter)

Comment 367

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9481769&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 368

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9486858&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 369

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9491416&tree=Firefox - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 370

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9498859&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 371

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9512272&tree=Profiling - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 372

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9529632&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 373

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9544421&tree=Profiling - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 374

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9550700&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 375

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9553331&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 376

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9578850&tree=Maple - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 378

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9615819&tree=Mozilla-Inbound - starting page conformance/more/functions/bufferSubDataBadArgs.html (and getting part way through it)
(Reporter)

Comment 379

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9617078&tree=Maple - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 380

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9624969&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 381

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9643901&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 382

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9656686&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 383

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9662707&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 384

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9696295&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 385

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9711069&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 386

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9682882&tree=Fx-Team - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 387

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9713144&tree=Fx-Team - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 388

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9726765&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 389

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9734945&tree=Mozilla-Inbound - starting page conformance/more/functions/copyTexImage2DBadArgs.html
(Reporter)

Comment 390

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9744579&tree=Mozilla-Inbound - starting page conformance/more/conformance/quickCheckAPI-B2.html
(Reporter)

Comment 391

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9748252&tree=Mozilla-Inbound - quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 392

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9749986&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9752157&tree=Mozilla-Inbound
conformance/more/conformance/quickCheckAPI-C.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 394

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9773100&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 395

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9766551&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 396

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9792180&tree=Mozilla-Inbound - conformance/more/functions/copyTexImage2D.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 397

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9801928&tree=Mozilla-Inbound - conformance/more/functions/copyTexImage2D.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9804679&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 399

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9804679&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 400

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9807864&tree=Firefox - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 401

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9813312&tree=Firefox - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 402

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9816248&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 404

7 years ago
 - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 405

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9843596&tree=Mozilla-Inbound - conformance/more/functions/deleteBufferBadArgs.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 406

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9869936&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 407

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9858303&tree=Mozilla-Inbound - conformance/more/functions/drawArrays.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 408

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9885306&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 409

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9893147&tree=Mozilla-Inbound - conformance/more/functions/copyTexSubImage2DBadArgs.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 410

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9895526&tree=Mozilla-Inbound - conformance/more/functions/copyTexSubImage2DBadArgs.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 411

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9899812&tree=Mozilla-Inbound - conformance/more/conformance/quickCheckAPI-B3.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 412

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9899694&tree=Mozilla-Inbound - conformance/more/functions/deleteBufferBadArgs.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 413

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9961160&tree=Mozilla-Inbound - conformance/more/functions/deleteBufferBadArgs.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 414

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9965204&tree=Firefox - conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
https://tbpl.mozilla.org/php/getParsedLog.php?id=9976476&tree=Mozilla-Inbound
conformance/more/conformance/quickCheckAPI-B2.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 416

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9976627&tree=Services-Central - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest) Starting test page
(Reporter)

Comment 417

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9961915&tree=Fx-Team - conformance/more/conformance/quickCheckAPI-D_G.html] (WebGL mochitest)
(Reporter)

Comment 418

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=9934518&tree=Fx-Team - conformance/more/functions/drawArrays.html] (WebGL mochitest)
(Reporter)

Comment 419

7 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=10003352&tree=Maple - conformance/more/functions/drawArrays.html] (WebGL mochitest)
This seems to have stopped around March 10. Let's call it WFM.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.