Closed Bug 1015200 Opened 11 years ago Closed 11 years ago

Frequent B2G emulator reftest timeouts

Categories

(Testing :: Reftest, defect)

Product:
Testing
Component:
Reftest
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(firefox33 unaffected, firefox34 fixed, firefox-esr31 unaffected)

Status:
RESOLVED FIXED
Milestone:
mozilla34
Tracking Flags:
Tracking Status
firefox33 --- unaffected
firefox34 --- fixed
firefox-esr31 --- unaffected

People

(Reporter: emorley, Assigned: ferjm)

References

Details

(Keywords: intermittent-failure)

We're seeing a sudden spike in B2G emulator reftest timeouts. This was blamed on bug 966646, which is now backed out, but I'm still seeing quite a few timeouts - is it perhaps caused by bug 922680?
b2g_emulator_vm mozilla-central opt test reftest-4 on 2014-05-22 18:24:31 PDT for push 2619a4def1b9 https://tbpl.mozilla.org/php/getParsedLog.php?id=40232948&tree=Mozilla-Central b2g_emulator_vm mozilla-central opt test reftest-4 on 2014-05-22 19:10:18 PDT for push e9b2b72f4e6c https://tbpl.mozilla.org/php/getParsedLog.php?id=40236070&tree=Mozilla-Central b2g_emulator_vm mozilla-central opt test reftest-8 on 2014-05-22 19:10:37 PDT for push e9b2b72f4e6c https://tbpl.mozilla.org/php/getParsedLog.php?id=40236447&tree=Mozilla-Central b2g_emulator_vm mozilla-central opt test reftest-14 on 2014-05-22 19:10:21 PDT for push e9b2b72f4e6c https://tbpl.mozilla.org/php/getParsedLog.php?id=40235887&tree=Mozilla-Central b2g_emulator_vm mozilla-central opt test crashtest-3 on 2014-05-22 19:10:36 PDT for push e9b2b72f4e6c https://tbpl.mozilla.org/php/getParsedLog.php?id=40236296&tree=Mozilla-Central
Did bug 922680 get re-landed at some point after comment 118?
No longer blocks: 922680
Oh I missed that backout; in which case I'm out of ideas...
Depends on: 1015178
I wasn't aware we weren't getting stacks on emulator reftest timeouts. The patch in bug 1015178 should fix that.
Depends on: 1015920
https://tbpl.mozilla.org/php/getParsedLog.php?id=40377085&tree=Mozilla-Central https://tbpl.mozilla.org/php/getParsedLog.php?id=40368864&tree=Mozilla-Central https://tbpl.mozilla.org/php/getParsedLog.php?id=40413596&tree=Mozilla-Central https://tbpl.mozilla.org/php/getParsedLog.php?id=40428005&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40426721&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40426251&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40426147&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40429666&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40422776&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40422937&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40420762&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40415383&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40413370&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40413383&tree=Mozilla-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40391083&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40426806&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40427308&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40428187&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40427318&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40427390&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40422955&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40420201&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40430253&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40429993&tree=B2g-Inbound https://tbpl.mozilla.org/php/getParsedLog.php?id=40430868&tree=Fx-Team https://tbpl.mozilla.org/php/getParsedLog.php?id=40429949&tree=Fx-Team https://tbpl.mozilla.org/php/getParsedLog.php?id=40430679&tree=Fx-Team https://tbpl.mozilla.org/php/getParsedLog.php?id=40426032&tree=Fx-Team https://tbpl.mozilla.org/php/getParsedLog.php?id=40424930&tree=Fx-Team https://tbpl.mozilla.org/php/getParsedLog.php?id=40423961&tree=Fx-Team https://tbpl.mozilla.org/php/getParsedLog.php?id=40423060&tree=Fx-Team https://tbpl.mozilla.org/php/getParsedLog.php?id=40413406&tree=Fx-Team
Depends on: 1016155
Depends on: 1016158
Depends on: 1016160
Just in case it got missed in here, bug 1015178 got fixed which means these have stacks now (from killAndGetStack).
So far, I only see a stack in the very last report: 19:09:04 INFO - Thread 0 (crashed) 19:09:04 INFO - 0 libc.so + 0xc31c 19:09:04 INFO - 1 libxul.so!js::gc::SystemPageAllocator::GetPageFaultCount() [Memory.cpp:ef053c67aac2 : 616 + 0x7] 19:09:04 INFO - 2 libxul.so!js::gcstats::Statistics::endSlice() [Statistics.cpp:ef053c67aac2 : 594 + 19:09:04 INFO - 3 libxul.so!js::gc::GCRuntime::collect(bool, long long, js::JSGCInvocationKind, JS::gcreason::Reason) [Statistics.h:ef053c67aac2 : 184 + 0x5] 19:09:04 INFO - 4 libxul.so!js::GCSlice [jsgc.cpp:ef053c67aac2 : 4947 + 0xb] 19:09:04 INFO - 5 libxul.so!JS::IncrementalGC(JSRuntime*, JS::gcreason::Reason, long long) [jsfriendapi.cpp:ef053c67aac2 : 208 + 0xb]
That code was refactored in bug 988486, which landed on inbound the same day this was filed. Though, it looks like it didn't hit m-c until 3 days later, and these timeouts were happening on 5/23 on m-c, so it can't be that...
So this appears to have started again on 7/24 on inbound shortly after a b2g-inbound merge (https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=b2g_emulator_vm.*crashtest&fromchange=9c3d8f8b46f7&tochange=0a9922138347). And thankfully, the first instance on b2g-inbound is in close proximity to the csets involved in that merge. In that merge, mwu's push looks maybe-suspicious. I'll try retriggering on b-i to try to confirm.
Actually, the retriggers are pointing at bug 903291. More retriggers running on the push prior to confirm. https://tbpl.mozilla.org/?tree=B2g-Inbound&jobname=b2g_emulator_vm.*crashtest&fromchange=30907d52c4c2&tochange=81f96f02263f
Retriggers are showing a ~10% failure rate. I plan to backout the responsible patch once confirmed as this is too high to meet visibility standards.
Backing out bug 903291. Hopefully we'll see these taper off once that merges around.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Resolution: FIXED → WORKSFORME
Assignee: nobody → ferjmoreno
Blocks: 903291
status-firefox33: --- → unaffected
status-firefox34: --- → fixed
status-firefox-esr31: --- → unaffected
Resolution: WORKSFORME → FIXED
Target Milestone: --- → mozilla34
You need to log in before you can comment on or make changes to this bug.