Intermittent crashtests/495875-1.html | application timed out after 330 seconds with no output followed by a spiral out of control

RESOLVED WORKSFORME

Status

()

RESOLVED WORKSFORME
8 years ago
6 years ago

People

(Reporter: philor, Unassigned)

Tracking

({intermittent-failure})

Trunk
mozilla7
x86
Windows 7
intermittent-failure
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

8 years ago
Not sure if this is really two (or three) separate things (crashtest timeout, minidump_stackwalk failure, buildbot failure to recover from that), or just one:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1292860409.1292861360.12942.gz
Rev3 WINNT 6.1 mozilla-central debug test crashtest on 2010/12/20 07:53:29
s: talos-r3-w7-023

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1292886810.1292887911.30068.gz
Rev3 WINNT 6.1 mozilla-central debug test crashtest on 2010/12/20 15:13:30
s: talos-r3-w7-025

REFTEST TEST-START | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/495875-1.html
++DOMWINDOW == 39 (0B163D90) [serial = 2586] [outer = 07E0FF28]
TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/495875-1.html | application timed out after 330 seconds with no output
INFO | automation.py | Application ran for: 0:12:44.692000
INFO | automation.py | Reading PID log: c:\users\cltbld\appdata\local\temp\tmp6laopwpidlog
PROCESS-CRASH | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/495875-1.html | application crashed (minidump found)
stderr from minidump_stackwalk:
2010-12-20 08:09:01: minidump_processor.cc:264: INFO: Processing minidump in file c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp
cygwin warning:
  MS-DOS style path detected: c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp
  Preferred POSIX equivalent is: /cygdrive/c/users/cltbld/appdata/local/temp/tmpx-n_r6/minidumps/01ea95a3-3824-401a-b385-6413087c90e5.dmp
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
2010-12-20 08:09:01: minidump.cc:3500: INFO: Minidump opened minidump c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp
2010-12-20 08:09:01: minidump.cc:3762: ERROR: ReadBytes: read 0/32
2010-12-20 08:09:01: minidump.cc:3519: ERROR: Minidump cannot read header
2010-12-20 08:09:01: minidump_processor.cc:268: ERROR: Minidump c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp could not be read
2010-12-20 08:09:01: minidump.cc:3472: INFO: Minidump closing minidump
2010-12-20 08:09:01: minidump_stackwalk.cc:509: ERROR: MinidumpProcessor::Process failed

minidump_stackwalk exited with return code 1
TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks!

REFTEST INFO | runreftest.py | Running tests: end.
Traceback (most recent call last):
  File "reftest/runreftest.py", line 284, in <module>
    main()
  File "reftest/runreftest.py", line 281, in main
    sys.exit(reftest.runTests(args[0], options))
  File "reftest/runreftest.py", line 167, in runTests
    self.cleanup(profileDir)
  File "reftest/runreftest.py", line 135, in cleanup
    shutil.rmtree(profileDir)
  File "C:\mozilla-build\python25\lib\shutil.py", line 169, in rmtree
    
  File "C:\mozilla-build\python25\lib\shutil.py", line 178, in rmtree
    
  File "C:\mozilla-build\python25\lib\shutil.py", line 176, in rmtree
    
WindowsError: [Error 41] The directory is not empty: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpx-n_r6\\extensions'
program finished with exit code 1
Comment hidden (Treeherder Robot)
The minidump_stackwalk spew certainly indicates that we wrote a zero-byte minidump here. Hey, don't we have a bug somewhere on our Win32 code to produce a crash from hung processes not working properly? If so, the resulting failure here is probably the same as there, just with a better error message.
(Reporter)

Comment 3

8 years ago
Unfortunately, if it's connected (wtf, "\\extensions"?), the Python error at the end winds up persuading buildbot to call it an exception, so we happily just ignored the first instance, since purple means not our problem.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Created attachment 541677 [details] [diff] [review]
ignore errors in removing profile directory at end of reftest run

I don't know what the root cause is here, but it looks like shutil.rmtree is failing to remove the profile directory for whatever reason, so I figured we could just work around that and make it not error on failure. The failure mode there would just become "leave a few files in %TEMP%" then.
Attachment #541677 - Flags: review?(philringnalda)
This is just a band-aid around the "spiral out of control", certainly.
Assignee: nobody → ted.mielczarek
(Reporter)

Comment 37

7 years ago
Comment on attachment 541677 [details] [diff] [review]
ignore errors in removing profile directory at end of reftest run

Now that I've seen just how big a wound that spiral is, putting a bandaid on it seems like it can't hurt anything.
Attachment #541677 - Flags: review?(philringnalda) → review+
Whiteboard: [orange] → [orange][inbound]
Comment hidden (Treeherder Robot)
http://hg.mozilla.org/mozilla-central/rev/1df1e9a3057b

I assume the bug should stay open based on comment 36, otherwise please mark it correctly.
Target Milestone: --- → mozilla7
Whiteboard: [orange][inbound] → [orange]
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Assignee: ted.mielczarek → nobody
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Updated

6 years ago
Keywords: intermittent-failure
(Assignee)

Updated

6 years ago
Whiteboard: [orange]
Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.