Closed Bug 620563 Opened 14 years ago Closed 11 years ago

Intermittent crashtests/495875-1.html | application timed out after 330 seconds with no output followed by a spiral out of control

Categories

(Core :: Layout, defect)

Product:
Core
Component:
Layout
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME
Milestone:
mozilla7

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

ignore errors in removing profile directory at end of reftest run
1.24 KB, patch
philor
: review+
Details | Diff | Splinter Review 
Not sure if this is really two (or three) separate things (crashtest timeout, minidump_stackwalk failure, buildbot failure to recover from that), or just one:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1292860409.1292861360.12942.gz
Rev3 WINNT 6.1 mozilla-central debug test crashtest on 2010/12/20 07:53:29
s: talos-r3-w7-023

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1292886810.1292887911.30068.gz
Rev3 WINNT 6.1 mozilla-central debug test crashtest on 2010/12/20 15:13:30
s: talos-r3-w7-025

REFTEST TEST-START | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/495875-1.html
++DOMWINDOW == 39 (0B163D90) [serial = 2586] [outer = 07E0FF28]
TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/495875-1.html | application timed out after 330 seconds with no output
INFO | automation.py | Application ran for: 0:12:44.692000
INFO | automation.py | Reading PID log: c:\users\cltbld\appdata\local\temp\tmp6laopwpidlog
PROCESS-CRASH | file:///c:/talos-slave/test/build/reftest/tests/layout/generic/crashtests/495875-1.html | application crashed (minidump found)
stderr from minidump_stackwalk:
2010-12-20 08:09:01: minidump_processor.cc:264: INFO: Processing minidump in file c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp
cygwin warning:
  MS-DOS style path detected: c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp
  Preferred POSIX equivalent is: /cygdrive/c/users/cltbld/appdata/local/temp/tmpx-n_r6/minidumps/01ea95a3-3824-401a-b385-6413087c90e5.dmp
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
2010-12-20 08:09:01: minidump.cc:3500: INFO: Minidump opened minidump c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp
2010-12-20 08:09:01: minidump.cc:3762: ERROR: ReadBytes: read 0/32
2010-12-20 08:09:01: minidump.cc:3519: ERROR: Minidump cannot read header
2010-12-20 08:09:01: minidump_processor.cc:268: ERROR: Minidump c:\users\cltbld\appdata\local\temp\tmpx-n_r6\minidumps\01ea95a3-3824-401a-b385-6413087c90e5.dmp could not be read
2010-12-20 08:09:01: minidump.cc:3472: INFO: Minidump closing minidump
2010-12-20 08:09:01: minidump_stackwalk.cc:509: ERROR: MinidumpProcessor::Process failed

minidump_stackwalk exited with return code 1
TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks!

REFTEST INFO | runreftest.py | Running tests: end.
Traceback (most recent call last):
  File "reftest/runreftest.py", line 284, in <module>
    main()
  File "reftest/runreftest.py", line 281, in main
    sys.exit(reftest.runTests(args[0], options))
  File "reftest/runreftest.py", line 167, in runTests
    self.cleanup(profileDir)
  File "reftest/runreftest.py", line 135, in cleanup
    shutil.rmtree(profileDir)
  File "C:\mozilla-build\python25\lib\shutil.py", line 169, in rmtree
    
  File "C:\mozilla-build\python25\lib\shutil.py", line 178, in rmtree
    
  File "C:\mozilla-build\python25\lib\shutil.py", line 176, in rmtree
    
WindowsError: [Error 41] The directory is not empty: 'c:\\users\\cltbld\\appdata\\local\\temp\\tmpx-n_r6\\extensions'
program finished with exit code 1
The minidump_stackwalk spew certainly indicates that we wrote a zero-byte minidump here. Hey, don't we have a bug somewhere on our Win32 code to produce a crash from hung processes not working properly? If so, the resulting failure here is probably the same as there, just with a better error message.
Unfortunately, if it's connected (wtf, "\\extensions"?), the Python error at the end winds up persuading buildbot to call it an exception, so we happily just ignored the first instance, since purple means not our problem.
I don't know what the root cause is here, but it looks like shutil.rmtree is failing to remove the profile directory for whatever reason, so I figured we could just work around that and make it not error on failure. The failure mode there would just become "leave a few files in %TEMP%" then.
Attachment #541677 - Flags: review?(philringnalda)
This is just a band-aid around the "spiral out of control", certainly.
Assignee: nobody → ted.mielczarek
Comment on attachment 541677 [details] [diff] [review]
ignore errors in removing profile directory at end of reftest run

Now that I've seen just how big a wound that spiral is, putting a bandaid on it seems like it can't hurt anything.
Attachment #541677 - Flags: review?(philringnalda) → review+
Whiteboard: [orange] → [orange][inbound]
http://hg.mozilla.org/mozilla-central/rev/1df1e9a3057b

I assume the bug should stay open based on comment 36, otherwise please mark it correctly.
Target Milestone: --- → mozilla7
Whiteboard: [orange][inbound] → [orange]
Assignee: ted.mielczarek → nobody
Whiteboard: [orange]
Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: