Closed Bug 537147 Opened 16 years ago Closed 15 years ago

Talos orange: "failed to initialize browser"; "sh: line 1:" ... "Terminated"

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cjones, Assigned: anodelman)

References

Details

(Keywords: intermittent-failure, Whiteboard: [talos])

Attachments

(2 files, 1 obsolete file)

[checked in]fix initalization woes on leopard
7.89 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
increase timeout, better error messages
1.51 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
See http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262104680.1262105624.16358.gz&fulltext=1 I can't tell why this build is going orange; the log is not very helpful. The most useful output seems to be Running test tp4: Started Tue, 29 Dec 2009 08:43:25 NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/641 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 235 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpvuhb44/profile getInfo.html > browser_output.txt Failed tp4: Stopped Tue, 29 Dec 2009 08:53:35 FAIL: Busted: tp4 FAIL: failed to initialize browser Completed test tp4: Stopped Tue, 29 Dec 2009 08:53:35 RETURN: cycle time: 00:10:10<br> I heard on IRC that this was a known problem, but for the life of me I can't find a bug on file. Apologies if this is a dupe.
(In reply to comment #0) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262104680.1262105624.16358.gz&fulltext=1 MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2009/12/29 08:38:00 (In reply to comment #1) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262130241.1262133368.4749.gz&fulltext=1 WINNT 6.0 mozilla-central talos dirty on 2009/12/29 15:44:01 http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262265032.1262269401.26182.gz MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2009/12/31 05:10:32 Running test ts_places_generated_max: Started Thu, 31 Dec 2009 06:01:17 NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/641 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 1654 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpqcBQ2d/profile getInfo.html > browser_output.txt Failed ts_places_generated_max: Stopped Thu, 31 Dec 2009 06:11:38 FAIL: Busted: ts_places_generated_max FAIL: failed to initialize browser Completed test ts_places_generated_max: Stopped Thu, 31 Dec 2009 06:11:38
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262246530.1262250346.799.gz MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2009/12/31 00:02:10 (who knows what's in the log - thanks to bug 535084 I'm not really willing to look)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262436838.1262437754.26036.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/02 04:53:58 sh: line 1: 225 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpasjQQF/profile getInfo.html > browser_output.txt Failed tp4: Stopped Sat, 02 Jan 2010 05:03:38 FAIL: Busted: tp4 FAIL: failed to initialize browser Does "failed to initialize browser" mean it didn't start up within a 10 minute timeout? http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262408235.1262414474.30697.gz MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/01/01 20:57:15 sh: line 1: 574 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpHML_hD/profile startup_test/startup_test.html?begin=1262410219920 > browser_output.txt Failed ts_cold_generated_max: Stopped Fri, 01 Jan 2010 21:32:55 FAIL: Busted: ts_cold_generated_max FAIL: timeout exceeded
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262870103.1262871847.27658.gz MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/07 05:15:03 sh: line 1: 203 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmp0Ql9a2/profile getInfo.html > browser_output.txt Failed ts: Stopped Thu, 07 Jan 2010 05:30:11 FAIL: Busted: ts FAIL: failed to initialize browser Completed test ts: Stopped Thu, 07 Jan 2010 05:30:11
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262940954.1262941881.2315.gz Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/08 00:55:54 talos-rev2-leopard06: Started Fri, 08 Jan 2010 00:55:28 Running test tp4: Started Fri, 08 Jan 2010 00:55:28 NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/641 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 226 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpn-tPW0/profile getInfo.html > browser_output.txt Failed tp4: Stopped Fri, 08 Jan 2010 01:05:42 FAIL: Busted: tp4 FAIL: failed to initialize browser Completed test tp4:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262935100.1262936149.5562.gz MacOSX Darwin 9.0.0 mozilla-central talos on 2010/01/07 23:18:20 Also happened on tdhtml, looks like the same type of failure: talos-rev2-leopard06: Started Thu, 07 Jan 2010 23:17:09 Running test tdhtml: Started Thu, 07 Jan 2010 23:17:09 NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/641 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 214 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpber59v/profile getInfo.html > browser_output.txt Failed tdhtml: Stopped Thu, 07 Jan 2010 23:27:15 FAIL: Busted: tdhtml FAIL: failed to initialize browser Completed test tdhtml: Stopped Thu, 07 Jan 2010 23:27:15
See also bug 536473, which was supposedly e10s-only but has the exact same symptoms. For what it's worth, each time I merge mozilla-central to e10s that bug doesn't reproduce, but on all subsequent e10s pushes it does... totally weird.
Oh, but bug 536473 was Windows-only.
Saw this again on tdhtml. There's got to be something bad going on here, I wish the logs weren't so opaque! MacOSX Darwin 9.0.0 mozilla-central talos on 2010/01/08 12:15:37 Completed sending results: Stopped Fri, 08 Jan 2010 12:27:36 program finished with exit code 0 TinderboxPrint:s: talos-rev2-leopard06 TinderboxPrint:id:20100108112111 TinderboxPrint:<a href = "http://hg.mozilla.org/mozilla-central/rev/36bd078ee596">rev:36bd078ee596</a> TinderboxPrint:FAIL: Busted: tdhtml TinderboxPrint:FAIL: failed to initialize browser TinderboxPrint: cycle time: 00:13:04<br>
This indicates that the browser has not managed to open, dump some stats and then immediately close. Talos gives the browser 10 minutes to complete this operation, otherwise it assumes that the browser is frozen and kills it. I would think that as this has always worked and there have been no talos code changes in range that this would be an issue with the browser getting frozen/stuck on startup intermittently.
Happened too with tjss. http://tinderbox.mozilla.org/showlog.cgi?log=Firefox MacOSX Darwin 9.0.0 mozilla-central talos jss on 2010/01/14 20:06:06 /1263528366.1263529255.32361.gz sh: line 1: 191 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpWBk90H/profile getInfo.html > browser_output.txt Failed tjss: Stopped Thu, 14 Jan 2010 20:20:48 FAIL: Busted: tjss FAIL: failed to initialize browser Completed test tjss: Stopped Thu, 14 Jan 2010 20:20:48 RETURN: cycle time: 00:10:05<br>
I'm no Talos expert, but I'll try to reproduce this on a machine and get to the bottom of it.
Assignee: nobody → bhearsum
I just ran dirty ts tests in a loop for 4 hours or so and haven't been able to reproduce the failure...I'm not sure where to go from here with this.
Assignee: bhearsum → nobody
So, we had a patch to bcontroller.py that landed on the 21st of December. bcontroller.py is involved in the initialization of the browser/profile before the test begins. I think this is strong evidence that this patch caused this issue. bug 529137 is the bug in question. Alice, do you think it could've caused this issue?
Alice said she'd have a look.
Assignee: nobody → anodelman
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264159818.1264160718.3452.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/22 03:30:18
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264199877.1264200811.24797.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/22 14:37:57
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264203911.1264204808.5326.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/22 15:45:11
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264241780.1264242772.3533.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/23 02:16:20
Missed one: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264107669.1264109094.29777.gz MacOSX Darwin 9.0.0 mozilla-central talos on 2010/01/21 13:01:09
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264342643.1264343570.1084.gz MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/24 06:17:23
This appears to be leopard only, occurring on several slaves.
Summary: Talos orange: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated → Talos orange on leopard: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated
Whiteboard: [talos]
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264455149.1264456873.8225.gz MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/25 13:32:29
Found some obvious issues in the code. Will stage this to ensure that it at least isn't harmful.
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264566514.1264572066.23944.gz MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/01/26 20:28:34
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264584974.1264585870.15979.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/27 01:36:14
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264622008.1264628644.9165.gz MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/01/27 11:53:28
Attachment #423699 - Flags: review?(catlee)
Attachment #423699 - Flags: review?(catlee) → review+
Depends on: 542411
OS: Mac OS X → All
Hardware: x86 → All
Summary: Talos orange on leopard: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated → Talos orange: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated
Comment on attachment 423699 [details] [diff] [review] [checked in]fix initalization woes on leopard Checking in ffprocess.py; /cvsroot/mozilla/testing/performance/talos/ffprocess.py,v <-- ffprocess.py new revision: 1.17; previous revision: 1.16 done Checking in ffsetup.py; /cvsroot/mozilla/testing/performance/talos/ffsetup.py,v <-- ffsetup.py new revision: 1.16; previous revision: 1.15 done Checking in ttest.py; /cvsroot/mozilla/testing/performance/talos/ttest.py,v <-- ttest.py new revision: 1.42; previous revision: 1.41 done
Attachment #423699 - Attachment description: work in progress: fix initalization woes on leopard → [checked in]fix initalization woes on leopard
Attachment #423699 - Flags: checked-in+
I'm still seeing instances of this bug, so my patch did not repair the issue. Will have to investigate further.
I'm having a hard time reproducing this bug, but here's another patch that fixes up some of the initialization logic. Basically, we were only taking one chance to read possible output from the browser based upon the return on process.poll(). process.poll() in python is a tricky business, so checking multiple times should be less risky.
Attachment #424134 - Flags: review?(catlee)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264959798.1264960749.18609.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/31 09:43:18 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265001933.1265003589.585.gz MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/31 21:25:33 FAIL: Busted: ts FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265020348.1265021299.31637.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/01 02:32:28 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265050936.1265051880.1680.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/01 11:02:16 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265081799.1265086617.28177.gz WINNT 5.1 mozilla-central talos dirty on 2010/02/01 19:36:39 FAIL: Busted: ts_places_generated_med FAIL: failed to initialize browser
Summary: Talos orange: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated → Talos orange: "failed to initialize browser"
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265136731.1265137655.28797.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/02 10:52:11 FAIL: Busted: tp4 FAIL: failed to initialize browser
Blocks: 438871
Whiteboard: [talos] → [talos][orange]
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265164379.1265165494.26473.gz MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/02 18:32:59 FAIL: Busted: tsvg FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265205522.1265206574.3884.gz MacOSX Darwin 9.0.0 mozilla-central talos nochrome on 2010/02/03 05:58:42 FAIL: Busted: tdhtml FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265218327.1265221594.20059.gz WINNT 6.0 mozilla-central talos dirty on 2010/02/03 09:32:07 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser FAIL: Busted: ts_places_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265254096.1265255011.19339.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/03 19:28:16 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265312391.1265313157.9314.gz WINNT 5.1 mozilla-central talos jss on 2010/02/04 11:39:51 FAIL: Busted: tjss FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265333420.1265334326.18791.gz MacOSX Darwin 9.0.0 mozilla-central talos jss on 2010/02/04 17:30:20 FAIL: Busted: tjss FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265400690.1265401638.26543.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/05 12:11:30 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265411152.1265412111.17420.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/05 15:05:52 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265455914.1265457170.3437.gz MacOSX Darwin 9.0.0 mozilla-central talos on 2010/02/06 03:31:54 FAIL: Busted: tdhtml FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265459770.1265463339.12443.gz MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/02/06 04:36:10 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265465061.1265466134.15229.gz MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/06 06:04:21 FAIL: Busted: tsvg FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265458498.1265463011.8623.gz MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/02/06 04:14:58 FAIL: Busted: ts FAIL: failed to initialize browser
And in Rev3: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265476732.1265481337.17485.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 09:18:52 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265465513.1265468597.12614.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 06:11:53 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265456669.1265459671.32479.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 03:44:29 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265486102.1265487055.18285.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/06 11:55:02 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265496037.1265499163.21989.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 14:40:37 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265505230.1265506315.3075.gz MacOSX Darwin 9.0.0 mozilla-central talos on 2010/02/06 17:13:50 FAIL: Busted: tdhtml FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265511280.1265514229.23825.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 18:54:40 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265561278.1265564207.8922.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 08:47:58 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265554324.1265555418.23599.gz MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/07 06:52:04 FAIL: Busted: tsvg FAIL: failed to initialize browser http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265555253.1265558403.5821.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 07:07:33 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265555838.1265559016.12808.gz WINNT 5.1 mozilla-central talos dirty on 2010/02/07 07:17:18 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser FAIL: Busted: ts_places_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265562756.1265565829.26668.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 09:12:36 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265604314.1265607261.6591.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 20:45:14 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265606876.1265609920.4073.gz WINNT 6.0 mozilla-central talos dirty on 2010/02/07 21:27:56 FAIL: Busted: ts_places_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265636963.1265637956.7397.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/08 05:49:23 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265638781.1265642137.25917.gz WINNT 5.1 mozilla-central talos dirty on 2010/02/08 06:19:41 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser FAIL: Busted: ts_places_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265669664.1265670565.5565.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/08 14:54:24 FAIL: Busted: tp4 FAIL: failed to initialize browser http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265671422.1265672540.29224.gz MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/08 15:23:42 FAIL: Busted: tsvg FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265686601.1265689520.32359.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/08 19:36:41 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265740276.1265741227.31353.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/09 10:31:16 s: talos-rev2-leopard12 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265766047.1265767135.2665.gz MacOSX Darwin 9.0.0 mozilla-central talos nochrome on 2010/02/09 17:40:47 s: talos-rev2-leopard12 FAIL: Busted: tdhtml FAIL: failed to initialize browser http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265765629.1265766575.28669.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/09 17:33:49 s: talos-rev2-leopard06 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265814927.1265817872.1656.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/10 07:15:27 s: talos-r3-leopard-006 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265828006.1265828948.2100.gz MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/10 10:53:26 s: talos-rev2-leopard12 FAIL: Busted: tp4 FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265884280.1265885193.19520.gz MacOSX Darwin 9.0.0 mozilla-central talos jss on 2010/02/11 02:31:20 FAIL: Busted: tjss FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265895992.1265899176.25531.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/11 05:46:32 s: talos-r3-leopard-002 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265911707.1265914799.14503.gz WINNT 5.1 mozilla-central talos dirty on 2010/02/11 10:08:27 s: talos-rev2-xp01 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265963414.1265966420.8744.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 00:30:14
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265965589.1265968582.828.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 01:06:29 s: talos-r3-leopard-008 http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265973565.1265976654.28234.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 03:19:25 s: talos-r3-leopard-020
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1266009141.1266012112.20932.gz&fulltext=1 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 13:12:21
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1266412328.1266413124.11277.gz&fulltext=1 Linux64 mozilla-1.9.2 talos nochrome on 2010/02/17 05:12:08 Busted tdhtml
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1266419400.1266423088.8831.gz&fulltext=1 WINNT 6.0 mozilla-1.9.2 talos dirty on 2010/02/17 07:10:00 ts_cold_generated_max
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1266522049.1266523153.20818.gz MacOSX Darwin 9.0.0 mozilla-central talos on 2010/02/18 11:40:49
Attachment #424134 - Flags: review?(catlee)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1267601420.1267604454.2448.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/03/02 23:30:20 s: talos-r3-leopard-012
Not currently working on this, putting back in the pool.
Assignee: anodelman → nobody
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269428669.1269429675.10361.gz Rev3 Fedora 12x64 mozilla-central talos tp4 on 2010/03/24 04:04:29 s: talos-r3-fed64-005
This seems to be less frequent now, if nothing else. Maybe the rev3 machines are less prone to this?
Priority: -- → P5
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269607003.1269610053.23073.gz Rev3 Fedora 12x64 mozilla-central talos dirty on 2010/03/26 05:36:43 s: talos-r3-fed64-005 Same slave as comment 86 and comment 84, though that could easily be coincidence.
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269863520.1269868571.7106.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/03/29 04:52:00 s: talos-r3-leopard-032
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1269899362.1269904599.21715.gz Rev3 Fedora 12x64 mozilla-1.9.2 talos dirty on 2010/03/29 14:49:22 s: talos-r3-fed64-005
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269949984.1269953662.4598.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/03/30 04:53:04 s: talos-r3-leopard-031
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270194885.1270198066.17974.gz Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central talos dirty on 2010/04/02 00:54:45 s: talos-r3-snow-011
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270350889.1270353867.25257.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/03 20:14:49 s: talos-r3-leopard-004
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270582557.1270587209.3938.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/06 12:35:57 s: talos-r3-leopard-019
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270589166.1270589899.14353.gz Rev3 Fedora 12 mozilla-central talos on 2010/04/06 14:26:06 s: talos-r3-fed-030
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270598598.1270599139.12192.gz Rev3 Fedora 12 mozilla-central talos dromaeo on 2010/04/06 17:03:18 s: talos-r3-fed-030 FAIL: Busted: dromaeo_css FAIL: Busted: dromaeo_dom FAIL: Busted: dromaeo_jslib FAIL: Busted: dromaeo_sunspider FAIL: Busted: dromaeo_v8 FAIL: Busted: dromaeo_basics
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270622571.1270622752.8536.gz Rev3 Fedora 12 mozilla-central talos cold [testfailed] Started 23:42, finished 23:46 FAIL: Busted: ts FAIL: failed to initialize browser FAIL: Busted: ts_cold FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270656902.1270660018.5911.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270672966.1270673371.14431.gz Rev3 Fedora 12 mozilla-central talos cold on 2010/04/07 13:42:46 s: talos-r3-fed-030 Stopped Wed, 07 Apr 2010 21:53:12 FAIL: Busted: ts Stopped Wed, 07 Apr 2010 21:53:14 FAIL: Busted: ts_cold
The following (but not the earlier "Fedora" ones) have in common that: sh: ../firefox/firefox-bin: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory NOISE: NOISE: __FAILbrowser non-zero return code (32256)__FAIL These logs have in common that: * the environment variables show that it's a 64-bit slave (I think, though not 100% sure) * they're running a 32-bit build * the error message is the kind of thing you'd get when running a 32-bit build on a 64-bit machine without any 32-bit libs (In reply to comment #94) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270589166.1270589899.14353.gz > Rev3 Fedora 12 mozilla-central talos on 2010/04/06 14:26:06 > s: talos-r3-fed-030 (In reply to comment #95) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270598598.1270599139.12192.gz > Rev3 Fedora 12 mozilla-central talos dromaeo on 2010/04/06 17:03:18 > s: talos-r3-fed-030 (In reply to comment #96) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270622571.1270622752.8536.gz > Rev3 Fedora 12 mozilla-central talos cold [testfailed] Started 23:42, finished > 23:46 (In reply to comment #98) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270672966.1270673371.14431.gz > Rev3 Fedora 12 mozilla-central talos cold on 2010/04/07 13:42:46 > s: talos-r3-fed-030
* they're all running on talos-r3-fed-030, which alice and catlee just found was a 64-bit slave mislabeled as a 32-bit one, and has pulled from the rotation.
Making the summary of this bug more specific; error messages that aren't of the form: sh: line 1: 235 Terminated , where the 235 can be any integer, should go in other bugs so that we have at least a chance of tracking one issue per bug.
Summary: Talos orange: "failed to initialize browser" → Talos orange: "failed to initialize browser"; "sh: line 1:" ... "Terminated"
(In reply to comment #97) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270656902.1270660018.5911.gz > Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty This was indeed of the form that goes in this bug, at least as I've managed to restrict it so far. (In reply to comment #93) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270582557.1270587209.3938.gz > Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/06 12:35:57 > s: talos-r3-leopard-019 ... as was this. (In reply to comment #92) > http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270350889.1270353867.25257.gz > Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/03 20:14:49 > s: talos-r3-leopard-004 ... and this. I haven't gone back further to check old logs.
We lost all the logs before March, but for the ones still available, that seems to have restricted this bug (once again) to being Mac-only and every-Mac-failure. Dunno whether that means the "sh: ... Terminated" is just an artifact of either the way Talos kills the browser on Mac, or what the Mac logs, or is actually a clue. I thought for a bit there might be a clue in the varying _NSAutoreleaseNoPool() and "Terminating app due to uncaught exception 'NSInvalidArgumentException':" spew, but those both seem to be present even in "successful" runs. http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270694689.1270699330.26628.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/07 19:44:49 s: talos-r3-leopard-007 Running test ts_cold_generated_max: Started Wed, 07 Apr 2010 20:21:27 2010-04-07 20:21:34.413 firefox-bin[570:10b] *** _NSAutoreleaseNoPool(): Object 0x617d70 of class NSThread autoreleased with no pool in place - just leaking Stack: (0x936e4f4f 0x935f1432 0x935f71a4 0x2db1550 0x2d9e4f2 0x2d9e5bc 0x2d7f377 0x2cd1a76 0x20fc78f 0x21005db) NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/643 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 570 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpQjnBCd/profile getInfo.html > browser_output.txt Failed ts_cold_generated_max: Stopped Wed, 07 Apr 2010 20:31:39
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270882543.1270885751.32446.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/09 23:55:43 s: talos-r3-leopard-016 Running test ts_cold_generated_max: Started Sat, 10 Apr 2010 00:08:42 2010-04-10 00:08:46.487 firefox-bin[568:10b] *** _NSAutoreleaseNoPool(): Object 0x617df0 of class NSThread autoreleased with no pool in place - just leaking Stack: (0x936e4f4f 0x935f1432 0x935f71a4 0x2da23a0 0x2d8f342 0x2d8f40c 0x2d701c7 0x2cc28c6 0x20ec13f 0x20eff8b) NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/643 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 568 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmp_Jh-MA/profile getInfo.html > browser_output.txt Failed ts_cold_generated_max: Stopped Sat, 10 Apr 2010 00:18:51
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271242373.1271246880.22571.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/14 03:52:53 s: talos-r3-leopard-007 Running test ts_cold_generated_max: Started Wed, 14 Apr 2010 04:27:15 2010-04-14 04:27:19.862 firefox-bin[578:10b] *** _NSAutoreleaseNoPool(): Object 0x617d70 of class NSThread autoreleased with no pool in place - just leaking Stack: (0x936e4f4f 0x935f1432 0x935f71a4 0x2db7a10 0x2da49b2 0x2da4a7c 0x2d85837 0x2cd7e46 0x21009bf 0x2103d4b) NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/643 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 578 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpoMVemh/profile getInfo.html > browser_output.txt Failed ts_cold_generated_max: Stopped Wed, 14 Apr 2010 04:37:24 FAIL: Busted: ts_cold_generated_max FAIL: failed to initialize browser Completed test ts_cold_generated_max: Stopped Wed, 14 Apr 2010 04:37:24
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271231755.1271237736.20655.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/14 00:55:55 s: talos-r3-leopard-032 Running test ts_places_generated_max: Started Wed, 14 Apr 2010 02:14:11 2010-04-14 02:14:15.850 firefox-bin[1610:10b] *** _NSAutoreleaseNoPool(): Object 0x617e30 of class NSThread autoreleased with no pool in place - just leaking Stack: (0x92483f4f 0x92390432 0x923961a4 0x2db7a10 0x2da49b2 0x2da4a7c 0x2d85837 0x2cd7e46 0x2100b1f 0x2103eab) NOISE: __metrics Screen width/height:1280/1024 NOISE: colorDepth:24 NOISE: Browser inner width/height: 1024/643 NOISE: Browser outer width/height: 1024/768__metrics sh: line 1: 1610 Terminated ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpWiCXYr/profile getInfo.html > browser_output.txt Failed ts_places_generated_max: Stopped Wed, 14 Apr 2010 02:24:21 FAIL: Busted: ts_places_generated_max FAIL: failed to initialize browser Completed test ts_places_generated_max: Stopped Wed, 14 Apr 2010 02:24:21
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271298012.1271301275.19403.gz Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central talos dirty on 2010/04/14 19:20:12
Assignee: nobody → anodelman
Priority: P5 → P2
Going to try and reproduce in staging environment.
This seems to be the browser taking longer than 10 minutes to initially open after a new max dirty profile is installed.
Increase the timeout to 20 minutes, also add better error messaging so that this can be tracked easier.
Attachment #424134 - Attachment is obsolete: true
Attachment #444527 - Flags: review?(catlee)
Attachment #444527 - Flags: review?(catlee) → review+
Blocks: 564914
Comment on attachment 444527 [details] [diff] [review] increase timeout, better error messages Checking in ffsetup.py; /cvsroot/mozilla/testing/performance/talos/ffsetup.py,v <-- ffsetup.py new revision: 1.18; previous revision: 1.17 done
Attachment #444527 - Flags: checked-in+
I presume this bug is FIXED with the last patch checked in.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
This is causing a lot of N900 oranges -- can we remove the final two lines? else: raise talosError("initialization timed out")
(In reply to comment #113) > This is causing a lot of N900 oranges -- can we remove the final two lines? > > else: > raise talosError("initialization timed out") Is that line the root of the problem or is it a symptom of the problem. From my reading of that code, it looks like it is a symptom.
(In reply to comment #114) > (In reply to comment #113) > > This is causing a lot of N900 oranges -- can we remove the final two lines? > > > > else: > > raise talosError("initialization timed out") > > Is that line the root of the problem or is it a symptom of the problem. From > my reading of that code, it looks like it is a symptom. What is the problem, other than an error is raised? Does Fennec behave the same way as Firefox re: this log? I'm probably going to create a tarball without these two lines and get smm2 to use it later this week.
Initialized timed out means that the browser did not close post first run (the first run is used to populate the empty profile) - I don't quite know what just ignoring the error will do. Has the profile been created successfully? Did the browser close?
I think we figured out this is bug 578343.
Whiteboard: [talos][orange] → [talos]
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: