Closed Bug 537147 Opened 15 years ago Closed 14 years ago

Talos orange: "failed to initialize browser"; "sh: line 1:" ... "Terminated"

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cjones, Assigned: anodelman)

References

Details

(Keywords: intermittent-failure, Whiteboard: [talos])

Attachments

(2 files, 1 obsolete file)

[checked in]fix initalization woes on leopard
7.89 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
increase timeout, better error messages
1.51 KB, patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review 
See http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262104680.1262105624.16358.gz&fulltext=1

I can't tell why this build is going orange; the log is not very helpful.  The most useful output seems to be


Running test tp4: 
		Started Tue, 29 Dec 2009 08:43:25
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/641
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:   235 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpvuhb44/profile getInfo.html > browser_output.txt
Failed tp4: 
		Stopped Tue, 29 Dec 2009 08:53:35
FAIL: Busted: tp4
FAIL: failed to initialize browser
Completed test tp4: 
		Stopped Tue, 29 Dec 2009 08:53:35
RETURN: cycle time: 00:10:10<br>


I heard on IRC that this was a known problem, but for the life of me I can't find a bug on file.  Apologies if this is a dupe.
(In reply to comment #0)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262104680.1262105624.16358.gz&fulltext=1

MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2009/12/29 08:38:00  

(In reply to comment #1)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262130241.1262133368.4749.gz&fulltext=1

WINNT 6.0 mozilla-central talos dirty on 2009/12/29 15:44:01  



http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262265032.1262269401.26182.gz
MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2009/12/31 05:10:32  

Running test ts_places_generated_max: 
		Started Thu, 31 Dec 2009 06:01:17
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/641
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:  1654 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpqcBQ2d/profile getInfo.html > browser_output.txt
Failed ts_places_generated_max: 
		Stopped Thu, 31 Dec 2009 06:11:38
FAIL: Busted: ts_places_generated_max
FAIL: failed to initialize browser
Completed test ts_places_generated_max: 
		Stopped Thu, 31 Dec 2009 06:11:38
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262246530.1262250346.799.gz
MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2009/12/31 00:02:10

(who knows what's in the log - thanks to bug 535084 I'm not really willing to look)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262436838.1262437754.26036.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/02 04:53:58  
sh: line 1:   225 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpasjQQF/profile getInfo.html > browser_output.txt
Failed tp4: 
		Stopped Sat, 02 Jan 2010 05:03:38
FAIL: Busted: tp4
FAIL: failed to initialize browser

Does "failed to initialize browser" mean it didn't start up within a 10 minute timeout?


http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262408235.1262414474.30697.gz
MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/01/01 20:57:15  
sh: line 1:   574 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpHML_hD/profile startup_test/startup_test.html?begin=1262410219920 > browser_output.txt
Failed ts_cold_generated_max: 
		Stopped Fri, 01 Jan 2010 21:32:55
FAIL: Busted: ts_cold_generated_max
FAIL: timeout exceeded
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262870103.1262871847.27658.gz
MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/07 05:15:03  

sh: line 1:   203 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmp0Ql9a2/profile getInfo.html > browser_output.txt
Failed ts: 
		Stopped Thu, 07 Jan 2010 05:30:11
FAIL: Busted: ts
FAIL: failed to initialize browser
Completed test ts: 
		Stopped Thu, 07 Jan 2010 05:30:11
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262940954.1262941881.2315.gz
Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/08 00:55:54

talos-rev2-leopard06: 
		Started Fri, 08 Jan 2010 00:55:28
Running test tp4: 
		Started Fri, 08 Jan 2010 00:55:28
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/641
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:   226 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpn-tPW0/profile getInfo.html > browser_output.txt
Failed tp4: 
		Stopped Fri, 08 Jan 2010 01:05:42
FAIL: Busted: tp4
FAIL: failed to initialize browser
Completed test tp4:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1262935100.1262936149.5562.gz
MacOSX Darwin 9.0.0 mozilla-central talos on 2010/01/07 23:18:20

Also happened on tdhtml, looks like the same type of failure:

talos-rev2-leopard06: 
		Started Thu, 07 Jan 2010 23:17:09
Running test tdhtml: 
		Started Thu, 07 Jan 2010 23:17:09
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/641
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:   214 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpber59v/profile getInfo.html > browser_output.txt
Failed tdhtml: 
		Stopped Thu, 07 Jan 2010 23:27:15
FAIL: Busted: tdhtml
FAIL: failed to initialize browser
Completed test tdhtml: 
		Stopped Thu, 07 Jan 2010 23:27:15
See also bug 536473, which was supposedly e10s-only but has the exact same symptoms. For what it's worth, each time I merge mozilla-central to e10s that bug doesn't reproduce, but on all subsequent e10s pushes it does... totally weird.
Oh, but bug 536473 was Windows-only.
Saw this again on tdhtml.  There's got to be something bad going on here, I wish the logs weren't so opaque!

MacOSX Darwin 9.0.0 mozilla-central talos on 2010/01/08 12:15:37  
Completed sending results: 
		Stopped Fri, 08 Jan 2010 12:27:36
program finished with exit code 0
TinderboxPrint:s: talos-rev2-leopard06
TinderboxPrint:id:20100108112111
TinderboxPrint:<a href = "http://hg.mozilla.org/mozilla-central/rev/36bd078ee596">rev:36bd078ee596</a>
TinderboxPrint:FAIL: Busted: tdhtml
TinderboxPrint:FAIL: failed to initialize browser
TinderboxPrint: cycle time: 00:13:04<br>
This indicates that the browser has not managed to open, dump some stats and then immediately close.  Talos gives the browser 10 minutes to complete this operation, otherwise it assumes that the browser is frozen and kills it.

I would think that as this has always worked and there have been no talos code changes in range that this would be an issue with the browser getting frozen/stuck on startup intermittently.
Happened too with tjss.

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox
MacOSX Darwin 9.0.0 mozilla-central talos jss on 2010/01/14 20:06:06  
/1263528366.1263529255.32361.gz
sh: line 1:   191 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpWBk90H/profile getInfo.html > browser_output.txt
Failed tjss: 
		Stopped Thu, 14 Jan 2010 20:20:48
FAIL: Busted: tjss
FAIL: failed to initialize browser
Completed test tjss: 
		Stopped Thu, 14 Jan 2010 20:20:48
RETURN: cycle time: 00:10:05<br>
I'm no Talos expert, but I'll try to reproduce this on a machine and get to the bottom of it.
Assignee: nobody → bhearsum
I just ran dirty ts tests in a loop for 4 hours or so and haven't been able to reproduce the failure...I'm not sure where to go from here with this.
Assignee: bhearsum → nobody
So, we had a patch to bcontroller.py that landed on the 21st of December. bcontroller.py is involved in the initialization of the browser/profile before the test begins. I think this is strong evidence that this patch caused this issue.

bug 529137 is the bug in question.

Alice, do you think it could've caused this issue?
Alice said she'd have a look.
Assignee: nobody → anodelman
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264159818.1264160718.3452.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/22 03:30:18
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264199877.1264200811.24797.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/22 14:37:57
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264203911.1264204808.5326.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/22 15:45:11
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264241780.1264242772.3533.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/23 02:16:20
Missed one:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264107669.1264109094.29777.gz
MacOSX Darwin 9.0.0 mozilla-central talos on 2010/01/21 13:01:09
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264342643.1264343570.1084.gz
MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/24 06:17:23
This appears to be leopard only, occurring on several slaves.
Summary: Talos orange: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated → Talos orange on leopard: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated
Whiteboard: [talos]
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264455149.1264456873.8225.gz
MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/25 13:32:29
Found some obvious issues in the code.  Will stage this to ensure that it at least isn't harmful.
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264566514.1264572066.23944.gz
MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/01/26 20:28:34
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264584974.1264585870.15979.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/27 01:36:14
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264622008.1264628644.9165.gz
MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/01/27 11:53:28
Attachment #423699 - Flags: review?(catlee)
Attachment #423699 - Flags: review?(catlee) → review+
Depends on: 542411
OS: Mac OS X → All
Hardware: x86 → All
Summary: Talos orange on leopard: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated → Talos orange: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated
Comment on attachment 423699 [details] [diff] [review]
[checked in]fix initalization woes on leopard

Checking in ffprocess.py;
/cvsroot/mozilla/testing/performance/talos/ffprocess.py,v  <--  ffprocess.py
new revision: 1.17; previous revision: 1.16
done
Checking in ffsetup.py;
/cvsroot/mozilla/testing/performance/talos/ffsetup.py,v  <--  ffsetup.py
new revision: 1.16; previous revision: 1.15
done
Checking in ttest.py;
/cvsroot/mozilla/testing/performance/talos/ttest.py,v  <--  ttest.py
new revision: 1.42; previous revision: 1.41
done
Attachment #423699 - Attachment description: work in progress: fix initalization woes on leopard → [checked in]fix initalization woes on leopard
Attachment #423699 - Flags: checked-in+
I'm still seeing instances of this bug, so my patch did not repair the issue.  Will have to investigate further.
I'm having a hard time reproducing this bug, but here's another patch that fixes up some of the initialization logic.  Basically, we were only taking one chance to read possible output from the browser based upon the return on process.poll().  process.poll() in python is a tricky business, so checking multiple times should be less risky.
Attachment #424134 - Flags: review?(catlee)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1264959798.1264960749.18609.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/01/31 09:43:18  
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265001933.1265003589.585.gz
MacOSX Darwin 9.0.0 mozilla-central talos cold on 2010/01/31 21:25:33  
FAIL: Busted: ts
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265020348.1265021299.31637.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/01 02:32:28  
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265050936.1265051880.1680.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/01 11:02:16  
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265081799.1265086617.28177.gz
WINNT 5.1 mozilla-central talos dirty on 2010/02/01 19:36:39  
FAIL: Busted: ts_places_generated_med
FAIL: failed to initialize browser
Summary: Talos orange: "Busted: tp4" "failed to initialize browser" with message about firefox being terminated → Talos orange: "failed to initialize browser"
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265136731.1265137655.28797.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/02 10:52:11
FAIL: Busted: tp4
FAIL: failed to initialize browser
Blocks: 438871
Whiteboard: [talos] → [talos][orange]
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265164379.1265165494.26473.gz
MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/02 18:32:59
FAIL: Busted: tsvg
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265205522.1265206574.3884.gz
MacOSX Darwin 9.0.0 mozilla-central talos nochrome on 2010/02/03 05:58:42  
FAIL: Busted: tdhtml
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265218327.1265221594.20059.gz
WINNT 6.0 mozilla-central talos dirty on 2010/02/03 09:32:07  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
FAIL: Busted: ts_places_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265254096.1265255011.19339.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/03 19:28:16  
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265312391.1265313157.9314.gz
WINNT 5.1 mozilla-central talos jss on 2010/02/04 11:39:51  
FAIL: Busted: tjss
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265333420.1265334326.18791.gz
MacOSX Darwin 9.0.0 mozilla-central talos jss on 2010/02/04 17:30:20  
FAIL: Busted: tjss
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265400690.1265401638.26543.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/05 12:11:30  
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265411152.1265412111.17420.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/05 15:05:52  
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265455914.1265457170.3437.gz
MacOSX Darwin 9.0.0 mozilla-central talos on 2010/02/06 03:31:54  
FAIL: Busted: tdhtml
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265459770.1265463339.12443.gz
MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/02/06 04:36:10  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265465061.1265466134.15229.gz
MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/06 06:04:21  
FAIL: Busted: tsvg
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265458498.1265463011.8623.gz
MacOSX Darwin 9.0.0 mozilla-central talos dirty on 2010/02/06 04:14:58  
FAIL: Busted: ts
FAIL: failed to initialize browser
And in Rev3:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265476732.1265481337.17485.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 09:18:52  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265465513.1265468597.12614.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 06:11:53  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265456669.1265459671.32479.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 03:44:29  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265486102.1265487055.18285.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/06 11:55:02  
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265496037.1265499163.21989.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 14:40:37  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265505230.1265506315.3075.gz
MacOSX Darwin 9.0.0 mozilla-central talos on 2010/02/06 17:13:50  
FAIL: Busted: tdhtml
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265511280.1265514229.23825.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/06 18:54:40  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265561278.1265564207.8922.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 08:47:58  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265554324.1265555418.23599.gz
MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/07 06:52:04  
FAIL: Busted: tsvg
FAIL: failed to initialize browser

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265555253.1265558403.5821.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 07:07:33  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265555838.1265559016.12808.gz
WINNT 5.1 mozilla-central talos dirty on 2010/02/07 07:17:18  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
FAIL: Busted: ts_places_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265562756.1265565829.26668.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 09:12:36  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265604314.1265607261.6591.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/07 20:45:14  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265606876.1265609920.4073.gz
WINNT 6.0 mozilla-central talos dirty on 2010/02/07 21:27:56  
FAIL: Busted: ts_places_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265636963.1265637956.7397.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/08 05:49:23
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265638781.1265642137.25917.gz
WINNT 5.1 mozilla-central talos dirty on 2010/02/08 06:19:41  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
FAIL: Busted: ts_places_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265669664.1265670565.5565.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/08 14:54:24  
FAIL: Busted: tp4
FAIL: failed to initialize browser

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265671422.1265672540.29224.gz
MacOSX Darwin 9.0.0 mozilla-central talos svg on 2010/02/08 15:23:42  
FAIL: Busted: tsvg
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265686601.1265689520.32359.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/08 19:36:41  
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265740276.1265741227.31353.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/09 10:31:16  
s: talos-rev2-leopard12
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265766047.1265767135.2665.gz
MacOSX Darwin 9.0.0 mozilla-central talos nochrome on 2010/02/09 17:40:47  
s: talos-rev2-leopard12
FAIL: Busted: tdhtml
FAIL: failed to initialize browser

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265765629.1265766575.28669.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/09 17:33:49  
s: talos-rev2-leopard06
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265814927.1265817872.1656.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/10 07:15:27  
s: talos-r3-leopard-006
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265828006.1265828948.2100.gz
MacOSX Darwin 9.0.0 mozilla-central talos tp4 on 2010/02/10 10:53:26  
s: talos-rev2-leopard12
FAIL: Busted: tp4
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265884280.1265885193.19520.gz
MacOSX Darwin 9.0.0 mozilla-central talos jss on 2010/02/11 02:31:20
FAIL: Busted: tjss
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265895992.1265899176.25531.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/11 05:46:32  
s: talos-r3-leopard-002
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265911707.1265914799.14503.gz
WINNT 5.1 mozilla-central talos dirty on 2010/02/11 10:08:27  
s: talos-rev2-xp01
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265963414.1265966420.8744.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 00:30:14
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265965589.1265968582.828.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 01:06:29  
s: talos-r3-leopard-008

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1265973565.1265976654.28234.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 03:19:25  
s: talos-r3-leopard-020
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1266009141.1266012112.20932.gz&fulltext=1
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/02/12 13:12:21
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1266412328.1266413124.11277.gz&fulltext=1
Linux64 mozilla-1.9.2 talos nochrome on 2010/02/17 05:12:08
Busted tdhtml
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1266419400.1266423088.8831.gz&fulltext=1
WINNT 6.0 mozilla-1.9.2 talos dirty on 2010/02/17 07:10:00
ts_cold_generated_max
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1266522049.1266523153.20818.gz
MacOSX Darwin 9.0.0 mozilla-central talos on 2010/02/18 11:40:49
Attachment #424134 - Flags: review?(catlee)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1267601420.1267604454.2448.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/03/02 23:30:20  
s: talos-r3-leopard-012
Not currently working on this, putting back in the pool.
Assignee: anodelman → nobody
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269428669.1269429675.10361.gz
Rev3 Fedora 12x64 mozilla-central talos tp4 on 2010/03/24 04:04:29  
s: talos-r3-fed64-005
This seems to be less frequent now, if nothing else. Maybe the rev3 machines are less prone to this?
Priority: -- → P5
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269607003.1269610053.23073.gz
Rev3 Fedora 12x64 mozilla-central talos dirty on 2010/03/26 05:36:43  
s: talos-r3-fed64-005

Same slave as comment 86 and comment 84, though that could easily be coincidence.
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269863520.1269868571.7106.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/03/29 04:52:00  
s: talos-r3-leopard-032
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1269899362.1269904599.21715.gz
Rev3 Fedora 12x64 mozilla-1.9.2 talos dirty on 2010/03/29 14:49:22  
s: talos-r3-fed64-005
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1269949984.1269953662.4598.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/03/30 04:53:04  
s: talos-r3-leopard-031
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270194885.1270198066.17974.gz
Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central talos dirty on 2010/04/02 00:54:45  
s: talos-r3-snow-011
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270350889.1270353867.25257.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/03 20:14:49  
s: talos-r3-leopard-004
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270582557.1270587209.3938.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/06 12:35:57  
s: talos-r3-leopard-019
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270589166.1270589899.14353.gz
Rev3 Fedora 12 mozilla-central talos on 2010/04/06 14:26:06  
s: talos-r3-fed-030
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270598598.1270599139.12192.gz
Rev3 Fedora 12 mozilla-central talos dromaeo on 2010/04/06 17:03:18  
s: talos-r3-fed-030
FAIL: Busted: dromaeo_css
FAIL: Busted: dromaeo_dom
FAIL: Busted: dromaeo_jslib
FAIL: Busted: dromaeo_sunspider
FAIL: Busted: dromaeo_v8
FAIL: Busted: dromaeo_basics
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270622571.1270622752.8536.gz
Rev3 Fedora 12 mozilla-central talos cold  [testfailed] Started 23:42, finished 23:46
FAIL: Busted: ts
FAIL: failed to initialize browser
FAIL: Busted: ts_cold
FAIL: failed to initialize browser
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270656902.1270660018.5911.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270672966.1270673371.14431.gz
Rev3 Fedora 12 mozilla-central talos cold on 2010/04/07 13:42:46  
s: talos-r3-fed-030
Stopped Wed, 07 Apr 2010 21:53:12
FAIL: Busted: ts
Stopped Wed, 07 Apr 2010 21:53:14
FAIL: Busted: ts_cold
The following (but not the earlier "Fedora" ones) have in common that:

sh: ../firefox/firefox-bin: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
NOISE: 
NOISE: __FAILbrowser non-zero return code (32256)__FAIL

These logs have in common that:
 * the environment variables show that it's a 64-bit slave (I think, though not 100% sure)
 * they're running a 32-bit build
 * the error message is the kind of thing you'd get when running a 32-bit build on a 64-bit machine without any 32-bit libs

(In reply to comment #94)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270589166.1270589899.14353.gz
> Rev3 Fedora 12 mozilla-central talos on 2010/04/06 14:26:06  
> s: talos-r3-fed-030

(In reply to comment #95)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270598598.1270599139.12192.gz
> Rev3 Fedora 12 mozilla-central talos dromaeo on 2010/04/06 17:03:18  
> s: talos-r3-fed-030

(In reply to comment #96)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270622571.1270622752.8536.gz
> Rev3 Fedora 12 mozilla-central talos cold  [testfailed] Started 23:42, finished
> 23:46

(In reply to comment #98)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270672966.1270673371.14431.gz
> Rev3 Fedora 12 mozilla-central talos cold on 2010/04/07 13:42:46  
> s: talos-r3-fed-030
 * they're all running on talos-r3-fed-030, which alice and catlee just found was a 64-bit slave mislabeled as a 32-bit one, and has pulled from the rotation.
Making the summary of this bug more specific; error messages that aren't of the form:
  sh: line 1:   235 Terminated             
, where the 235 can be any integer, should go in other bugs so that we have at least a chance of tracking one issue per bug.
Summary: Talos orange: "failed to initialize browser" → Talos orange: "failed to initialize browser"; "sh: line 1:" ... "Terminated"
(In reply to comment #97)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270656902.1270660018.5911.gz
> Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty

This was indeed of the form that goes in this bug, at least as I've managed to restrict it so far.

(In reply to comment #93)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270582557.1270587209.3938.gz
> Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/06 12:35:57  
> s: talos-r3-leopard-019

... as was this.

(In reply to comment #92)
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270350889.1270353867.25257.gz
> Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/03 20:14:49  
> s: talos-r3-leopard-004

... and this.


I haven't gone back further to check old logs.
We lost all the logs before March, but for the ones still available, that seems to have restricted this bug (once again) to being Mac-only and every-Mac-failure. Dunno whether that means the "sh: ... Terminated" is just an artifact of either the way Talos kills the browser on Mac, or what the Mac logs, or is actually a clue.

I thought for a bit there might be a clue in the varying _NSAutoreleaseNoPool() and "Terminating app due to uncaught exception 'NSInvalidArgumentException':" spew, but those both seem to be present even in "successful" runs.

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270694689.1270699330.26628.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/07 19:44:49  
s: talos-r3-leopard-007

Running test ts_cold_generated_max: 
		Started Wed, 07 Apr 2010 20:21:27
2010-04-07 20:21:34.413 firefox-bin[570:10b] *** _NSAutoreleaseNoPool(): Object 0x617d70 of class NSThread autoreleased with no pool in place - just leaking
Stack: (0x936e4f4f 0x935f1432 0x935f71a4 0x2db1550 0x2d9e4f2 0x2d9e5bc 0x2d7f377 0x2cd1a76 0x20fc78f 0x21005db)
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/643
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:   570 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpQjnBCd/profile getInfo.html > browser_output.txt
Failed ts_cold_generated_max: 
		Stopped Wed, 07 Apr 2010 20:31:39
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270882543.1270885751.32446.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/09 23:55:43
s: talos-r3-leopard-016

Running test ts_cold_generated_max: 
		Started Sat, 10 Apr 2010 00:08:42
2010-04-10 00:08:46.487 firefox-bin[568:10b] *** _NSAutoreleaseNoPool(): Object 0x617df0 of class NSThread autoreleased with no pool in place - just leaking
Stack: (0x936e4f4f 0x935f1432 0x935f71a4 0x2da23a0 0x2d8f342 0x2d8f40c 0x2d701c7 0x2cc28c6 0x20ec13f 0x20eff8b)
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/643
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:   568 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmp_Jh-MA/profile getInfo.html > browser_output.txt
Failed ts_cold_generated_max: 
		Stopped Sat, 10 Apr 2010 00:18:51
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271242373.1271246880.22571.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/14 03:52:53  
s: talos-r3-leopard-007

Running test ts_cold_generated_max: 
		Started Wed, 14 Apr 2010 04:27:15
2010-04-14 04:27:19.862 firefox-bin[578:10b] *** _NSAutoreleaseNoPool(): Object 0x617d70 of class NSThread autoreleased with no pool in place - just leaking
Stack: (0x936e4f4f 0x935f1432 0x935f71a4 0x2db7a10 0x2da49b2 0x2da4a7c 0x2d85837 0x2cd7e46 0x21009bf 0x2103d4b)
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/643
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:   578 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpoMVemh/profile getInfo.html > browser_output.txt
Failed ts_cold_generated_max: 
		Stopped Wed, 14 Apr 2010 04:37:24
FAIL: Busted: ts_cold_generated_max
FAIL: failed to initialize browser
Completed test ts_cold_generated_max: 
		Stopped Wed, 14 Apr 2010 04:37:24
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271231755.1271237736.20655.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty on 2010/04/14 00:55:55
s: talos-r3-leopard-032

Running test ts_places_generated_max: 
		Started Wed, 14 Apr 2010 02:14:11
2010-04-14 02:14:15.850 firefox-bin[1610:10b] *** _NSAutoreleaseNoPool(): Object 0x617e30 of class NSThread autoreleased with no pool in place - just leaking
Stack: (0x92483f4f 0x92390432 0x923961a4 0x2db7a10 0x2da49b2 0x2da4a7c 0x2d85837 0x2cd7e46 0x2100b1f 0x2103eab)
NOISE: __metrics	Screen width/height:1280/1024
NOISE: 	colorDepth:24
NOISE: 	Browser inner width/height: 1024/643
NOISE: 	Browser outer width/height: 1024/768__metrics
sh: line 1:  1610 Terminated              ../Minefield.app/Contents/MacOS/firefox-bin -foreground -profile /var/folders/Xr/Xr--yJnSEY0U11ET5NZuMU+++TM/-Tmp-/tmpWiCXYr/profile getInfo.html > browser_output.txt
Failed ts_places_generated_max: 
		Stopped Wed, 14 Apr 2010 02:24:21
FAIL: Busted: ts_places_generated_max
FAIL: failed to initialize browser
Completed test ts_places_generated_max: 
		Stopped Wed, 14 Apr 2010 02:24:21
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1271298012.1271301275.19403.gz
Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central talos dirty on 2010/04/14 19:20:12
Assignee: nobody → anodelman
Priority: P5 → P2
Going to try and reproduce in staging environment.
This seems to be the browser taking longer than 10 minutes to initially open after a new max dirty profile is installed.
Increase the timeout to 20 minutes, also add better error messaging so that this can be tracked easier.
Attachment #424134 - Attachment is obsolete: true
Attachment #444527 - Flags: review?(catlee)
Attachment #444527 - Flags: review?(catlee) → review+
Blocks: 564914
Comment on attachment 444527 [details] [diff] [review]
increase timeout, better error messages

Checking in ffsetup.py;
/cvsroot/mozilla/testing/performance/talos/ffsetup.py,v  <--  ffsetup.py
new revision: 1.18; previous revision: 1.17
done
Attachment #444527 - Flags: checked-in+
I presume this bug is FIXED with the last patch checked in.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
This is causing a lot of N900 oranges -- can we remove the final two lines?

else: 
    raise talosError("initialization timed out")
(In reply to comment #113)
> This is causing a lot of N900 oranges -- can we remove the final two lines?
> 
> else: 
>     raise talosError("initialization timed out")

Is that line the root of the problem or is it a symptom of the problem.  From my reading of that code, it looks like it is a symptom.
(In reply to comment #114)
> (In reply to comment #113)
> > This is causing a lot of N900 oranges -- can we remove the final two lines?
> > 
> > else: 
> >     raise talosError("initialization timed out")
> 
> Is that line the root of the problem or is it a symptom of the problem.  From
> my reading of that code, it looks like it is a symptom.

What is the problem, other than an error is raised? Does Fennec behave the same way as Firefox re: this log?

I'm probably going to create a tarball without these two lines and get smm2 to use it later this week.
Initialized timed out means that the browser did not close post first run (the first run is used to populate the empty profile) - I don't quite know what just ignoring the error will do.  Has the profile been created successfully?  Did the browser close?
I think we figured out this is bug 578343.
Whiteboard: [talos][orange] → [talos]
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: