Last Comment Bug 763166 - crash in mozilla::AndroidGeckoLayerClient::SetFirstPaintViewport
: crash in mozilla::AndroidGeckoLayerClient::SetFirstPaintViewport
Status: RESOLVED FIXED
[native-crash][startupcrash]
: crash, intermittent-failure, regression, topcrash
Product: Core
Classification: Components
Component: Widget: Android (show other bugs)
: 15 Branch
: ARM Android
: -- critical (vote)
: mozilla17
Assigned To: Kartikaya Gupta (email:kats@mozilla.com)
:
Mentors:
Depends on:
Blocks: 438871 747787 760152 778954
  Show dependency treegraph
 
Reported: 2012-06-09 01:34 PDT by Scoobidiver (away)
Modified: 2013-12-16 03:09 PST (History)
11 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
fixed
+
verified
fixed


Attachments
Add a GetJNIForCompositorThread and make the compositor JNI invocations use it (10.81 KB, patch)
2012-07-12 14:21 PDT, Kartikaya Gupta (email:kats@mozilla.com)
blassey.bugs: review+
Details | Diff | Splinter Review
patch to actually use tls (1.73 KB, patch)
2012-07-17 07:20 PDT, Brad Lassey [:blassey] (use needinfo?)
bugmail: review+
bugmail: checkin+
Details | Diff | Splinter Review
Remove race condition from AndroidBridge initialization (2.54 KB, patch)
2012-07-31 09:25 PDT, Kartikaya Gupta (email:kats@mozilla.com)
snorp: review+
bugmail: checkin+
Details | Diff | Splinter Review

Description Scoobidiver (away) 2012-06-09 01:34:30 PDT
There are only two crashes in 14.0b5 but there's a spike in startup crashes beginning from 16.0a1/20120606. The regression range for the spike is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a7a905fd70d5&tochange=6338a8988917
It's likely a regression from bug 758361 which was uplifted to Aurora.

Signature 	_JNIEnv::CallVoidMethod | mozilla::AndroidGeckoLayerClient::SetFirstPaintViewport More Reports Search
UUID	4a54c3c6-09ec-4376-b96b-87aa02120608
Date Processed	2012-06-08 21:40:28
Uptime	28
Last Crash	3.7 weeks before submission
Install Age	9.2 minutes since version was first installed.
Install Time	2012-06-08 21:31:13
Product	FennecAndroid
Version	16.0a1
Build ID	20120608030525
Release Channel	nightly
OS	Linux
OS Version	0.0.0 Linux 3.0.8+ #9 SMP PREEMPT Wed May 2 18:19:29 CEST 2012 armv7l
Build Architecture	arm
Build Architecture Info	
Crash Reason	SIGSEGV
Crash Address	0x0
App Notes 	
AdapterVendorID: archos, AdapterDeviceID: ARCHOS 80G9.
AdapterDescription: 'Model: 'ARCHOS 80G9', Product: 'G9A80', Manufacturer: 'archos', Hardware: 'archos''.
archos ARCHOS 80G9
archos/G9A80/A80:4.0.3/MR1/20120420.190002:user/release-keys
EMCheckCompatibility	True

Frame 	Module 	Signature 	Source
0 	libdvm.so 	libdvm.so@0x57a70 	
1 	libdvm.so 	libdvm.so@0x57a4f 	
2 	libxul.so 	_JNIEnv::CallVoidMethod 	jni.h:631
3 	libxul.so 	mozilla::AndroidGeckoLayerClient::SetFirstPaintViewport 	widget/android/AndroidJavaWrappers.cpp:664
4 	libxul.so 	mozilla::AndroidBridge::SetFirstPaintViewport 	widget/android/AndroidBridge.cpp:2047
5 	libxul.so 	mozilla::layers::CompositorParent::SetFirstPaintViewport 	gfx/layers/ipc/CompositorParent.cpp:429
6 	libxul.so 	mozilla::layers::CompositorParent::TransformShadowTree 	gfx/layers/ipc/CompositorParent.cpp:374
7 	libxul.so 	mozilla::layers::CompositorParent::Composite 	gfx/layers/ipc/CompositorParent.cpp:260
8 	libxul.so 	RunnableMethod<mozilla::layers::CompositorParent, void , Tuple0>::Run 	ipc/chromium/src/base/tuple.h:383
9 	libxul.so 	MessageLoop::RunTask 	ipc/chromium/src/base/message_loop.cc:318
10 	libxul.so 	MessageLoop::DeferOrRunPendingTask 	ipc/chromium/src/base/message_loop.cc:326
11 	libxul.so 	MessageLoop::DoWork 	ipc/chromium/src/base/message_loop.cc:426
12 	libxul.so 	base::MessagePumpDefault::Run 	ipc/chromium/src/base/message_pump_default.cc:23
13 	libxul.so 	MessageLoop::RunInternal 	ipc/chromium/src/base/message_loop.cc:208
14 	libxul.so 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:201
15 	libxul.so 	base::Thread::ThreadMain 	ipc/chromium/src/base/thread.cc:156
16 	libxul.so 	ThreadFunc 	ipc/chromium/src/base/platform_thread_posix.cc:31
17 	libc.so 	libc.so@0x12c4e 	
18 	libc.so 	libc.so@0x127a2 	
19 	libEGL.so 	libEGL.so@0x23e82 

More reports at:
https://crash-stats.mozilla.com/report/list?signature=_JNIEnv%3A%3ACallVoidMethod+|+mozilla%3A%3AAndroidGeckoLayerClient%3A%3ASetFirstPaintViewport
https://crash-stats.mozilla.com/report/list?signature=_JNIEnv%3A%3ACallVoidMethod
https://crash-stats.mozilla.com/report/list?signature=JNI_CreateJavaVM+|+_JNIEnv%3A%3ACallVoidMethod+|+mozilla%3A%3AAndroidGeckoLayerClient%3A%3ASetFirstPaintViewport
Comment 1 Scoobidiver (away) 2012-06-13 02:04:29 PDT
As 15.0a2 is unaffected, it's not caused by bug 758361.
Comment 2 Kartikaya Gupta (email:kats@mozilla.com) 2012-06-13 07:40:03 PDT
Given that this crash seems like a race condition on startup, I would guess that this change was the one that caused the spike in 16.0:

d945ae198516	Mike Hommey — Bug 760152 - Start library decompression earlier. r=blassey

Just a guess, though.
Comment 3 Naoki Hirata :nhirata (please use needinfo instead of cc) 2012-06-14 09:19:12 PDT
Setting as top crash : 6th on nightly, does not seem to appear in aurora, or beta
Comment 4 Kartikaya Gupta (email:kats@mozilla.com) 2012-06-14 10:35:59 PDT
CC'ing :glandium. Does bug 760152 only do library decompression earlier? Does it do any other gecko startup earlier?
Comment 5 Mike Hommey [:glandium] 2012-06-14 10:44:29 PDT
(In reply to Kartikaya Gupta (:kats) from comment #4)
> CC'ing :glandium. Does bug 760152 only do library decompression earlier?
> Does it do any other gecko startup earlier?

Gecko's actual initialization has not moved. Only decompression did.
Comment 6 Kartikaya Gupta (email:kats@mozilla.com) 2012-06-18 06:42:19 PDT
This crash is showing up a lot on tbpl; a lot of the android mochitest oranges are due to this. For example:

https://tbpl.mozilla.org/php/getParsedLog.php?id=12739494&tree=Firefox&full=1#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=12741230&tree=Firefox&full=1#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=12751669&tree=Firefox&full=1#error0
Comment 9 Phil Ringnalda (:philor, back in August) 2012-06-20 12:26:40 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=12834383&tree=Try
Comment 10 Phil Ringnalda (:philor, back in August) 2012-06-20 12:28:38 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=12836079&tree=Mozilla-Inbound
Comment 11 Phil Ringnalda (:philor, back in August) 2012-06-20 14:00:01 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=12838579&tree=Mozilla-Inbound
Comment 23 Phil Ringnalda (:philor, back in August) 2012-07-01 20:41:51 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=13154543&tree=Ionmonkey
Comment 24 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-12 14:21:32 PDT
Created attachment 641598 [details] [diff] [review]
Add a GetJNIForCompositorThread and make the compositor JNI invocations use it

This should be sufficient to fix this crash, assuming the problem is in fact the GetJNIForThread call. I can kill the GetJNIForThread() more in a separate bug.
Comment 25 Brad Lassey [:blassey] (use needinfo?) 2012-07-12 14:41:49 PDT
Comment on attachment 641598 [details] [diff] [review]
Add a GetJNIForCompositorThread and make the compositor JNI invocations use it

Review of attachment 641598 [details] [diff] [review]:
-----------------------------------------------------------------

::: widget/android/AndroidBridge.h
@@ +132,5 @@
> +    static JNIEnv* GetJNIForCompositorThread() {
> +        if (NS_LIKELY(sBridge)) {
> +            if (sBridge->mCompositorThread) {
> +                if ((void*)pthread_self() != sBridge->mCompositorThread) {
> +                    __android_log_print(ANDROID_LOG_ERROR, "AndroidBridge", "Non-compositor thread calling GetJNIForCompositorThread!");

this should probably be a NS_ABORT_IF_FALSE()

@@ +145,5 @@
> +            MutexAutoLock lock(sBridge->mCompositorJNICreationMutex);
> +
> +            if (sBridge->mCompositorThread) {
> +                // this means that another thread executed this function between the time we started executing
> +                // it and the time we acquired the mutex. fail.

this also means we have two threads that claim to be the compositor thread, which sounds like a disaster. Let's abort here too
Comment 26 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-13 06:40:26 PDT
Landed with NS_ABORT()s in the relevant places.

https://hg.mozilla.org/integration/mozilla-inbound/rev/083d36bafbc8
Comment 27 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-13 14:38:06 PDT
Backed out on suspicion of making all the android talos tests go red. Nothing useful in the log though.

https://hg.mozilla.org/integration/mozilla-inbound/rev/f3215d2dc286
Comment 28 :Ehsan Akhgari 2012-07-13 17:08:06 PDT
(In reply to comment #27)
> Backed out on suspicion of making all the android talos tests go red. Nothing
> useful in the log though.
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/f3215d2dc286

The backout seems to have fixed it.
Comment 29 Brad Lassey [:blassey] (use needinfo?) 2012-07-17 07:20:48 PDT
Created attachment 642940 [details] [diff] [review]
patch to actually use tls
Comment 30 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-17 07:22:59 PDT
Comment on attachment 642940 [details] [diff] [review]
patch to actually use tls

Review of attachment 642940 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with changing the if (status < 0) to if (status) as well.
Comment 31 Brad Lassey [:blassey] (use needinfo?) 2012-07-17 08:08:20 PDT
https://hg.mozilla.org/integration/mozilla-inbound/rev/2ee313f65ccf
Comment 32 Ed Morley [:emorley] 2012-07-18 05:57:29 PDT
https://hg.mozilla.org/mozilla-central/rev/2ee313f65ccf
Comment 34 Scoobidiver (away) 2012-07-27 23:13:15 PDT
14.0.1 and 15.0 Beta are affected.
Comment 35 Scoobidiver (away) 2012-07-29 11:21:16 PDT
Let's restrict this bug to top crashers.
It's #3 top crasher in 15.0b2. The Beta regression range is:
http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=695042299ec9&tochange=166ba24e4239
Bug 760152 is the only bug that belongs to the two regression windows.
Comment 37 Mike Hommey [:glandium] 2012-07-30 23:04:26 PDT
(In reply to Mike Hommey [:glandium] from comment #5)
> (In reply to Kartikaya Gupta (:kats) from comment #4)
> > CC'ing :glandium. Does bug 760152 only do library decompression earlier?
> > Does it do any other gecko startup earlier?
> 
> Gecko's actual initialization has not moved. Only decompression did.

actually, one thing moved: static initialization. Is there any static initialization that calls back into java?
Comment 40 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-31 09:24:14 PDT
(In reply to Mike Hommey [:glandium] from comment #37)
> actually, one thing moved: static initialization. Is there any static
> initialization that calls back into java?

I'm not aware of any offhand, but it's quite possible that there is. Looking at the code again, I found something that might be the problem here. Patch coming in a sec.
Comment 41 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-31 09:25:12 PDT
Created attachment 647566 [details] [diff] [review]
Remove race condition from AndroidBridge initialization

Pushed this to try at https://tbpl.mozilla.org/?tree=Try&rev=b40d1422b1cb, will request review if it comes back green.
Comment 42 Alex Keybl [:akeybl] 2012-07-31 10:01:38 PDT
I think I'd rather we just backed out bug 760152 on branches at this point, since it didn't have any obvious perf gains. Is that riskier?
Comment 43 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-31 10:30:38 PDT
Doing the backout is probably the better option. I don't know if my patch will actually fix this or not yet. If it does we can re-land it with the fix.
Comment 44 Alex Keybl [:akeybl] 2012-07-31 12:01:56 PDT
(In reply to Kartikaya Gupta (:kats) from comment #43)
> Doing the backout is probably the better option. I don't know if my patch
> will actually fix this or not yet. If it does we can re-land it with the fix.

Thanks, sending over to glandium in that case.
Comment 45 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-31 14:23:58 PDT
Comment on attachment 647566 [details] [diff] [review]
Remove race condition from AndroidBridge initialization

New try run is at https://tbpl.mozilla.org/?tree=Try&rev=e3e0e2f7ec4f (old one failed to build because i used a busted inbound changeset as a base). It looks about as green as android usually does, so requesting review.
Comment 46 Kartikaya Gupta (email:kats@mozilla.com) 2012-07-31 14:36:09 PDT
https://hg.mozilla.org/integration/mozilla-inbound/rev/87736e458d15
Comment 47 Ed Morley [:emorley] 2012-08-01 02:53:06 PDT
https://hg.mozilla.org/mozilla-central/rev/87736e458d15
Comment 48 Scoobidiver (away) 2012-08-02 01:40:55 PDT
Fixed in Aurora by the backout of bug 760152:
http://hg.mozilla.org/releases/mozilla-aurora/rev/e670dfc55dc8
Comment 49 Scoobidiver (away) 2012-08-03 01:22:20 PDT
There are still crashes in the trunk.
Comment 50 Mike Hommey [:glandium] 2012-08-03 01:43:02 PDT
(In reply to Scoobidiver from comment #49)
> There are still crashes in the trunk.

Do you have evidence of that? There are no crashes I can see with a build id from a build after the landing:
https://crash-stats.mozilla.com/report/list?range_value=7&range_unit=days&date=2012-08-02&signature=_JNIEnv%3A%3ACallVoidMethod%20|%20mozilla%3A%3AAndroidGeckoLayerClient%3A%3ASetFirstPaintViewport&version=FennecAndroid%3A17.0a1
Comment 52 Scoobidiver (away) 2012-08-04 02:14:40 PDT
There are no crashes after 16.0a2/20120801, i.e. the backout of bug 760152.
Comment 53 Naoki Hirata :nhirata (please use needinfo instead of cc) 2012-08-06 10:38:57 PDT
Crash on 8/3
https://crash-stats.mozilla.com/report/index/0e11bda2-b749-4f1f-944b-7430f2120806
Comment 54 Naoki Hirata :nhirata (please use needinfo instead of cc) 2012-08-06 10:47:17 PDT
https://crash-stats.mozilla.com/report/index/1fd05d7b-0f24-4063-9a84-ee7782120806
https://crash-stats.mozilla.com/report/index/e4673839-1a8f-4f07-ab22-956902120806

Note to clarify.. that's only 3 report that shows on/after 8/1 out of 2,838+634+1 Crash Reports.  Only 1 report after 8/1.

Should we close this out?  Or keep this and mark as a lower priority?  People just need to update it seems.
Comment 55 Scoobidiver (away) 2012-08-06 11:06:40 PDT
(In reply to Naoki Hirata :nhirata from comment #54)
> Should we close this out?  Or keep this and mark as a lower priority? 
No. When 17 Branch will go to Aurora, it will be #3 top crasher in 17.0a2 like it was in 16.0a2.
Comment 56 Phil Ringnalda (:philor, back in August) 2012-08-07 11:38:34 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14195251&tree=Mozilla-Inbound
Comment 57 Lukas Blakk [:lsblakk] use ?needinfo 2012-08-07 12:03:26 PDT
Bug 760152 was backed out of Beta 15 yesterday.
Comment 58 Phil Ringnalda (:philor, back in August) 2012-08-07 14:44:12 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14198783&tree=Mozilla-Inbound
Comment 59 Phil Ringnalda (:philor, back in August) 2012-08-07 14:45:15 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14196700&tree=Mozilla-Inbound
Comment 60 Phil Ringnalda (:philor, back in August) 2012-08-07 14:45:59 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14196485&tree=Mozilla-Inbound
Comment 61 Phil Ringnalda (:philor, back in August) 2012-08-07 14:48:03 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14200983&tree=Mozilla-Inbound
Comment 62 Phil Ringnalda (:philor, back in August) 2012-08-07 18:27:20 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14206103&tree=Mozilla-Inbound
Comment 63 Phil Ringnalda (:philor, back in August) 2012-08-08 01:09:45 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14216916&tree=Mozilla-Inbound
Comment 64 Ed Morley [:emorley] 2012-08-08 04:48:43 PDT
(In reply to Naoki Hirata :nhirata from comment #54)
> Should we close this out?  Or keep this and mark as a lower priority? 
> People just need to update it seems.

No. This is still occurring on trunk.
Comment 71 Phil Ringnalda (:philor, back in August) 2012-08-08 21:59:18 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14247039&tree=Mozilla-Inbound
Comment 74 Ed Morley [:emorley] 2012-08-09 05:26:05 PDT
Assigning to kats after speaking to glandium.
Comment 78 Geoff Brown [:gbrown] 2012-08-09 07:30:43 PDT
It looks like all of the recent logs are from mochitest runs -- why would that be?
Comment 80 Phil Ringnalda (:philor, back in August) 2012-08-09 10:13:40 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14263452&tree=Mozilla-Inbound
Comment 82 Phil Ringnalda (:philor, back in August) 2012-08-09 21:22:06 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14280678&tree=Firefox
Comment 89 Phil Ringnalda (:philor, back in August) 2012-08-10 07:36:43 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14293607&tree=Mozilla-Inbound
Comment 90 Phil Ringnalda (:philor, back in August) 2012-08-10 10:59:58 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14297906&tree=Mozilla-Inbound
Comment 91 Phil Ringnalda (:philor, back in August) 2012-08-10 21:33:31 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14308165&tree=Ionmonkey
Comment 92 Phil Ringnalda (:philor, back in August) 2012-08-11 12:12:26 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14318134&tree=Mozilla-Inbound
Comment 93 Phil Ringnalda (:philor, back in August) 2012-08-12 09:07:03 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14324223&tree=Firefox
Comment 94 Phil Ringnalda (:philor, back in August) 2012-08-12 14:37:44 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14327702&tree=Try
Comment 95 Phil Ringnalda (:philor, back in August) 2012-08-12 21:02:44 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14330324&tree=Mozilla-Inbound
Comment 96 Phil Ringnalda (:philor, back in August) 2012-08-12 22:09:18 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14331823&tree=Mozilla-Inbound
Comment 97 Phil Ringnalda (:philor, back in August) 2012-08-12 22:47:36 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14332284&tree=Mozilla-Inbound
Comment 99 Phil Ringnalda (:philor, back in August) 2012-08-13 12:05:54 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14346280&tree=Build-System
Comment 101 Ed Morley [:emorley] 2012-08-13 15:20:27 PDT
Kats, do you have any ideas for this?
Comment 102 Kartikaya Gupta (email:kats@mozilla.com) 2012-08-13 15:28:03 PDT
Unfortunately, no. I've looked through the code multiple times and don't see any other problems that could cause this.
Comment 103 Phil Ringnalda (:philor, back in August) 2012-08-13 17:39:34 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14340005&tree=Ionmonkey
Comment 104 Phil Ringnalda (:philor, back in August) 2012-08-13 21:58:53 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14359828&tree=Mozilla-Inbound
Comment 105 Phil Ringnalda (:philor, back in August) 2012-08-13 23:06:06 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14361228&tree=Mozilla-Inbound
Comment 106 Phil Ringnalda (:philor, back in August) 2012-08-14 00:01:20 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14361928&tree=Fx-Team
Comment 107 Phil Ringnalda (:philor, back in August) 2012-08-14 00:01:57 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14361872&tree=Fx-Team
Comment 110 Kartikaya Gupta (email:kats@mozilla.com) 2012-08-14 08:21:04 PDT
(In reply to Ed Morley [:edmorley] from comment #101)
> Kats, do you have any ideas for this?

Actually, we should back out bug 760152 on 17 as well. Backing that out seems to have fixed the error completely on 15 and 16, so it should help significantly on 17. Note that this error was happening even before bug 760152 landed (albeit at a very low crash volume) so I don't expect it to disappear completely, unless one of the other patches on this bug did actually fix something. (i.e. bug 760152 might have masked the original problem, which was fixed by one of the patches on this bug).

I posted a comment on bug 760152 requesting it be backed out on 17 as well.
Comment 111 Ed Morley [:emorley] 2012-08-14 08:28:16 PDT
Thank you :-)
Comment 116 Phil Ringnalda (:philor, back in August) 2012-08-15 17:12:05 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14415209&tree=Mozilla-Inbound
Comment 117 Phil Ringnalda (:philor, back in August) 2012-08-15 17:12:32 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14417380&tree=Mozilla-Inbound
Comment 118 Phil Ringnalda (:philor, back in August) 2012-08-15 17:13:01 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14416658&tree=Mozilla-Inbound
Comment 120 Phil Ringnalda (:philor, back in August) 2012-08-15 19:12:58 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14421092&tree=Mozilla-Inbound
Comment 121 Kartikaya Gupta (email:kats@mozilla.com) 2012-08-15 20:38:15 PDT
I pushed the backout of bug 760152 to inbound so hopefully this should stop happening as that propagates to the various trees. *fingers crossed*
Comment 122 Phil Ringnalda (:philor, back in August) 2012-08-15 23:28:50 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14423258&tree=Mozilla-Inbound (from before)
Comment 123 Phil Ringnalda (:philor, back in August) 2012-08-15 23:44:22 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14421893&tree=Firefox
Comment 125 Phil Ringnalda (:philor, back in August) 2012-08-16 07:36:54 PDT
https://tbpl.mozilla.org/php/getParsedLog.php?id=14429953&tree=Ionmonkey
Comment 126 Robert Kaiser 2012-08-20 04:50:40 PDT
This has stopped after the build from the 16th, so the backout worked. Should we mark this bug fixed?
Comment 127 Kartikaya Gupta (email:kats@mozilla.com) 2012-08-20 09:42:46 PDT
Sounds good to me.

Note You need to log in before you can comment on or make changes to this bug.