Closed Bug 1071217 Opened 10 years ago Closed 10 years ago

Intermittent testHomeBanner | application crashed [@ nsObserverService::AddObserver(nsIObserver*, char const*, bool)]

Categories

(Core :: Graphics: ImageLib, defect)

ARM
Android
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla36
Tracking Status
firefox34 --- unaffected
firefox35 + fixed
firefox36 + fixed
firefox-esr31 --- unaffected

People

(Reporter: RyanVM, Assigned: seth)

References

Details

(Keywords: crash, intermittent-failure, regression)

Attachments

(1 file, 2 obsolete files)

https://tbpl.mozilla.org/php/getParsedLog.php?id=48589299&tree=Fx-Team Android 2.3 Emulator fx-team opt test robocop-2 on 2014-09-22 07:57:23 PDT for push e42e0e8f3791 slave: tst-linux64-spot-992 08:41:59 WARNING - PROCESS-CRASH | testHomeBanner | application crashed [@ nsObserverService::AddObserver(nsIObserver*, char const*, bool)] 08:41:59 INFO - Crash dump filename: /tmp/tmpy0DidJ/1ca099d4-456d-462c-2e5acf2a-0cd1fb2d.dmp 08:41:59 INFO - Operating system: Android 08:41:59 INFO - 0.0.0 Linux 2.6.29-ge3d684d #1 Mon Dec 16 22:26:51 UTC 2013 armv7l generic/sdk/generic:2.3.7/GINGERBREAD/eng.ubuntu.20140123.014351:eng/test-keys 08:41:59 INFO - CPU: arm 08:41:59 INFO - 0 CPUs 08:41:59 INFO - 08:41:59 INFO - Crash reason: SIGSEGV 08:41:59 INFO - Crash address: 0x0 08:41:59 INFO - 08:41:59 INFO - Thread 10 (crashed) 08:41:59 INFO - 0 libxul.so!nsObserverService::AddObserver(nsIObserver*, char const*, bool) [nsObserverService.cpp:e42e0e8f3791 : 260 + 0x2] 08:41:59 INFO - r4 = 0x475b48e0 r5 = 0x4f61d945 r6 = 0x5559bf50 r7 = 0x00000000 08:41:59 INFO - r8 = 0x00000020 r9 = 0x47584030 r10 = 0x4f634e9a fp = 0x5559fb80 08:41:59 INFO - sp = 0x478ff7f8 lr = 0x4e3eff35 pc = 0x4e3df4be 08:41:59 INFO - Found by: given as instruction pointer in context 08:41:59 INFO - 1 libxul.so!mozilla::net::nsHttpAuthCache::nsHttpAuthCache() [nsHttpAuthCache.cpp:e42e0e8f3791 : 64 + 0xd] 08:41:59 INFO - r4 = 0x556a5e2c r5 = 0x00000000 r6 = 0x4e3df4ad r7 = 0x0000000a 08:41:59 INFO - r8 = 0x00000020 r9 = 0x47584030 r10 = 0x4f634e9a fp = 0x5559fb80 08:41:59 INFO - sp = 0x478ff810 pc = 0x4e49e58b 08:41:59 INFO - Found by: call frame info 08:41:59 INFO - 2 libxul.so!mozilla::net::nsHttpHandler::nsHttpHandler() [nsHttpHandler.cpp:e42e0e8f3791 : 205 + 0x5] 08:41:59 INFO - r4 = 0x556a5e00 r5 = 0x00000000 r6 = 0x00000001 r7 = 0x0000000a 08:41:59 INFO - r8 = 0x00000020 r9 = 0x47584030 r10 = 0x4f634e9a fp = 0x5559fb80 08:41:59 INFO - sp = 0x478ff828 pc = 0x4e4c53bb 08:41:59 INFO - Found by: call frame info 08:41:59 INFO - 3 libxul.so!mozilla::net::nsHttpHandlerConstructor [nsNetModule.cpp:e42e0e8f3791 : 242 + 0x3] 08:41:59 INFO - r4 = 0x556a5e00 r5 = 0x478ff898 r6 = 0x478ff898 r7 = 0x4f84e938 08:41:59 INFO - r8 = 0x00000000 r9 = 0x47584030 r10 = 0x4f634e9a fp = 0x5559fb80 08:41:59 INFO - sp = 0x478ff840 pc = 0x4e51369f 08:41:59 INFO - Found by: call frame info 08:41:59 INFO - 4 libxul.so!mozilla::GenericFactory::CreateInstance(nsISupports*, nsID const&, void**) [GenericFactory.cpp:e42e0e8f3791 : 17 + 0x9] 08:41:59 INFO - r4 = 0x4e513685 r5 = 0x478ff898 r6 = 0x00000000 r7 = 0x4f84e938 08:41:59 INFO - r8 = 0x00000000 r9 = 0x47584030 r10 = 0x4f634e9a fp = 0x5559fb80 08:41:59 INFO - sp = 0x478ff858 pc = 0x4e3fa6cd 08:41:59 INFO - Found by: call frame info 08:41:59 INFO - 5 libxul.so!nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) [nsComponentManager.cpp:e42e0e8f3791 : 1190 + 0xb] 08:41:59 INFO - r4 = 0x4e3fa6c1 r5 = 0x478ff898 r6 = 0x00000000 r7 = 0x4f84e938 08:41:59 INFO - r8 = 0x00000000 r9 = 0x47584030 r10 = 0x4f634e9a fp = 0x5559fb80 08:41:59 INFO - sp = 0x478ff860 pc = 0x4e3ecf93
Component: Networking → DOM
Hmm, there seem to be several different stacks here. But most of them seem to be 21:56:37 INFO - 0 libxul.so!nsObserverService::AddObserver(nsIObserver*, char const*, bool) [nsObserverService.cpp:408e9201cebb : 260 + 0x2] 21:56:37 INFO - r4 = 0x4ef0bdb5 r5 = 0x474be940 r6 = 0x56da3e08 r7 = 0x00000000 21:56:37 INFO - r8 = 0x00000000 r9 = 0x4742d120 r10 = 0x4f0c8632 fp = 0x4f1c28f0 21:56:37 INFO - sp = 0x5c3ff878 lr = 0x4dc5463d pc = 0x4dc4344c 21:56:37 INFO - Found by: given as instruction pointer in context 21:56:37 INFO - 1 libxul.so!mozilla::image::ShutdownTracker::Initialize() [ShutdownTracker.cpp:408e9201cebb : 65 + 0x3] 21:56:37 INFO - r4 = 0x474be940 r5 = 0x4dc43425 r6 = 0x5c3ff8ec r7 = 0x00000000 21:56:37 INFO - r8 = 0x00000000 r9 = 0x4742d120 r10 = 0x4f0c8632 fp = 0x4f1c28f0 21:56:37 INFO - sp = 0x5c3ff890 pc = 0x4e0201d3 21:56:37 INFO - Found by: call frame info 21:56:37 INFO - 2 libxul.so!mozilla::image::InitModule() [nsImageModule.cpp:408e9201cebb : 94 + 0x3] 21:56:37 INFO - r4 = 0x4f59a0e0 r5 = 0x00000000 r6 = 0x5c3ff8ec r7 = 0x00000000 21:56:37 INFO - r8 = 0x00000000 r9 = 0x4742d120 r10 = 0x4f0c8632 fp = 0x4f1c28f0 21:56:37 INFO - sp = 0x5c3ff8a8 pc = 0x4e02b34f 21:56:37 INFO - Found by: call frame info 21:56:37 INFO - 3 libxul.so!nsComponentManagerImpl::KnownModule::Load() [nsComponentManager.cpp:408e9201cebb : 858 + 0x1] 21:56:37 INFO - r4 = 0x474b61c0 r5 = 0x00000000 r6 = 0x5c3ff8ec r7 = 0x00000000 21:56:37 INFO - r8 = 0x00000000 r9 = 0x4742d120 r10 = 0x4f0c8632 fp = 0x4f1c28f0 21:56:37 INFO - sp = 0x5c3ff8b8 pc = 0x4dc4fa1d 21:56:37 INFO - Found by: call frame info
Aha, we're using observer service in a wrong thread. This is not good.
Group: core-security
This is a regression from bug 1070340.
Blocks: 1070340
Component: DOM → ImageLib
(But no need to be security sensitive since we explicitly crash when using observer service in wrong thread.)
Group: core-security
And I was looking at the stacks for test_imagecapture.html issue. If bug 1070340 works as expected, then bug 916643 does something wrong with threads.
[Tracking Requested - why for this release]: Nasty crashes.
Keywords: regression
This became a perma-orange with an attempted landing of bug 1087560. It should be easy to fix this, for someone who knows the code. Just don't use the observer service off the main thread anymore. I'm not sure how this isn't perma-crashing, so maybe that's a complicating factor.
Blocks: 1087560
Flags: needinfo?(seth)
We're init'd a module off of the main thread. Is that allowed?
(In reply to Timothy Nikkel (:tn) from comment #19) > We're init'd a module off of the main thread. Is that allowed? My impression is that it's not. Am I wrong? The whole point of bug 1070340 was to avoid touching the observer service off-main-thread!
Flags: needinfo?(seth)
Flags: needinfo?(ayang)
If we are Init'd the image module then we are also going to be calling imgLoader::GlobalInit which also touches the observer service, and that code has been there since at least 2012. So this seems a little odd that bug 1070340 would cause this.
So, ImageEncoder uses on main thread only?
Flags: needinfo?(ayang)
(In reply to Alfredo Yang from comment #22) > So, ImageEncoder uses on main thread only? No, that should work. (We should consider it a bug if it doesn't.) But we should be initializing the image module only on the main thread. That's what I don't get; why does GetImageEncoder() calling do_CreateInstance for an encoder result in us initializing the entire image module? Surely it should already be initialized by the point this test gets run?
Benjamin, do you understand why this is happening? I'm not sure what we're doing wrong here.
Flags: needinfo?(benjamin)
(In reply to Seth Fowler [:seth] from comment #24) > Benjamin, do you understand why this is happening? I'm not sure what we're > doing wrong here. AddObserver MOZ_CRASH()es when called off-main-thread. So don't call AddObserver off-main-thread. :)
In code, we have a crash at line 260, which is: > NS_ENSURE_VALIDCALL This is the macro definition: > #define NS_ENSURE_VALIDCALL \ > if (!NS_IsMainThread()) { \ > MOZ_CRASH("Using observer service off the main thread!"); \ > return NS_ERROR_UNEXPECTED; \ > } \ > if (mShuttingDown) { \ > NS_ERROR("Using observer service after XPCOM shutdown!"); \ > return NS_ERROR_ILLEGAL_DURING_SHUTDOWN; \ > } I'm not sure why the MOZ_CRASH text doesn't appear.
> I don't get; why does GetImageEncoder() calling do_CreateInstance for an > encoder result in us initializing the entire image module? Surely it should > already be initialized by the point this test gets run? That seems to be the crux of the issue. Why do you assume that the image module is already initialized at this point? That doesn't look like a safe assumption to me. It depends on what work the main thread may have already done, and that may or may not have included any image-decoding work, especially if we're early in startup, right? It may be that the main thread is still laying out the initial document and hasn't started to render it yet (or the document doesn't have any images, so the image module wasn't needed). The main thread stack at the time of the crash is: 21:56:37 INFO - Thread 10 21:56:37 INFO - 0 libxul.so!do_QueryFrame::operator nsIScrollableFrame*<nsIScrollableFrame> [nsQueryFrame.h:408e9201cebb : 89 + 0x0] 21:56:37 INFO - 1 libxul.so!nsLayoutUtils::GetScrollableFrameFor(nsIFrame const*) [nsLayoutUtils.cpp:408e9201cebb : 1597 + 0x3] 21:56:37 INFO - 2 libxul.so!mozilla::ContainerState::ComputeOpaqueRect(nsDisplayItem*, nsIFrame const*, nsIFrame const*, mozilla::DisplayItemClip const&, nsDisplayList*, bool*, bool*) [FrameLayerBuilder.cpp:408e9201cebb : 2728 + 0x5] 21:56:37 INFO - 3 libxul.so!mozilla::ContainerState::ProcessDisplayItems(nsDisplayList*) [FrameLayerBuilder.cpp:408e9201cebb : 3074 + 0x21] 21:56:37 INFO - 4 libxul.so!mozilla::FrameLayerBuilder::BuildContainerLayerFor(nsDisplayListBuilder*, mozilla::layers::LayerManager*, nsIFrame*, nsDisplayItem*, nsDisplayList*, mozilla::ContainerLayerParameters const&, mozilla::gfx::Matrix4x4 const*, unsigned int) [FrameLayerBuilder.cpp:408e9201cebb : 4032 + 0x7] 21:56:37 INFO - 5 libxul.so!nsDisplayList::PaintForFrame(nsDisplayListBuilder*, nsRenderingContext*, nsIFrame*, unsigned int) [nsDisplayList.cpp:408e9201cebb : 1368 + 0x7] 21:56:37 INFO - 6 libxul.so!nsDisplayList::PaintRoot(nsDisplayListBuilder*, nsRenderingContext*, unsigned int) [nsDisplayList.cpp:408e9201cebb : 1286 + 0xd] 21:56:37 INFO - 7 libxul.so!nsLayoutUtils::PaintFrame(nsRenderingContext*, nsIFrame*, nsRegion const&, unsigned int, unsigned int) [nsLayoutUtils.cpp:408e9201cebb : 3104 + 0x7] 21:56:37 INFO - 8 libxul.so!PresShell::RenderDocument(nsRect const&, unsigned int, unsigned int, gfxContext*) [nsPresShell.cpp:408e9201cebb : 4935 + 0x15] 21:56:37 INFO - 9 libxul.so!mozilla::AndroidBridge::CaptureThumbnail(nsIDOMWindow*, int, int, int, _jobject*, bool&) [AndroidBridge.cpp:408e9201cebb : 1869 + 0x13] 21:56:37 INFO - 10 libxul.so!ThumbnailRunnable::Run() [nsAppShell.cpp:408e9201cebb : 101 + 0x17] 21:56:37 INFO - 11 libxul.so!RunnableMethod<FdWatcher, void (FdWatcher::*)(), Tuple0>::Run() [tuple.h:408e9201cebb : 383 + 0x13] 21:56:37 INFO - 12 libxul.so!MessageLoop::RunTask(Task*) [message_loop.cc:408e9201cebb : 361 + 0x5] 21:56:37 INFO - 13 libxul.so!MessageLoop::ProcessNextDelayedNonNestableTask() [message_loop.cc:408e9201cebb : 249 + 0x7] 21:56:37 INFO - 14 libxul.so!MessageLoop::DoIdleWork() [message_loop.cc:408e9201cebb : 478 + 0x3] 21:56:37 INFO - 15 libxul.so!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [MessagePump.cpp:408e9201cebb : 132 + 0x7] 21:56:37 INFO - 16 libxul.so!MessageLoop::RunInternal() [message_loop.cc:408e9201cebb : 233 + 0x5] ...XRE_main The crashing thread is clearly initializing the module: 21:56:37 INFO - Thread 51 (crashed) 21:56:37 INFO - 0 libxul.so!nsObserverService::AddObserver(nsIObserver*, char const*, bool) [nsObserverService.cpp:408e9201cebb : 260 + 0x2] 21:56:37 INFO - 1 libxul.so!mozilla::image::ShutdownTracker::Initialize() [ShutdownTracker.cpp:408e9201cebb : 65 + 0x3] 21:56:37 INFO - 2 libxul.so!mozilla::image::InitModule() [nsImageModule.cpp:408e9201cebb : 94 + 0x3] 21:56:37 INFO - 3 libxul.so!nsComponentManagerImpl::KnownModule::Load() [nsComponentManager.cpp:408e9201cebb : 858 + 0x1] 21:56:37 INFO - 4 libxul.so!nsFactoryEntry::GetFactory() [nsComponentManager.cpp:408e9201cebb : 1915 + 0x3] 21:56:37 INFO - 5 libxul.so!nsComponentManagerImpl::CreateInstanceByContractID(char const*, nsISupports*, nsID const&, void**) [nsComponentManager.cpp:408e9201cebb : 1196 + 0x5] 21:56:37 INFO - 6 libxul.so!CallCreateInstance(char const*, nsISupports*, nsID const&, void**) [nsComponentManagerUtils.cpp:408e9201cebb : 151 + 0xb] 21:56:37 INFO - 7 libxul.so!nsCreateInstanceByContractID::operator()(nsID const&, void**) const [nsComponentManagerUtils.cpp:408e9201cebb : 197 + 0xb] 21:56:37 INFO - 8 libxul.so!nsCOMPtr_base::assign_from_helper(nsCOMPtr_helper const&, nsID const&) [nsCOMPtr.cpp:408e9201cebb : 125 + 0xb] 21:56:37 INFO - 9 libxul.so!mozilla::dom::ImageEncoder::GetImageEncoder(nsAString_internal&) [nsCOMPtr.h:408e9201cebb : 662 + 0x3] 21:56:37 INFO - 10 libxul.so!mozilla::dom::ImageEncoder::ExtractDataFromLayersImageAsync(nsAString_internal&, nsAString_internal const&, bool, mozilla::layers::Image*, mozilla::dom::EncodeCompleteCallback*) [ImageEncoder.cpp:408e9201cebb : 254 + 0x3] 21:56:37 INFO - 11 libxul.so!mozilla::CaptureTask::NotifyQueuedTrackChanges(mozilla::MediaStreamGraph*, int, int, long long, unsigned int, mozilla::MediaSegment const&) [CaptureTask.cpp:408e9201cebb : 147 + 0xd] 21:56:37 INFO - 12 libxul.so!mozilla::TrackUnionStream::CopyTrackData(mozilla::StreamBuffer::Track*, unsigned int, long long, long long, bool*) [TrackUnionStream.cpp:408e9201cebb : 371 + 0x25] 21:56:37 INFO - 13 libxul.so!mozilla::TrackUnionStream::ProcessInput(long long, long long, unsigned int) [TrackUnionStream.cpp:408e9201cebb : 108 + 0x25] 21:56:37 INFO - 14 libxul.so!mozilla::MediaStreamGraphImpl::Process(long long, long long) [MediaStreamGraph.cpp:408e9201cebb : 1336 + 0x19] 21:56:37 INFO - 15 libxul.so!mozilla::MediaStreamGraphImpl::OneIteration(long long, long long, long long, long long) [MediaStreamGraph.cpp:408e9201cebb : 1417 + 0x11] 21:56:37 INFO - 16 libxul.so!mozilla::ThreadedDriver::RunThread() [GraphDriver.cpp:408e9201cebb : 305 + 0x21] 21:56:37 INFO - 17 libxul.so!mozilla::MediaStreamGraphInitThreadRunnable::Run() [GraphDriver.cpp:408e9201cebb : 216 + 0x5] 21:56:37 INFO - 18 libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:408e9201cebb : 830 + 0x5]
Flags: needinfo?(benjamin)
(In reply to Jeff Gilbert [:jgilbert] from comment #25) > (In reply to Seth Fowler [:seth] from comment #24) > > Benjamin, do you understand why this is happening? I'm not sure what we're > > doing wrong here. > > AddObserver MOZ_CRASH()es when called off-main-thread. So don't call > AddObserver off-main-thread. :) That's not the question.
So after talking with Benjamin more on IRC, it seems that the problem is that mochitests are split up and this test happens to be the first one run in its 'group' right now. It does make sense, then, that this code is initializing the image module. I'd really prefer not to force us to create encoders on the main thread just because of this initialization order problem. We're much better off forcing the image module to be initialized on the main thread, instead of relying on lazily initializing the module. Benjamin, is there an accepted pattern for doing that?
Flags: needinfo?(benjamin)
I don't think there's a simple answer to that question. It might depend on the API surface and how clients typically use it.
Flags: needinfo?(benjamin)
Could we call the image module init from the init of the modules that use it? Layout, gfx, and the stacks in this bug look like media.
(In reply to Olli Pettay [:smaug] from comment #32) > Maybe initialize it in > http://mxr.mozilla.org/mozilla-central/source/layout/build/nsLayoutStatics. > cpp?rev=28cd206a24fc#145 I think that's a good idea. Note that we already explicitly *shut down* the image module from layout; see here: http://mxr.mozilla.org/mozilla-central/source/image/build/nsImageModule.cpp#123 So this would be consistent with that.
This patch implements Olli's suggestion. I don't know if there is a more explicit way to do this; let me know if there is.
Attachment #8515277 - Flags: review?(benjamin)
Assignee: nobody → seth
Status: NEW → ASSIGNED
(In reply to Timothy Nikkel (:tn) from comment #31) > Could we call the image module init from the init of the modules that use > it? Layout, gfx, and the stacks in this bug look like media. Do you think layout is enough, or should we do it in all these cases?
(In reply to Seth Fowler [:seth] from comment #35) > (In reply to Timothy Nikkel (:tn) from comment #31) > > Could we call the image module init from the init of the modules that use > > it? Layout, gfx, and the stacks in this bug look like media. > > Do you think layout is enough, or should we do it in all these cases? Do we have a guarantee that layout is initialized before media (the cause of this bug)? I don't know much about initialization.
You know that layout is initialized at startup because xpconnect is part of layout and is required to load JS components.
I am a little concerned that this doesn't cause us to early-initialize the gfx module, though. graphics has a weird startup dependency that it isn't initialized until after the profile is loaded, because it loads graphics prefs once and doesn't (can't) honor dynamic pref changes.
(In reply to Benjamin Smedberg [:bsmedberg] from comment #38) > I am a little concerned that this doesn't cause us to early-initialize the > gfx module, though. graphics has a weird startup dependency that it isn't > initialized until after the profile is loaded, because it loads graphics > prefs once and doesn't (can't) honor dynamic pref changes. Unfortunately imagelib now uses the gfxPrefs mechanism; we just recently made that switch. So imagelib has the same issue. Can we ensure that the profile is loaded even earlier? I'm not sure how I would achieve that.
Flags: needinfo?(benjamin)
No, we can't.
Flags: needinfo?(benjamin)
Comment on attachment 8515277 [details] [diff] [review] Explicitly initialize the image module from layout Cancelling this on the presumption that your note about gfxPrefs means that we can't use this approach. Please re-request if I misunderstood.
Attachment #8515277 - Flags: review?(benjamin)
Disabled test_imagecapture.html on Android after it spontaneous went perma-fail on a merge from m-c today (no other branches appear to have been affected). https://hg.mozilla.org/integration/mozilla-inbound/rev/cf41d3155b06
Whiteboard: [test disabled on Android][leave open]
Alright, given that ImageEncoder relies on the gfx module anyway it should be sufficient to initialize the image module from gfxPlatform::Init. That will also ensure that it's safe to use gfxPrefs. Changing the reviewer to Bas since the initialization code has moved. This issue spiked on inbound again today and the test got disabled, but that change hasn't made it to central yet. The final version of the patch will also reenable 'test_imagecapture.html'.
Attachment #8520920 - Flags: review?(bas)
Attachment #8515277 - Attachment is obsolete: true
Actually, it would've been better to include Android in that try job! Here's a second run with only Android: https://tbpl.mozilla.org/?tree=Try&rev=d7a020f1361a
Comment on attachment 8520920 [details] [diff] [review] Explicitly initialize the image module from gfxPlatform::Init Review of attachment 8520920 [details] [diff] [review]: ----------------------------------------------------------------- Looks fine to me.
Attachment #8520920 - Flags: review?(bas) → review+
Thanks for the review! Here's the final version of the patch which also enables the test.
Attachment #8520920 - Attachment is obsolete: true
We'll want this on Aurora too.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: [test disabled on Android][leave open]
Target Milestone: --- → mozilla36
Comment on attachment 8521066 [details] [diff] [review] Explicitly initialize the image module from gfxPlatform::Init Approval Request Comment [Feature/regressing bug #]: Bug 1070340 or bug 916643, depending on your point of view. [User impact if declined]: It's not clear that users could ever trigger this issue in a real browser session. However, uplifting this patch will help out our sheriffs and our test coverage by eliminating either intermittent oranges or the need to disable 'test_imagecapture.html' on the Aurora branch. [Describe test coverage new/current, TBPL]: This is about fixing an issue uncovered by an existing test, 'test_imagecapture.html'. The patch landed on central yesterday and has resolved the problem and allowed us to reenable the test. [Risks and why]: This is a very low risk patch. [String/UUID change made/needed]: None.
Attachment #8521066 - Flags: approval-mozilla-aurora?
Attachment #8521066 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: