Closed Bug 984274 Opened 11 years ago Closed 10 years ago

Intermittent test_sandbox_permission.html | Test timed out.

Categories

(Core :: Permission Manager, defect)

x86
Gonk (Firefox OS)
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla33
Tracking Status
firefox31 --- wontfix
firefox32 --- fixed
firefox33 --- fixed
firefox-esr24 --- unaffected
b2g-v1.4 --- fixed
b2g-v2.0 --- fixed
b2g-v2.1 --- fixed

People

(Reporter: cbook, Assigned: ayang)

References

()

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 11 obsolete files)

2.53 KB, patch
fabrice
: review+
fabrice
: feedback+
Details | Diff | Splinter Review
46 bytes, text/x-github-pull-request
fabrice
: review+
Details | Review
b2g_emulator_vm b2g-inbound opt test mochitest-1 on 2014-03-16 20:29:21 PDT for push 26c82a370b06 slave: tst-linux64-spot-942 https://tbpl.mozilla.org/php/getParsedLog.php?id=36248186&tree=B2g-Inbound 37 INFO TEST-UNEXPECTED-FAIL | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | Test timed out.
The tests are actually passed, but somehow it blocks for a long time before running tests. 09:08:26 INFO - 32 INFO TEST-END | /tests/b2g/components/test/mochitest/test_permission_deny.html | finished in 3554ms 09:08:26 INFO - 33 INFO TEST-START | /tests/b2g/components/test/mochitest/test_sandbox_permission.html 09:08:29 INFO - ############################### browserElementPanning.js loaded 09:08:29 INFO - ######################## BrowserElementChildPreload.js loaded 09:08:33 INFO - *** UTM:SVC TimerManager:notify - notified @mozilla.org/b2g/webapps-update-timer;1 09:08:42 INFO - LoadPlugin: failed to initialize shared library libXt.so [libXt.so: cannot open shared object file: No such file or directory] 09:08:42 INFO - LoadPlugin: failed to initialize shared library libXext.so [libXext.so: cannot open shared object file: No such file or directory] 09:08:42 INFO - LoadPlugin: failed to initialize shared library /usr/lib/mozilla/plugins/libtotem-narrowspace-plugin.so [/usr/lib/mozilla/plugins/libtotem-narrowspace-plugin.so: wrong ELF class: ELFCLASS64] 09:08:42 INFO - LoadPlugin: failed to initialize shared library /usr/lib/mozilla/plugins/librhythmbox-itms-detection-plugin.so [/usr/lib/mozilla/plugins/librhythmbox-itms-detection-plugin.so: wrong ELF class: ELFCLASS64] 09:08:42 INFO - LoadPlugin: failed to initialize shared library /usr/lib/mozilla/plugins/libtotem-mully-plugin.so [/usr/lib/mozilla/plugins/libtotem-mully-plugin.so: wrong ELF class: ELFCLASS64] 09:08:42 INFO - LoadPlugin: failed to initialize shared library /usr/lib/mozilla/plugins/libtotem-cone-plugin.so [/usr/lib/mozilla/plugins/libtotem-cone-plugin.so: wrong ELF class: ELFCLASS64] 09:08:42 INFO - LoadPlugin: failed to initialize shared library /usr/lib/mozilla/plugins/libtotem-gmp-plugin.so [/usr/lib/mozilla/plugins/libtotem-gmp-plugin.so: wrong ELF class: ELFCLASS64] 09:08:51 INFO - ###################################### forms.js loaded 09:08:51 INFO - ############################### browserElementPanning.js loaded 09:08:51 INFO - ######################## BrowserElementChildPreload.js loaded 09:10:33 INFO - *** UTM:SVC TimerManager:notify - notified timerID: user-agent-updates-timer 09:10:34 INFO - *** UTM:SVC TimerManager:registerTimer - id: user-agent-updates-timer 09:13:53 INFO - 34 INFO TEST-INFO | dumping last 11 message(s) 09:13:53 INFO - 35 INFO TEST-INFO | if you need more context, please use SimpleTest.requestCompleteLog() in your test 09:13:53 INFO - 36 INFO TEST-INFO | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | request permissions for ["video-capture"] 09:13:53 INFO - 37 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected number of permissions 09:13:53 INFO - 38 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected permission type 09:13:53 INFO - 39 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected permission option 09:13:53 INFO - 40 INFO TEST-INFO | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | request permissions for ["audio-capture","video-capture"] 09:13:53 INFO - 41 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected number of permissions 09:13:53 INFO - 42 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected permission type 09:13:53 INFO - 43 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected permission option 09:13:54 INFO - 44 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected permission type 09:13:54 INFO - 45 INFO TEST-PASS | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | expected permission option 09:13:54 INFO - 46 INFO TEST-INFO | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | request permissions for ["audio-capture"] 09:13:54 INFO - 47 INFO TEST-UNEXPECTED-FAIL | /tests/b2g/components/test/mochitest/test_sandbox_permission.html | Test timed out.
Test harness print out the queued test logs while test is finished/timeout, so the log time doesn't mean anything. [1] http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js#337
Component: WebRTC → WebRTC: Audio/Video
Can't reproduce it after adding logs. I will try again with a clean build. https://tbpl.mozilla.org/?tree=Try&rev=89972f692275
This seems to have significantly spiked recently. Can you please take another look? :)
Flags: needinfo?(ayang)
Component: WebRTC: Audio/Video → Permission Manager
My initial attempts to bisect this down appear to be pointing at this merge to m-c: https://hg.mozilla.org/mozilla-central/pushloghtml?startID=26870&endID=26871 Bob, the test landing from bug 985135 is in there. I'm wondering if we're getting cross-pollution from that or something. I'll try to keep bisecting on inbound to further narrow down the range.
Flags: needinfo?(bobowencode)
Actually, I've now been able to reproduce on an older m-c rev which includes the original patch for bug 985135 instead :P
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #129) > My initial attempts to bisect this down appear to be pointing at this merge > to m-c: > https://hg.mozilla.org/mozilla-central/pushloghtml?startID=26870&endID=26871 > > Bob, the test landing from bug 985135 is in there. I'm wondering if we're > getting cross-pollution from that or something. I'll try to keep bisecting > on inbound to further narrow down the range. I'm not familiar with that test, but it's possible. I'll take a look, not sure how much time I'll have though.
Flags: needinfo?(bobowencode)
Retriggers on m-c are pointing to the below merge as when it started. Bug 985135 definitely stands out there, but I can try to bisect down to a specific push on inbound if it's felt there's any value in doing so. https://hg.mozilla.org/mozilla-central/pushloghtml?startID=26867&endID=26868
this is now getting close to perma/very frequent orange
It is easy to reproduce on emulator with audio+video gUM request. It asserts at http://dxr.mozilla.org/mozilla-central/source/dom/camera/GonkCameraControl.cpp?from=GonkCameraControl.cpp#1044 in my local test. In normal situation, nsGonkCameraControl::StopImpl() should be called in camera thread. But it's possbile to be called in mail thread via MediaEngineWebRTCVideoSource::DeallocImpl() -> nsGonkCameraControl::~nsGonkCameraControl(). I'll do more tests on try to confirm it is the root cause.
Flags: needinfo?(ayang)
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #141) > Retriggers on m-c are pointing to the below merge as when it started. Bug > 985135 definitely stands out there, but I can try to bisect down to a > specific push on inbound if it's felt there's any value in doing so. > https://hg.mozilla.org/mozilla-central/pushloghtml?startID=26867&endID=26868 I can't tell why the hit-rate is getting high in these pushes. I'm afraid it could be other bugs except for comment 176.
I've run the test 5 times on try, with and without the change for bug 985135. It failed 2 times without the fix and 3 times with it. Obviously this isn't conclusive, but I'm not all that sure that my patch has caused this increase.
The retriggers on m-c seem to quite conclusively point at that merge as the culprit, but I've done more retriggers on the push prior to be sure. If bug 985135 isn't at fault, we'll need to bisect within that range on inbound to narrow it down further. Thanks for following up, Bob!
Conveniently, rev 1a00dde01343 on inbound already shows one of those failures too, which predates bug 985135 landing. Inbound retriggers going :)
The rapid failed ratio caused by testcase of bug 978660. It opens the buggy emulator camera hal and deadlock easily. Then the following testcase will be timeout. Here is the try without the testcase, https://tbpl.mozilla.org/?tree=Try&rev=07f332c0b6ef. :schien will update the testcase of bug 978660 soon.
It's sad that we cannot write any test that initiate camera on emulator. I'll disable part of test_permission_gum_remember.html in Bug 1019572 and file another bug for the camera HAL on b2g emulator.
Summary: Intermittent | test_sandbox_permission.html | Test timed out. → Intermittent test_sandbox_permission.html | Test timed out.
Blocks: 1019572
No longer blocks: CVE-2014-1552
No longer blocks: 1019572
Depends on: 1019572
This is the #1 orange on TBPL by a 3x margin. Disabled on B2G. https://hg.mozilla.org/integration/mozilla-inbound/rev/e62ac37ca662
Whiteboard: [test disabled on B2G][leave open]
Mike: see comment 228 and comment 231 This is *really* bad but has been disabled on tbpl
Flags: needinfo?(mhabicher)
(In reply to Randell Jesup [:jesup] from comment #350) > > Mike: see comment 228 and comment 231 > This is *really* bad but has been disabled on tbpl Yeah, I had a patch backed out as collateral damage over the weekend. I'm working on a fix in 1022705.
Flags: needinfo?(mhabicher)
So, this isn't even remotely fixed by bug 1022705...
Flags: needinfo?(mhabicher)
Whiteboard: [test disabled on B2G][leave open]
The frequency seems down, though, so it helped. Mike, can you push a try with the previous test disabled, then retrigger M1 a bunch to verify if that's still the cause? Unless you think we're still pretty certain something with b2g camera is causing it.
Disabled test_sandbox_permission.html for too many intermittent failures: https://hg.mozilla.org/integration/mozilla-inbound/rev/5ea391573087
Keywords: leave-open
Whiteboard: [test disabled]
There are two issues in this bug. 1. The camera HAL is deadlock when closing camera HAL, bug 1021429. 2. The audio-capture and video-capture ALLOW_ACTION permissions pushed in this test are reset by WebappsUpdater to UNKNOWN_ACTION, so the test is failed to get prompt. The WebappsUpdater causes Webapp.jsm to reset the app permission in [1]. Because [1] runs on Task.async(), it could overwrite the new added permissions during testing. I try to disable the WebappsUpdateTimer in test and it works [2]. But I wonder that's a bug in WebappUpdater or PermissionsInstall.jsm. Hi :baku, for issue 2. The new add permissions in this test are overwritten by Webapps.jsm::updateHostedApp due to WebappsUpdate. Is it a bug or any way waiting for PermissionsInstall is completed? Thanks. [1] http://dxr.mozilla.org/mozilla-central/source/dom/apps/src/Webapps.jsm?from=Webapps.jsm#1956 [2] https://tbpl.mozilla.org/?tree=Try&rev=c3f4ae701c79
Flags: needinfo?(amarchesini)
Attached patch disable_app_update (obsolete) — Splinter Review
Disable AppUpdateTimer
Assignee: nobody → ayang
Comment on attachment 8443279 [details] [diff] [review] disable_app_update Review of attachment 8443279 [details] [diff] [review]: ----------------------------------------------------------------- ::: b2g/components/test/mochitest/test_sandbox_permission.html @@ +90,5 @@ > +SpecialPowers.pushPrefEnv({ > + "set": [ > + ["media.navigator.permission.disabled", false], > + ["app.update.enabled", true], > + ["app.update.interval", 86400] Shouldn't we we doing this at the testsuite level, like we do for others suites?
(In reply to Ed Morley [:edmorley UTC+0] from comment #439) > Comment on attachment 8443279 [details] [diff] [review] > disable_app_update > > Review of attachment 8443279 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: b2g/components/test/mochitest/test_sandbox_permission.html > @@ +90,5 @@ > > +SpecialPowers.pushPrefEnv({ > > + "set": [ > > + ["media.navigator.permission.disabled", false], > > + ["app.update.enabled", true], > > + ["app.update.interval", 86400] > > Shouldn't we we doing this at the testsuite level, like we do for others > suites? Yes, bug 1023357 suffers the same issue according to :shien try run in [1]. I'll make it testsuite level. [1] https://tbpl.mozilla.org/?tree=Try&rev=eb77a7e3506b
Attached patch disable_webapp_update_timer (obsolete) — Splinter Review
Disable b2g/components/WebappsUpdateTimer.js. It will reset the permissions added by tests.
Attachment #8446319 - Flags: review?(jgriffin)
Attachment #8443279 - Attachment is obsolete: true
Flags: needinfo?(amarchesini)
Attached patch refactory_sandbox_test (obsolete) — Splinter Review
Refactory, simplified the test script by posting event to notify iframe script.
Attachment #8446323 - Flags: review?(amarchesini)
Comment on attachment 8446319 [details] [diff] [review] disable_webapp_update_timer Review of attachment 8446319 [details] [diff] [review]: ----------------------------------------------------------------- Hmm...I think we should disable both system update and webapp update in test suite. I feel there will be another set of intermittent failures after turning on system update.
After discussing with schien, it's probably better to let fabrice to review it.
Attachment #8446319 - Flags: review?(jgriffin) → review?(fabrice)
Attachment #8446319 - Flags: review?(fabrice) → review+
Comment on attachment 8446319 [details] [diff] [review] disable_webapp_update_timer Review of attachment 8446319 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/profiles/prefs_b2g_unittest.js @@ +10,5 @@ > user_pref("marionette.force-local", true); > + > +// WebappsUpdateTimer will be triggerred without following preferences. That causes app updates during testing. > +user_pref("app.update.enabled", true); > +user_pref("app.update.interval", 86400); Won't that trigger the general update component that also does a network access? Or is the update channel set up so that we actually don't go to the network?
Attachment #8446319 - Flags: review+
(In reply to Fabrice Desré [:fabrice] from comment #445) > Comment on attachment 8446319 [details] [diff] [review] > disable_webapp_update_timer > > Review of attachment 8446319 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: testing/profiles/prefs_b2g_unittest.js > @@ +10,5 @@ > > user_pref("marionette.force-local", true); > > + > > +// WebappsUpdateTimer will be triggerred without following preferences. That causes app updates during testing. > > +user_pref("app.update.enabled", true); > > +user_pref("app.update.interval", 86400); > > Won't that trigger the general update component that also does a network > access? Or is the update channel set up so that we actually don't go to the > network? Yes, it's possible to trigger general update. I'll find other way.
Attachment #8446319 - Attachment is obsolete: true
Attachment #8446323 - Attachment is obsolete: true
Attachment #8446323 - Flags: review?(amarchesini)
Attached patch remove_webappsupdatetimer (obsolete) — Splinter Review
Remove the WebappsUpdateTimer entry from category manager in mochitest b2g_start_script.js. Run a try if any intermittent failure. https://tbpl.mozilla.org/?tree=Try&rev=b9a01f7eb547
Disable webapp update by setting a super long update interval. Try looks green. https://tbpl.mozilla.org/?tree=Try&rev=61e9fe1e8678
Attachment #8448542 - Flags: feedback?(fabrice)
Attachment #8448542 - Flags: feedback?(ayang)
(In reply to Shih-Chiang Chien [:schien] (UTC+8) from comment #449) > Created attachment 8448542 [details] [diff] [review] > disable-webapp-update-timer.patch > > Disable webapp update by setting a super long update interval. Try looks > green. > > https://tbpl.mozilla.org/?tree=Try&rev=61e9fe1e8678 I should said "test cases under /b2g shows stable green".
Another way, remove update-timers from category manager. 2 choices for fabrice to decide. Try is good. https://tbpl.mozilla.org/?tree=Try&rev=3293ee247f04 (The one failure is due to issue 1 in comment 437.)
Attachment #8447089 - Attachment is obsolete: true
Attachment #8448551 - Flags: feedback?(schien)
Attachment #8448551 - Flags: feedback?(fabrice)
I forgot to re-enable test_sandbox_permission.html in my previous patch. Looks like test_sandbox_premission.html is still bad. :( https://tbpl.mozilla.org/?tree=Try&rev=b129ef85da90
Attachment #8448542 - Attachment is obsolete: true
Attachment #8448542 - Flags: feedback?(fabrice)
Attachment #8448542 - Flags: feedback?(ayang)
Comment on attachment 8448551 [details] [diff] [review] remove_webappsupdatetimer Review of attachment 8448551 [details] [diff] [review]: ----------------------------------------------------------------- That looks like a smart idea!
Attachment #8448551 - Flags: feedback?(fabrice) → feedback+
Comment on attachment 8448551 [details] [diff] [review] remove_webappsupdatetimer Thank you, fabrice. I'll add full try run later.
Attachment #8448551 - Flags: feedback?(schien) → review?(fabrice)
Attachment #8448551 - Flags: review?(fabrice) → review+
Comment on attachment 8448551 [details] [diff] [review] remove_webappsupdatetimer Review of attachment 8448551 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/mochitest/b2g_start_script.js @@ +21,5 @@ > > let homescreen = document.getElementById('systemapp'); > let container = homescreen.contentWindow.document.getElementById('test-container'); > > +// Disable udpate timers which cause failure in b2g permisson prompt tests. s/udpate/update and s/failure/failures
And s/permisson/permission
Bonus, this patch fixes the last remaining issue we were hitting for getting bug 995417 landed on b2g30 as well!
Unfortunately, this caused a couple B2G mochitests to perma-fail on b2g30, so I had to back it out. Will be interesting to see if the same failures happen on a trunk Try push. https://tbpl.mozilla.org/php/getParsedLog.php?id=42993183&tree=Mozilla-B2g30-v1.4 https://tbpl.mozilla.org/php/getParsedLog.php?id=42992104&tree=Mozilla-B2g30-v1.4
The failures happen consistently on trunk as well (plus test_fs_appendFile.html, which is new to Gecko 33): https://tbpl.mozilla.org/?tree=Try&rev=1634ae0b61ec
The test failure of test_mozaudiochannel.html is because setting mozAudioChannelType failed by no permission for audio-channel-alarm [1]. It's very weird because the audio-channel-alarm is default allowed for packaged/certified app. I'm trying to figure out why we lost this permission entry. [1] http://dxr.mozilla.org/mozilla-central/source/content/html/content/src/HTMLMediaElement.cpp#2319
The reason of these failures is because the default settings in PermissionsTable.jsm is not in mochitest app. I'm not familiar with mochitest app but I guess it could be mochitest app doesn't initialize PermissionsInstaller.
BTW, the update timers will install the PermissionsTable.jsm once it got notified. So test_mozaudiochannel.html works well in whole b2g try. If you run it alone, it is high failure rate because the update timer is not notified yet.
Actually, it looks like bug 848411.
Attached patch fix-mozaudiochannel-test.patch (obsolete) — Splinter Review
Here is the patch for fixing test_mozaudiochannel.html. Try result shows stable green on mochitest-3 (chuck contains test_mozaudiochannel.html) with this patch.
(In reply to Shih-Chiang Chien [:schien] (UTC+8) from comment #465) > Created attachment 8452077 [details] [diff] [review] > fix-mozaudiochannel-test.patch > > Here is the patch for fixing test_mozaudiochannel.html. Try result shows > stable green on mochitest-3 (chuck contains test_mozaudiochannel.html) with > this patch. Forget to provide the link of try result: https://tbpl.mozilla.org/?tree=Try&rev=dc3b4e1b39ff
Attached patch fix-devicestorage-test.patch (obsolete) — Splinter Review
patch for test_overrideDir.html and test_fs_appendFile.html. try result looks good so far after applying all three patches: https://tbpl.mozilla.org/?tree=Try&rev=1578ad882567
Agreed, everything looks green on the Aurora/b2g30 backport Try runs I ran too :)
Flags: needinfo?(mhabicher)
Keywords: leave-open
Attached patch fix-mozaudiochannel-test.patch (obsolete) — Splinter Review
Mochitest on b2g is executed like a hosted app. We need to explicitly add permission for audio-channel-alarm and audio-channel-content before setting any mozaudiochannel attribute.
Attachment #8452077 - Attachment is obsolete: true
Attachment #8452823 - Flags: review?(amarchesini)
Attached patch fix-devicestorage-test.patch (obsolete) — Splinter Review
Mochitest on b2g is executed like a hosted app. We need to add permission for device-storage:sdcard explicitly before executing test steps.
Attachment #8452195 - Attachment is obsolete: true
Attachment #8452824 - Flags: review?(dhylands)
The problem is PermissionsInstaller.jsm will initialize the permissions in PermissionsTable.jsm for b2g app during normal installation or upgrade; however, mochitest-app doesn't do it. I think schien's patch is good for this bug since I can't figure it out where to add PermissionsInstaller fix on mochitest-app.
So I added sdcard permissions to the mochitest app in the gecko tree (as part of bug 959591): http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/manifest.webapp#19 Perhaps we just need to do the same thing to the mochitest app in the gaia tree? https://github.com/mozilla-b2g/gaia/blob/master/dev_apps/mochitest/manifest.webapp#L18
Attachment #8452824 - Flags: review?(dhylands)
Hmm. Failing under linux, so obviously a gaia change won't have any impact. If the sdcard permissions aren't working, then why are the other ones working? I guess I don't buy the fix. I'd rather see the correct manifest fixed.
I was misreading the TBPL failures. It looks like all of theses failures were on the emulator, which uses the permissions from the manifest.webapp in the gaia tree. So I think it would be better to fix the manifest.webapp in the gaia tree rather than modifying the tests
Thanks to @dhyland's suggestion, tests passed locally after adding the required permissions in Mochitest app. However I cannot find a way to test b2g mochitest with my version of gaia on TBPL, modifying gaia.json [1] only works for b2g-desktop. Any suggestion for how to test it on TBPL? [1] https://wiki.mozilla.org/ReleaseEngineering/TryServer#Using_a_custom_Gaia
Attachment #8452823 - Attachment is obsolete: true
Attachment #8452824 - Attachment is obsolete: true
Attachment #8452823 - Flags: review?(amarchesini)
Attachment #8453588 - Flags: review?(fabrice)
Attachment #8453588 - Flags: review?(fabrice) → review+
(In reply to Shih-Chiang Chien [:schien] (UTC+8) from comment #475) > Created attachment 8453588 [details] [review] > pointer to pull request: add permission for mochitest > > Thanks to @dhyland's suggestion, tests passed locally after adding the > required permissions in Mochitest app. However I cannot find a way to test > b2g mochitest with my version of gaia on TBPL, modifying gaia.json [1] only > works for b2g-desktop. Any suggestion for how to test it on TBPL? > > [1] https://wiki.mozilla.org/ReleaseEngineering/TryServer#Using_a_custom_Gaia I think you'd need to talk to somebody on #ateam, like jgriffin.
Flags: needinfo?(jgriffin)
Just had a discussion with @ahal and @edmorley on IRC, it seems no way we can run b2g emulator tests with customized gaia on try. My gaia patch is just adding permission to mochitest app. @RyanVM, do you think it is appropriate to go ahead landing the pull request first?
Flags: needinfo?(ryanvm)
Worth a shot!
Flags: needinfo?(ryanvm)
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla33
I don't think we're completely out of the woods yet, but bug 1019572 is probably better suited for remaining work at this point rather than overloading this bug.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: