Open Bug 984012 Opened 11 years ago Updated 1 year ago

Consider removing the security.turn_off_all_security_so_that_viruses_can_take_over_this_computer pref

Categories

(Core :: XPConnect, defect)

Product:
Core
Component:
XPConnect
Type:
defect

Tracking

()

People

(Reporter: jandem, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: sec-audit)

Attachments

(1 file, 1 obsolete file)

less aggressive patch which leaves forcePCOS/forcePCOW alone
17.09 KB, patch
Details | Diff | Splinter Review
The pwn2own exploit in bug 982974 sets the following pref after it can read/write arbitrary memory: security.turn_off_all_security_so_that_viruses_can_take_over_this_computer Is this pref really necessary for our release builds? Without this pref he would have found another way, but we should do what we can to make attacks harder or less reliable IMO.
Component: Security → XPConnect
951948
Depends on: 951948
Keywords: sec-audit
(In reply to Jan de Mooij [:jandem] from comment #0) > Is this pref really necessary for our release builds? Assuming we want to be able to run our test suite on release builds, yes. > Without this pref he > would have found another way, but we should do what we can to make attacks > harder or less reliable IMO. At the point when the attacker is reading and writing arbitrary memory with enough precision to set prefs, I think we've clearly lost. Any attacker who gets that far isn't going to be deterred by finding one of the millions of ways to execute arbitrary code in that situation.
Time for another episode of "Jesse argues with himself"! > It sounds dangerous to have a pref like this. At least it's clear what the pref does. If an exploit used another pref, people might mistakenly believe that it demonstrated a second vulnerability in Firefox, rather than just a cross-platform way to turn [a pair of arbitrary memory read and write] into privilege escalation. > If I try out an exploit that uses this pref, does it leave the pref set, > leaving me vulnerable until I go to about:config and unset it manually? Probably. But that would be true for any other abused pref. > Why does this pref exist? Because some tests still use enablePrivilege: https://bugzilla.mozilla.org/showdependencytree.cgi?id=546848&hide_resolved=1 > Tests run with this pref? Does that mean it's dangerous to run tests, > since they might have bugs that cause them to load web content? Switching from enablePrivilege to SpecialPowers would not fix this, because SpecialPowers offers overpowered APIs.
Group: core-security
Depends on: 546848
I wonder if there's some lightweight way we could make this harder to take advantage of, like at startup decide if we're going to respect this pref (via environment var or whatever), then store it in a var, and make the page containing the var read only. Maybe that doesn't help any. > Any attacker who gets that far isn't going to be deterred by finding one of the millions of ways to execute arbitrary code in that situation. Sure, but you could make this argument for almost any security bug. ;)
Depends on: 788653
I count only 29 tests in m-c that still use enablePrivilege: dom/plugins/test/mochitest/cocoa_focus.html dom/plugins/test/mochitest/cocoa_window_focus.html dom/xml/test/old/load/loadauth.html dom/xslt/tests/XSLTMark/XSLTMark-view.js dom/xslt/tests/XSLTMark/XSLTMark-static.js dom/tests/mochitest/bugs/test_bug346659.html dom/tests/mochitest/bugs/bug346659-opener-echoer.html dom/tests/mochitest/bugs/bug346659-parent-echoer.html dom/tests/mochitest/bugs/test_bug61098.html dom/tests/mochitest/ajax/offline/460353_iframe_ownmanifest.html dom/tests/mochitest/ajax/offline/offlineTests.js dom/tests/mochitest/ajax/offline/test_fallback.html embedding/test/test_bug449141.html extensions/cookie/test/file_testloadflags.js gfx/tests/crashtests/358732-iframe.html gfx/tests/crashtests/358732-3.html js/xpconnect/tests/mochitest/test_bug504877.html layout/tools/tests/regression_tests.js layout/tools/tests/content_dumping.html layout/tools/tests/debug_utils.html layout/generic/crashtests/370174-3.html layout/generic/crashtests/370174-4.html netwerk/wifi/tests/wifi_access_point_test.html testing/web-platform/tests/2dcontext/tools/gentest.py toolkit/components/satchel/test/subtst_form_submission_1.html toolkit/components/satchel/test/test_form_submission.html uriloader/exthandler/tests/mochitest/test_unsafeBidiChars.xhtml widget/tests/test_bug565392.html xpfe/test/winopen.js None of these appear to be terribly important. If we disabled them all in release builds, we could restrict enablePrivilege to debug builds, which (as suggested in comment 1) would make it useless for exploits "in the wild".
I think some of these aren't even "real" tests that we run in automation (i.e., they're just files lying around in the tree). In particular, I'm thinking of these, although I might be slightly off: (In reply to Zack Weinberg (:zwol) from comment #6) > dom/xml/test/old/load/loadauth.html > dom/xslt/tests/XSLTMark/XSLTMark-view.js > dom/xslt/tests/XSLTMark/XSLTMark-static.js > layout/tools/tests/regression_tests.js > layout/tools/tests/content_dumping.html > layout/tools/tests/debug_utils.html > netwerk/wifi/tests/wifi_access_point_test.html > xpfe/test/winopen.js
FWIW, something like bug 440908 would allow to make the pref unmodifiable to normal users and exploits, while still allowing it to be changed for tests.
Note that Talos also makes use of enablePrivilege. There were abandoned efforts to fix that in bug 715588.
Depends on: 920013
Depends on: 871323
Depends on: 1145553
I was working on removing enablePrivilege calls in the tests. Indeed, a couple of them are just files that could be removed, I think. Other rewrite/removal enablePrivilege bugs: > dom/plugins/test/mochitest/cocoa_focus.html > dom/plugins/test/mochitest/cocoa_window_focus.html That is bug 920013. > dom/tests/mochitest/ajax/offline/460353_iframe_ownmanifest.html > dom/tests/mochitest/ajax/offline/offlineTests.js > dom/tests/mochitest/ajax/offline/test_fallback.html Bug 871323. > gfx/tests/crashtests/358732-iframe.html > gfx/tests/crashtests/358732-3.html > layout/generic/crashtests/370174-3.html > layout/generic/crashtests/370174-4.html Filed bug 1145553.
On further investigation, SpecialPowers itself appears to be doing something not entirely unlike enablePrivilege (and gated on the same pref): getRawComponents() calls Cu.forcePrivilegedComponentsForScope(), which calls CrashIfNotInAutomation(), which crashes the browser if security.turn_off_all_security_so_that_viruses_can_take_over_this_computer isn't set. If the pref *is* set, it appears to poke the chrome Cu/Ci/Cc/Cr globals into the content scope, which is basically what enablePrivilege("UniversalXPConnect") used to do. https://dxr.mozilla.org/mozilla-central/source/testing/specialpowers/content/specialpowersAPI.js#61 https://dxr.mozilla.org/mozilla-central/source/js/xpconnect/src/XPCComponents.cpp#3155 https://dxr.mozilla.org/mozilla-central/source/js/xpconnect/src/xpcprivate.h#3766 This appears to have been added in bug 951948, which is ostensibly about cleaning up JS-engine wrappers, which are black magic to me. :bholley, does this actually imply what I'm worried it does, namely that there is a much larger set of tests that would break if this functionality were compiled out in release builds? Tracing in the other direction, *at least* everything that refers to "SpecialPowers.Components" is potentially affected: https://dxr.mozilla.org/mozilla-central/search?q=SpecialPowers.Components&case=true&redirect=true
Flags: needinfo?(bobbyholley)
Rather than speculate further I prepared a patch that removes all remaining vestiges of enablePrivilege from the core, and I've pushed it to try, and let's find out just how orange it burns. In addition to netscape.security.PrivilegeManager, this zaps forcePermissiveCOWs and forcePrivilegedComponentsForScope, and everything related to UniversalXPConnect inside js/. As such I fully expect this to break a lot of tests that *appear* to have been converted to SpecialPowers (but were only relying on the dark secret inside), as well as the small number of legacy tests that haven't. https://treeherder.mozilla.org/#/jobs?repo=try&revision=faba58f0a0d8
(In reply to Zack Weinberg (:zwol) from comment #11) > On further investigation, SpecialPowers itself appears to be doing something > not entirely unlike enablePrivilege (and gated on the same pref): forcePrivilegedComponentsForScope and forcePermissiveCOWs are somewhat different in the sense that chrome initiates the action (not content). > getRawComponents() calls Cu.forcePrivilegedComponentsForScope(), which calls > CrashIfNotInAutomation(), which crashes the browser if > security.turn_off_all_security_so_that_viruses_can_take_over_this_computer > isn't set. Yes. This doesn't mean the pref has anything the do with enablePrivilege - it just means that the feature is dangerous and I only wanted us to use it for testing, so I put in a belt-and-suspenders check to make sure nobody was calling it from code we ship. We would also be secure without the CrashIfNotInAutomation call - we'd still need to actively shoot ourselves in the foot for this to be a problem. > If the pref *is* set, it appears to poke the chrome Cu/Ci/Cc/Cr > globals into the content scope, which is basically what > enablePrivilege("UniversalXPConnect") used to do. Not really, no. enablePrivileged had dozens of odd side-effects, implications, and weird semantics. > This appears to have been added in bug 951948, which is ostensibly about > cleaning up JS-engine wrappers, which are black magic to me. That bug was infinitely more relevant to security than this one. > :bholley, does > this actually imply what I'm worried it does, namely that there is a much > larger set of tests that would break if this functionality were compiled out > in release builds? What is "this functionality"? Again forcePermissiveCOWs etc have nothing to do with enablePrivilege. (In reply to Zack Weinberg (:zwol) from comment #12) > Rather than speculate further I prepared a patch that removes all remaining > vestiges of enablePrivilege from the core, and I've pushed it to try, and > let's find out just how orange it burns. It will burn a lot. > In addition to netscape.security.PrivilegeManager, this zaps > forcePermissiveCOWs and forcePrivilegedComponentsForScope, and everything > related to UniversalXPConnect inside js/. As such I fully expect this to > break a lot of tests that *appear* to have been converted to SpecialPowers > (but were only relying on the dark secret inside) What is this dichotomy between "conversion to SpecialPowers" and "dark secrets"? SpecialPowers is set of chrome-privileged code (basically a pessimally insecure addon) that we expose to content during automation. There are thousands of ways that chrome code can compromise the browser (indeed, running arbitrary chrome code is one definition of compromising the browser). In general, this bug is a huge waste of time. If somebody can execute arbitrary chrome code, they can already run whatever they want, and don't need forcePermissiveCOWs etc to do it. If somebody is writing arbitrary memory, they can just as easily set the principal of their compartment to gSystemPrincipal. People are just using the enablePrivilege trick because it's cute and marginally easier than the myriad of other options. Trying to remove it is a game of whack-a-mole that you will not win, because the moles have root and are running you in a debugger.
Flags: needinfo?(bobbyholley)
On that note - if any C++ hackers want to work on things that will actually help us significantly with future pwn2owns, please let me know.
the key point here is if flipping the pref is really "marginally easier" or its "a lot easier." several years ago we had a whole class of attacks from gaming type of sites that took advantage of similar prefs to attack users. the scenario went something like this. - entice users to start playing some type of game or engage with the attacking web site. - interrupt the game making it look like the browser was running into some kind of problem or slow down. - offer a fix to the user in the form of instructions to reset the super-power pref to an unsafe value using about config, or installation of rogue addon. both types of methods might be used starting with manual setting change. - once the super power setting was changed, use it to reliably exploit the system. as jesse mentioned its actually good that the pref name says what it does as one last line of defense if a user can be enticed into the phishing for about:config changes style attack. security.turn_off_all_security_so_that_viruses_can_take_over_this_computer almost is so over the top that attackers might joke about this. e.g. your kidding right, why would firefox developers provide this setting? its really ok, and what it really does is it makes the game you are playing run faster... If we really do want this kind of significantly more dangerous pref lying around in release builds we might also think about informing the user they some how got into this dangerous mode with visual signals in the chrome. At least that would help advanced users sort out and diagnose problems that novice users might have created for themselves.
(In reply to chris hofmann from comment #15) > the key point here is if flipping the pref is really "marginally easier" or > its "a lot easier." If we're talking about what exploits do, I think it's the former, unless someone can convince me why mucking with the pref service is materially easier than mucking with the compartment private (or anything else that shellcode can do once it can peek and poke). > several years ago we had a whole class of attacks from gaming type of sites > that took advantage of similar prefs to attack users. I'm not really worried about that, given the name of the pref. > as jesse mentioned its actually good that the pref name says what it does Yep, this was intentional. The downside is that once a year somebody is surprised by it and asks why it's there and everybody comes running back to this bug to discuss it all over again. > security.turn_off_all_security_so_that_viruses_can_take_over_this_computer > almost is so over the top that attackers might joke about this. e.g. your > kidding right, why > would firefox developers provide this setting? its really ok, and what it > really does > is it makes the game you are playing run faster... This doesn't feel like a compelling threat to me, though I could be wrong. > If we really do want this kind of significantly more dangerous pref lying > around in release builds we might also think about informing the user they > some how got into this dangerous mode with visual signals in the chrome. At > least that would help advanced users sort out and diagnose problems that > novice users might have created for themselves. I don't think we want it lying around per se - it's just that this issue doesn't deserve the attention it's getting, and that engineering resources could be a lot better spent on other things (security-related and not). And for this same reason, I don't think a visual indicator is time well-spent.
An interesting defense to the attack in comment 15 might be to add it to safe browsing checks. At least that would give *us* the chance to control the message before the user was phished with dangerous instructions. If a page contains any words about security.turn_off_all_security_so_that_viruses_can_take_over_this_computer throw up the safe browsing warning. Scanning the web for page that talked about the old dangerous prefs were actually the way we found out about the attack scenario, and which sites were using it. We should probably keep an eye on https://search.yahoo.com/yhs/search?p=turn_off_all_security_so_that_viruses_can_take_over_this_computer&ei=UTF-8&hspart=mozilla&hsimp=yhs-002 and https://www.google.com/search?q=turn_off_all_security_so_that_viruses_can_take_over_this_computer&ie=utf-8&oe=utf-8 for the next few weeks to watch for rouge mentions of the use of the pref since this is now getting some good press attention and the wood is burning in the minds of the script kittys. I did a bit of a check just now and did see all the press, developer discussion, and security researcher discussion, but nothing showing up as rogue use AFAICT. The best comment was on https://news.ycombinator.com/item?id=9239028 jayzalowitz 3 hours ago I just wanted to state that I for one welcome our virus overloards
> > security.turn_off_all_security_so_that_viruses_can_take_over_this_computer > > almost is so over the top that attackers might joke about this. e.g. your > > kidding right, why > > would firefox developers provide this setting? its really ok, and what it > > really does > > is it makes the game you are playing run faster... > > This doesn't feel like a compelling threat to me, though I could be wrong. The name of the pref does provide some level of security, but there is the potential for being over the top as mentioned. There is also another far greater risk here since the pref name is in English, and only provides some level of defense to those with at least some level of understanding of English. That leaves 70% of our users possibly not understanding what the intention or what pref does at all. The attack here appears like this to a Simplified Chinese speaking user. The are first present with background and instruction" 为了使我们的网站正常工作，请更改如下所示为true Firefox的县如下面的示例 (translation: To make our site work properly please change the firefox pref shown below to true as shown in the following example - excuse the google translate) The they are provide the pref setting in english turn_off_all_security_so_that_viruses_can_take_over_this_computer=true (user without understanding of english basically sees this part as giberish) Reversing the situation if I asked you: To make our site work properly please change the firefox pref named "关掉所有的安全，这样的病毒可以接管这台电脑" or إيقاف تشغيل كافة الأجهزة الأمنية بحيث الفيروسات يمكن أن تتخذ على هذا الكمبيوتر to "true", would you understand the risks you might be taking?
Is there a way to lock the pref to avoid it being changed by the user via about:config?
guessing there might be, but that might require some special case handling in both the "add a new pref" and/or "change an existing pref" code. wonder who owns about:config these days?
(In reply to Bobby Holley (:bholley) from comment #19) > Is there a way to lock the pref to avoid it being changed by the user via > about:config? Maybe, instead of being a boolean pref, it could be an integer pref whose value had to equal the value of an environment variable? That's no problem for the test automation, but now a social-engineering attacker has to instruct the victim to set a pref *and* muck around with Windows shortcuts, or shell profiles, or whatever the Mac equivalent is. It's only raising the bar, but it seems like it might be raising the bar in a useful way. More mysterious steps = more hassle = more people decide they don't want to see the dancing baby *that* much. Bonus points if the value has to be something that's difficult for the victim to look up, although I confess I don't see a good way to accomplish that and still keep it easy to set in automation.
Attachment #8580719 - Attachment is obsolete: true
I tried a less aggressive patch which removes enablePrivilege but not forcePrivilegedComponentsForScope and forcePermissiveCOWs. That broke only the ~30 remaining tests that explicitly use enablePrivilege, plus Talos. https://treeherder.mozilla.org/#/jobs?repo=try&revision=df5d9c21f0c0 If I understand what was said in comment 13 correctly, unlike enablePrivilege, forcePrivilegedComponentsForScope and forcePermissiveCOWs do *not* facilitate privilege escalation because you have to already be chrome to invoke them. So if we could get to the point where we could land *this* patch, we could probably declare victory. The long pole here, I think, is Talos. Time to reopen bug 715588?
(In reply to Bobby Holley (:bholley) from comment #19) > Is there a way to lock the pref to avoid it being changed by the user via > about:config? (In reply to chris hofmann from comment #20) > guessing there might be, but that might require some special case handling > in both the "add a new pref" and/or "change an existing pref" code. wonder > who owns about:config these days? There was a related issue about marionette a while ago too. In the end, marionette stuff was allowed to be present if Firefox was run with a special run-time flag on startup, e.g. `firefox.exe -marionette` I'd imagine that it might be possible to run automation with an equivalent flag enabled, yet keep end-users safe(r) as they are unlikely to have run Firefox that way. It's just raising the bar.
(In reply to Zack Weinberg (:zwol) from comment #22) > If I understand what was said in comment 13 correctly, unlike > enablePrivilege, forcePrivilegedComponentsForScope and forcePermissiveCOWs > do *not* facilitate privilege escalation because you have to already be > chrome to invoke them. But you _also_ have to be chrome in order to set a pref. You can twiddle with the bits behind setBoolPref with a memory exploit, just like you can twiddle the bits behind forcePermissiveCOWs etc. The non-english-speaker social engineering angle on the pref is indeed a concern (though a small one, I think). > The long pole here, I think, is Talos. Time to reopen bug 715588? I think fixing Talos for this would be welcome cleanup, but I think the a-team has more important work to do. You're welcome to fix it if that interests you.
(In reply to Bobby Holley (:bholley) from comment #19) > Is there a way to lock the pref to avoid it being changed by the user via > about:config? See comment 8.
(In reply to Mike Hommey [:glandium] from comment #25) > (In reply to Bobby Holley (:bholley) from comment #19) > > Is there a way to lock the pref to avoid it being changed by the user via > > about:config? > > See comment 8. (That patch has been in use in Debian for 7 years...)
(In reply to Bobby Holley (:bholley) from comment #24) > (In reply to Zack Weinberg (:zwol) from comment #22) > > If I understand what was said in comment 13 correctly, unlike > > enablePrivilege, forcePrivilegedComponentsForScope and forcePermissiveCOWs > > do *not* facilitate privilege escalation because you have to already be > > chrome to invoke them. > > But you _also_ have to be chrome in order to set a pref. You can twiddle > with the bits behind setBoolPref with a memory exploit, just like you can > twiddle the bits behind forcePermissiveCOWs etc. In terms of memory exploits I would be much more concerned with someone twiddling CompartmentPrivate::universalXPConnectEnabled and/or ::forcePermissiveCOWs. (You said upthread that someone could equally set JSCompartment::principal to the system principal, but to do that the attacker has to learn a pointer to the system principal, which I hope you will agree is, although not impossible for a memory exploit, *more difficult* than flipping one bit.) This particular pref is not present in the prefs table when it's off; diddling the hash table *correctly* strikes me as on a par with learning a pointer to the system principal. (Somehow learn a pointer to PREF_SetBoolPref.) > The non-english-speaker social engineering angle on the pref is indeed a > concern (though a small one, I think). I concur. > I think fixing Talos for this would be welcome cleanup, but I think the > a-team has more important work to do. You're welcome to fix it if that > interests you. Unfortunately not something I have time to do myself just now. :-/
(In reply to Zack Weinberg (:zwol) from comment #27) > In terms of memory exploits I would be much more concerned with someone > twiddling CompartmentPrivate::universalXPConnectEnabled and/or > ::forcePermissiveCOWs. (You said upthread that someone could equally set > JSCompartment::principal to the system principal, but to do that the > attacker has to learn a pointer to the system principal, which I hope you > will agree is, although not impossible for a memory exploit, *more > difficult* than flipping one bit.) I wouldn't be surprised if the system principal ended up being in a consistent memory location in practice. > This particular pref is not present in the prefs table when it's off; > diddling the hash table *correctly* strikes me as on a par with learning a > pointer to the system principal. (Somehow learn a pointer to > PREF_SetBoolPref.) Right, which is totally doable. Messing with the compartment private also requires you to find it, but it's findable. > > I think fixing Talos for this would be welcome cleanup, but I think the > > a-team has more important work to do. You're welcome to fix it if that > > interests you. > > Unfortunately not something I have time to do myself just now. :-/ I think that's the story for everyone else involved here too. ;-)
Depends on: 1149966
Depends on: 911573
See Also: → 440908
See Also: → 1435113
Depends on: 1435113

I killed enablePrivilege (bug 1448967) and forcePermissiveCOWs (bug 1481640), but now IsInAutomation usage proliferates all over the tree to implement test-only danger features ☹

(In reply to Masatoshi Kimura [:emk] from comment #29)

I killed enablePrivilege (bug 1448967) and forcePermissiveCOWs (bug 1481640),

Awesome work!

but now IsInAutomation usage proliferates all over the tree to implement test-only danger features ☹

Indeed! I didn't realize how many people had started using it. That said, a cursory glance suggests that a lot of the use-cases are just about behaving differently under testing, rather than doing something directly unsafe. Would be worth looking through the list more carefully and identifying any cases that look particularly dangerous.

(In reply to Masatoshi Kimura [:emk] from comment #31)

I think the following usages are particularly unsafe.

  • Enables unsigned add-ons, WebExtensions experiments, and add-on sideloading:

Not so worried about these. The threat model here is that an attacker has limited ability to flip some bits in memory. If they're in the position to position a malicious addon to be loaded by Firefox, we've probably lost anyway.

This seems like it could more-plausibly be used as part of a sandbox escape. Gijs, how easy would it be to get rid of this?

Flags: needinfo?(gijskruitbosch+bugs)

(In reply to Bobby Holley (:bholley) from comment #32)

(In reply to Masatoshi Kimura [:emk] from comment #31)

I think the following usages are particularly unsafe.

  • Enables unsigned add-ons, WebExtensions experiments, and add-on sideloading:

Not so worried about these. The threat model here is that an attacker has limited ability to flip some bits in memory. If they're in the position to position a malicious addon to be loaded by Firefox, we've probably lost anyway.

This seems like it could more-plausibly be used as part of a sandbox escape. Gijs, how easy would it be to get rid of this?

It would require fixing all the tests that flip security.allow_unsafe_parent_loads . It's not impossible, though some of those are tedious. Relevant bugs:

bug 1565545
bug 1565276
bug 1565279

Though it looks like we might need some more.

I'm not sure how easy this is for webextension code. My understanding is that we're running some tests with parent-process loads in part to verify that we're not breaking the webextensions code if they run in the parent process, because that is how things work in Fenix. Given e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1626349 , I don't know how hard it is to stop doing that, or what test coverage we lose if we stop running the specific tests that flip these prefs in the parent process configuration.

Flags: needinfo?(gijskruitbosch+bugs)

(In reply to :Gijs (he/him) from comment #33)

It would require fixing all the tests that flip security.allow_unsafe_parent_loads . It's not impossible, though some of those are tedious. Relevant bugs:

bug 1565545
bug 1565276
bug 1565279

Though it looks like we might need some more.

I'm not sure how easy this is for webextension code. My understanding is that we're running some tests with parent-process loads in part to verify that we're not breaking the webextensions code if they run in the parent process, because that is how things work in Fenix. Given e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1626349 , I don't know how hard it is to stop doing that, or what test coverage we lose if we stop running the specific tests that flip these prefs in the parent process configuration.

Shane, do you know what the state of the webextension tests here is?

Flags: needinfo?(mixedpuppy)
Depends on: 1644287

I just added a patch to remove allow_unsafe_parent_loads in the few places it's used in extensions. There is still a mozaddonmanager api test, as well as tests in devtools. Probably best to ping others for those. I'll see if I can dig into the AOM api issue.

Flags: needinfo?(mixedpuppy)
Depends on: 1647410
See Also: → 1729327
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: