Closed Bug 1323277 Opened 8 years ago Closed 5 years ago

Crash in InvalidArrayIndex_CRASH | nsTArray_Impl<T>::ElementAt | mozilla::safebrowsing::HashStore::ReadAddPrefixes

Categories

(Firefox for Android Graveyard :: General, defect, P5)

Unspecified
Android
defect

Tracking

(fennec51+, firefox50 affected, firefox51+ wontfix, firefox52 wontfix, firefox53 wontfix, firefox54 affected)

RESOLVED WORKSFORME
Tracking Status
fennec 51+ ---
firefox50 --- affected
firefox51 + wontfix
firefox52 --- wontfix
firefox53 --- wontfix
firefox54 --- affected

People

(Reporter: kbrosnan, Unassigned)

References

Details

(Keywords: crash, topcrash-android-armv7)

Crash Data

Attachments

(1 file)

[Tracking Requested - why for this release]: Top crash on 51.0b5 This bug was filed from the Socorro interface and is report bp-f099ca09-a78e-452a-ac0a-ac5fd2161213. ============================================================= Mfg Model And Ver ABI # samsung SM-G930V 23 (REL) armeabi-v7a 88 13.8% samsung SM-G930P 23 (REL) armeabi-v7a 43 6.8% samsung SM-G935V 23 (REL) armeabi-v7a 43 6.8% samsung SM-G935T 23 (REL) armeabi-v7a 33 5.2% samsung SAMSUNG-SM-G930A 23 (REL) armeabi-v7a 32 5.0%
Track 51+ as top crash in 51. Hi :kanru, Can you help take a look at this one?
Flags: needinfo?(kchen)
I had a quick look here but it's not obvious. The crash stacks have some levels of inlining so they don't pinpoint the exact place causing the problem. Some browsing through the code indicates all arrays are initialized to fixed lengths (via SetLength or SetCapacity) first and then iterated over with the same count as used for the init, so they don't seem like they can have such out of bounds accesses. The file being read is checksummed so data corruption is also not very likely. I checked for this signature on desktop, but it's very low frequency there: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=nsTArray_Impl%3CT%3E%3A%3AAppendElements%3CT%3E%20%7C%20mozilla%3A%3Asafebrowsing%3A%3AHashStore%3A%3AReadAddPrefixes&date=%3E%3D2016-12-09T15%3A21%3A00.000Z&date=%3C2016-12-16T15%3A21%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_sort=-date&page=1#reports These two caught my attentionh, being from recent Firefoxes: https://crash-stats.mozilla.com/report/index/b737663d-031c-4376-9be9-eff092161213 https://crash-stats.mozilla.com/report/index/6ab3e22e-e117-4e2d-b37a-081232161213 if (!mAddPrefixes.SetCapacity(count, fallible)) { return NS_ERROR_OUT_OF_MEMORY; } for (uint32_t i = 0; i < count; i++) { AddPrefix *add = mAddPrefixes.AppendElement(fallible); <=== add->prefix.FromUint32(0); add->addChunk = chunks[i]; <*** } The crashes claim an invalid write in the indicated (===) line, which is very strange given the SetCapacity right above - it's also why there's no null check on the returned pointer.
This seems to be device specific. The Galaxy S7 & S7 Edge from the North American market.
Assignee: nobody → gpascutto
tracking-fennec: ? → 51+
I checked the crashing devices that are not Galaxy S7 (USA)'s. The top 20 or so that I looked at are all, without exception, Snapdragon 820s with the 2x2 Kryo cores. I didn't see any other chipset or core in the list of crashes. Note that the European Galaxy S7 has a different core config (with other known bugs :-). This makes me rather confident that the Qualcomm Kryo core has a bug that this code is hitting. Not sure if there's anything actionable for us here. Maybe jchen can help with a disassembly of the crashing code, and this may give some insight to what is happening. Because of the above this doesn't look actionable for me, though.
Flags: needinfo?(nchen)
Hmm, looking at the crash stack may not reveal much: we're crashing because our own runtime array bounds checks are catching a memory or register corruption, and the reliable stack before this crash is ReadAddPrefixes which is a few levels higher up. So there's probably going to be too much asm inbetween these two stations to even wager a guess what's wrong with the CPU core.
Assignee: gpascutto → nobody
Correlations tab in crash reports confirms this: (99.43% in signature vs 02.49% overall) adapter_device_id = Adreno (TM) 530 (97.71% in signature vs 02.32% overall) CPU Info = ARMv1 Qualcomm part(0x51002150) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt
Flags: needinfo?(kchen)
Crash happens at [1] when accessing `chunks[i]`. Does look like a corruption of some kind. [1] https://hg.mozilla.org/releases/mozilla-beta/annotate/4dacdeda1d7f/toolkit/components/url-classifier/HashStore.cpp#l837
Flags: needinfo?(nchen)
I reached out and managed to get a hold of a few relevant engineers at Qualcomm, whom I've pointed to this bug. I don't think there's anything actionable or us here without understanding the specifics of the bug in the CPU. There are no crash reports from 52, so either we get lucky with the compiler/alignment there or the recent work for Protover v4 support in nearby code made the failure case go away. I'm going to WONTFIX this on our side. The correlation report is enough proof that this is a device problem, not a Firefox problem.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
According to comment #8, mark 51 won't fix as this is a device problem.
Steps to force the crashing code to run: Install Firefox 50 or 51 (beta) on an Android device. It'll have to be rooted, or some of the following commands won't be possible. adb shell su cd /data/data/org.mozilla.firefox_beta (no _beta if using release) cd files/mozilla ls -l (note the "xxxxxx.default" dir, which is your profile dir) cd <profile dir> ls -l safebrowsing This will display the databases with malware and phishing information. The crash happens when the Google (goog) .sbstore files are periodically updated (the Mozilla ones won't trigger the offending codepath). We need the data to be on disk, but not complete. Note the file sizes. Last modified times here are misleading as this directory is copied/moved on update. Because of this you also shouldn't try to cd into this directory, it'll be deleted from under your feet. -rw------- u0_a90 u0_a90 134920 2017-01-13 21:00 goog-badbinurl-shavar.pset -rw------- u0_a90 u0_a90 57302 2017-01-13 21:00 goog-badbinurl-shavar.sbstore -rw------- u0_a90 u0_a90 369956 2017-01-13 21:00 goog-malware-shavar.pset -rw------- u0_a90 u0_a90 481159 2017-01-13 21:00 goog-malware-shavar.sbstore -rw------- u0_a90 u0_a90 319374 2017-01-13 21:00 goog-phish-shavar.pset -rw------- u0_a90 u0_a90 612536 2017-01-13 21:00 goog-phish-shavar.sbstore -rw------- u0_a90 u0_a90 181260 2017-01-13 21:00 goog-unwanted-shavar.pset -rw------- u0_a90 u0_a90 132052 2017-01-13 21:00 goog-unwanted-shavar.sbstore Force quit Firefox by some combination of ps + kill or swiping it away in the phone's task manager. cat prefs.js Look for the lines: user_pref("browser.safebrowsing.provider.google.lastupdatetime", "1484335676102"); user_pref("browser.safebrowsing.provider.google.nextupdatetime", "1484339276102"); We need to change these, this can be done: 1) By launching the browser, going to about:config, editing them, and force-quitting Firefox again. 2) If you are lucky and your rooted Android device has a vi or sed on-system, by editing in place. 3) By copying the file out, editing, and pushing it back. Assuming (2) or (3), edit to: user_pref("browser.safebrowsing.provider.google.lastupdatetime", "-1"); user_pref("browser.safebrowsing.provider.google.nextupdatetime", "-1"); and push it back. (In the case of my Nexus 4, this required a detour via /sdcard/Download as I couldn't pull from /data/data) Now relaunch Firefox and start monitoring the safebrowsing dir again in the shell. After 5 to 10 seconds or so you should see the files update: -rw------- u0_a90 u0_a90 134948 2017-01-13 21:09 goog-badbinurl-shavar.pset -rw------- u0_a90 u0_a90 57367 2017-01-13 21:09 goog-badbinurl-shavar.sbstore -rw------- u0_a90 u0_a90 526982 2017-01-13 21:09 goog-malware-shavar.pset -rw------- u0_a90 u0_a90 1109131 2017-01-13 21:09 goog-malware-shavar.sbstore -rw------- u0_a90 u0_a90 414302 2017-01-13 21:09 goog-phish-shavar.pset -rw------- u0_a90 u0_a90 1043020 2017-01-13 21:09 goog-phish-shavar.sbstore -rw------- u0_a90 u0_a90 288542 2017-01-13 21:09 goog-unwanted-shavar.pset -rw------- u0_a90 u0_a90 281327 2017-01-13 21:09 goog-unwanted-shavar.sbstore Which means we read the contents from the disk before merging and writing out. It's the reading part where this crash is happening. Note that after iterating this a few times in succession, the databases will be complete and the update won't run. Nuke the safebrowsing folder and force another update (the first iteration now won't run the relevant code because there's nothing to read).
There are almost no crash in release 52 but we've 3706 crashes in release 53 (3647 for 53.0.2). All of them have adapter_device_id = Adreno (TM) 530 [100.0% vs 16.90% if adapter_vendor_id = Qualcomm].
Noting, we still have ~5000 crashes per week for Fennec 52.0.2 with this signature.
I have a user who is seeing this on a google Pixel device in FF53. Is there anything he can do to work around the problem? Or are there any questions we want to ask him about his config? Is this bug really WONTFIX?
Flags: needinfo?(gpascutto)
It looks like the code that was using the hash table here was changed in bug 1339760. I wonder if this helps the situation.
See Also: → 1339760
So far there have not been any crash instances in FF55, although our pre-release population on fennec is a bit small.
(In reply to Ben Kelly [reviewing, but slowly][:bkelly] from comment #14) > I have a user who is seeing this on a google Pixel device in FF53. Is there > anything he can do to work around the problem? Or are there any questions > we want to ask him about his config? > > Is this bug really WONTFIX? The Google Pixel also uses Qualcomm Kryo CPU cores. There is nothing we can do about this, we don't even know the exact code sequence or alignment the CPU breaks on. The user could disable SafeBrowsing and hope that the broken CPU in those devices doesn't cause any other crashes. I don't think we can infer from betas that release will work, as if it's alignment or register allocation or whatever sensitive, any random change in Firefox can make the problem start or stop appearing.
Flags: needinfo?(gpascutto)
Uh, so we're just going to leave a large crasher in a large amount of phones? Pixel is google flagship device right now. Hopefully the changes in bug 1339760 avoid the problem, but it seems strange to just accept these crashes as more devices use these CPUs.
(In reply to Ben Kelly [reviewing, but slowly][:bkelly] from comment #19) > Uh, so we're just going to leave a large crasher in a large amount of > phones? Pixel is google flagship device right now. This isn't limited to the Pixel, it's in every device with a Qualcomm Snapdragon 820/821 chipset, which includes many flagship Samsung Galaxy phones. > Hopefully the changes in bug 1339760 avoid the problem, Very unlikely, they don't change the code near the crash at all, although it might end up running on a different thread. Given that we don't know what the CPU bug is, this might have some effect, but likely it won't, or it might just randomly hide it until an unrelated change makes things crash again. > but it seems strange to just accept these crashes as more devices use these CPUs. What do you propose? We don't even have a repro case, we don't know what's wrong, we only know that when this bug triggers we will *eventually* detect corruption of the CPU state that would cause us to corrupt security related data-structures (this is the thing that triggers the crash). About the only thing I can think of is to put a check similar to the security check that's forcing the crash, and forcing that to bail out with something like a NS_ERROR_OUT_OF_MEMORY. This might catch the bug, it might hide it, or it might cause the crash to move to another unrelated place (i.e. the next code that runs). I'm willing to try that.
Assignee: nobody → gpascutto
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
See Also: 1339760
Do we have any contact with the manufacturer? Maybe they could help us work around it. I mean, why don't other apps crash all the time on these devices? Also, we have a correlation with the chipset to the crash, but do we any any idea in what way the CPU is causing this? I mean, its possible we are doing something illegal and we're only triggering it with the timing on these devices. I'm asking the person with the Pixel if they have noticed any STR.
(In reply to Ben Kelly [reviewing, but slowly][:bkelly] from comment #21) > Do we have any contact with the manufacturer? Maybe they could help us work > around it. See above, comment 8 and 10. They weren't aware of this flaw until we informed them. I've received no updates whether they've been able to reproduce or pinpoint the issue. I can try pinging them again. >I mean, why don't other apps crash all the time on these devices? Because if they did, Qualcomm would have noticed it during development. > Also, we have a correlation with the chipset to the crash, but do we any any > idea in what way the CPU is causing this? As already said above, not at all. If we did, we would have something to work with. > I mean, its possible we are doing something illegal and we're only triggering it with the timing on these > devices. Anything is possible, but I audited the code soon after this bug was filed and didn't find any obvious (or less obvious) ways in which it could go wrong in the way the crash reports indicated. Combine this with the crash only ever being seen on this specific, new-to-the-market CPU core...
I asked :asuth if he saw this since he has a Pixel. He has not, but he noted he does not have a case on his phone. The user I am talking to who gets the crash semi-frequently does have a case on his phone. I guess thermal issues support the bugged CPU issue. The thing I don't understand is why the users are triggering this code path so much. Do we end up leaving the filesystem directory in a state such that we're constantly trying to reload safebrowsing stuff? I didn't think we checked it that often.
(In reply to Ben Kelly [reviewing, but slowly][:bkelly] from comment #23) > The thing I don't understand is why the users are triggering this code path > so much. Do we end up leaving the filesystem directory in a state such that > we're constantly trying to reload safebrowsing stuff? I didn't think we > checked it that often. We check it every 30-45 minutes. There won't always be an update from the server (which triggers the code), but there are very often.
I wonder if it would be possible or worth it to decrease the frequency of checks on these devices.
Adding 54 as affected since I do see some crashes in crash-stats.
(In reply to Ben Kelly [reviewing, but slowly][:bkelly] from comment #25) > I wonder if it would be possible or worth it to decrease the frequency of > checks on these devices. It's not allowed to do so - it's a requirement of the service. There are some plans to make use of the service built into the OS (modern Androids or Google Services added this), but I don't know the state of our efforts there.
For the record, Firefox for Android is compiled with -Os. This doesn't enable general function inlining, but ReadAddPrefixes only has one caller, and: https://gcc.gnu.org/onlinedocs/gcc-4.9.4/gcc/Optimize-Options.html#Optimize-Options -finline-functions-called-once Consider all static functions called once for inlining into their caller even if they are not marked inline. If a call to a given function is integrated, then the function is not output as assembler code in its own right. Enabled at levels -O1, -O2, -O3 and -Os. Note that this *is* enabled at -Os. The function isn't literally static, but GCC's analysis might conclude it is, especially with LTO, and I believed that's what I saw in the crash dumps, where ReadAddPrefixes doesn't have a proper stack and hence no indicated crash line. However, looking closer at the raw crash dump reveals that ReadAddPrefixes might not be inlined, and I confirmed this by disassembling the Firefox 53.0.2 binaries. So this means disabling inlining won't get us anything. We'll have to add an actual guard for the crash conditions. However, this means the corruption can be isolated to this loop: 0x01290ab2 <+62>: cmp r4, r7 0x01290ab4 <+64>: beq.n 0x1290ade <_ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv+106> 0x01290ab6 <+66>: movs r1, #1 0x01290ab8 <+68>: mov r0, r6 0x01290aba <+70>: bl 0x128f164 <_ZN13nsTArray_ImplIN7mozilla12safebrowsing9AddPrefixE25nsTArrayFallibleAllocatorE14AppendElementsIS3_EEPS2_j> 0x01290abe <+74>: mov r1, r9 0x01290ac0 <+76>: movs r2, #4 0x01290ac2 <+78>: str.w r8, [sp, #4] 0x01290ac6 <+82>: mov r10, r0 0x01290ac8 <+84>: blx 0x33d4a4 <memcpy@plt> 0x01290acc <+88>: mov r1, r4 0x01290ace <+90>: mov r0, r5 0x01290ad0 <+92>: bl 0x34d07a <_ZNK13nsTArray_ImplIj27nsTArrayInfallibleAllocatorE9ElementAtEj> 0x01290ad4 <+96>: adds r4, #1 0x01290ad6 <+98>: ldr r3, [r0, #0] 0x01290ad8 <+100>: str.w r3, [r10, #4] 0x01290adc <+104>: b.n 0x1290ab2 <_ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv+62> Where r4 must be "i" and r7 must be "count", and r5 is "chunks". Now, note r7 isn't transmitted to ElementAt(), so it must know the maximum size from the data stored in the FallibleTArray itself, i.e. behind the r5 pointer. This means either r4 is corrupted (causing a wrong index), or r7 is corrupted (which would cause the loop not to bail when it should). The manipulations on r4 in this loop look rather trivial (and are done repeatedly!), so to me the crash is more suggestive of r7 being corrupted (any value higher that the actual chunks size will cause the exact error seen, whereas it requires slightly more coincidence for r4 to corrupt itself to the loop bound). So let's assume the corruption is in r7. 1) It gets corrupted in the loop above. Possible, but we can't say much about this. The loop itself doesn't touch it. Interestingly, the disassembly of "_ZNK13nsTArray_ImplIj27nsTArrayInfallibleAllocatorE9ElementAtEj" is: Dump of assembler code for function _ZNK13nsTArray_ImplIj27nsTArrayInfallibleAllocatorE9ElementAtEj: 0x0034d07a <+0>: push {r3, lr} 0x0034d07c <+2>: mov r3, r1 0x0034d07e <+4>: ldr r0, [r0, #0] 0x0034d080 <+6>: ldr r1, [r0, #0] 0x0034d082 <+8>: cmp r3, r1 0x0034d084 <+10>: bcc.n 0x34d08c <_ZNK13nsTArray_ImplIj27nsTArrayInfallibleAllocatorE9ElementAtEj+18> 0x0034d086 <+12>: mov r0, r3 0x0034d088 <+14>: bl 0x3a5ae4 <_Z23InvalidArrayIndex_CRASHjj> 0x0034d08c <+18>: add.w r0, r0, r3, lsl #2 0x0034d090 <+22>: adds r0, #8 0x0034d092 <+24>: pop {r3, pc} Which means it doesn't touch r7. Our stack unwinding claims r7 is equal to count at this point (that would be the correct value), but I'm not sure how reliable that is in this case, i.e. if it correctly recovers the registers of this frame (it does clobber r7 in _Z23InvalidArrayIndex_CRASHj). 2) It gets corrupted before entry into the loop. What happens to it, and where does it come from? Dump of assembler code for function _ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv: 0x01290a74 <+0>: stmdb sp!, {r0, r1, r4, r5, r6, r7, r8, r9, r10, lr} 0x01290a78 <+4>: add r5, sp, #8 0x01290a7a <+6>: ldr r7, [r0, #16] <----- indexed from r0, which assumedly must be mHeader, meaning this is correctly mHeader.numAddPrefixes 0x01290a7c <+8>: mov r6, r0 0x01290a7e <+10>: ldr r3, [pc, #124] ; (0x1290afc <_ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv+136>) 0x01290a80 <+12>: ldr r0, [r0, #52] ; 0x34 0x01290a82 <+14>: add r3, pc ===================================================================================================================== 0x01290a84 <+16>: mov r2, r7 <----- r7 used to read "count" elements and set "chunks" size 0x01290a86 <+18>: str.w r3, [r5, #-8]! 0x01290a8a <+22>: mov r1, r5 0x01290a8c <+24>: bl 0x1290938 <_ZN7mozilla12safebrowsingL13ByteSliceReadEP14nsIInputStreamP14FallibleTArrayIjEj> 0x01290a90 <+28>: ands.w r4, r0, #2147483648 ; 0x80000000 0x01290a94 <+32>: bne.n 0x1290ae2 <_ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv+110> 0x01290a96 <+34>: adds r6, #72 ; 0x48 0x01290a98 <+36>: mov r1, r7 <----- r7 used to prep mAddPrefixes 0x01290a9a <+38>: movs r2, #8 0x01290a9c <+40>: mov r0, r6 0x01290a9e <+42>: bl 0x350284 <_ZN13nsTArray_baseI27nsTArrayInfallibleAllocator25nsTArray_CopyWithMemutilsE14EnsureCapacityI25nsTArrayFallibleAllocatorEENT_15ResultTypeProxyEjj> 0x01290aa2 <+46>: tst.w r0, #255 ; 0xff 0x01290aa6 <+50>: bne.n 0x1290aac <_ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv+56> 0x01290aa8 <+52>: ldr r4, [pc, #76] ; (0x1290af8 <_ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv+132>) 0x01290aaa <+54>: b.n 0x1290ae4 <_ZN7mozilla12safebrowsing9HashStore15ReadAddPrefixesEv+112> 0x01290aac <+56>: add.w r9, sp, #4 0x01290ab0 <+60>: mov r8, r4 0x01290ab2 <+62>: cmp r4, r7 ===================================================================================================================== So this would mean the corruption would happen between the two lines. There are at least 2 calls inbetween that save and restore the register, so a clobbering of the stack could produce the same problem.
Attachment #8880036 - Attachment is obsolete: true
Attachment #8880036 - Flags: review?(snorp)
By the way, when trying to test this on Nightly, I only got updates for the Mozilla tables, and the Google code gave back an invalid updateURL error, making this problematic to test on Nightly. Is this intentional/known?
Flags: needinfo?(francois)
Samuel Neves pointed out to me that the call to ByteSliceRead contains a zlib uncompress call. If this is a call to the system lib, it means a potential difference that is system specific. I thought we shipped our own zlib exactly to avoid these problems, but inspecting our sources seems to indicate this is not true: https://dxr.mozilla.org/mozilla-central/source/mobile/android/config/mozconfigs/common?q=mobile%2Fandroid%2Fconfig%2Fmozconfigs%2Fcommon&redirect_type=direct#42 Is it possible the SnapDragons are coming with a zlib that is busted in some corner case and clobbering part of the stack?
Flags: needinfo?(snorp)
Depends on: 1375859
(In reply to Gian-Carlo Pascutto [:gcp] from comment #32) > By the way, when trying to test this on Nightly, I only got updates for the > Mozilla tables, and the Google code gave back an invalid updateURL error, > making this problematic to test on Nightly. Is this intentional/known? Are you using a real Nightly build or your own? For your own builds, you need to put the Google API key in ~/.gapi.data otherwise V4 updates will fail. mozilla-central is now V4-only.
Flags: needinfo?(francois)
Comment on attachment 8880036 [details] Bug 1323277 - Narrow down/avoid device specific SafeBrowsing bugs on Android. https://reviewboard.mozilla.org/r/151364/#review157694 Impressive findings in https://bugzilla.mozilla.org/show_bug.cgi?id=1323277#c30 I've learned a lot. Thanks gcp! (BTW, according your inference, r7 might be corrupted so |count| isn't trustworthy in the loop but we trust |chunks.Length()| so we count on that as the exit condition. With this assumption, do you think it would be more informative to have an assertion to compare |count| and |chunks.Length()|?)
Attachment #8880036 - Flags: review?(hchang) → review+
(In reply to Henry Chang [:hchang] from comment #35) > (BTW, according your inference, r7 might be corrupted so |count| isn't > trustworthy > in the loop but we trust |chunks.Length()| so we count on that as the exit > condition. > With this assumption, do you think it would be more informative to have an > assertion to compare |count| and |chunks.Length()|?) I'm not sure that won't be entirely compiled out by the optimizer. FWIW I'm not going to land this patch unless we continue seeing this issue now that bug 1375859 has landed. (Always using our own zlib)
To clarify, in the first assertion I believe the optimizer will likely generate the same code for count vs mHeader.numAddPrefixes. In the second one it provides no benefit to check because it would be caught by the first check.
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195 Needinfo :susheel if you think this bug should be re-triaged.
Priority: -- → P5
Assignee: gpascutto → nobody

Closing because no crashes reported for 12 weeks.

Status: REOPENED → RESOLVED
Closed: 8 years ago5 years ago
Resolution: --- → WORKSFORME
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: