775237 - Before shipping ARMv6 to Google Play (release product), sync device exclusions for unsupported ARMv6 devices

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Alex Keybl [:akeybl]]

Before shipping ARMv6 to Google Play, the ARMv6 device blacklist needs to be in order.

Comment 1

[JP Rosevear [:jpr]]

I think we want a very small whitelist to test the waters - only phones that generally meet the ARMv7 minimum requirements and that our QA has tried.  I'd really just like to gauge the kind of ratings reaction we get.

Comment 2

[Alex Keybl [:akeybl]]

(In reply to JP Rosevear [:jpr] from comment #1)
> I think we want a very small whitelist to test the waters - only phones that
> generally meet the ARMv7 minimum requirements and that our QA has tried. 
> I'd really just like to gauge the kind of ratings reaction we get.

Including the Mobile QA team to help us understand which devices have gotten testing (and could get testing).

On a side note, this may become another gargantuan task of trying to find all ARMv6 phones and blackisting them, so I'm pretty hesitant to commit to doing this in the next beta.

Comment 3

[Irina Sandu]

According to https://wiki.mozilla.org/QA/Fennec/Armv6Compatibility we have these devices that passed:

* Samsung Galaxy Gio
* HTC Wildfire S
* Samsung Galaxy Pro 
 
and these confirmed by the community:

* Samsung Galaxy Ace
* LG Optimus L55C

None of these devices meet our ARMv7 min sys reqs. I can find the ARMv6 models that meet them , but that number is rather limited and if we see that we run well on these that are lower, maybe we can go with these, too. These will be the ones to really tell us how we would be doing if we went on ARMv6 territory, because they represent the most popular hardware spec in that area. 

Alex, is there an option where you can blacklist everything else that we don't have supported now and then just select the ARMv6 ones we want to test? The list of ARMv6 models is indeed rather big, longer than ARMv7.

Comment 4

[Aaron Train [:aaronmt]]

Noting here that:

* Unofficially tested by community means someone reported that the build started and works and that's about it.

Side note on quality -- maybe it's just me, but I am hesitant to jump from merely getting preliminary pre-alpha Nightly builds to pushing them to the the market ever so soon.

Comment 5

[JP Rosevear [:jpr]]

(In reply to Aaron Train [:aaronmt] from comment #4)
> Side note on quality -- maybe it's just me, but I am hesitant to jump from
> merely getting preliminary pre-alpha Nightly builds to pushing them to the
> the market ever so soon.

Understood, but I'd like to get a sense of where the ratings are going to be at sooner rather than later.

Comment 6

[Karen Rudnitski [:kar]]

I also understand the worry about quality, although if we're going to try and release this in FF15, we had best get as much quality feedback as possible - and as soon as issues are identified, make it a priority that we fix those to limit bad ratings. 

I would therefore be inclined to whitelist the 5 devices listed in comment #3 as a start (I like to think big!) with the wording in the release notes stating that we're just starting down the road of supporting these devices and feedback is encouraged.

Comment 7

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I agree with Karen and JPR... if we can release and get some feedback on it on the beta channel, it would be great.  We can aim for a higher rating for the release.

Comment 8

[Armen [:armenzg]]

We currently don't have any test coverage for armv6 devices (neither on tegras nor on pandas).
Are we sure we want to do a release in such conditions? With no way of catching regression?

On another note I *just* landed armv6 beta builds (not beta *release* code).
Tomorrow I will do a reconfiguration to enable it on production.

On another note, there's been zero work done for armv6 builds with our release automation.

If we have to get on this ASAP then we also have to see what other releng work needs to be dropped.

Comment 9

[JP Rosevear [:jpr]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #8)
> We currently don't have any test coverage for armv6 devices (neither on
> tegras nor on pandas).
> Are we sure we want to do a release in such conditions? With no way of
> catching regression?

I understand that, which is why we should do this limited and with QA tested devices only.  An ETA on ARMv6 testing running on ARMv7 machines would be good.

Comment 10

[Armen [:armenzg]]

(In reply to JP Rosevear [:jpr] from comment #9)
> (In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #8)
> > We currently don't have any test coverage for armv6 devices (neither on
> > tegras nor on pandas).
> > Are we sure we want to do a release in such conditions? With no way of
> > catching regression?
> 
> I understand that, which is why we should do this limited and with QA tested
> devices only.  An ETA on ARMv6 testing running on ARMv7 machines would be
> good.

If the work started today, we would need at least 3-6 weeks and that is by having to shut down the android xul tests (we're out of capacity) and probably not the whole set of tests because nothing will come up green at first.

Unfortunately there are too many high priorities right now:
* we need to setup more foopies which drive the tegras
** we need more capacity
* we need to work towards foopy-less setup so we can start driving panda boards and increase reliability of the setup (the current foopy setup has a lot of stability issues that makes test failures hard to catch)

I could go on and on about things that are in releng's side (and I will follow up with blassey and elancaster) but we're drowning with the amount of things in releng side.

I would like this to be dropped until Q4. We need more people and we don't currently have them.

Comment 11

[Tony Chung [:tchung]]

(In reply to Naoki Hirata :nhirata from comment #7)
> I agree with Karen and JPR... if we can release and get some feedback on it
> on the beta channel, it would be great.  We can aim for a higher rating for
> the release.

Before we jump the gun, i propose we have a group discussion on this outside of the bug.   Armen's already pointed out releng concerns on unfinished business and priority work.  

For QA, eric's testing on these devices was basically smoketesting plus a few light BFTs on each device, spending no more than 30 minutes total.   and this was on a nightly build.  He did tell me that overall it fares comparable in performance to the stock browser.   (pretty lousy -- takes about 3-4x longer to load a webpage than on armv7 phone)  But 30 minutes of dogfooding a nightly, doesnt sit enough with me if we're talking about popping it out to production.  (And to Be fair, thats what we asked of him for this task).  Remember, we spent the last 6 months banging hard on Fennec 14 to get a quality product out... so half an hour here makes me nervous.

There's also the question if socorro has crash support for Armv6 yet.  Can anyone answer that question?

I'm all for getting beta testers and expanding coverage, but lets please discuss before making any judgement calls in the bug.

Comment 12

[Aki Sasaki (not active)]

https://bugzilla.mozilla.org/show_bug.cgi?id=775232#c8

Comment 13

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Socorro is getting armv6 crashes: Samsung GT i5700 (ie Samsung Spica is Armv6 I believe):

https://crash-analysis.mozilla.com/rkaiser/2012-07-19/2012-07-19.fennecandroid.nightly.devices.weekly.html#dev-samsung_gt_i5700

Unfortunately Socorro at this time is not separating v6 from v7; the separation is by device names.

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Also note a logged crash bug w/ comment : https://bugzilla.mozilla.org/show_bug.cgi?id=771774#c13

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

Since it's been agreed that we will be targeting Armv6 for Firefox 16, I'm adjusting the tracking flags accordingly.

Comment 16

[Alex Keybl [:akeybl]]

We'll use failed phones on https://wiki.mozilla.org/QA/Fennec/Armv6Compatibility to implement this blocklist.

Comment 17

[Kevin Brosnan [Ex-Mozilla]]

Our initial plan was to deliver builds to armv6 devices that meet the specs 800mhz and 512mb ram.

Comment 18

[Alex Keybl [:akeybl]]

(In reply to Kevin Brosnan [:kbrosnan] from comment #17)
> Our initial plan was to deliver builds to armv6 devices that meet the specs
> 800mhz and 512mb ram.

Correct, and that may end up being what we do for Release. But we'd like to see the feedback that we get on Beta first rather than coming up with an incredibly large device blocklist.

Comment 19

[Erin Lancaster [:elan]]

Checking: is the proposed plan to blocklist everything besides these:
Samsung Galaxy Gio
* HTC Wildfire S
* Samsung Galaxy Pro 
 
and these confirmed by the community:

* Samsung Galaxy Ace
* LG Optimus L55C

Or is it to not block list anything?

Comment 20

[Alex Keybl [:akeybl]]

(In reply to Erin Lancaster [:elancaster] from comment #19)
> Checking: is the proposed plan to blocklist everything besides these:
> Samsung Galaxy Gio
> * HTC Wildfire S
> * Samsung Galaxy Pro 
>  
> and these confirmed by the community:
> 
> * Samsung Galaxy Ace
> * LG Optimus L55C
> 
> Or is it to not block list anything?

Originally we were thinking of blacklisting only the known not working phones. Now we're compiling a blocklist for all phones outside the minimum system requirements at https://etherpad.mozilla.org/armv6blocklist.

Comment 21

[Alex Keybl [:akeybl]]

Bug 789125 tracked doing this on Beta. Let's use this bug for final release.

Comment 22

[Benoit Jacob [:bjacob] (mostly away)]

The single most important cause (over 90%) of ARMv6 startup crashes is lack of OpenGL ES 2 support. Among these crashes, the single most important cause (over 90%) is faulty drivers/hardware with Vivante GC600 GPUs found in rk29board and in imapx200 chipsets (this can be seen in the 'Hardware' field in AppNotes in crash reports). See this table (see bug 771774 comment 28, and earlier tables in bug 778175).

This table records only version 16+ of Firefox.

Date | ARM v7  | ARM v7   | ARMv6   | ARMv6    | ARMv6   | ARMv6
     | crashes | no-GLES2 | crashes | no-GLES2 | Vivante | Vivante
     |         | crashes  |         | crashes  | crashes | no-GLES2
     |         |          |         |          |         | crashes
-----+---------+----------+---------+----------+---------+---------+
0908 |    8375 |       87 |     144 |      128 |     125 |   125
0909 |    8160 |       72 |     214 |      192 |     181 |   181
0910 |    7117 |       71 |     103 |       83 |      76 |    76
0911 |    7151 |       77 |     157 |      134 |     121 |   121
0912 |    7321 |       67 |     122 |      107 |      90 |    90
0913 |    7283 |       75 |     158 |      134 |     134 |   134

So we must blacklist all the devices that have Vivante GC600 GPUs, and that includes all the devices based on rk29board and on imapx200 chipsets.

Comment 23

[Benoit Jacob [:bjacob] (mostly away)]

Here is a command telling which phone models we need to blacklist. Here, 'a' is a file concatenating a week's worth of crash reports, from Sept. 7 to Sept. 13. The first colu

$ cat a | grep armv6 | egrep rk29board\|imapx200 | sed 's|.*Model|Model|g' | sed 's/|.*//g' | sort | uniq -c | sort -nr | sed 's|     ||g'
58 Model: MID7105SR, Product: MID7105SR, Manufacturer: unknown, Hardware: imapx200' 
57 Model: KIRA N10021, Product: KIRA_N10021, Manufacturer: AIRIS, Hardware: imapx200' 
51 Model: KIRA N10021, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
48 Model: DISCO10, Product: DISCO10, Manufacturer: unknown, Hardware: imapx200' 
43 Model: mid, Product: mid, Manufacturer: Infotmic, Hardware: imapx200' 
40 Model: sy0103HB, Product: sy0103HC, Manufacturer: Infotmic, Hardware: imapx200' 
32 Model: N7000, Product: N7000, Manufacturer: AIRIS, Hardware: imapx200' 
31 Model: p7901a, Product: p7901a, Manufacturer: unknown, Hardware: imapx200' 
27 Model: m7131, Product: m7131, Manufacturer: Infotmic, Hardware: imapx200' 
23 Model: M799CA, Product: M799CA, Manufacturer: Infotmic, Hardware: imapx200' 
19 Model: m799f, Product: m799f, Manufacturer: Infotmic, Hardware: imapx200' 
18 Model: M799RB, Product: M799RB, Manufacturer: Infotmic, Hardware: imapx200' 
18 Model: GOCLEVER NETBOOK I102, Product: AIRIS, Manufacturer: Goclever, Hardware: imapx200' 
17 Model: M1099RD, Product: M1099RD, Manufacturer: Infotmic, Hardware: imapx200' 
17 Model: IS701R, Product: INFOT, Manufacturer: infotm, Hardware: imapx200' 
16 Model: Z909, Product: Z909, Manufacturer: Infotmic, Hardware: imapx200' 
16 Model: venus, Product: venus, Manufacturer: unknown, Hardware: imapx200' 
16 Model: AIRIS, Product: AIRIS, Manufacturer: COBY, Hardware: imapx200' 
15 Model: NB_10_HD, Product: NB_10_HD, Manufacturer: unknown, Hardware: imapx200' 
14 Model: KIRA_N9000, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
13 Model: SY0103HC, Product: SY0103HC, Manufacturer: Infotmic, Hardware: imapx200' 
11 Model: NETBOOK_I102, Product: AIRIS, Manufacturer: Goclever, Hardware: imapx200' 
11 Model: m702sa, Product: m702sa, Manufacturer: Infotmic, Hardware: imapx200' 
10 Model: dk799rb, Product: dk799rb, Manufacturer: Infotmic, Hardware: imapx200' 
 9 Model: PAD7, Product: INFOT, Manufacturer: infotm, Hardware: imapx200' 
 9 Model: P745SD, Product: P745SD, Manufacturer: Infotmic, Hardware: imapx200' 
 9 Model: mid, Product: mid, Manufacturer: Sinovideo, Hardware: imapx200' 
 8 Model: OnePAD 705, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
 8 Model: dk1032c, Product: dk1032c, Manufacturer: Infotmic, Hardware: imapx200' 
 7 Model: MID, Product: MID, Manufacturer: unknown, Hardware: imapx200' 
 7 Model: m1099ca, Product: m1099ca, Manufacturer: Infotmic, Hardware: imapx200' 
 6 Model: OnePAD 720, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
 6 Model: M010F, Product: M010F, Manufacturer: Infotmic, Hardware: imapx200' 
 6 Model: i7, Product: i7, Manufacturer: unknown, Hardware: imapx200' 
 5 Model: unknown, Product: dk799erb, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: TPC7003R, Product: TPC7003R, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: QWTB9010, Product: QWTB9010, Manufacturer: unknown, Hardware: imapx200' 
 5 Model: MID, Product: MID, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: MID7010, Product: MID7010, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: KIRA N10020, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
 4 Model: sy0103HB, Product: KIRA_N10020, Manufacturer: Infotmic, Hardware: imapx200' 
 4 Model: Nexus S, Product: soju, Manufacturer: samsung, Hardware: imapx200' 
 3 Model: zt180, Product: zt180, Manufacturer: ZT, Hardware: imapx200' 
 3 Model: SYTABEX7-2, Product: P7901, Manufacturer: unknown, Hardware: imapx200' 
 3 Model: P790RN, Product: m799f, Manufacturer: Infotmic, Hardware: imapx200' 
 3 Model: P745, Product: P745, Manufacturer: Infotmic, Hardware: imapx200' 
 3 Model: P0106RN, Product: P0106RN, Manufacturer: Infotmic, Hardware: imapx200' 
 3 Model: MID7015A, Product: MID7010, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: unknown, Product: m799f, Manufacturer: Infotmic, Hardware: imapx200' 
 2 Model: TAB_I70, Product: TAB_I70, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: TAB101F, Product: TAB101F, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: PC7251, Product: PC7251, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: P001, Product: P001, Manufacturer: Infotmic, Hardware: imapx200' 
 2 Model: MOX_PAD724, Product: MOX_PAD724, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: M799, Product: M799, Manufacturer: unknown, Hardware: imapx200' 
 1 Model: T10, Product: T10, Manufacturer: Infotmic, Hardware: imapx200' 
 1 Model: Superpad2, Product: Superpad2, Manufacturer: unknown, Hardware: imapx200' 
 1 Model: AN10G2, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board'

Comment 24

[Benoit Jacob [:bjacob] (mostly away)]

Is it possible to blacklist according to the 'Hardware' field (like imapx200) rather than having to list all device models?

Not only this would be far easier here, but this would also prevent some oddities, i.e. look at the Samsung Soju entry in the above list: Samsung Soju is supposed to be a fine ARMv7 + PowerVR phone, but somehow some faulty phones with imapx200 chipsets identified as Samsung Soju.

Comment 25

[Aaron Train [:aaronmt]]

Wouldn't it be easier to filter for OpenGL ES 2.0 support in addition making sure that the filter for minimum Android 2.2 (Open GL ES2.0 support) is working? Adding a Google Play filter for GL_OES_compressed_ETC1_RGB8_texture (http://developer.android.com/guide/topics/manifest/supports-gl-texture-element.html) ?

Comment 26

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Aaron Train [:aaronmt] from comment #25)
> Wouldn't it be easier to filter for OpenGL ES 2.0 support in addition making
> sure that the filter for minimum Android 2.2 (Open GL ES2.0 support) is
> working?

If we can do that, that would of course be best as we would then be checking exactly for what we need. Notice that there is an important distinction here between theoretically supporting OpenGL ES 2, and actually supporting it. The Vivante GC600 GPUs that are causing the 90% of the trouble discussed above, theoretically support OpenGL ES 2 but in practice our OpenGL test code fails to create a OpenGL ES 2 context. This (Java) code is here:
  https://hg.mozilla.org/mozilla-central/file/fcd9acafa3ff/mobile/android/base/gfx/GfxInfoThread.java
That is what would be nice to use as a test here, if the Google Play store allow such programmatic tests.


> Adding a Google Play filter for GL_OES_compressed_ETC1_RGB8_texture
> (http://developer.android.com/guide/topics/manifest/supports-gl-texture-
> element.html) ?

Notice that this particular OpenGL ES extension is not relevant to us. I would rather just do a plain OpenGL ES 2 context creation  ( + eglMakeCurrent and glGetString ) test like the one linked to above, which is what determines the startup crashes that we are trying to avoid here. It is true that Firefox requires a couple of OpenGL ES 2 extensions as well to run, but we don't know that we are having any trouble with that.

Comment 27

[Benoit Jacob [:bjacob] (mostly away)]

The list of devices in comment 23 was limited to ARMv6+Vivante devices. Instead, the better approach is to list all devices that gave the no-OpenGL-ES2-available startup crash. This is achieved by the following command. This finds several mode devices to block:

$ cat a | grep 'OpenGL-accelerated layers are a hard requirement on this platform' | sed 's|.*Model|Model|g' | sed 's/|.*//g' | sort | uniq -c | sort -nr | sed 's|     ||g'
58 Model: MW0812, Product: TABLET, Manufacturer: unknown, Hardware: rk29board' 
58 Model: MID7105SR, Product: MID7105SR, Manufacturer: unknown, Hardware: imapx200' 
57 Model: KIRA N10021, Product: KIRA_N10021, Manufacturer: AIRIS, Hardware: imapx200' 
51 Model: KIRA N10021, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
48 Model: DISCO10, Product: DISCO10, Manufacturer: unknown, Hardware: imapx200' 
43 Model: mid, Product: mid, Manufacturer: Infotmic, Hardware: imapx200' 
40 Model: sy0103HB, Product: sy0103HC, Manufacturer: Infotmic, Hardware: imapx200' 
36 Model: AN10G2, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
32 Model: N7000, Product: N7000, Manufacturer: AIRIS, Hardware: imapx200' 
31 Model: p7901a, Product: p7901a, Manufacturer: unknown, Hardware: imapx200' 
27 Model: m7131, Product: m7131, Manufacturer: Infotmic, Hardware: imapx200' 
26 Model: HTC Desire S, Product: htc_saga, Manufacturer: HTC, Hardware: saga' 
23 Model: M799CA, Product: M799CA, Manufacturer: Infotmic, Hardware: imapx200' 
23 Model: HTC Wildfire, Product: htc_buzz, Manufacturer: HTC, Hardware: buzz' 
19 Model: m799f, Product: m799f, Manufacturer: Infotmic, Hardware: imapx200' 
19 Model: Hero, Product: full hero, Manufacturer: HTC, Hardware: hero' 
18 Model: M799RB, Product: M799RB, Manufacturer: Infotmic, Hardware: imapx200' 
18 Model: HTC Desire HD A9191, Product: htc_ace, Manufacturer: HTC, Hardware: spade' 
18 Model: GOCLEVER NETBOOK I102, Product: AIRIS, Manufacturer: Goclever, Hardware: imapx200' 
18 Model: Desire HD, Product: htc_ace, Manufacturer: HTC, Hardware: spade' 
17 Model: VTAB1008, Product: VTAB1008, Manufacturer: Vizio, Hardware: brownstone' 
17 Model: TR719, Product: TR719, Manufacturer: unknown, Hardware: rk29board' 
17 Model: M1099RD, Product: M1099RD, Manufacturer: Infotmic, Hardware: imapx200' 
17 Model: IS701R, Product: INFOT, Manufacturer: infotm, Hardware: imapx200' 
16 Model: Z909, Product: Z909, Manufacturer: Infotmic, Hardware: imapx200' 
16 Model: venus, Product: venus, Manufacturer: unknown, Hardware: imapx200' 
16 Model: AIRIS, Product: AIRIS, Manufacturer: COBY, Hardware: imapx200' 
15 Model: NB_10_HD, Product: NB_10_HD, Manufacturer: unknown, Hardware: imapx200' 
14 Model: SK17i, Product: SK17i_1250-0889, Manufacturer: Sony Ericsson, Hardware: semc' 
14 Model: KIRA_N9000, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
13 Model: SY0103HC, Product: SY0103HC, Manufacturer: Infotmic, Hardware: imapx200' 
13 Model: sdk, Product: sdk, Manufacturer: unknown, Hardware: goldfish' 
13 Model: N12FT_A, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
13 Model: LT26i, Product: LT26i_1257-8897, Manufacturer: Sony Ericsson, Hardware: semc' 
12 Model: SO-03D, Product: SO-03D_1256-7200, Manufacturer: Sony Ericsson, Hardware: semc' 
12 Model: QBEN QB700, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
12 Model: N-04D, Product: N-04D, Manufacturer: NEC, Hardware: sdk' 
12 Model: MS-N7Y1, Product: WindPad Enjoy 7, Manufacturer: MSI, Hardware: rk29board' 
11 Model: NETBOOK_I102, Product: AIRIS, Manufacturer: Goclever, Hardware: imapx200' 
11 Model: m702sa, Product: m702sa, Manufacturer: Infotmic, Hardware: imapx200' 
10 Model: sdkDemo, Product: sdkDemo, Manufacturer: unknown, Hardware: rk29board' 
10 Model: dk799rb, Product: dk799rb, Manufacturer: Infotmic, Hardware: imapx200' 
 9 Model: PAD7, Product: INFOT, Manufacturer: infotm, Hardware: imapx200' 
 9 Model: P745SD, Product: P745SD, Manufacturer: Infotmic, Hardware: imapx200' 
 9 Model: mid, Product: mid, Manufacturer: Sinovideo, Hardware: imapx200' 
 9 Model: imito am801, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 9 Model: ADR6300, Product: inc, Manufacturer: HTC, Hardware: inc' 
 8 Model: SHV-E160L, Product: SHV-E160L, Manufacturer: samsung, Hardware: SHV-E160L' 
 8 Model: OnePAD 705, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
 8 Model: GT-I5800, Product: GT-I5800, Manufacturer: samsung, Hardware: unknown' 
 8 Model: dk1032c, Product: dk1032c, Manufacturer: Infotmic, Hardware: imapx200' 
 7 Model: zt180, Product: zt180, Manufacturer: ZT, Hardware: imapx200' 
 7 Model: MW0821, Product: TABLET, Manufacturer: unknown, Hardware: rk29board' 
 7 Model: MID, Product: MID, Manufacturer: unknown, Hardware: imapx200' 
 7 Model: mercury, Product: mercury, Manufacturer: unknown, Hardware: imapx200' 
 7 Model: m1099ca, Product: m1099ca, Manufacturer: Infotmic, Hardware: imapx200' 
 6 Model: PC36100, Product: htc_supersonic, Manufacturer: HTC, Hardware: supersonic' 
 6 Model: OnePAD 720, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
 6 Model: M010F, Product: M010F, Manufacturer: Infotmic, Hardware: imapx200' 
 6 Model: i7, Product: i7, Manufacturer: unknown, Hardware: imapx200' 
 6 Model: ADR6350, Product: htc_vivow, Manufacturer: HTC, Hardware: vivow' 
 6 Model: 001HT, Product: htc_ace, Manufacturer: HTC, Hardware: spade' 
 5 Model: unknown, Product: dk799erb, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: U8150, Product: u8150, Manufacturer: Huawei, Hardware: u8150' 
 5 Model: TPC7003R, Product: TPC7003R, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: ST8000, Product: BOULM803HC, Manufacturer: unknown, Hardware: rk29board' 
 5 Model: QWTB9010, Product: QWTB9010, Manufacturer: unknown, Hardware: imapx200' 
 5 Model: MID, Product: MID, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: MID7010, Product: MID7010, Manufacturer: Infotmic, Hardware: imapx200' 
 5 Model: MB501, Product: zeppelin, Manufacturer: Motorola, Hardware: zeppelin' 
 5 Model: KIRA N10020, Product: AIRIS, Manufacturer: AIRIS, Hardware: imapx200' 
 5 Model: AN7G2, Product: sdkDemo, Manufacturer: unknown, Hardware: rk29board' 
 4 Model: Wildfire, Product: cm_buzz, Manufacturer: HTC, Hardware: buzz' 
 4 Model: sy0103HB, Product: KIRA_N10020, Manufacturer: Infotmic, Hardware: imapx200' 
 4 Model: Nexus S, Product: soju, Manufacturer: samsung, Hardware: imapx200' 
 4 Model: N3T, Product: TR719, Manufacturer: unknown, Hardware: rk29board' 
 4 Model: MOMO, Product: TR810, Manufacturer: unknown, Hardware: rk29board' 
 4 Model: Ideos, Product: Ideos, Manufacturer: HUAWEI, Hardware: huawei' 
 4 Model: G-BOX, Product: sdkDemo, Manufacturer: unknown, Hardware: rk29board' 
 4 Model: DATAM803HC, Product: DATAM803HC, Manufacturer: unknown, Hardware: rk29board' 
 3 Model: U8120, Product: soju, Manufacturer: Huawei, Hardware: u8120' 
 3 Model: SYTABEX7-2, Product: P7901, Manufacturer: unknown, Hardware: imapx200' 
 3 Model: P790RN, Product: m799f, Manufacturer: Infotmic, Hardware: imapx200' 
 3 Model: P745, Product: P745, Manufacturer: Infotmic, Hardware: imapx200' 
 3 Model: P0106RN, Product: P0106RN, Manufacturer: Infotmic, Hardware: imapx200' 
 3 Model: ODYS-Xpress, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 3 Model: MOTWX435KT, Product: Caymus_VRGUS, Manufacturer: Motorola, Hardware: qcom' 
 3 Model: MID7015A, Product: MID7010, Manufacturer: unknown, Hardware: imapx200' 
 3 Model: HTC Sapphire, Product: passion, Manufacturer: HTC, Hardware: sapphire' 
 3 Model: HTC Hero, Product: passion, Manufacturer: HTC, Hardware: hero' 
 3 Model: GT-I9100, Product: GT-I9100, Manufacturer: samsung, Hardware: smdkc210' 
 3 Model: GASEM803HC, Product: GASEM803HC, Manufacturer: unknown, Hardware: rk29board' 
 3 Model: DS821B, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 3 Model: CUBE U9GT 2, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: WT19a, Product: WT19a_1254-4430, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: unknown, Product: m799f, Manufacturer: Infotmic, Hardware: imapx200' 
 2 Model: U9GT_S, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: U8120, Product: cm_u8120, Manufacturer: Huawei, Hardware: qcom' 
 2 Model: T-Mobile G2, Product: htc_vision, Manufacturer: HTC, Hardware: vision' 
 2 Model: TAB_I70, Product: TAB_I70, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: TAB101F, Product: TAB101F, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: ST15a, Product: ST15a_1249-9321, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: SO-02D, Product: SO-02D_1256-6356, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: SK17a, Product: SK17a_1250-1736, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: R800a, Product: R800a_1255-9904, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: R800a, Product: R800a_1247-6272, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: PC7251, Product: PC7251, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: P001, Product: P001, Manufacturer: Infotmic, Hardware: imapx200' 
 2 Model: N50DT, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: N12, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: MOX_PAD724, Product: MOX_PAD724, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: MID, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: M799, Product: M799, Manufacturer: unknown, Hardware: imapx200' 
 2 Model: LT26i, Product: LT26i_1257-6750, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: LT18i, Product: LT18i_1254-2259, Manufacturer: Sony Ericsson, Hardware: semc' 
 2 Model: JOYPAD_C81, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: IDEOS S7 Slim, Product: SLIM_XL_ID, Manufacturer: HUAWEI, Hardware: qsd8k_slim' 
 2 Model: IDEOS S7 Slim, Product: SLIM_HW_PL, Manufacturer: HUAWEI, Hardware: qsd8k_slim' 
 2 Model: ICE, Product: U8500, Manufacturer: HUAWEI, Hardware: huawei' 
 2 Model: HTC Tattoo, Product: htc_click, Manufacturer: HTC, Hardware: bahamas' 
 2 Model: HTC Glacier, Product: htc_glacier, Manufacturer: HTC, Hardware: glacier' 
 2 Model: Excellent8, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: E15i, Product: E15i, Manufacturer: Sony Ericsson, Hardware: delta' 
 2 Model: android, Product: sdkDemo, Manufacturer: unknown, Hardware: rk29board' 
 2 Model: 740, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 1 Model: YAN 7.1C, Product: rk2906, Manufacturer: unknown, Hardware: rk29board' 
 1 Model: wwe10, Product: wwe10, Manufacturer: unknown, Hardware: imapx200' 
 1 Model: VONO, Product: TR719, Manufacturer: unknown, Hardware: rk29board' 
 1 Model: U9GTS_A, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 1 Model: T10, Product: T10, Manufacturer: Infotmic, Hardware: imapx200' 
 1 Model: Superpad2, Product: Superpad2, Manufacturer: unknown, Hardware: imapx200' 
 1 Model: ST18a, Product: ST18a_1252-6824, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: SO-02C, Product: SO-02C_1249-5055, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: SK17i, Product: SK17i_1250-0861, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: SEBM912HC, Product: soju, Manufacturer: Samsung, Hardware: rk29board' 
 1 Model: rk29sdk, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 1 Model: R800a, Product: R800a_1249-4482, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: MT15a, Product: MT15a_1247-1542, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: MT11i, Product: MT11i_1254-5772, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: MS-N0Y1, Product: WindPad Enjoy 10, Manufacturer: MSI, Hardware: rk29board' 
 1 Model: MID, Product: MID, Manufacturer: VIMICRO, Hardware: vimicro' 
 1 Model: LT28i, Product: LT28i_1264-3077, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: LT26i, Product: LT26i_1257-7434, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: LT26i, Product: LT26i_1257-6954, Manufacturer: Sony Ericsson, Hardware: semc' 
 1 Model: LG-P930, Product: lge_iproj, Manufacturer: LGE, Hardware: qcom' 
 1 Model: LG-MS770, Product: l0_MPCS_US, Manufacturer: lge, Hardware: l0' 
 1 Model: LG-E610, Product: m4_pls_pl, Manufacturer: LGE, Hardware: m4' 
 1 Model: kekkoyasui ioPad6, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 1 Model: JY-G2, Product: JY-G2, Manufacturer: alps, Hardware: mt6575' 
 1 Model: HTC Incredible S, Product: htc_vivo, Manufacturer: HTC, Hardware: vivo' 
 1 Model: HTC Desire, Product: htc_bravo, Manufacturer: HTC, Hardware: bravo' 
 1 Model: HERO200, Product: cyanogen_heroc, Manufacturer: HTC, Hardware: heroc' 
 1 Model: GT-N7000, Product: GT-N7000, Manufacturer: samsung, Hardware: smdkc210' 
 1 Model: GT-I5800, Product: cm_apollo, Manufacturer: Samsung, Hardware: gt-i5801' 
 1 Model: GT-I5700, Product: soju, Manufacturer: samsung, Hardware: spica' 
 1 Model: Behold BeTAB, Product: sdkDemo, Manufacturer: Beholder, Hardware: rk29board' 
 1 Model: AN10G2-LZ, Product: rk29sdk, Manufacturer: unknown, Hardware: rk29board' 
 1 Model: ADR6330VW, Product: htc_blissc, Manufacturer: HTC, Hardware: blissc' 
 1 Model: A70HT3, Product: A70HT3, Manufacturer: unknown, Hardware: rk29board'

Comment 28

[Benoit Jacob [:bjacob] (mostly away)]

...though I'm not so sure anymore now that I look at the devices listed in comment 27: for example near the top we find the HTC Desire S which should really work (has a Adreno 205 GPU). If we could do an actual runtime check as Aaron suggests, that would be best.

Comment 29

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: List of devices sorted by percentage of occurence of the no-OpenGL-ES2 crash

This file was obtained as follows:

$ cat a | grep 'OpenGL-accelerated layers are a hard requirement on this platform' | sed 's|.*Model|Model|g' | sed "s/' |.*//g" | sort | uniq | sed 's|     ||g' > devices

$ while read d; do echo "$d, `grep "$d" a | wc -l`, `grep "$d" a | grep 'OpenGL-accelerated layers are a hard requirement on this platform' | wc -l`"; done < devices | tee devices.csv

+ some manual editing in LibreOffice, adding the Percentages in column G, and sorting.

The devices with Ratio 100% are devices where all the crash reports we received, were this startup crash. They are where we should start blacklisting, as they likely never allow getting past this startup crash. Further down there also are some devices with less than 100% but still large percentage of occurence of this crash.

Comment 30

[Alex Keybl [:akeybl]]

(In reply to Benoit Jacob [:bjacob] from comment #29)
> Created attachment 661837 [details]
> List of devices sorted by percentage of occurence of the no-OpenGL-ES2 crash
> 
> This file was obtained as follows:
> 
> $ cat a | grep 'OpenGL-accelerated layers are a hard requirement on this
> platform' | sed 's|.*Model|Model|g' | sed "s/' |.*//g" | sort | uniq | sed
> 's|     ||g' > devices
> 
> $ while read d; do echo "$d, `grep "$d" a | wc -l`, `grep "$d" a | grep
> 'OpenGL-accelerated layers are a hard requirement on this platform' | wc
> -l`"; done < devices | tee devices.csv
> 
> + some manual editing in LibreOffice, adding the Percentages in column G,
> and sorting.
> 
> The devices with Ratio 100% are devices where all the crash reports we
> received, were this startup crash. They are where we should start
> blacklisting, as they likely never allow getting past this startup crash.
> Further down there also are some devices with less than 100% but still large
> percentage of occurence of this crash.

This is awesome work! The ratios you provided are especially telling. One question - would it be possible to limit this list to devices with crashes that are occurring on beta builds? We're most concerned about blocklisting beta/release (users can try builds on Nightly/Aurora as they choose), and I've seen a number of devices that appear to be unsupported on beta in this list (for instance DISCO10).

Comment 31

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: List of devices sorted by percentage of occurence of this crash (Firefox 16 only)

There you go. It doesn't seem to make a huge difference overall, but indeed some devices are gone. Obtained by adding "| grep FennecAndroid.16" to my grep commands, so what this really filters for is version 16, not beta channel per se.

Comment 32

[Alex Keybl [:akeybl]]

Thanks - I've gone through and compared the first 50 or so to the Google Play Store devices at https://docs.google.com/spreadsheet/ccc?key=0Av9ejaFxk72tdG9PdllkVmozd0tCMGk0bjNfOEJsaWc

We'll want to additionally block

rk29sdk
TR719
mid

I've already done this in the beta product. The rest of the top devices are not recognized by the store, and therefore can't be blocked from installing. Working theory is that users are downloading the beta builds directly from FTP, but we don't yet have a way of verifying.

Comment 33

[Alex Keybl [:akeybl]]

(of course they could also be Fennec 16 users on Nightly/Aurora)

Comment 34

[Scoobidiver (away)]

(In reply to Benoit Jacob [:bjacob] from comment #23)
> from Sept. 7 to Sept. 13.
I think the period must be extended to four weeks in order to take into account those who tested their device when 16.0 Beta became live.

Comment 35

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Scoobidiver from comment #34)
> (In reply to Benoit Jacob [:bjacob] from comment #23)
> > from Sept. 7 to Sept. 13.
> I think the period must be extended to four weeks in order to take into
> account those who tested their device when 16.0 Beta became live.

See bug 771774 comment 28: the volume of ARMv6 crashers was near-zero until around September 6-7.

Comment 36

[Scoobidiver (away)]

(In reply to Benoit Jacob [:bjacob] from comment #35)
> See bug 771774 comment 28: the volume of ARMv6 crashers was near-zero until
> around September 6-7.
Yes, but although this bug is titled ARMv6, I think OpenGL ES 2 support also concerns ARMv7 devices with many startup crashes before September 6.

Comment 37

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Scoobidiver from comment #36)
> (In reply to Benoit Jacob [:bjacob] from comment #35)
> > See bug 771774 comment 28: the volume of ARMv6 crashers was near-zero until
> > around September 6-7.
> Yes, but although this bug is titled ARMv6, I think OpenGL ES 2 support also
> concerns ARMv7 devices with many startup crashes before September 6.

For what it's worth, the command lines I gave above (and resulting spreadsheet) are not armv6 specific.

Comment 38

[Alex Keybl [:akeybl]]

(In reply to Benoit Jacob [:bjacob] from comment #37)
> (In reply to Scoobidiver from comment #36)
> > (In reply to Benoit Jacob [:bjacob] from comment #35)
> > > See bug 771774 comment 28: the volume of ARMv6 crashers was near-zero until
> > > around September 6-7.
> > Yes, but although this bug is titled ARMv6, I think OpenGL ES 2 support also
> > concerns ARMv7 devices with many startup crashes before September 6.
> 
> For what it's worth, the command lines I gave above (and resulting
> spreadsheet) are not armv6 specific.

I didn't filter anything out, so we've blocked as much as we can. See comment 32.

Comment 39

[Scoobidiver (away)]

(In reply to Alex Keybl [:akeybl] from comment #38)
> I didn't filter anything out, so we've blocked as much as we can. See
> comment 32.
Yes, indeed they are ARMv6&ARMv7 devices in Beta, but Benoit's table is two weeks old and you may have missed devices tested at the end of the Beta phase.

Comment 40

[Alex Keybl [:akeybl]]

ARMv6 is staying on beta for one more cycle. Tracking for FF17.

Comment 41

[Lukas Blakk [:lsblakk] use ?needinfo]

Assigning this to Aaron who now has access & control over device exclusions in the GP store.  Aaron, the list of excluded devices for the Firefox product will need to be updated to match what we exclude from Firefox Beta once the new 17.0 apk has been published.

Comment 42

[Aaron Train [:aaronmt]]

Is the current exclusion list reflective of current requirements and testing? There are 374 devices manually excluded that I'd rather not tamper with unless they are valid.

Comment 43

[Alex Keybl [:akeybl]]

(In reply to Aaron Train [:aaronmt] from comment #42)
> Is the current exclusion list reflective of current requirements and
> testing? There are 374 devices manually excluded that I'd rather not tamper
> with unless they are valid.

The entirety of the beta exclusion list is valid, and should just be reflected in the release list.

Comment 44

[Aaron Train [:aaronmt]]

This is now completed after rigorous copy pasting. On Firefox product, unlike Firefox Beta, under device exclusions I don't know why it will not list the unsupported devices due to manifest (on the right side column, once the popup window is open). I am checking to see individually if each one is matched as unsupported to manifest.