1037056 - [Flame] We need a way to FOTA our Gecko/Gaia.

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Vendcom, defect, P1)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

When testing the URL override with the straight 122 OEM build, the override did not override to our server.  I had to flash a v1.3 gaia on top in order for the override to work.

We need a way to switch back to check their OTA server for gonk updates.

This might be a two part bug?  Not sure.

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I filed this bug to figure out the mechanism where we need to get OTA updates from two different locations.

David, is there a good way to achieve this?

Comment 2

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: logcat.txt

To note : 
https://developer.mozilla.org/en-US/docs/Mozilla/Setting_up_an_update_server
I am referring to the app.update.url.override

These are the steps that I took in regards to getting the logcat attached:
1. flash the OEM v122 build ; turned wifi on through ftu
2. 
ran the following commands : 
./change_channel.sh -v aurora
./change_ota_url.sh -u https://aus4.mozilla.org/update/3/B2G/28.0/20140709000201/flame/en-US/aurora/Boot2Gecko%202.0.0.0-prerelease/default/default/update.xml?force=1

[ note these are scripts from : https://github.com/Mozilla-TWQA/B2G-flash-tool ]
3. adb logcat > logcat.txt
4. on the phone : settings -> device info -> system updates -> Check for updates

First time I tried check for updates, it says check again later.  Second time I tried it says the system was already up to date.

If I flash my own v1.3 gaia on top, I am able to update to 2.0 with the same steps above.

Comment 3

[Dave Hylands [:dhylands]]

The phone really only supports getting updates from a single location.

You can mix FOTA and OTA on the same server from the same URL.

Although it's almost like we want to have 2 different update checks.

Currently, we use the gecko/gaia build id and look for new gecko/gaia updates.

It's almost like we need a parallel update channel to look for gonk updates.

So not only change URLs, but also change how the version is calculated.

So you'd have URL1 using buildid and URL2 using some gonkid

Comment 4

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

We can find the gonk id currently with : adb shell getprop t2m.sw.version

Comment 5

[[e-mail] (:bhearsum)]

Naoki and I chatted on IRC about this a bit. For starters, the reason that setting app.update.url.override doesn't work in a T2M image is because our updater doesn't run as part of that (verified through the lack of GeckoUpdater or AUS:SVC in the attached logcat).

At least our Gaia needs to be flashed before our updater runs - though it seems like a good idea to do a matching Gecko as well.

In an ideal world, I would suggest that the best way to deal with updates that require Gonk is to ship a FOTA containing Gonk+Gecko+Gaia from one update server. Doing that is the simplest thing from a management standpoint, and also provides better UX.

In a world where we're not allowed to distribute Gonk bits, we could, in theory, have our update server point to a Gonk MAR hosted elsewhere, and a Gecko+Gaia MAR hosted by us. That would require update server changes that are unlikely to happen for awhile. The other way to live in this world is to have the partner do Gonk+Gecko+Gaia updates instead.

In any case, right now the reality is that when you need to take a Gonk change, you need to flash the partner image, then flash our Gecko/Gaia. Then you'll receive updates from us again.

Comment 6

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Thanks bhearsum, dhylands!

It seems like if there's a way for T2M to have a Gonk.mar, checking via the adb shell getprop t2m.sw.version
and they allow the F/OTA gecko/gaia to point to : https://aus4.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%PRODUCT_DEVICE%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

That might work.  I think we also would need to have the gaia/gecko changes that they make to accommodate two URLs like this?

Francis, could you chat with T2M to see if something like this might be possible since we can't distribute the Gonk?  This may make things easier for dogfooders.

Comment 7

[[e-mail] (:bhearsum)]

I feel like this should be discussed between releng and some b2g folks more before we actually try to do anything here. I can't volunteer myself at this moment, but I'm happy to help when my schedule clears up a bit.

There's lots of tricky issues at play when we're talking about the updater, and I don't want us to go halfway into a fix only to discover it won't work.

Comment 8

[Asa Dotzler [:asa]]

This deserves a meeting, soon. Who do we need in the room? Michael Treese can probably help organize.

Comment 9

[Chris AtLee [:catlee]]

When I spoke with Fabrice and Mwu about this a few months ago we agreed that the client side would be responsible for knowing if it could safely apply a gecko update given its current gonk state.

Agreed on need for a meeting, sign me up.

Comment 10

[[e-mail] (:bhearsum)]

If we do need to put the Gonk version in the update URL, I would suggest that it belongs in OS_VERSION somewhere. This would be similar-ish to how we put OS + Service pack information in there for desktop OS'. It looks like that could be done with a new ifdef in https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#340

But I agree that we need a meeting to figure out what's actually needed here.

Comment 11

[Diane Tate]

guys - I am not sure why this meeting is not happening so I will set up a meeting for next week - I will assume it's for the group on this bug but please add to the meeting attendee list as you see fit.

Comment 12

[Michael:mtreese]

Calling for this meeting without a clear problem statement is premature.  First, depending on what we are trying to sovle, MWu and Fabrice may be needed. Most of the things Asa and I discussed are engineering questions. A second meeting will still be needed to work out our FOTA options and potential engineering roadmap and the Monday meeting in my opinion should be cancelled.

I will work on the problem statement and set up a meeting later in the week.

Comment 13

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think we can't do a FOTA for the gonk layer; having said that we still need to do a gecko/gaia FOTA for Flame as the system partition doesn't have enough space for multiple OTAing.

Changing the title to match.

Comment 14

[:gerard-majax]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #13)
> I think we can't do a FOTA for the gonk layer; having said that we still
> need to do a gecko/gaia FOTA for Flame as the system partition doesn't have
> enough space for multiple OTAing.
> 
> Changing the title to match.

Producing such a package is simple: ./build.sh gecko-update-fota

Comment 15

[:gerard-majax]

From bug 1201540 comment 42 and discussion on IRC, :bhearsum suggested we allow setting "update_type" also in Gecko config JSON. The file is loaded by http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/b2g_build.py#l358 from |b2g/configs/<device>/config.json|.

So a plan would be to:
 - add update_type override support in b2g_build.py, like https://hg.mozilla.org/try/rev/e75147e495c0#l3.18 but reading also from gecko_config
 - add update_type definition to fota value in the proper flame-kk ota definition

We will still have to fix the balrog pushing. We also need to make sure there is no other way to set update_type value for a Flame KK OTA build.

Comment 16

[:gerard-majax]

Attachment: Flame FOTA r=...

Comment 17

[:gerard-majax]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=daaba0e2584d

Comment 18

[:gerard-majax]

OR .... http://hg.mozilla.org/build/buildbot-configs/file/56bcb5e87c33/mozilla/b2g_config.py#l778 ? With ./testing/mozharness/configs/b2g/releng-fota-updates.py instead of ./testing/mozharness/configs/b2g/releng-private-updates.py

Comment 19

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #18)
> OR ....
> http://hg.mozilla.org/build/buildbot-configs/file/56bcb5e87c33/mozilla/
> b2g_config.py#l778 ? With
> ./testing/mozharness/configs/b2g/releng-fota-updates.py instead of
> ./testing/mozharness/configs/b2g/releng-private-updates.py

At least on TaskCluster we have something that now seems to do the trick:
https://tools.taskcluster.net/task-graph-inspector/#CtoB30EjReCc16jeQJvKWw/HTlVe5e0QziqJ_eBYdJ0eA/0

See the "fota-flame-update.mar" artifact, this is what we want !

Comment 20

[:gerard-majax]

All green: https://tools.taskcluster.net/task-graph-inspector/#GZeqRuMsQRypZ0SwBmXrAg/aqmfK-IaRXO7xF_ODDHH0A/

Comment 21

[:gerard-majax]

Attachment: Flame FOTA r=...

Comment 22

[:gerard-majax]

Comment on attachment 8668501 [details] [diff] [review]
Flame FOTA r=...

One last round of review: this is doing what I want and what was suggested.

Next step is making sure Balrog communication is fine, but that is out of my scope!

Comment 23

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Note to self : need to test

Comment 24

[Wander Lairson Costa]

Comment on attachment 8668501 [details] [diff] [review]
Flame FOTA r=...

Review of attachment 8668501 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

::: testing/taskcluster/tasks/builds/b2g_flame_kk_ota_base.yml
@@ +6,5 @@
>    payload:
>      env:
>        TARGET: 'flame-kk'
>        DEBUG: 0
> +      MOZHARNESS_CONFIG: b2g/taskcluster-phone-fota.py

Shouldn't fota have its own task file?

Comment 25

[:gerard-majax]

So we create testing/taskcluster/tasks/builds/b2g_flame_kk_fota_base.yml ?

Comment 26

[:gerard-majax]

And we add a device build "flame-kk-fota" next to "flame-kk-ota" ? Why not.

Comment 27

[:gerard-majax]

Attachment: Flame FOTA r=...

Comment 28

[:gerard-majax]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=73414b379f91

Comment 29

[Wander Lairson Costa]

(In reply to Alexandre LISSY :gerard-majax from comment #26)
> And we add a device build "flame-kk-fota" next to "flame-kk-ota" ? Why not.

Yeah, although ota builds are currently done in bb, plans are to move to TC in future, so the OTA tasks should be preserved.

Comment 30

[:gerard-majax]

Attachment: Buildbot config PR

Comment 31

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #28)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=73414b379f91

All good.

Let's check we do not break anything: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6a891ea3a0e

Comment 32

[:gerard-majax]

Attachment: Adding support for producing Gecko/Gaia FOTA for Flame

Rebased and cleaned up the hacks for try

Comment 33

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #32)
> Created attachment 8668897 [details] [diff] [review]
> Adding support for producing Gecko/Gaia FOTA for Flame
> 
> Rebased and cleaned up the hacks for try

https://tools.taskcluster.net/task-graph-inspector/#YtX0zNSqSpeq53f7r-W0Ag/

Comment 34

[:gerard-majax]

Comment on attachment 8668897 [details] [diff] [review]
Adding support for producing Gecko/Gaia FOTA for Flame

Carrying r+ from :wcosta

Comment 35

[:gerard-majax]

Still looking good: https://tools.taskcluster.net/task-graph-inspector/#Ancw92gORzWq0wOYvDbLXg/

Comment 36

[Jordan Lund (:jlund)]

Comment on attachment 8668897 [details] [diff] [review]
Adding support for producing Gecko/Gaia FOTA for Flame

Review of attachment 8668897 [details] [diff] [review]:
-----------------------------------------------------------------

looks good.

is the only change between testing/mozharness/configs/b2g/taskcluster-phone-fota.py and testing/mozharness/configs/b2g/taskcluster-phone-ota.py the addition of `"update_type": "fota"` ?

if so and if you expect these two config files to continue sharing any future changes, it might be better to make 'update_type' a command line option that gets passed by taskcluster

e.g.
`hg mv testing/mozharness/configs/b2g/taskcluster-phone-ota.py testing/mozharness/configs/b2g/taskcluster-phone-update.py`
# add --update-type command line option and then ...
fota case: `--config b2g/taskcluster-phone-update.py --update-type fota`
ota case: `--config b2g/taskcluster-phone-update.py --update-type ota`

Comment 37

[Chris AtLee [:catlee]]

Comment on attachment 8668877 [details] [review]
Buildbot config PR

committed as https://hg.mozilla.org/build/buildbot-configs/rev/f4e8891b9849

Comment 38

[Chris AtLee [:catlee]]

We're going to be disabling the flame-kk-ota builds in TC in bug 1188992 to try and reduce confusion.

Comment 39

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/f4e8891b9849

Comment 40

[:gerard-majax]

Can we also land the patch for TaskCluster? I know it's not being used yet, but I would prefer we keep everything in sync at that point.

Comment 41

[Chris AtLee [:catlee]]

No, I'd prefer not to land in TC, since your patch will schedule new builds that will also be publishing updates.

The buildbot patch however, is not working. The build is failing at some point late in the process; I'm digging into this today.

Comment 42

[Chris AtLee [:catlee]]

Attachment: Add java to flame build environment

Comment 43

[Chris AtLee [:catlee]]

https://hg.mozilla.org/integration/b2g-inbound/rev/9ccd468f27bd86b7d0267cb975d0e0e8c511422c
Bug 1037056: Add java to flame-kk builds r=rail

Comment 44

[Chris AtLee [:catlee]]

https://hg.mozilla.org/integration/b2g-inbound/rev/9ccd468f27bd

Comment 45

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/c6ede6f30f3d

Comment 46

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c6ede6f30f3d

Comment 47

[Dylan Oliver [:doliver]]

(In reply to Wes Kocher (:KWierso) from comment #46)
> https://hg.mozilla.org/mozilla-central/rev/c6ede6f30f3d

I don't think we're ready to resolve yet -- this is just for the java dependency, right?

Comment 48

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Oh. Happy to reopen if that's the case.

Comment 49

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Slight correction : Build bot patch + java dependency.  

Having said that, we still need to have it working for Taskcluster as we're moving towards taskcluster as the long term solution.  So yes, we still need this bug open.

Comment 50

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/9ccd468f27bd

Comment 51

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Carsten, this isn't completed yet.  Please read comment 49.

Comment 52

[Dylan Oliver [:doliver]]

Too many patches in here causing some confusion, tagging leave-open.

Comment 53

[Chris AtLee [:catlee]]

Attachment: Silence extraneous l10n warnings

Comment 54

[Chris AtLee [:catlee]]

Comment on attachment 8671660 [details] [diff] [review]
Silence extraneous l10n warnings

My local builds are working, and yet the ones in buildbot are failing, but I can't see why because the log exceeds the maximum.

This patch should remove quite a bit of l10n warnings, and hopefully let me see what's going wrong with the builds.

Comment 55

[Chris AtLee [:catlee]]

https://hg.mozilla.org/integration/b2g-inbound/rev/6e02bb301c4d9dbd1ae185f23310ad58d85f0257
Bug 1037056: Silence extraneous l10n warnings r=rail

Comment 56

[Chris AtLee [:catlee]]

Comment on attachment 8671660 [details] [diff] [review]
Silence extraneous l10n warnings

https://hg.mozilla.org/integration/b2g-inbound/rev/6e02bb301c4d

Comment 57

[Chris AtLee [:catlee]]

https://hg.mozilla.org/integration/b2g-inbound/rev/a891728a0947ef7aa96a8229c62d19b6e45abce8
Bug 1037056 - Adding support for producing Gecko/Gaia FOTA for Flame r=wcosta,jlund,catlee

Comment 58

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/a891728a0947

Comment 59

[:gerard-majax]

Attachment: Adding support for producing Gecko/Gaia FOTA for Flame

Added FOTA_FINGERPRINTS in sync with bug 1213824

Comment 60

[:gerard-majax]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=22972293353f

Comment 61

[:gerard-majax]

Attachment: Adding support for producing Gecko/Gaia FOTA for Flame

Removed fingerprints

Comment 62

[:gerard-majax]

Triggered new build: https://tools.taskcluster.net/task-graph-inspector/#PF5WRfaCQsG0lSGzbjQkIg/

Comment 63

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: IMG_1238.JPG

I tried testing via upgrading build 20151009150221 and I got the signature error again.

Comment 64

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: recovery_logcat.zip

Recovery logs and logcat file attached.

Comment 65

[:gerard-majax]

Naoki, I cannot debug this without the full logs, namely I need to make sure your system partition matches. The Edify code we make use of from AOSP:
>    if not fp:
>      raise ValueError("must specify some fingerprints")
>    cmd = (
>           ' ||\n    '.join([('file_getprop("/system/build.prop", '
>                         '"ro.build.fingerprint") == "%s"')
>                        % i for i in fp]) +
>           ' ||\n    abort("Package expects build fingerprint of %s; this '
>           'device has " + getprop("ro.build.fingerprint") + ".");'
>           ) % (" or ".join(fp),)
>    self.script.append(cmd)

So I guess the fingerprint is really wrong on system partition.

Comment 66

[:gerard-majax]

I have retested on my side, and it is working as expected. Naoki, that fingerprint will only match the v18D_v4 build. It will fail for anything else. Please make sure you tested with a recovery and system partition from v18D_v4 build.

Comment 67

[Chris AtLee [:catlee]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #64)
> Created attachment 8673360 [details]
> recovery_logcat.zip
> 
> Recovery logs and logcat file attached.

I can't help debug this, sorry.

Comment 68

[:gerard-majax]

(In reply to Chris AtLee [:catlee] from comment #67)
> (In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from
> comment #64)
> > Created attachment 8673360 [details]
> > recovery_logcat.zip
> > 
> > Recovery logs and logcat file attached.
> 
> I can't help debug this, sorry.

It's okay, we checked with Naoki on IRC, he tested applying a FOTA with fingerprint check enabled on an invalid base system. So the error was expected as I said in comment 66.

Comment 69

[Nicole Yee (:nicoleyee)]

[Blocking Requested - why for this release]:

Comment 70

[:gerard-majax]

So, as we spoke on IRC, maybe we do not need a new base image. That depends on the changes needed for bluetooth of course. But, if we only need to unpack/add new binaries we do build on the system partition, and to flash new init scripts that are part of the boot partition, then we can do all of that as part of a FOTA Gecko/Gaia update package: B2G_FOTA_PARTS env variable will control that.

So Thomas, what are all the changes that we need for Bluetooth on Flame?

Comment 71

[Thomas Zimmermann [:tzimmermann] [:tdz]]

None. (?) I don't think there have been any Bluetooth changes recently that require base-image updates.

Maybe you're talking about Sensors, but this work is delayed by the technical discussion anyway.

Comment 72

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Thomas, we're showing that there's a difference between base build of flame 18Dv4 and the fullflash (fastboot flash) of a recent flame build in bug 1204617.

This is why we think we may need a new base build and was exploring to see if we could potentially update using a FOTA that doesn't contain proprietary parts.

Based on what you are saying, it sounds like we should see about getting a new base build.  Is that correct?

Comment 73

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2r/rev/6b4e563acaf9

Comment 74

[Thomas Zimmermann [:tzimmermann] [:tdz]]

Naoki, I left a comment in the other bug report to explain the current behavior.

Comment 76

[Mihai Barbat]

I just had my first System Update on my Flame. I'm on a the nightly channel :D Does this mean the bug is fixed?

Comment 77

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Based on Thomas's comment, we can close this bug out as FOTA is fixed via buildbot.
We still need to branch out for the longer term solution for taskcluster.

Comment 78

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

to note : The FOTA builds being made by taskcluster will be handled in bug 1218544

Comment 79

[:gerard-majax]

Attachment: Producing FOTA packages for Flame

The Flame device updates have been broken for a while because of updates
starting to be too big. Because of the way applying OTA works, there is
no good solution except switching to applying Gecko/Gaia updates in
recovery mode. This was done in bug 1037056 on the Buildbot instances,
but we need to support this on TaskCluster also.

Comment 80

[Wander Lairson Costa]

Comment on attachment 8682403 [details] [diff] [review]
Producing FOTA packages for Flame

Review of attachment 8682403 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/configs/b2g/taskcluster-phone-ota.py
@@ +37,5 @@
>      "env": {
>          "GAIA_OPTIMIZE": "1",
>          "WGET_OPTS": "-c -q"
>      },
> +    "update_types": [ "ota", "fota" ],

Although this script currently runs only for flame-kk, this is intended as a generic config file for OTA builds, are you sure you want to build fota for every device?

Comment 81

[:gerard-majax]

(In reply to Wander Lairson Costa [:wcosta] from comment #80)
> Comment on attachment 8682403 [details] [diff] [review]
> Producing FOTA packages for Flame
> 
> Review of attachment 8682403 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: testing/mozharness/configs/b2g/taskcluster-phone-ota.py
> @@ +37,5 @@
> >      "env": {
> >          "GAIA_OPTIMIZE": "1",
> >          "WGET_OPTS": "-c -q"
> >      },
> > +    "update_types": [ "ota", "fota" ],
> 
> Although this script currently runs only for flame-kk, this is intended as a
> generic config file for OTA builds, are you sure you want to build fota for
> every device?

Well that's something we can change if we need, but I guess it will do no harm to produce OTA and FOTA. Do you see any reason we should not? Just from a QA point of view I guess it's useful to keep both on all devices, since it can help QA to check behavior of both.

Comment 82

[Wander Lairson Costa]

(In reply to Alexandre LISSY :gerard-majax from comment #81)
> (In reply to Wander Lairson Costa [:wcosta] from comment #80)
> > Comment on attachment 8682403 [details] [diff] [review]
> > Producing FOTA packages for Flame
> > 
> > Review of attachment 8682403 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: testing/mozharness/configs/b2g/taskcluster-phone-ota.py
> > @@ +37,5 @@
> > >      "env": {
> > >          "GAIA_OPTIMIZE": "1",
> > >          "WGET_OPTS": "-c -q"
> > >      },
> > > +    "update_types": [ "ota", "fota" ],
> > 
> > Although this script currently runs only for flame-kk, this is intended as a
> > generic config file for OTA builds, are you sure you want to build fota for
> > every device?
> 
> Well that's something we can change if we need, but I guess it will do no
> harm to produce OTA and FOTA. Do you see any reason we should not? Just from
> a QA point of view I guess it's useful to keep both on all devices, since it
> can help QA to check behavior of both.

I see no problem with that :)

Comment 83

[:gerard-majax]

https://tools.taskcluster.net/task-graph-inspector/#Pjwd3f3tTxaiKYdHktvVew/

Comment 84

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #83)
> https://tools.taskcluster.net/task-graph-inspector/#Pjwd3f3tTxaiKYdHktvVew/

See bug 1201540 comment 86: OTA and FOTA produced out of the same Flame KK build

Comment 85

[:gerard-majax]

checkin-needed for attachment 8682403 [details] [diff] [review]: that should wait bug 1201540 to land attachment 8683110 [details] [diff] [review]

Comment 86

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/5a3d69c1b2ad

Comment 87

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5a3d69c1b2ad

Comment 88

[Nick Thomas [:nthomas] (UTC+12)]

http://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/002fa417cf7c