958202 - Switch update server for Buri from OTA to FOTA by using the solution seen in bug 935059

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jason Smith [:jsmith]]

We now have a FOTA solution we can use for internal testing in bug 935059. We switch over the update server to use the new FOTA solution instead of the old OTA solution.

Comment 1

[Alexandre LISSY :gerard-majax]

FYI, this is valid for not only buri, but it should work on all devices ; at least when I tested I successfully used: keon, peak, nexus s, nexus 4 and inari. Some might be dependent on bug 952535.

Comment 2

[Jason Smith [:jsmith]]

This is a followup to a 1.3 blocker that's required to get FOTA working for internal testing, so this needs to be treated as a 1.3 blocker. For tracking purposes, we can treat this as NPOTB technically though, as this is internal infrastructure.

Comment 3

[Jason Smith [:jsmith]]

Question that came up during a data migration testing discussion:

When the new FOTA solution gets stood up, can we get a URL to the location of the server?

Comment 4

[Dave Hylands [:dhylands]]

The URL shouldn't change.

The contents of the updates is the only thing that's different between FOTA and OTA, so there is absolutely no reason to bring up a new server.

Comment 5

[Preeti Raghunath(:Preeti)]

Aki,

ni for FOTA.

Comment 6

[Aki Sasaki (not active)]

I'll take it.

(In reply to Dave Hylands [:dhylands] from comment #4)
> The URL shouldn't change.
> 
> The contents of the updates is the only thing that's different between FOTA
> and OTA, so there is absolutely no reason to bring up a new server.

I think the snippet xml needs to change to serve the FOTA, correct?
On all 3 update servers we have, we assume certain snippet behavior (update.boot2gecko.org builds the xml from a wsgi app; aus and balrog have strict attribute whitelists that they allow, aiui), so there will need to be server side changes as well.

Comment 7

[Aki Sasaki (not active)]

Is FOTA available on all branches, or only on 1.3 and newer?

Comment 8

[Dave Hylands [:dhylands]]

FOTA has been available on the phones for quite some time (probably 1.0 or newer).

However, the scripts to generate the FOTA package may need to be backported

Comment 9

[Aki Sasaki (not active)]

Ok. Yeah, I think it'll be easier if we can use the same update process for all live branches (1.1, 1.1hd, 1.2, 1.3, master).  Otherwise we'll need to support both until 1.2 EOL.

Comment 10

[Gabriele Svelto [:gsvelto]]

The scripts used to generate the FOTA package were backported to 1.3 but not to the older branches. I'll try to uplift them to the older branches tomorrow.

Comment 11

[Aki Sasaki (not active)]

:16:41     INFO -  Generating FOTA update package
23:16:41     INFO -  mkdir -p `dirname out/target/product/hamachi/fota/partial/update.zip` || true
23:16:41    ERROR -  Traceback (most recent call last):
23:16:41     INFO -    File "tools/update-tools/build-flash-fota.py", line 19, in <module>
23:16:41     INFO -      import argparse
23:16:41     INFO -  ImportError: No module named argparse
23:16:41     INFO -  make: *** [out/target/product/hamachi/fota/partial/update.zip] Error 1
23:16:41     INFO -  real	4m52.868s
23:16:41     INFO -  user	3m36.363s
23:16:41     INFO -  sys	0m38.385s
23:16:41     INFO -  
23:16:41     INFO -  > Build failed! <
23:16:41     INFO -  Build with |./build.sh -j1| for better messages
23:16:41     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
23:16:41    ERROR - Return code: 2

Has this introduced new dependencies?

Comment 12

[Aki Sasaki (not active)]

Hm, we have python 2.7 installed, and argparse is built in.  I wonder if we're calling the wrong python.

Comment 13

[Gabriele Svelto [:gsvelto]]

(In reply to Aki Sasaki [:aki] from comment #12)
> I wonder if we're calling the wrong python.

Sounds weird, I've quickly checked my python installation and argparse is installed by default in both 2.7 and 3.3. I'm not familiar with the language difference between 2.7 and 3+, could it be possible that the script doesn't work on python versions >3?

Comment 14

[Aki Sasaki (not active)]

(In reply to Gabriele Svelto [:gsvelto] from comment #13)
> (In reply to Aki Sasaki [:aki] from comment #12)
> > I wonder if we're calling the wrong python.
> 
> Sounds weird, I've quickly checked my python installation and argparse is
> installed by default in both 2.7 and 3.3. I'm not familiar with the language
> difference between 2.7 and 3+, could it be possible that the script doesn't
> work on python versions >3?

We're using /usr/bin/python, which is python 2.6 and doesn't have argparse.

I re-started with python 2.7 at the front of PATH, but it seems to ignore it.  Does the script start with #!/usr/bin/env python, or is it called with a hardcoded /usr/bin/python?

Comment 15

[Gabriele Svelto [:gsvelto]]

(In reply to Aki Sasaki [:aki] from comment #14)
> I re-started with python 2.7 at the front of PATH, but it seems to ignore
> it.  Does the script start with #!/usr/bin/env python, or is it called with
> a hardcoded /usr/bin/python?

All scripts related to FOTA are using |#!/usr/bin/env python| except for update_tools.py:

https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/update_tools.py

Can you try adding |#!/usr/bin/env python| at the beginning of the file and see if it fixes the problem? If it does open a bug and I'll land the fix immediately.

Comment 16

[Aki Sasaki (not active)]

(In reply to Gabriele Svelto [:gsvelto] from comment #15)
> (In reply to Aki Sasaki [:aki] from comment #14)
> > I re-started with python 2.7 at the front of PATH, but it seems to ignore
> > it.  Does the script start with #!/usr/bin/env python, or is it called with
> > a hardcoded /usr/bin/python?
> 
> All scripts related to FOTA are using |#!/usr/bin/env python| except for
> update_tools.py:
> 
> https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/
> update_tools.py
> 
> Can you try adding |#!/usr/bin/env python| at the beginning of the file and
> see if it fixes the problem? If it does open a bug and I'll land the fix
> immediately.

I added #!/usr/bin/env python , but it still broke since I wasn't setting PATH appropriately.  Once I set PATH appropriately, it appears to work with or without the #!/usr/bin/env python .  I think it would be cleaner to add the hash-bang, but I don't think it strictly blocks.

I'll patch b2g_build.py to set the PATH appropriately and try again from the top.

Comment 17

[Aki Sasaki (not active)]

Attachment: fota_wsgi

This will turn all updates served from update.boot2gecko.org into FOTA updates.

Alternately we could have a fork of /data/dogfood live elsewhere, and a subset of the platform/version directories point at the FOTA fork.

I *think* this is the correct string to add to turn them into FOTA updates, according to https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/update_tools.py#L567 ?

Comment 18

[Aki Sasaki (not active)]

Finally tracked down the issue with PATH: http://hg.mozilla.org/build/mozharness/file/697adf40550e/scripts/b2g_build.py#l887 was actually affecting self.env, which affected later self.query_env() calls.  Fixed in staging.

Latest build failed on update.xml creation; fixing.

Comment 19

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #17)
> Created attachment 8364643 [details] [diff] [review]
> fota_wsgi
> 
> This will turn all updates served from update.boot2gecko.org into FOTA
> updates.
> 
> Alternately we could have a fork of /data/dogfood live elsewhere, and a
> subset of the platform/version directories point at the FOTA fork.
> 
> I *think* this is the correct string to add to turn them into FOTA updates,
> according to
> https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/
> update_tools.py#L567 ?

Yep, that's the string I recall using too, back when I was creating those things manually.

Comment 20

[Aki Sasaki (not active)]

Attachment: mozharness - fota updates

This switches all b2g builds to fota updates.

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8365401 [details] [diff] [review]
mozharness - fota updates

Review of attachment 8365401 [details] [diff] [review]:
-----------------------------------------------------------------

I have a couple of questions here.

::: configs/b2g/releng-beta.py
@@ +81,5 @@
>          "SYMBOL_SERVER_SSH_KEY": "/home/mock_mozilla/.ssh/b2gbld_dsa",
>          "SYMBOL_SERVER_PATH": "/mnt/netapp/breakpad/symbols_b2g/",
>          "POST_SYMBOL_UPLOAD_CMD": "/usr/local/bin/post-symbol-upload.py",
>          "WGET_OPTS": "-c -q",
> +        "PATH": "/tools/python27/bin:%(PATH)s",

Will %(PATH)s be replaces by current path somehow?

::: mozharness/mozilla/signing.py
@@ +86,5 @@
>              replace_dict['sha512_hash'] = sha512_hash
>          else:
>              replace_dict['sha512_hash'] = self.query_sha512sum(binary_path)
> +        if extra_update_attrs:
> +            replace_dict['extra_update_attrs'] = ' %s' % extra_update_attrs

I think you need
else:
  replace_dict['extra_update_attrs'] = ''

here, or something like that to set that dict entry so interpolation doesn't fail.

Comment 22

[Aki Sasaki (not active)]

(In reply to Rail Aliiev [:rail] from comment #21)
> Comment on attachment 8365401 [details] [diff] [review]
> mozharness - fota updates
> 
> Review of attachment 8365401 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I have a couple of questions here.
> 
> ::: configs/b2g/releng-beta.py
> @@ +81,5 @@
> >          "SYMBOL_SERVER_SSH_KEY": "/home/mock_mozilla/.ssh/b2gbld_dsa",
> >          "SYMBOL_SERVER_PATH": "/mnt/netapp/breakpad/symbols_b2g/",
> >          "POST_SYMBOL_UPLOAD_CMD": "/usr/local/bin/post-symbol-upload.py",
> >          "WGET_OPTS": "-c -q",
> > +        "PATH": "/tools/python27/bin:%(PATH)s",
> 
> Will %(PATH)s be replaces by current path somehow?

Yeah, in query_exe():
http://hg.mozilla.org/build/mozharness/file/4a2b05714936/mozharness/base/script.py#l544

> ::: mozharness/mozilla/signing.py
> @@ +86,5 @@
> >              replace_dict['sha512_hash'] = sha512_hash
> >          else:
> >              replace_dict['sha512_hash'] = self.query_sha512sum(binary_path)
> > +        if extra_update_attrs:
> > +            replace_dict['extra_update_attrs'] = ' %s' % extra_update_attrs
> 
> I think you need
> else:
>   replace_dict['extra_update_attrs'] = ''
> 
> here, or something like that to set that dict entry so interpolation doesn't
> fail.

Good catch!  I tried to account for that by setting it to '' instead of None but then failed to put it in the replace_dict.

Comment 23

[Aki Sasaki (not active)]

Attachment: with the else

Comment 24

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #22)
> (In reply to Rail Aliiev [:rail] from comment #21)
> Yeah, in query_exe():
> http://hg.mozilla.org/build/mozharness/file/4a2b05714936/mozharness/base/
> script.py#l544

er, query_env()

Comment 25

[Aki Sasaki (not active)]

Comment on attachment 8365420 [details] [diff] [review]
with the else

https://hg.mozilla.org/build/mozharness/rev/ef0cb3ea6ab0

Comment 26

[Aki Sasaki (not active)]

Busted in inari-eng stage_update.py (bug 963894)
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566133&tree=Mozilla-Central&full=1

Helix, helix-eng, leo, and leo-eng busted during ./build.sh gecko-update-fota :
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566420&tree=Mozilla-Central&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566158&tree=Mozilla-Central&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566272&tree=Mozilla-Central&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566412&tree=Mozilla-Central&full=1
21:39:53     INFO -  Created filesystem with 911/32768 inodes and 43595/131072 blocks
21:39:53     INFO -  Install system fs image: out/target/product/helix/system.img
21:40:01     INFO -  out/target/product/helix/system.img+ total size is 172030120
21:40:01     INFO -  Using ext4 filesystem
21:40:01     INFO -  gonk-misc/Android.mk:437: Ext FS but no recovery fstab. Using values specified by env: SYSTEM_PARTITION and DATA_PARTITION:
21:40:01     INFO -  gonk-misc/Android.mk:437: SYSTEM_PARTITION @
21:40:01     INFO -  gonk-misc/Android.mk:437: DATA_PARTITION @
21:40:01    ERROR -  gonk-misc/Android.mk:437: *** No SYSTEM_PARTITION.  Stop.
21:40:01     INFO -  real	5m46.346s
21:40:01     INFO -  user	4m19.489s
21:40:01     INFO -  sys	1m14.291s
21:40:01     INFO -  
21:40:01     INFO -  > Build failed! <
21:40:01     INFO -  Build with |./build.sh -j1| for better messages
21:40:01     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
21:40:01    ERROR - Return code: 2
21:40:01    FATAL - failed to create complete update
21:40:01    FATAL - Running post_fatal callback...
21:40:01    FATAL - Exiting 2

Comment 27

[Aki Sasaki (not active)]

And mako (nexus 4):
21:51:20     INFO -  Generating FOTA update package
21:51:20     INFO -  mkdir -p `dirname out/target/product/mako/fota/partial/update.zip` || true
21:51:24     INFO -  Error: java is required to be on your PATH for signing
21:51:24     INFO -  make: *** [out/target/product/mako/fota/partial/update.zip] Error 1
21:51:24     INFO -  real	6m18.492s
21:51:24     INFO -  user	3m52.090s
21:51:24     INFO -  sys	1m6.779s
21:51:24     INFO -  
21:51:24     INFO -  > Build failed! <
21:51:24     INFO -  Build with |./build.sh -j1| for better messages
21:51:24     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
21:51:24    ERROR - Return code: 2
21:51:24    FATAL - failed to create complete update
21:51:24    FATAL - Running post_fatal callback...
21:51:24    FATAL - Exiting 2

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 8365420 [details] [diff] [review]
with the else

Backed out:
http://hg.mozilla.org/build/mozharness/rev/05051454e5b1

Comment 29

[Aki Sasaki (not active)]

Filed bug 963896 for the busted gecko-update-fota devices.

Work is blocked until bug 963896 and bug 963894 are fixed.

Comment 30

[Jason Smith [:jsmith]]

(In reply to Aki Sasaki [:aki] from comment #29)
> Filed bug 963896 for the busted gecko-update-fota devices.
> 
> Work is blocked until bug 963896 and bug 963894 are fixed.

So both of those bugs deal with cases that don't involve the Buri device.

Couldn't we just enable this only for Buri? Or are those dependent bugs going to happen on Buri as well?

Comment 31

[Aki Sasaki (not active)]

(In reply to Jason Smith [:jsmith] from comment #30)
> (In reply to Aki Sasaki [:aki] from comment #29)
> > Filed bug 963896 for the busted gecko-update-fota devices.
> > 
> > Work is blocked until bug 963896 and bug 963894 are fixed.
> 
> So both of those bugs deal with cases that don't involve the Buri device.
> 
> Couldn't we just enable this only for Buri? Or are those dependent bugs
> going to happen on Buri as well?

Sure, we could re-scope to only do FOTA for Buri.

Comment 32

[Gabriele Svelto [:gsvelto]]

(In reply to Jason Smith [:jsmith] from comment #30)
> (In reply to Aki Sasaki [:aki] from comment #29)
> Couldn't we just enable this only for Buri? Or are those dependent bugs
> going to happen on Buri as well?

Buri will work fine. We'll have to fix other devices on a case-by-case basis as we found out they often have different partition schemes or build system quirks that require some changes for the FOTA image builds to work properly.

Comment 33

[Aki Sasaki (not active)]

Attachment: fota-buri-only

Running this in staging.

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 8366220 [details] [diff] [review]
fota-buri-only

Ok, in staging, leo and leo-eng didn't break (using OTA) and hamachi built FOTA just fine.  I'm running a hamachi-eng nightly as well.

Comment 35

[Aki Sasaki (not active)]

Interdiff: https://bugzilla.mozilla.org/attachment.cgi?oldid=8365420&action=interdiff&newid=8366220&headers=1

Comment 36

[Aki Sasaki (not active)]

Attachment: fota_configs

Switch hamachi and hamachi-eng to the new config files.
I suppose I could have made this a cmdline option as well...

Comment 37

[Rail Aliiev [:rail]]

Comment on attachment 8366220 [details] [diff] [review]
fota-buri-only

lgtm

Comment 38

[Aki Sasaki (not active)]

Comment on attachment 8366220 [details] [diff] [review]
fota-buri-only

https://hg.mozilla.org/build/mozharness/rev/36c2a89a53bf

Comment 39

[Aki Sasaki (not active)]

Comment on attachment 8366381 [details] [diff] [review]
fota_configs

https://hg.mozilla.org/build/buildbot-configs/rev/cf9b89e7c09a

Comment 40

[Aki Sasaki (not active)]

Bug re-scoped.
I'll merge/reconfig/kick off nightlies/make the server changes tomorrow.

Comment 41

[Aki Sasaki (not active)]

FOTA updates are live for buri 1.4.0 and 1.3.0.

Comment 42

[Aki Sasaki (not active)]

Has this landed on b2g28_v1_2 ?
https://tbpl.mozilla.org/php/getParsedLog.php?id=33720121&tree=Mozilla-B2g26-v1.2&full=1

32:31    ERROR -  make: *** No rule to make target `gecko-update-fota'.  Stop.
20:32:31     INFO -  real	0m9.551s
20:32:31     INFO -  user	0m7.614s
20:32:31     INFO -  sys	0m1.715s
20:32:31     INFO -  
20:32:31     INFO -  > Build failed! <
20:32:31     INFO -  Build with |./build.sh -j1| for better messages
20:32:31     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
20:32:31    ERROR - Return code: 2

Comment 43

[Jordan Lund (:jlund)]

buildbot-config patch was in production as of ~3pm PT! But looks like you already knew that. Sorry for tardiness on updating bug :)

Comment 44

[Gabriele Svelto [:gsvelto]]

(In reply to Aki Sasaki [:aki] from comment #42)
> Has this landed on b2g28_v1_2 ?

Not yet, sorry about that. Now it has landed:

https://github.com/mozilla-b2g/gonk-misc/commit/1093d10c20be64a9c79787096f69f096b07da5a2

Do you need it on v1-train too?

Comment 45

[Aki Sasaki (not active)]

(In reply to Gabriele Svelto [:gsvelto] from comment #44)
> (In reply to Aki Sasaki [:aki] from comment #42)
> > Has this landed on b2g28_v1_2 ?
> 
> Not yet, sorry about that. Now it has landed:
> 
> https://github.com/mozilla-b2g/gonk-misc/commit/
> 1093d10c20be64a9c79787096f69f096b07da5a2

Thanks!
We're green on 1.2!  FOTA updates live for 1.2 as well.

> Do you need it on v1-train too?

Jason: do we need FOTA for buri on 1.1.0 and 1.1hd?  Looks like it's just a git merge, a one-line buildbot-configs patch, a reconfig, and a server-side wsgi edit away.  Or we can resolve this bug now that we have FOTA updates for buri on >= 1.2.0.

Comment 46

[Jason Smith [:jsmith]]

(In reply to Aki Sasaki [:aki] from comment #45)
> (In reply to Gabriele Svelto [:gsvelto] from comment #44)
> > (In reply to Aki Sasaki [:aki] from comment #42)
> > > Has this landed on b2g28_v1_2 ?
> > 
> > Not yet, sorry about that. Now it has landed:
> > 
> > https://github.com/mozilla-b2g/gonk-misc/commit/
> > 1093d10c20be64a9c79787096f69f096b07da5a2
> 
> Thanks!
> We're green on 1.2!  FOTA updates live for 1.2 as well.
> 
> > Do you need it on v1-train too?
> 
> Jason: do we need FOTA for buri on 1.1.0 and 1.1hd?  Looks like it's just a
> git merge, a one-line buildbot-configs patch, a reconfig, and a server-side
> wsgi edit away.  Or we can resolve this bug now that we have FOTA updates
> for buri on >= 1.2.0.

Buri 1.1 = Yes
Buri 1.1 HD = No

Comment 47

[Gabriele Svelto [:gsvelto]]

OK, pushed to v1-train too, SHA1 e4aef220cca5d33fd90d84af651a7b700723f8bb

Comment 48

[Aki Sasaki (not active)]

Attachment: fota_1.1

I'm just changing the config file to use the releng-fota-updates.py config file.

Comment 49

[Jordan Lund (:jlund)]

Comment on attachment 8367653 [details] [diff] [review]
fota_1.1

lgtm

Comment 50

[Aki Sasaki (not active)]

Comment on attachment 8367653 [details] [diff] [review]
fota_1.1

https://hg.mozilla.org/build/buildbot-configs/rev/2775da4bf0da

Comment 51

[Aki Sasaki (not active)]

Should be live for 1.1 as well.
Resolving.

Comment 52

[Aki Sasaki (not active)]

Comment on attachment 8364643 [details] [diff] [review]
fota_wsgi

For posterity:  Should be isOSUpdate, not isOsUpdate.
(Fixed on update.boot2gecko.org)