Closed Bug 958202 Opened 9 years ago Closed 8 years ago

Switch update server for Buri from OTA to FOTA by using the solution seen in bug 935059

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(blocking-b2g:1.3+)

RESOLVED FIXED
blocking-b2g 1.3+

People

(Reporter: jsmith, Assigned: aki)

References

Details

(Whiteboard: [NPOTB])

Attachments

(4 files, 2 obsolete files)

We now have a FOTA solution we can use for internal testing in bug 935059. We switch over the update server to use the new FOTA solution instead of the old OTA solution.
Blocks: 935059
FYI, this is valid for not only buri, but it should work on all devices ; at least when I tested I successfully used: keon, peak, nexus s, nexus 4 and inari. Some might be dependent on bug 952535.
This is a followup to a 1.3 blocker that's required to get FOTA working for internal testing, so this needs to be treated as a 1.3 blocker. For tracking purposes, we can treat this as NPOTB technically though, as this is internal infrastructure.
blocking-b2g: --- → 1.3?
Whiteboard: [NPOTB]
Question that came up during a data migration testing discussion:

When the new FOTA solution gets stood up, can we get a URL to the location of the server?
The URL shouldn't change.

The contents of the updates is the only thing that's different between FOTA and OTA, so there is absolutely no reason to bring up a new server.
Aki,

ni for FOTA.
blocking-b2g: 1.3? → 1.3+
Flags: needinfo?(aki)
I'll take it.

(In reply to Dave Hylands [:dhylands] from comment #4)
> The URL shouldn't change.
> 
> The contents of the updates is the only thing that's different between FOTA
> and OTA, so there is absolutely no reason to bring up a new server.

I think the snippet xml needs to change to serve the FOTA, correct?
On all 3 update servers we have, we assume certain snippet behavior (update.boot2gecko.org builds the xml from a wsgi app; aus and balrog have strict attribute whitelists that they allow, aiui), so there will need to be server side changes as well.
Assignee: nobody → aki
Flags: needinfo?(aki)
Is FOTA available on all branches, or only on 1.3 and newer?
Flags: needinfo?(dhylands)
FOTA has been available on the phones for quite some time (probably 1.0 or newer).

However, the scripts to generate the FOTA package may need to be backported
Flags: needinfo?(dhylands)
Ok. Yeah, I think it'll be easier if we can use the same update process for all live branches (1.1, 1.1hd, 1.2, 1.3, master).  Otherwise we'll need to support both until 1.2 EOL.
The scripts used to generate the FOTA package were backported to 1.3 but not to the older branches. I'll try to uplift them to the older branches tomorrow.
:16:41     INFO -  Generating FOTA update package
23:16:41     INFO -  mkdir -p `dirname out/target/product/hamachi/fota/partial/update.zip` || true
23:16:41    ERROR -  Traceback (most recent call last):
23:16:41     INFO -    File "tools/update-tools/build-flash-fota.py", line 19, in <module>
23:16:41     INFO -      import argparse
23:16:41     INFO -  ImportError: No module named argparse
23:16:41     INFO -  make: *** [out/target/product/hamachi/fota/partial/update.zip] Error 1
23:16:41     INFO -  real	4m52.868s
23:16:41     INFO -  user	3m36.363s
23:16:41     INFO -  sys	0m38.385s
23:16:41     INFO -  
23:16:41     INFO -  > Build failed! <
23:16:41     INFO -  Build with |./build.sh -j1| for better messages
23:16:41     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
23:16:41    ERROR - Return code: 2

Has this introduced new dependencies?
Hm, we have python 2.7 installed, and argparse is built in.  I wonder if we're calling the wrong python.
(In reply to Aki Sasaki [:aki] from comment #12)
> I wonder if we're calling the wrong python.

Sounds weird, I've quickly checked my python installation and argparse is installed by default in both 2.7 and 3.3. I'm not familiar with the language difference between 2.7 and 3+, could it be possible that the script doesn't work on python versions >3?
(In reply to Gabriele Svelto [:gsvelto] from comment #13)
> (In reply to Aki Sasaki [:aki] from comment #12)
> > I wonder if we're calling the wrong python.
> 
> Sounds weird, I've quickly checked my python installation and argparse is
> installed by default in both 2.7 and 3.3. I'm not familiar with the language
> difference between 2.7 and 3+, could it be possible that the script doesn't
> work on python versions >3?

We're using /usr/bin/python, which is python 2.6 and doesn't have argparse.

I re-started with python 2.7 at the front of PATH, but it seems to ignore it.  Does the script start with #!/usr/bin/env python, or is it called with a hardcoded /usr/bin/python?
Flags: needinfo?(gsvelto)
(In reply to Aki Sasaki [:aki] from comment #14)
> I re-started with python 2.7 at the front of PATH, but it seems to ignore
> it.  Does the script start with #!/usr/bin/env python, or is it called with
> a hardcoded /usr/bin/python?

All scripts related to FOTA are using |#!/usr/bin/env python| except for update_tools.py:

https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/update_tools.py

Can you try adding |#!/usr/bin/env python| at the beginning of the file and see if it fixes the problem? If it does open a bug and I'll land the fix immediately.
Flags: needinfo?(gsvelto)
(In reply to Gabriele Svelto [:gsvelto] from comment #15)
> (In reply to Aki Sasaki [:aki] from comment #14)
> > I re-started with python 2.7 at the front of PATH, but it seems to ignore
> > it.  Does the script start with #!/usr/bin/env python, or is it called with
> > a hardcoded /usr/bin/python?
> 
> All scripts related to FOTA are using |#!/usr/bin/env python| except for
> update_tools.py:
> 
> https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/
> update_tools.py
> 
> Can you try adding |#!/usr/bin/env python| at the beginning of the file and
> see if it fixes the problem? If it does open a bug and I'll land the fix
> immediately.

I added #!/usr/bin/env python , but it still broke since I wasn't setting PATH appropriately.  Once I set PATH appropriately, it appears to work with or without the #!/usr/bin/env python .  I think it would be cleaner to add the hash-bang, but I don't think it strictly blocks.

I'll patch b2g_build.py to set the PATH appropriately and try again from the top.
Attached patch fota_wsgiSplinter Review
This will turn all updates served from update.boot2gecko.org into FOTA updates.

Alternately we could have a fork of /data/dogfood live elsewhere, and a subset of the platform/version directories point at the FOTA fork.

I *think* this is the correct string to add to turn them into FOTA updates, according to https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/update_tools.py#L567 ?
Attachment #8364643 - Flags: review?(jgriffin)
Finally tracked down the issue with PATH: http://hg.mozilla.org/build/mozharness/file/697adf40550e/scripts/b2g_build.py#l887 was actually affecting self.env, which affected later self.query_env() calls.  Fixed in staging.

Latest build failed on update.xml creation; fixing.
Attachment #8364643 - Flags: review?(jgriffin) → review+
(In reply to Aki Sasaki [:aki] from comment #17)
> Created attachment 8364643 [details] [diff] [review]
> fota_wsgi
> 
> This will turn all updates served from update.boot2gecko.org into FOTA
> updates.
> 
> Alternately we could have a fork of /data/dogfood live elsewhere, and a
> subset of the platform/version directories point at the FOTA fork.
> 
> I *think* this is the correct string to add to turn them into FOTA updates,
> according to
> https://github.com/mozilla-b2g/B2G/blob/master/tools/update-tools/
> update_tools.py#L567 ?

Yep, that's the string I recall using too, back when I was creating those things manually.
Attached patch mozharness - fota updates (obsolete) — Splinter Review
This switches all b2g builds to fota updates.
Attachment #8365401 - Flags: review?(rail)
Comment on attachment 8365401 [details] [diff] [review]
mozharness - fota updates

Review of attachment 8365401 [details] [diff] [review]:
-----------------------------------------------------------------

I have a couple of questions here.

::: configs/b2g/releng-beta.py
@@ +81,5 @@
>          "SYMBOL_SERVER_SSH_KEY": "/home/mock_mozilla/.ssh/b2gbld_dsa",
>          "SYMBOL_SERVER_PATH": "/mnt/netapp/breakpad/symbols_b2g/",
>          "POST_SYMBOL_UPLOAD_CMD": "/usr/local/bin/post-symbol-upload.py",
>          "WGET_OPTS": "-c -q",
> +        "PATH": "/tools/python27/bin:%(PATH)s",

Will %(PATH)s be replaces by current path somehow?

::: mozharness/mozilla/signing.py
@@ +86,5 @@
>              replace_dict['sha512_hash'] = sha512_hash
>          else:
>              replace_dict['sha512_hash'] = self.query_sha512sum(binary_path)
> +        if extra_update_attrs:
> +            replace_dict['extra_update_attrs'] = ' %s' % extra_update_attrs

I think you need
else:
  replace_dict['extra_update_attrs'] = ''

here, or something like that to set that dict entry so interpolation doesn't fail.
(In reply to Rail Aliiev [:rail] from comment #21)
> Comment on attachment 8365401 [details] [diff] [review]
> mozharness - fota updates
> 
> Review of attachment 8365401 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I have a couple of questions here.
> 
> ::: configs/b2g/releng-beta.py
> @@ +81,5 @@
> >          "SYMBOL_SERVER_SSH_KEY": "/home/mock_mozilla/.ssh/b2gbld_dsa",
> >          "SYMBOL_SERVER_PATH": "/mnt/netapp/breakpad/symbols_b2g/",
> >          "POST_SYMBOL_UPLOAD_CMD": "/usr/local/bin/post-symbol-upload.py",
> >          "WGET_OPTS": "-c -q",
> > +        "PATH": "/tools/python27/bin:%(PATH)s",
> 
> Will %(PATH)s be replaces by current path somehow?

Yeah, in query_exe():
http://hg.mozilla.org/build/mozharness/file/4a2b05714936/mozharness/base/script.py#l544

> ::: mozharness/mozilla/signing.py
> @@ +86,5 @@
> >              replace_dict['sha512_hash'] = sha512_hash
> >          else:
> >              replace_dict['sha512_hash'] = self.query_sha512sum(binary_path)
> > +        if extra_update_attrs:
> > +            replace_dict['extra_update_attrs'] = ' %s' % extra_update_attrs
> 
> I think you need
> else:
>   replace_dict['extra_update_attrs'] = ''
> 
> here, or something like that to set that dict entry so interpolation doesn't
> fail.

Good catch!  I tried to account for that by setting it to '' instead of None but then failed to put it in the replace_dict.
Attached patch with the else (obsolete) — Splinter Review
Attachment #8365401 - Attachment is obsolete: true
Attachment #8365401 - Flags: review?(rail)
Attachment #8365420 - Flags: review?(rail)
(In reply to Aki Sasaki [:aki] from comment #22)
> (In reply to Rail Aliiev [:rail] from comment #21)
> Yeah, in query_exe():
> http://hg.mozilla.org/build/mozharness/file/4a2b05714936/mozharness/base/
> script.py#l544

er, query_env()
Attachment #8365420 - Flags: review?(rail) → review+
Depends on: 963894
Busted in inari-eng stage_update.py (bug 963894)
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566133&tree=Mozilla-Central&full=1

Helix, helix-eng, leo, and leo-eng busted during ./build.sh gecko-update-fota :
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566420&tree=Mozilla-Central&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566158&tree=Mozilla-Central&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566272&tree=Mozilla-Central&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=33566412&tree=Mozilla-Central&full=1
21:39:53     INFO -  Created filesystem with 911/32768 inodes and 43595/131072 blocks
21:39:53     INFO -  Install system fs image: out/target/product/helix/system.img
21:40:01     INFO -  out/target/product/helix/system.img+ total size is 172030120
21:40:01     INFO -  Using ext4 filesystem
21:40:01     INFO -  gonk-misc/Android.mk:437: Ext FS but no recovery fstab. Using values specified by env: SYSTEM_PARTITION and DATA_PARTITION:
21:40:01     INFO -  gonk-misc/Android.mk:437: SYSTEM_PARTITION @
21:40:01     INFO -  gonk-misc/Android.mk:437: DATA_PARTITION @
21:40:01    ERROR -  gonk-misc/Android.mk:437: *** No SYSTEM_PARTITION.  Stop.
21:40:01     INFO -  real	5m46.346s
21:40:01     INFO -  user	4m19.489s
21:40:01     INFO -  sys	1m14.291s
21:40:01     INFO -  
21:40:01     INFO -  > Build failed! <
21:40:01     INFO -  Build with |./build.sh -j1| for better messages
21:40:01     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
21:40:01    ERROR - Return code: 2
21:40:01    FATAL - failed to create complete update
21:40:01    FATAL - Running post_fatal callback...
21:40:01    FATAL - Exiting 2
And mako (nexus 4):
21:51:20     INFO -  Generating FOTA update package
21:51:20     INFO -  mkdir -p `dirname out/target/product/mako/fota/partial/update.zip` || true
21:51:24     INFO -  Error: java is required to be on your PATH for signing
21:51:24     INFO -  make: *** [out/target/product/mako/fota/partial/update.zip] Error 1
21:51:24     INFO -  real	6m18.492s
21:51:24     INFO -  user	3m52.090s
21:51:24     INFO -  sys	1m6.779s
21:51:24     INFO -  
21:51:24     INFO -  > Build failed! <
21:51:24     INFO -  Build with |./build.sh -j1| for better messages
21:51:24     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
21:51:24    ERROR - Return code: 2
21:51:24    FATAL - failed to create complete update
21:51:24    FATAL - Running post_fatal callback...
21:51:24    FATAL - Exiting 2
No longer blocks: 935059
Depends on: 935059
Depends on: 963896
Filed bug 963896 for the busted gecko-update-fota devices.

Work is blocked until bug 963896 and bug 963894 are fixed.
Whiteboard: [NPOTB] → [NPOTB][Blocked]
(In reply to Aki Sasaki [:aki] from comment #29)
> Filed bug 963896 for the busted gecko-update-fota devices.
> 
> Work is blocked until bug 963896 and bug 963894 are fixed.

So both of those bugs deal with cases that don't involve the Buri device.

Couldn't we just enable this only for Buri? Or are those dependent bugs going to happen on Buri as well?
(In reply to Jason Smith [:jsmith] from comment #30)
> (In reply to Aki Sasaki [:aki] from comment #29)
> > Filed bug 963896 for the busted gecko-update-fota devices.
> > 
> > Work is blocked until bug 963896 and bug 963894 are fixed.
> 
> So both of those bugs deal with cases that don't involve the Buri device.
> 
> Couldn't we just enable this only for Buri? Or are those dependent bugs
> going to happen on Buri as well?

Sure, we could re-scope to only do FOTA for Buri.
(In reply to Jason Smith [:jsmith] from comment #30)
> (In reply to Aki Sasaki [:aki] from comment #29)
> Couldn't we just enable this only for Buri? Or are those dependent bugs
> going to happen on Buri as well?

Buri will work fine. We'll have to fix other devices on a case-by-case basis as we found out they often have different partition schemes or build system quirks that require some changes for the FOTA image builds to work properly.
Attached patch fota-buri-onlySplinter Review
Running this in staging.
Attachment #8365420 - Attachment is obsolete: true
Comment on attachment 8366220 [details] [diff] [review]
fota-buri-only

Ok, in staging, leo and leo-eng didn't break (using OTA) and hamachi built FOTA just fine.  I'm running a hamachi-eng nightly as well.
Attachment #8366220 - Flags: review?(rail)
Whiteboard: [NPOTB][Blocked] → [NPOTB]
Attached patch fota_configsSplinter Review
Switch hamachi and hamachi-eng to the new config files.
I suppose I could have made this a cmdline option as well...
Attachment #8366381 - Flags: review?(rail)
Comment on attachment 8366220 [details] [diff] [review]
fota-buri-only

lgtm
Attachment #8366220 - Flags: review?(rail) → review+
Attachment #8366381 - Flags: review?(rail) → review+
Bug re-scoped.
I'll merge/reconfig/kick off nightlies/make the server changes tomorrow.
Summary: Switch update server for FxOS from OTA to FOTA by using the solution seen in bug 935059 → Switch update server for Buri from OTA to FOTA by using the solution seen in bug 935059
FOTA updates are live for buri 1.4.0 and 1.3.0.
Has this landed on b2g28_v1_2 ?
https://tbpl.mozilla.org/php/getParsedLog.php?id=33720121&tree=Mozilla-B2g26-v1.2&full=1

32:31    ERROR -  make: *** No rule to make target `gecko-update-fota'.  Stop.
20:32:31     INFO -  real	0m9.551s
20:32:31     INFO -  user	0m7.614s
20:32:31     INFO -  sys	0m1.715s
20:32:31     INFO -  
20:32:31     INFO -  > Build failed! <
20:32:31     INFO -  Build with |./build.sh -j1| for better messages
20:32:31     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
20:32:31    ERROR - Return code: 2
Flags: needinfo?(lissyx+mozillians)
Flags: needinfo?(gsvelto)
buildbot-config patch was in production as of ~3pm PT! But looks like you already knew that. Sorry for tardiness on updating bug :)
(In reply to Aki Sasaki [:aki] from comment #42)
> Has this landed on b2g28_v1_2 ?

Not yet, sorry about that. Now it has landed:

https://github.com/mozilla-b2g/gonk-misc/commit/1093d10c20be64a9c79787096f69f096b07da5a2

Do you need it on v1-train too?
Flags: needinfo?(gsvelto)
(In reply to Gabriele Svelto [:gsvelto] from comment #44)
> (In reply to Aki Sasaki [:aki] from comment #42)
> > Has this landed on b2g28_v1_2 ?
> 
> Not yet, sorry about that. Now it has landed:
> 
> https://github.com/mozilla-b2g/gonk-misc/commit/
> 1093d10c20be64a9c79787096f69f096b07da5a2

Thanks!
We're green on 1.2!  FOTA updates live for 1.2 as well.

> Do you need it on v1-train too?

Jason: do we need FOTA for buri on 1.1.0 and 1.1hd?  Looks like it's just a git merge, a one-line buildbot-configs patch, a reconfig, and a server-side wsgi edit away.  Or we can resolve this bug now that we have FOTA updates for buri on >= 1.2.0.
Flags: needinfo?(lissyx+mozillians) → needinfo?(jsmith)
(In reply to Aki Sasaki [:aki] from comment #45)
> (In reply to Gabriele Svelto [:gsvelto] from comment #44)
> > (In reply to Aki Sasaki [:aki] from comment #42)
> > > Has this landed on b2g28_v1_2 ?
> > 
> > Not yet, sorry about that. Now it has landed:
> > 
> > https://github.com/mozilla-b2g/gonk-misc/commit/
> > 1093d10c20be64a9c79787096f69f096b07da5a2
> 
> Thanks!
> We're green on 1.2!  FOTA updates live for 1.2 as well.
> 
> > Do you need it on v1-train too?
> 
> Jason: do we need FOTA for buri on 1.1.0 and 1.1hd?  Looks like it's just a
> git merge, a one-line buildbot-configs patch, a reconfig, and a server-side
> wsgi edit away.  Or we can resolve this bug now that we have FOTA updates
> for buri on >= 1.2.0.

Buri 1.1 = Yes
Buri 1.1 HD = No
Flags: needinfo?(jsmith)
OK, pushed to v1-train too, SHA1 e4aef220cca5d33fd90d84af651a7b700723f8bb
Attached patch fota_1.1Splinter Review
I'm just changing the config file to use the releng-fota-updates.py config file.
Attachment #8367653 - Flags: review?(jlund)
Comment on attachment 8367653 [details] [diff] [review]
fota_1.1

lgtm
Attachment #8367653 - Flags: review?(jlund) → review+
Should be live for 1.1 as well.
Resolving.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Depends on: 966962
Comment on attachment 8364643 [details] [diff] [review]
fota_wsgi

For posterity:  Should be isOSUpdate, not isOsUpdate.
(Fixed on update.boot2gecko.org)
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.