816275 - Take over nightly+beta updates from jgriffin

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P2)

Description

[Rail Aliiev [:rail]]

We generate update.mar for b2g builds but we don't publish any metadata which can be used by update servers. As a option releng can generate and publish update.xml file.

Comment 1

[Aki Sasaki (not active)]

This will be copied to http://update.boot2gecko.org/, but we don't currently have access.

Comment 2

[Rail Aliiev [:rail]]

Attachment: b2g updates

to be tested

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 686354 [details] [diff] [review]
b2g updates

>+# TODO: pass licenseURL and detailsURL
>+UPDATE_XML_TEMPLATE = """
>+<?xml version="1.0"?>
>+<updates>
>+  <update type="minor" appVersion="%(version)s" version="%(version)s" extensionVersion="%(version)s" buildID="%(buildid)s" licenseURL="http://www.mozilla.com/test/sample-eula.html" detailsURL="http://www.mozilla.com/test/sample-details.html">
>+      <patch type="complete" URL="%(url)s?build_id=%(buildid)s&amp;version=%(version)s" hashFunction="SHA512" hashValue="%(sha512_hash)s" size="%(size)d"/>
>+  </update>
>+</updates>
>+"""

I see that this is based on http://update.boot2gecko.org/nightly/update.xml, but it could be better. There's a mixture of old and new-style version parameters, so I'd suggest using displayVersion, appVersion, and platformVersion (the modern ones). FWIW we don't pass a licenseURL or detailsURL for desktop nightlies, and those test values are both 404s.

Comment 4

[Rail Aliiev [:rail]]

Attachment: upload update.xml

will need this as well

Comment 5

[Rail Aliiev [:rail]]

Attachment: b2g updates

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 686456 [details] [diff] [review]
b2g updates

>+    def query_version(self):
>+        dirs = self.query_abs_dirs()
>+        application_ini = os.path.join(dirs['work_dir'], 'out', 'target',
>+                                       'product', self.config['target'],
>+                                       'system', 'b2g', 'application.ini')
>+        data = open(application_ini).read()

maybe self.read_from_file() ?

Comment 7

[Rail Aliiev [:rail]]

Attachment: b2g updates

* use mixin methods to read/write to files
interdiff is https://gist.github.com/4171578

Comment 8

[Aki Sasaki (not active)]

Morphing bug.

* get ssh key to jgriffin
* publish mar + app.ini to update.boot2gecko.org for nightlies
* configs for stable builds on stable update channel with a few more tweaks (figure these out)
* investigate aus (not dec10)

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 686736 [details] [diff] [review]
b2g updates

I'm going to make some changes. No need to review for now.

Comment 10

[Rail Aliiev [:rail]]

jgriffin,

can you provide the following information:

* hostname where application.ini and MAR files will be uploaded
* directory(ies) where they should be uploaded

I'll attach ssh public keys in a bit.

Comment 11

[Jonathan Griffin (:jgriffin)]

Host: ec2-184-73-70-191.compute-1.amazonaws.com 

For nightly updates, the path is: /data/update-channels/nightly.

The script which auto-generates update.xml expects the app.ini and .mar files to be named using the format shown in the existing files:  http://update.boot2gecko.org/nightly/.

For stable updates, the path is: /data/update-channels/stable, but please don't push any updates here without posting notice as that will potentially break dogfooders.

Comment 12

[Aki Sasaki (not active)]

Per #b2g

[10:57]	<lsblakk>	jgriffin: the stable channel should just 'die' at this last update (11-27)
[10:58]	<jgriffin>	lsblakk: you mean you don't care about preserving the update path for dogfood users who haven't flashed?
[10:58]	<lsblakk>	correct
[10:58]	<lsblakk>	we will have to flash and get them onto beta, knowing they have new kernel
[10:58]	<lsblakk>	aki: we have to do this to get around not being able to ship a proprietary FOTA update
[10:59]	<lsblakk>	will not block getting testers on new kernel on expecting ZTE to provide updates
[10:59]	<lsblakk>	they are nowhere near ready to do so
[10:59]	<aki>	sounds like i should update https://bugzilla.mozilla.org/show_bug.cgi?id=816275 to providing a nightly+beta channel, not nightly+stable?[10:59]	<lsblakk>	aki: correct
[10:59]	<lsblakk>	we had to change name to 'beta' for crash stats purposes

Comment 13

[Aki Sasaki (not active)]

John emailed Alex and Dietrich on Nov30, to ask if beta updates should be

a) desktop mozilla-aurora style (nightly builds if something has changed, people are updated unless we explicitly block that channel, which generally happens on merge day), or

b) desktop mozilla-beta style (beta release builds that only get built and pushed to the channel after explicit go & qa sign-off)

Comment 14

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #13)
> John emailed Alex and Dietrich on Nov30, to ask if beta updates should be
> 
> a) desktop mozilla-aurora style (nightly builds if something has changed,
> people are updated unless we explicitly block that channel, which generally
> happens on merge day), or
> 
> b) desktop mozilla-beta style (beta release builds that only get built and
> pushed to the channel after explicit go & qa sign-off)

You might want to add lsblakk to that conversation; she's managing the dogfood program which is the only consumer of the beta updates atm.  AFAIK, we want option b)

Comment 15

[Aki Sasaki (not active)]

Lukas: comment 14 :)

I was guessing that since the beta-channel builds would be off the new beta branch, they would in theory be more stable than nightly, then realized that "nightly" could also mean nightly-off-beta.

Your call.

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 686451 [details] [diff] [review]
upload update.xml

This patch isn't required anymore

Comment 17

[Rail Aliiev [:rail]]

Attachment: b2g updates

The current patch implements generation of update.xml file. It:
* copies mar file to b2g_update_${buildid}.mar
* generates update.xml
* copies update.xml to update_${buildid}.xml
* during the first pass of upload it rsyncs everything except update.xml to make sure we don't serve updates for failed uploads
* second pass of upload rsyncs everything, uncluding update.xml

I changed the naming schema to use build ids instead of dates, so it'll be easier to find files if something goes wrong.

Nick, could you take a look at update.xml template?

Example output:

<?xml version="1.0"?>
<updates>
  <update type="minor" displayVersion="20.0a1" appVersion="20.0a1" platformVersion="20.0a1" buildID="20121130165443">
      <patch type="complete" URL="http://update.boot2gecko.org/nightly/b2g_update_20121130165443.mar?build_id=20121130165443&amp;version=20.0a1" hashFunction="SHA512" hashValue="c1da984f14eb0ae90c17958bd380ea017031e0d82da0fd72d452912db11ce292f7c74a878c8f83bd9f3638e1646c0d741d6bcced1819d069faa3660dcdcd7f65" size="47991360"/>
  </update>
</updates>

Comment 18

[Aki Sasaki (not active)]

Comment on attachment 687423 [details] [diff] [review]
b2g updates

Lgtm.

My main concern here is whether update.boot2gecko.org will work with the update.xml, or if we should be uploading the mar+app.ini ?

If it works with update.xml, awesome.

Comment 19

[Rail Aliiev [:rail]]

(In reply to Aki Sasaki [:aki] from comment #18)
> Comment on attachment 687423 [details] [diff] [review]
> b2g updates
> 
> Lgtm.
> 
> My main concern here is whether update.boot2gecko.org will work with the
> update.xml, or if we should be uploading the mar+app.ini ?
> 
> If it works with update.xml, awesome.

I believe update.xml is generated by jenkins from app.ini files using http://update.boot2gecko.org/nightly/makeupdate.sh. Once we take over that job, there is no reason to make it a 2-step operation.

jgriffin: 
when do you prefer to shut down the old method and turn on this one?

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 687423 [details] [diff] [review]
b2g updates

The XML template looks ok to me. It'll look a little odd in the logs when there are two "Upload successful" messages, maybe 'Upload 1 of 2 successful" + "Upload 2 of 2 successful" instead.

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

The 'beta' channel should be more like:

b) desktop mozilla-beta style (beta release builds that only get built and pushed to the channel after explicit go & qa sign-off)

which is how the 'stable' channel currently operates.  If we can have automated, 24-hour delayed builds & updates available for turning on/off that would be amazing, but without that option just being able to manually trigger beta channel automation once a particular nightly has been signed off on would be sufficient.  The Jenkins set up does a build for stable when given a build number for a nightly run, could the releng automation for b2g nightly/beta builds do something like this too based on buildid or changeset?

Comment 22

[Rail Aliiev [:rail]]

Sounds like we're trying to reinvent and reimplement AUS! :)

How about this scenario:

* add another "target" (or platform in buildbot): unagi-beta
* enable nightly build on it (not sure how simple will be disabling CI builds)
* publish dated MAR files and dated update.xml (update_${buildid}.xml) to the "stable" directory
* QA signs off updates against a dated update.xml
* RelEng copies dated update.xml to update.xml

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Rail Aliiev [:rail] from comment #22)
> Sounds like we're trying to reinvent and reimplement AUS! :)
> 
> How about this scenario:
> 
> * add another "target" (or platform in buildbot): unagi-beta
> * enable nightly build on it (not sure how simple will be disabling CI
> builds)
> * publish dated MAR files and dated update.xml (update_${buildid}.xml) to
> the "stable" directory
> * QA signs off updates against a dated update.xml
> * RelEng copies dated update.xml to update.xml

This looks fine.  Esp if that means that once we're routinely passing smoketests we can do away with explicit QA sign off and just automatically copy the dated update.xml every 24 hours after build.

Comment 24

[Jonathan Griffin (:jgriffin)]

(In reply to Rail Aliiev [:rail] from comment #19)
> (In reply to Aki Sasaki [:aki] from comment #18)
> > Comment on attachment 687423 [details] [diff] [review]
> > b2g updates
> > 
> > Lgtm.
> > 
> > My main concern here is whether update.boot2gecko.org will work with the
> > update.xml, or if we should be uploading the mar+app.ini ?
> > 
> > If it works with update.xml, awesome.
> 
> I believe update.xml is generated by jenkins from app.ini files using
> http://update.boot2gecko.org/nightly/makeupdate.sh. Once we take over that
> job, there is no reason to make it a 2-step operation.
> 
> jgriffin: 
> when do you prefer to shut down the old method and turn on this one?

For nightlies we can turn it on today if you'd like.

Comment 25

[Rail Aliiev [:rail]]

Attachment: b2g updates

Another iteration:
* added configs/b2g/releng-beta.py which uploads and publishes updates automatically
* configs/b2g/releng-beta-stable.py which uploads and doesn't publishe updates automatically ("autopublish": False)
* b2g_build.py pushes update.xml only if self.config['update'].get('autopublish') set

buildbot-config changes incoming

Comment 26

[Rail Aliiev [:rail]]

Attachment: bb-configs

I hate loops in configs but... The diff is here https://gist.github.com/4198617

To be tested later tonight.

Comment 27

[Aki Sasaki (not active)]

Comment on attachment 687952 [details] [diff] [review]
b2g updates

I think Catlee might use the commented out staging config lines.
Ideally we get staging_config.py to point at staging config files, but I'm not going to block on that.

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 687954 [details] [diff] [review]
bb-configs

Do we need any sort of merge day comment or w/e here?

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 687952 [details] [diff] [review]
b2g updates

http://hg.mozilla.org/build/mozharness/rev/9dc88777c40b

Comment 30

[Rail Aliiev [:rail]]

Comment on attachment 687954 [details] [diff] [review]
bb-configs

http://hg.mozilla.org/build/buildbot-configs/rev/d5596fcbc6db

Comment 31

[Rail Aliiev [:rail]]

In production. I'll trigger builds in a bit.

Comment 32

[Rail Aliiev [:rail]]

Beta stable update.xml is here:
http://update.boot2gecko.org/stable/update_20121204093930.xml

Beta nightly has been published to http://update.boot2gecko.org/nightly/update_20121204093945.xml, which works fine, but http://update.boot2gecko.org/nightly/update.xml returns HTTP/500 for some reason. Any idea why it may happen?

Comment 33

[Rail Aliiev [:rail]]

[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98] mod_wsgi (pid=15196): Exception occurred processing WSGI script '/data/update-channels/nightly/update.wsgi'.
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98] Traceback (most recent call last):
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98]   File "/data/dogfood/dogdish/dispatcher.py", line 195, in __call__
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98]     res = handler()
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98]   File "/data/dogfood/dogdish/dispatcher.py", line 136, in __call__
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98]     application = current_update.application()
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98]   File "/data/dogfood/dogdish/dispatcher.py", line 70, in application
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98]     assert os.path.exists(application_ini)
[Tue Dec 04 15:24:18 2012] [error] [client 66.207.208.98] AssertionError

Comment 34

[Rail Aliiev [:rail]]

I think, once we remove WSGI* statements from /etc/httpd/conf.d/update.conf we're all set.

Comment 35

[Rail Aliiev [:rail]]

Attachment: add more logging

I think it'd be great to dump snippet/update.xml contents to simplify debugging.

Comment 36

[Rail Aliiev [:rail]]

Comment on attachment 688277 [details] [diff] [review]
add more logging

http://hg.mozilla.org/build/mozharness/rev/e343b7797c52

Comment 37

[Jonathan Griffin (:jgriffin)]

(In reply to Rail Aliiev [:rail] from comment #34)
> I think, once we remove WSGI* statements from /etc/httpd/conf.d/update.conf
> we're all set.

Note that if we stop using the update WSGI, lsblakk will lose dogfood id's in the mar requests.  lsblakk, is that ok?

Comment 38

[Rail Aliiev [:rail]]

Attachment: b2g updates 2

We talked with jgriffin and decided to use the existing dogfood app so we can track dogfood ids. This requires:

* upload only 3 files:
  update_${buildid}.xml
  application_${buildid}.ini 
  b2g_update_${buildid}.mar
* No need to autopublish update.xml (single pass rsync)

jgriffin: you'll need to update the app to use buildids instead of timestamps. I think, buildids make more sense since they are connected to real builds, so it'll be easier to find the corresponding logs, etc.

Comment 39

[Rail Aliiev [:rail]]

Comment on attachment 688332 [details] [diff] [review]
b2g updates 2

<aki> rail-buildduty: i think that looks good, but does this mean beta-stable will get pushed to update.b2g.org and get served as updates right away?

Yeah...

Any good idea how we should manage the stable channel without losing the dogfood ids?

Comment 40

[Aki Sasaki (not active)]

Push our beta-stable bits to a staging dir that we move into the real dir when it's signed off ?

Comment 41

[JP Rosevear [:jpr]]

Since we have a working system, not a blocker, but if this makes sense organizationally, please do.

Comment 42

[Rail Aliiev [:rail]]

Attachment: b2g updates 2

I *think* this one should work with the current webapp:

* upload application_${buildid}.ini alongside with other files
* don't autopublish update.xml to the nightly directory, the webapp should handle this
* autopublish all files to stable-stage directory, which is not handled by the webapp (for now?). This directory (channel!) will be used by QA

Once QA signs off the stable-stage channel the only thing required to publish a particular buildid would be copying mar and ini files to ../stable

Comment 43

[Rail Aliiev [:rail]]

Comment on attachment 688393 [details] [diff] [review]
b2g updates 2

http://hg.mozilla.org/build/mozharness/rev/6f2b9bf381f6

Comment 44

[Rail Aliiev [:rail]]

http://update.boot2gecko.org/stable-stage/update.xml and http://update.boot2gecko.org/nightly/update.xml look good. I'll check the real nightly build tomorrow morning before I close the bug.

Comment 45

[Rail Aliiev [:rail]]

Lukas,

right now the script uploads beta updates to http://update.boot2gecko.org/stable-stage/update.xml. Do you want me to upload those somewhere else? e.g. http://update.boot2gecko.org/beta-stage/update.xml or http://update.boot2gecko.org/betatest/update.xml

Comment 46

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Rail Aliiev [:rail] from comment #45)

> http://update.boot2gecko.org/betatest/update.xml

would be best - let's not use 'stable' anywhere anymore.



(In reply to Rail Aliiev [:rail] from comment #34)
> I think, once we remove WSGI* statements from /etc/httpd/conf.d/update.conf
> we're all set.

Note that if we stop using the update WSGI, lsblakk will lose dogfood id's in the mar requests.  lsblakk, is that ok?


--- This is really not great.  Getting this data is helpful for maintaining a high level of participation/data collection in testing.  

Rail - can anything be done to keep the dogfood id in the mar requests with the new automation?  If it's really impossible, then at least we have the info in the update pings, which is still very useful, but being able to watch the complete update process for a particular user gives us some pretty excellent visibility in any bugs in that area.

Comment 47

[JP Rosevear [:jpr]]

Err, as per #41, this was actually supposed to be a minus.  Please do if it makes sense, but not a blocker.

Comment 48

[Rail Aliiev [:rail]]

(In reply to Lukas Blakk [:lsblakk] from comment #46)
> (In reply to Rail Aliiev [:rail] from comment #45)
> 
> > http://update.boot2gecko.org/betatest/update.xml
> 
> would be best - let's not use 'stable' anywhere anymore.

OK, I'll prep a patch for that.

> 
> (In reply to Rail Aliiev [:rail] from comment #34)
> > I think, once we remove WSGI* statements from /etc/httpd/conf.d/update.conf
> > we're all set.
> 
> Note that if we stop using the update WSGI, lsblakk will lose dogfood id's
> in the mar requests.  lsblakk, is that ok?
> 
> 
> --- This is really not great.  Getting this data is helpful for maintaining
> a high level of participation/data collection in testing.  
> 
> Rail - can anything be done to keep the dogfood id in the mar requests with
> the new automation?  If it's really impossible, then at least we have the
> info in the update pings, which is still very useful, but being able to
> watch the complete update process for a particular user gives us some pretty
> excellent visibility in any bugs in that area.

The current solution uses the webapp for the nightly channel, and doesn't change anything for the beta channel. If we want to keep track of dogfood IDs for the betatest channel it would need some changes on update.b2g.o machine. Let me know if it's needed.

Comment 49

[Rail Aliiev [:rail]]

Attachment: s/stable/beta/g

Comment 50

[Rail Aliiev [:rail]]

Comment on attachment 688754 [details] [diff] [review]
s/stable/beta/g

http://hg.mozilla.org/build/mozharness/rev/412d94e7f6fe

Comment 51

[Rail Aliiev [:rail]]

Resolving this a fixed.

I tested generated updates by changing app.update.url in the dogfood-setup.sh to:

pref("app.update.url", "http://update.boot2gecko.org/betatest/update.xml?build_id=%BUILD_ID%&version=%VERSION%&dogfood_id=%DISTRIBUTION%");

and 

pref("app.update.url", "http://update.boot2gecko.org/nightly/update.xml?build_id=%BUILD_ID%&version=%VERSION%&dogfood_id=%DISTRIBUTION%");

Both worked fine.

The first one sets the update channel to "beta" and there is no tracking for dogfood IDs for MARs. Please file another bug to enable it. It would require 1 change in the releng configs and jgriffin to set up it on the server.

The second URL sets the channel to "nightly".

Comment 52

[Tony Chung [:tchung]]

working on https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/ now