832074 - expose buildid in B2G manifests

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Robert Helmer [:rhelmer]]

Socorro uses the old .txt files to get information about builds e.g.:
ftp://ftp.mozilla.org/pub/b2g/nightly/latest-mozilla-central/b2g-21.0a1.en-US.android-arm.txt

However these don't exist for device builds, I am told that they are replaced with manifest files such as:
ftp://ftp.mozilla.org/pub/b2g/manifests/2013-01-17/source_otoro_2013-01-17.xml

We are happy to support these, but would need the buildid to be present. Can these be added?

Comment 1

[Aki Sasaki (not active)]

Attachment: (mozharness) b2g buildid in manifest + 2 pep8 fixes

Comment 2

[Laura Thomson :laura]

When will this land?  We'll need it to do b2g crash reporting.

Comment 3

[Selena Deckelmann :selenamarie :selena]

In addition to BuildID we also need: 

version
build_type
beta_number (if it's a beta)

To be provided in this XML file. 

Here's the complete list of information we need for each build: 
   product_name, version, platform, build_id, build_type, beta_number, repository

My mapping for these currently are: 

product_name = 'b2g'
version = ?
platform = $device ('unagi', 'unagi-stable', 'otoro')
build_id = BuildID
build_type = ? 
beta_number = ? 
repository = $repo ('mozilla-beta', 'mozilla-releases')

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 703924 [details] [diff] [review]
(mozharness) b2g buildid in manifest + 2 pep8 fixes

Review of attachment 703924 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_build.py
@@ +525,5 @@
>  
>          for line in sources.splitlines():
>              new_sources.append(line)
>              if 'Gonk specific things' in line:
> +                new_sources.append('  <!-- BuildID: %s -->' % self.buildbot_properties["buildid"])

self.buildbot_properties["buildid"] doesn't always exist. in particular, if a force-build happens, then there's generally no buildid set at this point in the build.

we have a query_buildid() method, but it won't do anything until _after_ the build happens since it relies on parsing platform.ini. Even so, I think the canonical location for the buildid is platform.ini.

Comment 5

[Aki Sasaki (not active)]

Hm, should I move the manifest creation to after the build finishes?

Comment 6

[Chris AtLee [:catlee]]

That would probably work. I remember talking about this before with Nick, and we were worried that the build system consumed sources.xml somehow. If that's not the case, then we could definitely munge it afterwards.

Comment 7

[Aki Sasaki (not active)]

John, do you either a) know whether we use sources.xml in the b2g build itself (not repo, but the compile/package steps), or b) know who would know?

Comment 8

[Aki Sasaki (not active)]

(In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> In addition to BuildID we also need: 
> 
> version
> build_type
> beta_number (if it's a beta)
> 
> To be provided in this XML file. 
> 
> Here's the complete list of information we need for each build: 
>    product_name, version, platform, build_id, build_type, beta_number,
> repository
> 
> My mapping for these currently are: 
> 
> product_name = 'b2g'
> version = ?
> platform = $device ('unagi', 'unagi-stable', 'otoro')
> build_id = BuildID
> build_type = ? 
> beta_number = ? 
> repository = $repo ('mozilla-beta', 'mozilla-releases')

Ok.
This, along with the r- above, make this significantly larger than a one- or three- line patch.

What is build_type?

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to Aki Sasaki [:aki] from comment #8)
> (In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> > In addition to BuildID we also need: 
> > 
> > version
> > build_type
> > beta_number (if it's a beta)
> > 
> > To be provided in this XML file. 
> > 
> > Here's the complete list of information we need for each build: 
> >    product_name, version, platform, build_id, build_type, beta_number,
> > repository
> > 
> > My mapping for these currently are: 
> > 
> > product_name = 'b2g'
> > version = ?
> > platform = $device ('unagi', 'unagi-stable', 'otoro')
> > build_id = BuildID
> > build_type = ? 
> > beta_number = ? 
> > repository = $repo ('mozilla-beta', 'mozilla-releases')
> 
> Ok.
> This, along with the r- above, make this significantly larger than a one- or
> three- line patch.
> 
> What is build_type?

The channel the build ships with - Release, Beta, Aurora, Nightly

Comment 10

[Selena Deckelmann :selenamarie :selena]

Oops. I fail at using bugzilla. Adding John back and clearing my needsinfo as rhelmer answered the Q about build_type.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

We generate a sources.xml file as part of the build, but do not consume it outside of possibly use as an argument to config.sh.

Putting all of this information into the manifest xml either means we're corrupting the repo manifest XML according to the DTD, or we're encoding tons of information as comments, both less than ideal.  Instead, let's generate a second xml file which optionally has the contents of a repo manifest as a node.

Comment 12

[Aki Sasaki (not active)]

(In reply to John Ford [:jhford] from comment #11)
> We generate a sources.xml file as part of the build, but do not consume it
> outside of possibly use as an argument to config.sh.

Ok.

> Putting all of this information into the manifest xml either means we're
> corrupting the repo manifest XML according to the DTD, or we're encoding
> tons of information as comments, both less than ideal.

We're already putting tons of comments there, where tons is 2 + number of gaia locales + number of gecko locales.  I'm not sure a handful more is going to be a lot worse.

> Instead, let's
> generate a second xml file which optionally has the contents of a repo
> manifest as a node.

This would defeat the purpose of using sources.xml; we might as well use a .txt file for socorro at that point.  Which is definitely an option now with comment 3.

Comment 13

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #12)

> This would defeat the purpose of using sources.xml; we might as well use a
> .txt file for socorro at that point.  Which is definitely an option now with
> comment 3.

That's fine with us. JSON or YAML would be nice.

Some background: The other software Socorro accepts crashes for have embedded much of the information we need in the FTP file paths. The only special file that we access in for other software is for the build_id. This evolved over several years, of course.

Comment 14

[Aki Sasaki (not active)]

Ok.  I think another file means we need in-tree configs to change, which will push this fix out even further.

Current ETA: approximately EOW.

Comment 15

[Aki Sasaki (not active)]

Attachment: (mozharness) [needs testing] create socorro.json

This may actually work, without in-tree configs.

:catlee - does this look feasible?
:rhelmer - does this look like it would work for you? (see make_socorro_json() method)

Planning to test this tomorrow if this passes the eyeball test.

Comment 16

[Robert Helmer [:rhelmer]]

Comment on attachment 705246 [details] [diff] [review]
(mozharness) [needs testing] create socorro.json

lgtm, maybe should be named something other than socorro since anyone could consume it right? That's very much a nit though, selena is driving this so deferring to her

Comment 17

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #15)
> Created attachment 705246 [details] [diff] [review]
> (mozharness) [needs testing] create socorro.json
> 
> This may actually work, without in-tree configs.
> 
> :catlee - does this look feasible?
> :rhelmer - does this look like it would work for you? (see
> make_socorro_json() method)
> 
> Planning to test this tomorrow if this passes the eyeball test.

lgtm!  Agree with :rhelmer's nit that you may wish to name it something other than socorro.

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 705246 [details] [diff] [review]
(mozharness) [needs testing] create socorro.json

Review of attachment 705246 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_build.py
@@ +893,5 @@
> +        manifest_config = self.config.get('manifest', {})
> +        socorro_dict = {
> +            'buildid': self.query_buildid(),
> +            'version': self.query_version(),
> +            'build_type': manifest_config.get('update_channel'),

can this be update_channel instead? build_type is confusing (to me at least)

@@ +949,5 @@
>              os.path.join(upload_dir, xmlfilename)
>          )
> +        socorro_json = os.path.join(dirs['work_dir'], 'socorro.json')
> +        if os.path.exists(socorro_json):
> +            self.copy_to_upload_dir(socorro_json, os.path.join(upload_dir, xmlfilename))

s/xmlfilename/'socorro.json'/?

Comment 19

[Aki Sasaki (not active)]

(In reply to Chris AtLee [:catlee] from comment #18)
> Comment on attachment 705246 [details] [diff] [review]
> (mozharness) [needs testing] create socorro.json
> 
> Review of attachment 705246 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: scripts/b2g_build.py
> @@ +893,5 @@
> > +        manifest_config = self.config.get('manifest', {})
> > +        socorro_dict = {
> > +            'buildid': self.query_buildid(),
> > +            'version': self.query_version(),
> > +            'build_type': manifest_config.get('update_channel'),
> 
> can this be update_channel instead? build_type is confusing (to me at least)

I'd guess no, per comment 3.

> @@ +949,5 @@
> >              os.path.join(upload_dir, xmlfilename)
> >          )
> > +        socorro_json = os.path.join(dirs['work_dir'], 'socorro.json')
> > +        if os.path.exists(socorro_json):
> > +            self.copy_to_upload_dir(socorro_json, os.path.join(upload_dir, xmlfilename))
> 
> s/xmlfilename/'socorro.json'/?

Good catch.

Comment 20

[Robert Kaiser]

(In reply to Aki Sasaki [:aki] from comment #19)
> (In reply to Chris AtLee [:catlee] from comment #18)
> > can this be update_channel instead? build_type is confusing (to me at least)
> 
> I'd guess no, per comment 3.

AFAIK Selena/Rob can easily map whatever names you're using to those internal Socorro names. I personally think that build_type is bad naming that would be good not to spread outside of Socorro if possible. It's called "channel", "update channel" or "release channel" everywhere else (even crash reports have it annotated as "ReleaseChannel"), so it's IMHO best to be consistent with that where possible.

Comment 21

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #19)
> (In reply to Chris AtLee [:catlee] from comment #18)
> > Comment on attachment 705246 [details] [diff] [review]
> > (mozharness) [needs testing] create socorro.json
> > 
> > Review of attachment 705246 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: scripts/b2g_build.py
> > @@ +893,5 @@
> > > +        manifest_config = self.config.get('manifest', {})
> > > +        socorro_dict = {
> > > +            'buildid': self.query_buildid(),
> > > +            'version': self.query_version(),
> > > +            'build_type': manifest_config.get('update_channel'),
> > 
> > can this be update_channel instead? build_type is confusing (to me at least)
> 
> I'd guess no, per comment 3.

It can be update_channel!

I'm just going by what ended up in the other files. For us, it is a very simple matter to map it.

I'm very excited to be getting this in the form of a consistent file! Much nicer than scraping data from filesystem paths. :)  Thanks so much.

Comment 22

[Aki Sasaki (not active)]

Attachment: Create+upload socorro.json

{"buildid": "20130124140754", "update_channel": "nightly", "version": "18.0"}

Worked, failed on various upload bits in staging, but that's because our b2g mozharness staging configs are nonexistent atm.

Got to http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/manifests/socorro.json at least.

I have the nagging suspicion this will require organize.py changes :'( :'(

Comment 23

[Aki Sasaki (not active)]

Attachment: check organize.py into build/tools/stage/

Don't know if this is checked in anywhere.
I'm about to edit it, so let's check it in before editing.

Comment 24

[Aki Sasaki (not active)]

Woo!

http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/manifests/2013-01-24-16/
http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/manifests/2013/01/2013-01-24-16/
http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/manifests/latest/socorro_unagi_2013-01-24-16.json
{"buildid": "20130124163231", "update_channel": "nightly", "version": "18.0"}

Additional patches incoming.

Comment 25

[Aki Sasaki (not active)]

Attachment: allow for optional -HH in organize.py

Making this optional means we can replace the existing organize.py without breaking uploads for builds-in-progress.

Comment 26

[Aki Sasaki (not active)]

Attachment: (mozharness) create+upload dated socorro.json

Should land after organize.py is updated on stage.mozilla.org:~b2gbld/ .

Comment 27

[Aki Sasaki (not active)]

Re: naming nits,

a) I'm open to suggestions
b) it's formatted for, and provided specifically for Socorro so it seems like a decent name to me, but see (a) :)
c) should be easy to change

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 706045 [details] [diff] [review]
check organize.py into build/tools/stage/

http://hg.mozilla.org/build/tools/rev/7ef9f55c29c5

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 706055 [details] [diff] [review]
allow for optional -HH in organize.py

http://hg.mozilla.org/build/tools/rev/8f4c153c071d

Comment 30

[Aki Sasaki (not active)]

Updated organize.py on stage.mozilla.org via

wget http://hg.mozilla.org/build/tools/raw-file/8f4c153c071d/stage/organize.py
mv organize.py.1 organize.py

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 706056 [details] [diff] [review]
(mozharness) create+upload dated socorro.json

http://hg.mozilla.org/build/mozharness/rev/e879dd6ece6e

Comment 32

[Aki Sasaki (not active)]

Kicked off mozilla-b2g18 nightlies for testing.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Aki Sasaki [:aki] from comment #24)
> Woo!
> {"buildid": "20130124163231", "update_channel": "nightly", "version": "18.0"}

Will this say 'betatest' for the b2g_mozilla-b2g18_unagi_stable_nightly builds ? Those builds actually end up on the beta channel.

Comment 34

[Aki Sasaki (not active)]

(In reply to Nick Thomas [:nthomas] from comment #33)
> (In reply to Aki Sasaki [:aki] from comment #24)
> > Woo!
> > {"buildid": "20130124163231", "update_channel": "nightly", "version": "18.0"}
> 
> Will this say 'betatest' for the b2g_mozilla-b2g18_unagi_stable_nightly
> builds ? Those builds actually end up on the beta channel.

Nope,
http://hg.mozilla.org/build/mozharness/file/e879dd6ece6e/configs/b2g/releng-beta-stable.py#l57

Comment 35

[Aki Sasaki (not active)]

Yay!
http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/

Verified, http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/socorro_unagi-stable_2013-01-24-14.json says "beta".

Comment 36

[Robert Kaiser]

"beta" sounds like what we want as that's also what crash reports come in with. ;-)

Comment 37

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #35)
> Yay!
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/
> 
> Verified,
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/socorro_unagi-
> stable_2013-01-24-14.json says "beta".

So -- I'm only seeing .json created for unagi, not for unagi_stable or otoro -- is this correct?

Comment 38

[Aki Sasaki (not active)]

I see unagi and unagi_stable here:
http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/

Do you need it for otoro?  Those builds are pretty locked down (no dogfooding etc.), but it's not difficult to add the json for otoro if needed.

Comment 39

[Aki Sasaki (not active)]

Also, these files may need to change locations and/or get renamed if we start building these on multiple branches, or one branch will overwrite the other's.

Comment 40

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #39)
> Also, these files may need to change locations and/or get renamed if we
> start building these on multiple branches, or one branch will overwrite the
> other's.

Totally fine with us. I'll just be crawling through looking for JSON.

Comment 41

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #38)
> I see unagi and unagi_stable here:
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/

AH! I just misread. 
 
> Do you need it for otoro?  Those builds are pretty locked down (no
> dogfooding etc.), but it's not difficult to add the json for otoro if needed.

If we will accept crashes from them, we will need them... so I think yes, even if the volume of crashes expected is close to zero.

Comment 42

[Aki Sasaki (not active)]

Attachment: Create otoro socorro json

Comment 43

[Aki Sasaki (not active)]

Comment on attachment 706500 [details] [diff] [review]
Create otoro socorro json

http://hg.mozilla.org/build/mozharness/rev/8a07b04ce511

Should get otoro json's now.

Comment 44

[Aki Sasaki (not active)]

Kicked off b2g18 nightlies.

Comment 45

[Aki Sasaki (not active)]

Voila, otoro json: http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/2013-01-25-11/socorro_otoro_2013-01-25-11.json

These don't have an update channel aiui.

Comment 46

[Robert Kaiser]

I wouldn't say "no dogfooding" for otoro, for a number of people (like me) the otoro is the only device they have, so that's what they're testing with. AFAIK may also still be the reference device in terms of performance as it's possibly even closer to what will be shipped than the unagi is. It's just that for the larger dogfooding program, we couldn't get the amount of otoro devices we needed so we're using unagi instead.

And, BTW, otoro uses the "nightly" update channel.

Comment 47

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #45)
> Voila, otoro json:
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/2013-01-25-11/
> socorro_otoro_2013-01-25-11.json
> 
> These don't have an update channel aiui.

Can we get update_channel changed from 'null' to 'nightly'? 

We call this 'release_channel' in Socorro. This code is quite strict and would require significant refactoring to accommodate null or unknown release channels.

Comment 48

[Selena Deckelmann :selenamarie :selena]

Also noting that http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/ is not the same as stage. 

Is this intentional?

Comment 49

[Aki Sasaki (not active)]

(In reply to Selena Deckelmann :selenamarie :selena from comment #48)
> Also noting that http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/ is
> not the same as stage. 
> 
> Is this intentional?

Stage is essentially ftp.m.o, just different heads serving the same content.

If you're commenting on how the manifests all moved under 1.0.0/ and 1.0.1/, that's me from bug 834892.

Comment 50

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #49)
> (In reply to Selena Deckelmann :selenamarie :selena from comment #48)
> > Also noting that http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/ is
> > not the same as stage. 
> > 
> > Is this intentional?
> 
> Stage is essentially ftp.m.o, just different heads serving the same content.
> 
> If you're commenting on how the manifests all moved under 1.0.0/ and 1.0.1/,
> that's me from bug 834892.

Ah! Thanks for the explanation.

Is it possible to derive a beta number from any of this new information? Builds with a update_channel of 'beta' do not have a beta_number associated with them, and we also need this.

Comment 51

[Aki Sasaki (not active)]

(In reply to Selena Deckelmann :selenamarie :selena from comment #50)
> Is it possible to derive a beta number from any of this new information?
> Builds with a update_channel of 'beta' do not have a beta_number associated
> with them, and we also need this.

There is no concept of beta_number for these beta-channel builds.

beta_number corresponds with "go build a release-build type VERSION beta N", and we build that.

For b2g beta channel builds, they're built alongside every nightly, which happen at least 2x/day.  There is no "beta_number" tied into these builds or directories.  We promote the build that QA signs off on to the beta channel.

I could certainly insert the buildid into the beta_number, and you'd get an increasing number that could be used to identify the build, but that doesn't really correspond to our existing idea of beta_number.

Comment 52

[Robert Kaiser]

IMHO, in "rapid beta" mode we should not count on having a beta number - in "traditional beta" mode, it's good to count on that.

Comment 53

[Aki Sasaki (not active)]

(In reply to Robert Kaiser (:[e-mail]) from comment #52)
> IMHO, in "rapid beta" mode we should not count on having a beta number - in
> "traditional beta" mode, it's good to count on that.

Makes sense to me.
In this case, we would be in "rapid beta" mode.

So: I think I've got to get otoro.json to say "nightly", and we're done here?

Comment 54

[Aki Sasaki (not active)]

Attachment: force otoro socorro json to specify "nightly" channel

It's kind of ugly that we're doing this for a shared config file (non-otoro-specific), but not sure how else to do this.  I suppose we could use releng-beta.py for otoro b2g18* builds ?

Comment 55

[Robert Kaiser]

(In reply to Aki Sasaki [:aki] from comment #53)
> So: I think I've got to get otoro.json to say "nightly", and we're done here?

Yes, I think so.
(Also note that if we really would not have a channel set explicitely, the build would fall back to using "default" as the channel. I know from flashing builds onto an otoro myself and seeing its data that we actually have the otoro builds set the "nightly" channel, though.)

Comment 56

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki [:aki] from comment #51)
> (In reply to Selena Deckelmann :selenamarie :selena from comment #50)
> > Is it possible to derive a beta number from any of this new information?
> > Builds with a update_channel of 'beta' do not have a beta_number associated
> > with them, and we also need this.
> 
> There is no concept of beta_number for these beta-channel builds.
> 
> beta_number corresponds with "go build a release-build type VERSION beta N",
> and we build that.
> 
> For b2g beta channel builds, they're built alongside every nightly, which
> happen at least 2x/day.  There is no "beta_number" tied into these builds or
> directories.  We promote the build that QA signs off on to the beta channel.
> 
> I could certainly insert the buildid into the beta_number, and you'd get an
> increasing number that could be used to identify the build, but that doesn't
> really correspond to our existing idea of beta_number.

I see! I believe we're turning on rapid beta in socorro for this Real Soon Now, so as long as this isn't expected to be treated like "normal" betas, it's ok as it is.

Comment 57

[Aki Sasaki (not active)]

Comment on attachment 708317 [details] [diff] [review]
force otoro socorro json to specify "nightly" channel

http://hg.mozilla.org/build/mozharness/rev/2e81b866b155
http://hg.mozilla.org/build/mozharness/rev/7ff8e2bda5f6

Comment 58

[Aki Sasaki (not active)]

http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/1.0.0/latest/socorro_otoro.json says nightly.

Closing.