Closed Bug 832074 Opened 10 years ago Closed 9 years ago

expose buildid in B2G manifests

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: aki)

References

Details

Attachments

(5 files, 3 obsolete files)

Socorro uses the old .txt files to get information about builds e.g.:
ftp://ftp.mozilla.org/pub/b2g/nightly/latest-mozilla-central/b2g-21.0a1.en-US.android-arm.txt

However these don't exist for device builds, I am told that they are replaced with manifest files such as:
ftp://ftp.mozilla.org/pub/b2g/manifests/2013-01-17/source_otoro_2013-01-17.xml

We are happy to support these, but would need the buildid to be present. Can these be added?
Blocks: 821989
Blocks: 832083
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
Assignee: nobody → aki
Attachment #703924 - Flags: review?(catlee)
When will this land?  We'll need it to do b2g crash reporting.
In addition to BuildID we also need: 

version
build_type
beta_number (if it's a beta)

To be provided in this XML file. 

Here's the complete list of information we need for each build: 
   product_name, version, platform, build_id, build_type, beta_number, repository

My mapping for these currently are: 

product_name = 'b2g'
version = ?
platform = $device ('unagi', 'unagi-stable', 'otoro')
build_id = BuildID
build_type = ? 
beta_number = ? 
repository = $repo ('mozilla-beta', 'mozilla-releases')
Comment on attachment 703924 [details] [diff] [review]
(mozharness) b2g buildid in manifest + 2 pep8 fixes

Review of attachment 703924 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_build.py
@@ +525,5 @@
>  
>          for line in sources.splitlines():
>              new_sources.append(line)
>              if 'Gonk specific things' in line:
> +                new_sources.append('  <!-- BuildID: %s -->' % self.buildbot_properties["buildid"])

self.buildbot_properties["buildid"] doesn't always exist. in particular, if a force-build happens, then there's generally no buildid set at this point in the build.

we have a query_buildid() method, but it won't do anything until _after_ the build happens since it relies on parsing platform.ini. Even so, I think the canonical location for the buildid is platform.ini.
Attachment #703924 - Flags: review?(catlee) → review-
Hm, should I move the manifest creation to after the build finishes?
That would probably work. I remember talking about this before with Nick, and we were worried that the build system consumed sources.xml somehow. If that's not the case, then we could definitely munge it afterwards.
John, do you either a) know whether we use sources.xml in the b2g build itself (not repo, but the compile/package steps), or b) know who would know?
Flags: needinfo?(jhford)
(In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> In addition to BuildID we also need: 
> 
> version
> build_type
> beta_number (if it's a beta)
> 
> To be provided in this XML file. 
> 
> Here's the complete list of information we need for each build: 
>    product_name, version, platform, build_id, build_type, beta_number,
> repository
> 
> My mapping for these currently are: 
> 
> product_name = 'b2g'
> version = ?
> platform = $device ('unagi', 'unagi-stable', 'otoro')
> build_id = BuildID
> build_type = ? 
> beta_number = ? 
> repository = $repo ('mozilla-beta', 'mozilla-releases')

Ok.
This, along with the r- above, make this significantly larger than a one- or three- line patch.

What is build_type?
Flags: needinfo?(sdeckelmann)
(In reply to Aki Sasaki [:aki] from comment #8)
> (In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> > In addition to BuildID we also need: 
> > 
> > version
> > build_type
> > beta_number (if it's a beta)
> > 
> > To be provided in this XML file. 
> > 
> > Here's the complete list of information we need for each build: 
> >    product_name, version, platform, build_id, build_type, beta_number,
> > repository
> > 
> > My mapping for these currently are: 
> > 
> > product_name = 'b2g'
> > version = ?
> > platform = $device ('unagi', 'unagi-stable', 'otoro')
> > build_id = BuildID
> > build_type = ? 
> > beta_number = ? 
> > repository = $repo ('mozilla-beta', 'mozilla-releases')
> 
> Ok.
> This, along with the r- above, make this significantly larger than a one- or
> three- line patch.
> 
> What is build_type?

The channel the build ships with - Release, Beta, Aurora, Nightly
Flags: needinfo?(jhford)
Oops. I fail at using bugzilla. Adding John back and clearing my needsinfo as rhelmer answered the Q about build_type.
Flags: needinfo?(sdeckelmann) → needinfo?(jhford)
We generate a sources.xml file as part of the build, but do not consume it outside of possibly use as an argument to config.sh.

Putting all of this information into the manifest xml either means we're corrupting the repo manifest XML according to the DTD, or we're encoding tons of information as comments, both less than ideal.  Instead, let's generate a second xml file which optionally has the contents of a repo manifest as a node.
Flags: needinfo?(jhford)
(In reply to John Ford [:jhford] from comment #11)
> We generate a sources.xml file as part of the build, but do not consume it
> outside of possibly use as an argument to config.sh.

Ok.

> Putting all of this information into the manifest xml either means we're
> corrupting the repo manifest XML according to the DTD, or we're encoding
> tons of information as comments, both less than ideal.

We're already putting tons of comments there, where tons is 2 + number of gaia locales + number of gecko locales.  I'm not sure a handful more is going to be a lot worse.

> Instead, let's
> generate a second xml file which optionally has the contents of a repo
> manifest as a node.

This would defeat the purpose of using sources.xml; we might as well use a .txt file for socorro at that point.  Which is definitely an option now with comment 3.
(In reply to Aki Sasaki [:aki] from comment #12)

> This would defeat the purpose of using sources.xml; we might as well use a
> .txt file for socorro at that point.  Which is definitely an option now with
> comment 3.

That's fine with us. JSON or YAML would be nice.

Some background: The other software Socorro accepts crashes for have embedded much of the information we need in the FTP file paths. The only special file that we access in for other software is for the build_id. This evolved over several years, of course.
Ok.  I think another file means we need in-tree configs to change, which will push this fix out even further.

Current ETA: approximately EOW.
This may actually work, without in-tree configs.

:catlee - does this look feasible?
:rhelmer - does this look like it would work for you? (see make_socorro_json() method)

Planning to test this tomorrow if this passes the eyeball test.
Attachment #703924 - Attachment is obsolete: true
Attachment #705246 - Flags: feedback?(rhelmer)
Attachment #705246 - Flags: feedback?(catlee)
Comment on attachment 705246 [details] [diff] [review]
(mozharness) [needs testing] create socorro.json

lgtm, maybe should be named something other than socorro since anyone could consume it right? That's very much a nit though, selena is driving this so deferring to her
Attachment #705246 - Flags: feedback?(sdeckelmann)
Attachment #705246 - Flags: feedback?(rhelmer)
Attachment #705246 - Flags: feedback+
(In reply to Aki Sasaki [:aki] from comment #15)
> Created attachment 705246 [details] [diff] [review]
> (mozharness) [needs testing] create socorro.json
> 
> This may actually work, without in-tree configs.
> 
> :catlee - does this look feasible?
> :rhelmer - does this look like it would work for you? (see
> make_socorro_json() method)
> 
> Planning to test this tomorrow if this passes the eyeball test.

lgtm!  Agree with :rhelmer's nit that you may wish to name it something other than socorro.
Attachment #705246 - Flags: feedback?(sdeckelmann) → feedback+
Comment on attachment 705246 [details] [diff] [review]
(mozharness) [needs testing] create socorro.json

Review of attachment 705246 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_build.py
@@ +893,5 @@
> +        manifest_config = self.config.get('manifest', {})
> +        socorro_dict = {
> +            'buildid': self.query_buildid(),
> +            'version': self.query_version(),
> +            'build_type': manifest_config.get('update_channel'),

can this be update_channel instead? build_type is confusing (to me at least)

@@ +949,5 @@
>              os.path.join(upload_dir, xmlfilename)
>          )
> +        socorro_json = os.path.join(dirs['work_dir'], 'socorro.json')
> +        if os.path.exists(socorro_json):
> +            self.copy_to_upload_dir(socorro_json, os.path.join(upload_dir, xmlfilename))

s/xmlfilename/'socorro.json'/?
Attachment #705246 - Flags: feedback?(catlee) → feedback+
(In reply to Chris AtLee [:catlee] from comment #18)
> Comment on attachment 705246 [details] [diff] [review]
> (mozharness) [needs testing] create socorro.json
> 
> Review of attachment 705246 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: scripts/b2g_build.py
> @@ +893,5 @@
> > +        manifest_config = self.config.get('manifest', {})
> > +        socorro_dict = {
> > +            'buildid': self.query_buildid(),
> > +            'version': self.query_version(),
> > +            'build_type': manifest_config.get('update_channel'),
> 
> can this be update_channel instead? build_type is confusing (to me at least)

I'd guess no, per comment 3.

> @@ +949,5 @@
> >              os.path.join(upload_dir, xmlfilename)
> >          )
> > +        socorro_json = os.path.join(dirs['work_dir'], 'socorro.json')
> > +        if os.path.exists(socorro_json):
> > +            self.copy_to_upload_dir(socorro_json, os.path.join(upload_dir, xmlfilename))
> 
> s/xmlfilename/'socorro.json'/?

Good catch.
(In reply to Aki Sasaki [:aki] from comment #19)
> (In reply to Chris AtLee [:catlee] from comment #18)
> > can this be update_channel instead? build_type is confusing (to me at least)
> 
> I'd guess no, per comment 3.

AFAIK Selena/Rob can easily map whatever names you're using to those internal Socorro names. I personally think that build_type is bad naming that would be good not to spread outside of Socorro if possible. It's called "channel", "update channel" or "release channel" everywhere else (even crash reports have it annotated as "ReleaseChannel"), so it's IMHO best to be consistent with that where possible.
(In reply to Aki Sasaki [:aki] from comment #19)
> (In reply to Chris AtLee [:catlee] from comment #18)
> > Comment on attachment 705246 [details] [diff] [review]
> > (mozharness) [needs testing] create socorro.json
> > 
> > Review of attachment 705246 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: scripts/b2g_build.py
> > @@ +893,5 @@
> > > +        manifest_config = self.config.get('manifest', {})
> > > +        socorro_dict = {
> > > +            'buildid': self.query_buildid(),
> > > +            'version': self.query_version(),
> > > +            'build_type': manifest_config.get('update_channel'),
> > 
> > can this be update_channel instead? build_type is confusing (to me at least)
> 
> I'd guess no, per comment 3.

It can be update_channel!

I'm just going by what ended up in the other files. For us, it is a very simple matter to map it.

I'm very excited to be getting this in the form of a consistent file! Much nicer than scraping data from filesystem paths. :)  Thanks so much.
Attached patch Create+upload socorro.json (obsolete) — Splinter Review
{"buildid": "20130124140754", "update_channel": "nightly", "version": "18.0"}

Worked, failed on various upload bits in staging, but that's because our b2g mozharness staging configs are nonexistent atm.

Got to http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/manifests/socorro.json at least.

I have the nagging suspicion this will require organize.py changes :'( :'(
Don't know if this is checked in anywhere.
I'm about to edit it, so let's check it in before editing.
Attachment #706045 - Flags: review?(catlee)
Making this optional means we can replace the existing organize.py without breaking uploads for builds-in-progress.
Attachment #706055 - Flags: review?(catlee)
Should land after organize.py is updated on stage.mozilla.org:~b2gbld/ .
Attachment #705246 - Attachment is obsolete: true
Attachment #705990 - Attachment is obsolete: true
Attachment #706056 - Flags: review?(catlee)
Re: naming nits,

a) I'm open to suggestions
b) it's formatted for, and provided specifically for Socorro so it seems like a decent name to me, but see (a) :)
c) should be easy to change
Attachment #706045 - Flags: review?(catlee) → review+
Attachment #706055 - Flags: review?(catlee) → review+
Comment on attachment 706045 [details] [diff] [review]
check organize.py into build/tools/stage/

http://hg.mozilla.org/build/tools/rev/7ef9f55c29c5
Attachment #706045 - Flags: checked-in+
Comment on attachment 706055 [details] [diff] [review]
allow for optional -HH in organize.py

http://hg.mozilla.org/build/tools/rev/8f4c153c071d
Attachment #706055 - Flags: checked-in+
Attachment #706056 - Flags: review?(catlee) → review+
Updated organize.py on stage.mozilla.org via

wget http://hg.mozilla.org/build/tools/raw-file/8f4c153c071d/stage/organize.py
mv organize.py.1 organize.py
Comment on attachment 706056 [details] [diff] [review]
(mozharness) create+upload dated socorro.json

http://hg.mozilla.org/build/mozharness/rev/e879dd6ece6e
Attachment #706056 - Flags: checked-in+
Kicked off mozilla-b2g18 nightlies for testing.
(In reply to Aki Sasaki [:aki] from comment #24)
> Woo!
> {"buildid": "20130124163231", "update_channel": "nightly", "version": "18.0"}

Will this say 'betatest' for the b2g_mozilla-b2g18_unagi_stable_nightly builds ? Those builds actually end up on the beta channel.
(In reply to Nick Thomas [:nthomas] from comment #33)
> (In reply to Aki Sasaki [:aki] from comment #24)
> > Woo!
> > {"buildid": "20130124163231", "update_channel": "nightly", "version": "18.0"}
> 
> Will this say 'betatest' for the b2g_mozilla-b2g18_unagi_stable_nightly
> builds ? Those builds actually end up on the beta channel.

Nope,
http://hg.mozilla.org/build/mozharness/file/e879dd6ece6e/configs/b2g/releng-beta-stable.py#l57
"beta" sounds like what we want as that's also what crash reports come in with. ;-)
(In reply to Aki Sasaki [:aki] from comment #35)
> Yay!
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/
> 
> Verified,
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/socorro_unagi-
> stable_2013-01-24-14.json says "beta".

So -- I'm only seeing .json created for unagi, not for unagi_stable or otoro -- is this correct?
I see unagi and unagi_stable here:
http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/

Do you need it for otoro?  Those builds are pretty locked down (no dogfooding etc.), but it's not difficult to add the json for otoro if needed.
Also, these files may need to change locations and/or get renamed if we start building these on multiple branches, or one branch will overwrite the other's.
(In reply to Aki Sasaki [:aki] from comment #39)
> Also, these files may need to change locations and/or get renamed if we
> start building these on multiple branches, or one branch will overwrite the
> other's.

Totally fine with us. I'll just be crawling through looking for JSON.
(In reply to Aki Sasaki [:aki] from comment #38)
> I see unagi and unagi_stable here:
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/

AH! I just misread. 
 
> Do you need it for otoro?  Those builds are pretty locked down (no
> dogfooding etc.), but it's not difficult to add the json for otoro if needed.

If we will accept crashes from them, we will need them... so I think yes, even if the volume of crashes expected is close to zero.
Attachment #706500 - Flags: review?(catlee)
Attachment #706500 - Flags: review?(catlee) → review+
Comment on attachment 706500 [details] [diff] [review]
Create otoro socorro json

http://hg.mozilla.org/build/mozharness/rev/8a07b04ce511

Should get otoro json's now.
Attachment #706500 - Flags: checked-in+
Kicked off b2g18 nightlies.
I wouldn't say "no dogfooding" for otoro, for a number of people (like me) the otoro is the only device they have, so that's what they're testing with. AFAIK may also still be the reference device in terms of performance as it's possibly even closer to what will be shipped than the unagi is. It's just that for the larger dogfooding program, we couldn't get the amount of otoro devices we needed so we're using unagi instead.

And, BTW, otoro uses the "nightly" update channel.
(In reply to Aki Sasaki [:aki] from comment #45)
> Voila, otoro json:
> http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/2013-01-25-11/
> socorro_otoro_2013-01-25-11.json
> 
> These don't have an update channel aiui.

Can we get update_channel changed from 'null' to 'nightly'? 

We call this 'release_channel' in Socorro. This code is quite strict and would require significant refactoring to accommodate null or unknown release channels.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Also noting that http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/ is not the same as stage. 

Is this intentional?
(In reply to Selena Deckelmann :selenamarie :selena from comment #48)
> Also noting that http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/ is
> not the same as stage. 
> 
> Is this intentional?

Stage is essentially ftp.m.o, just different heads serving the same content.

If you're commenting on how the manifests all moved under 1.0.0/ and 1.0.1/, that's me from bug 834892.
(In reply to Aki Sasaki [:aki] from comment #49)
> (In reply to Selena Deckelmann :selenamarie :selena from comment #48)
> > Also noting that http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/ is
> > not the same as stage. 
> > 
> > Is this intentional?
> 
> Stage is essentially ftp.m.o, just different heads serving the same content.
> 
> If you're commenting on how the manifests all moved under 1.0.0/ and 1.0.1/,
> that's me from bug 834892.

Ah! Thanks for the explanation.

Is it possible to derive a beta number from any of this new information? Builds with a update_channel of 'beta' do not have a beta_number associated with them, and we also need this.
(In reply to Selena Deckelmann :selenamarie :selena from comment #50)
> Is it possible to derive a beta number from any of this new information?
> Builds with a update_channel of 'beta' do not have a beta_number associated
> with them, and we also need this.

There is no concept of beta_number for these beta-channel builds.

beta_number corresponds with "go build a release-build type VERSION beta N", and we build that.

For b2g beta channel builds, they're built alongside every nightly, which happen at least 2x/day.  There is no "beta_number" tied into these builds or directories.  We promote the build that QA signs off on to the beta channel.

I could certainly insert the buildid into the beta_number, and you'd get an increasing number that could be used to identify the build, but that doesn't really correspond to our existing idea of beta_number.
IMHO, in "rapid beta" mode we should not count on having a beta number - in "traditional beta" mode, it's good to count on that.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #52)
> IMHO, in "rapid beta" mode we should not count on having a beta number - in
> "traditional beta" mode, it's good to count on that.

Makes sense to me.
In this case, we would be in "rapid beta" mode.

So: I think I've got to get otoro.json to say "nightly", and we're done here?
It's kind of ugly that we're doing this for a shared config file (non-otoro-specific), but not sure how else to do this.  I suppose we could use releng-beta.py for otoro b2g18* builds ?
Attachment #708317 - Flags: review?(catlee)
(In reply to Aki Sasaki [:aki] from comment #53)
> So: I think I've got to get otoro.json to say "nightly", and we're done here?

Yes, I think so.
(Also note that if we really would not have a channel set explicitely, the build would fall back to using "default" as the channel. I know from flashing builds onto an otoro myself and seeing its data that we actually have the otoro builds set the "nightly" channel, though.)
(In reply to Aki Sasaki [:aki] from comment #51)
> (In reply to Selena Deckelmann :selenamarie :selena from comment #50)
> > Is it possible to derive a beta number from any of this new information?
> > Builds with a update_channel of 'beta' do not have a beta_number associated
> > with them, and we also need this.
> 
> There is no concept of beta_number for these beta-channel builds.
> 
> beta_number corresponds with "go build a release-build type VERSION beta N",
> and we build that.
> 
> For b2g beta channel builds, they're built alongside every nightly, which
> happen at least 2x/day.  There is no "beta_number" tied into these builds or
> directories.  We promote the build that QA signs off on to the beta channel.
> 
> I could certainly insert the buildid into the beta_number, and you'd get an
> increasing number that could be used to identify the build, but that doesn't
> really correspond to our existing idea of beta_number.

I see! I believe we're turning on rapid beta in socorro for this Real Soon Now, so as long as this isn't expected to be treated like "normal" betas, it's ok as it is.
Attachment #708317 - Flags: review?(catlee) → review+
http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/1.0.0/latest/socorro_otoro.json says nightly.

Closing.
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.