890093 - Upload sources.xml for device builds to ftp.mozilla.org

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We'd like to enable sheriffs to use a tool to diff the commits between two builds' sources.xml (bug 890086), but in order to do this, it would work best if sources.xml were uploaded to the same dir as the build log (ftp.mozilla.org) to make locating those from TBPL easy.

Currently, sources.xml is only uploaded to the pvtbuilds location, but figuring out the correct pvtbuilds location from TBPL is a pain.

Could we get sources.xml uploaded to ftp.mozilla.org?

Comment 1

[Andrew Halberstadt [:ahal]]

I think it would be super cool if we wrote this as an hg extension such that 'hg diff -r <rev>' would look for and download the sources.xml of <rev> and compare it to the sources.xml of <tip> (only if a special flag is passed or something) and format the output in a familiar way.

Actually, come to think of it we should just write the script as a module first and then it would be simple for an hg extension to consume it later on.

Comment 2

[Andrew Halberstadt [:ahal]]

Oops, meant to put that comment on bug 890086. Sorry for spam.

Comment 3

[cmtalbert]

Do we have any ETA on when we can expect this? We're starting work on the tool that will use these uploaded sources.xml files now (over in bug 890086).

Comment 4

[Aki Sasaki (not active)]

We already have sources.xml uploaded to ftp.m.o: http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests as well as pvtbuilds.

I'm not entirely sure atm how mozharness (on the client) is going to figure out where buildbot (on the master) is going to upload the logs at the end of the job.

Does the existing b2g manifest location work for you?  That would make the above problem moot.

Comment 5

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #0)
> We'd like to enable sheriffs to use a tool to diff the commits between two
> builds' sources.xml (bug 890086), but in order to do this, it would work
> best if sources.xml were uploaded to the same dir as the build log
> (ftp.mozilla.org) to make locating those from TBPL easy.
> 
> Currently, sources.xml is only uploaded to the pvtbuilds location, but
> figuring out the correct pvtbuilds location from TBPL is a pain.
> 
> Could we get sources.xml uploaded to ftp.mozilla.org?

Also, would it help the sheriffs to have a TinderboxPrint'ed link to the pvtbuilds location?  Would that resolve this bug?

Comment 6

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #4)
> We already have sources.xml uploaded to ftp.m.o:
> http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests as well as pvtbuilds.
> 
> I'm not entirely sure atm how mozharness (on the client) is going to figure
> out where buildbot (on the master) is going to upload the logs at the end of
> the job.
> 
> Does the existing b2g manifest location work for you?  That would make the
> above problem moot.

It looks like we're only uploading nightly manifests to that location; and we need per-commit manifests for sheriffing.

Comment 7

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #5)
> (In reply to Jonathan Griffin (:jgriffin) from comment #0)
> > We'd like to enable sheriffs to use a tool to diff the commits between two
> > builds' sources.xml (bug 890086), but in order to do this, it would work
> > best if sources.xml were uploaded to the same dir as the build log
> > (ftp.mozilla.org) to make locating those from TBPL easy.
> > 
> > Currently, sources.xml is only uploaded to the pvtbuilds location, but
> > figuring out the correct pvtbuilds location from TBPL is a pain.
> > 
> > Could we get sources.xml uploaded to ftp.mozilla.org?
> 
> Also, would it help the sheriffs to have a TinderboxPrint'ed link to the
> pvtbuilds location?  Would that resolve this bug?

Yes, I think that would probably be sufficient.

Comment 8

[Aki Sasaki (not active)]

Hm, we disable uploads on depend device builds across the board.
Maybe a combination of turning the ftp.m.o manifest upload to be for both depend+nightly, and tinderboxprinting the link.

Comment 9

[Jonathan Griffin (:jgriffin)]

That would be perfect, and would make it easier to handle these files in TBPLv2 in the future.

Comment 10

[Aki Sasaki (not active)]

Attachment: (mozharness) [needs testing] upload dep manifests with tinderboxprint

Comment 11

[Aki Sasaki (not active)]

Woot: http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/mozilla-central/

Attaching a new patch with the latest fixes.

Comment 12

[Aki Sasaki (not active)]

Attachment: manifest upload

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 775742 [details] [diff] [review]
manifest upload

Review of attachment 775742 [details] [diff] [review]:
-----------------------------------------------------------------

The code seems fine but I'm wondering about the config and the end result on ftp.m.o. Based on your tests we'll have 
 .../b2g/manifests/<version>/YYYY-MM-DD-HH/source_<platform>_YYYY-MM-DD-HH.xml 
for nightly builds (plus symlinks without the timestamp), and then
 .../b2g/manifests/<branch>/<platform>/sources-YYYYMMDDHHmmss.xml
That is likely to confusing if you don't know what's going on. It would be helpful to have .../b2g/manifests/{nightly,tinderbox-builds} at least, assuming we can get the consumers of the nightly manifests to adjust. If you don't like tinderbox-builds (historical cruft!) then using 'depend' or 'on-checkin' or similar is fine by me.

Are we OK if rebuilds overwrite the xml file ? Can the sources change while the rebuild means the buildID is preserved ? If so we need something to make the filename unique.

We'd also need some sort of cleanup job dep builds will be needed in cron.

::: scripts/b2g_build.py
@@ +1374,1 @@
>              self.info("Upload successful")

The various if/else/elif here are bit hard to read. Perhaps it would be clearer if there was one if and self.info("Upload successful").

Comment 14

[Aki Sasaki (not active)]

Attachment: manifest upload, take 2

Testing in staging.

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 777597 [details] [diff] [review]
manifest upload, take 2

http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/nightly/
http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/depend/

Comment 16

[Aki Sasaki (not active)]

Boo, partners scrape b2g/manifests.

I can:

a) softlink b2g/manifests/nightly/* into b2g/manifests
b) announce b2g/manifests/* is moving into b2g/manifests/nightly/
c) not move the nightly bits, but create the depend directory

Comment 17

[Aki Sasaki (not active)]

mvines: do you have a preference among the above choices?  (comment 16)

Comment 18

[Michael Vines [:m1] [:evilmachines]]

Please just do whatever is cleanest from your perspective, the changes to adapt over here are pretty minor. Just give us a couple hours notice before this goes live.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 777597 [details] [diff] [review]
manifest upload, take 2

lgtm

Comment 20

[Aki Sasaki (not active)]

:laura, :rhelmer -- Are you ok with this b2g/manifests path change as well?

I'm about to post to dev.b2g; I'd make this path change next week, most likely, assuming everyone's ok with it.

Comment 21

[Aki Sasaki (not active)]

(Asking because of bug 832074)

Comment 22

[Robert Helmer [:rhelmer]]

Selena, will this break anything in socorro's ftpscraper (bug bug 832074)?

Comment 23

[Aki Sasaki (not active)]

Comment on attachment 777597 [details] [diff] [review]
manifest upload, take 2

Landed: https://hg.mozilla.org/build/mozharness/rev/d555b3fd5d3c

When we merge this to production, we'll start using the new paths, and downstream users will be affected after the next nightly.

Comment 24

[Aki Sasaki (not active)]

Merged to production.
I'm going to move the old nightly manifests into b2g/manifests/nightly/ manually.

Comment 25

[Aki Sasaki (not active)]

Files are moved.
I'm going to verify, and create a cron job to clean up.

Comment 26

[Aki Sasaki (not active)]

Attachment: b2g_manifest_cron

manifest depend cleanup crontab

from productdelivery svn.
Not sure how to get this live?

Comment 27

[Diego Wilson [:diego]]

I updated our partner builds to point to the new location. Thanks for the heads up!

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 779973 [details] [diff] [review]
b2g_manifest_cron

lgtm. If you land this then puppet will take care of deploying it.

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 779973 [details] [diff] [review]
b2g_manifest_cron

(bb08)deathduck:/src/clean/svn/productdelivery/files/cron [14:44:35]
1479$ svn ci
Adding         b2gbld
Transmitting file data .
Committed revision 71807.

Comment 30

[Aki Sasaki (not active)]

I see the TinderboxPrints, and I see depend sources.xml being uploaded into http://ftp.mozilla.org/pub/mozilla.org/b2g/manifests/depend/.

Resolving fixed.