834892 - uploaded manifests in latest directory should have static filename

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

The manifests in http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/latest/

currently have filenames like source_unagi-stable_2013-01-25-11.xml

they should have static filenames instead/also so it's easier for automation to consume them.

Comment 1

[Aki Sasaki (not active)]

organize.py could strip out the date and softlink ?

I was thinking we should have branch names in the manifests/ directory tree so we can upload more than one set.

Comment 2

[Michael Vines [:m1] [:evilmachines]]

Yes, branch names would definitely be desired.  

It also might be nice to have a source manifest for each branch that  is not device specific.  For example not everybody uses unagi/otoro for v1.0.0 development.  Those folks really just need the source manifest for the bits in g.m.o that are common between all the devices.

Eg:
 http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/v1.0.0/source_generic.xml
 http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/v1.0.0/source_generic-stable.xml
 http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/v1.0.1/source_generic.xml
 http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/v1.0.1/source_generic-stable.xml
 http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/v1.1.0/source_generic.xml
 http://stage.mozilla.org/pub/mozilla.org/b2g/manifests/v1.1.0/source_generic-stable.xml

Although now that we've stripped out the actual build ID from the filename, we would probably want that information either in the file name or derive somehow else.   The build "GUID" will become important when looking up test results for a given build, and matching with downstream partner builds.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

from irc, the work in this bug is also needed to support posting manifests from multiple/different branches to ftp.m.o.

Comment 4

[Aki Sasaki (not active)]

Attachment: (tools) [needs testing] short_symlink support in organize.py

Comment 5

[Aki Sasaki (not active)]

(In reply to Michael Vines [:m1] [:evilmachines] from comment #2)
> It also might be nice to have a source manifest for each branch that  is not
> device specific.

This is problematic in that:

* we should, in general, get the same gecko revision for all nightlies on a certain branch.  The exceptions would be if a specific nightly is kicked off by hand without others, or without a revision specified.

* however, gaia and l10n revisions are then pulled during the build according to branch, not specified revision.  If one build happens later than the others, and a checkin happens in the meantime, then the revisions there will differ.

We can call one of the nightly device types as the authoritative one per branch as a way around this if wanted.

Comment 6

[Aki Sasaki (not active)]

Attachment: (tools) short_symlink support in organize.py

Minor cleanup.

Comment 7

[Aki Sasaki (not active)]

Attachment: (mozharness) upload manifests to a versioned directory

Currently this specifies mozilla-b2g18: 1.0.1, mozilla-b2g18_v1_0_0: 1.0.0

I can put the 'v' in there pretty easily.
Not sure if I should have made mozilla-b2g18 1.0.x, but that can be modified pretty easily too.

This should be sufficient for testing.

Comment 8

[Aki Sasaki (not active)]

That works:

http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/manifests/1.0.0/latest/

Index of /pub/mozilla.org/b2g/manifests/1.0.0/latest
[ICO]	Name	Last modified	Size	Description
[DIR]	Parent Directory	 	-
[ ]	socorro_unagi.json	29-Jan-2013 14:54 	77
[ ]	socorro_unagi_2013-01-29-17.json	29-Jan-2013 14:54 	77
[TXT]	source_unagi.xml	29-Jan-2013 14:54 	20K
[TXT]	source_unagi_2013-01-29-17.xml	29-Jan-2013 14:54 	20K


:evilmachines - does this work for you?

We upload a

b2g/manifests/1.0.0/YEAR/DATE/

with

source_DEVICE_DATE.xml
source_DEVICE.xml softlink to dated xml
socorro_DEVICE_DATE.json
socorro_DEVICE.json softlink to dated json

and softlink the entire directory to 1.0.0/latest ?

Bikeshedding on the version (1.0.0 vs v1.0.0 vs. mozilla-b2g18_v1_0_0) should happen now if we want to avoid changing it again in the future, but again it's not hard to change.

Also, not sure if mozilla-b2g18 should be 1.0.1, 1.0.x, v1-train, 1.1, or what.

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 707751 [details] [diff] [review]
(tools) short_symlink support in organize.py

dev-stage01 already has this.

http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/manifests/1.0.0/latest/

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 707756 [details] [diff] [review]
(mozharness) upload manifests to a versioned directory

With the knowledge that the branches manifest "version" naming will probably change.

If we have to do even more stuff in here we might make these nested dicts, e.g.

"branches": {
    "mozilla-b2g18": {
        "upload_version": "v1-train",
        "generic_platform": "otoro",
    },
},

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 707756 [details] [diff] [review]
(mozharness) upload manifests to a versioned directory

Review of attachment 707756 [details] [diff] [review]:
-----------------------------------------------------------------

r+ once we figure out what the correct version #'s should be.

::: configs/b2g/releng-beta-stable.py
@@ +52,5 @@
>          "ssh_user": "b2gbld",
>          "target_suffix": "-stable",
> +        "branches": {
> +            'mozilla-b2g18_v1_0_0': '1.0.0',
> +            'mozilla-b2g18': '1.0.1',

I think mozilla-b2g18_v1_0_1 is 1.0.1.

Not sure what mozilla-b2g18 should be called...It's for leo now I think?

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 707751 [details] [diff] [review]
(tools) short_symlink support in organize.py

http://hg.mozilla.org/build/tools/rev/e20897708cd7

Comment 14

[Aki Sasaki (not active)]

Updated organize.py on stage.m.o.  We should now be getting short [non-dated] symlinks in the current directory structure.

Once we get a decision on naming I can land the mozharness portion, and the manifests will have a 1.0.0 or 1.0.1 (or v1.0.0 or v1.0.1 or ...) in the directory tree.  And bug 835556 should be fixed.

Comment 15

[Aki Sasaki (not active)]

[11:17]	<joduinn-mtg>	aki yes, b2g18 being 1.0.1 until 15th feb.

Comment 16

[Aki Sasaki (not active)]

Comment on attachment 707756 [details] [diff] [review]
(mozharness) upload manifests to a versioned directory

Since mozilla-b2g18 is, in fact, 1.0.1, I'm going to land as-is.

default: http://hg.mozilla.org/build/mozharness/rev/abc7e3482d7f
production: http://hg.mozilla.org/build/mozharness/rev/ce8901b71fed

Kicking off b2g18 and b2g18_v1_0_0 nightlies.

Comment 17

[Michael Vines [:m1] [:evilmachines]]

New manifests on f.m.o look good.  

However the hole in the moz manifests is now very apparent -- the SHA1 used for the http://git.mozilla.org/?p=b2g/B2G.git project is not recorded.  This is an artifact of how this project in the Mozilla builds is magical and exists outside of repo.   

Moz builds are bootstrapped by a |git clone|, while CAF builds are bootstrapped with |repo init| precisely to avoid this issue.

Fixing this properly is surely out of scope for v1.0.0 but maybe v1.0.1 and is a separate bug.

For now I'll just set up our builds to float at the top of b2g/B2G.git and cross my fingers.  As short-term damage control might just want to output the SHA1 in a .txt file on f.m.o or something.

Comment 18

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

The b2g repo isn't required for bootstrapping anymore.

Where do you put the checked-out b2g repo?  We can just add it to our manifests and put it at that location, and start the (post-v1) process of deprecating the old setup.

Comment 19

[Michael Vines [:m1] [:evilmachines]]

device/qcom/b2g_common/mozilla-b2g/ right now but that can be easily moved.

Only reason why we even have it around still is because run-gdb.sh and flash.sh are nice.

Comment 20

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

It has quite a few tools that everyone should be using, like scripts to interact with the profiler, get memory reports, build update packages, ...  Naughty me for letting things get this bad but there are only so many battles that can be fought.

I guess all we need to do on our side is ensure it gets tagged along with other repos.

Comment 21

[Aki Sasaki (not active)]

Moved the historical manifests, and the new ones are uploading and linking correctly.

I think we're done here; please reopen (or file a new bug for new issues) if there's work left to be done.

Comment 22

[Michael Vines [:m1] [:evilmachines]]

Bug 834098 is for the rest.