Closed Bug 979554 Opened 10 years ago Closed 8 years ago

revamp b2g upload configs

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jgriffin, Unassigned)

References

Details

(Whiteboard: [see comment 6])

We're currently working on adding the ability to perform bisection against on-device test runs, but we lack a way to find the B2G device build for a given gecko commit.

There are many different ways we could expose this; the simplest is probably to add the build_url's to http://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz.  That file already contains emulator build url's, but not the url's for B2G device builds like the hamachi.

Once we have that, we could either query Treeherder for this data (if it exposes an API for that), or we could stand up a barebones service that would perform the mapping.
Many of our device builds don't actually upload their results anywhere ATM. See bug 966412.
(In reply to Chris AtLee [:catlee] from comment #1)
> Many of our device builds don't actually upload their results anywhere ATM.
> See bug 966412.

I know; I think hamachi builds are the most relevant right now, though, and those do get uploaded.  We'd have to work with releng to tweak which builds get uploaded over time as our automation targets change.
Assignee: nobody → aki
Which package would we make the build_url ?  The gecko package?  We don't have a hamachi.zip anymore...
Flags: needinfo?(jgriffin)
(In reply to Aki Sasaki [:aki] from comment #3)
> Which package would we make the build_url ?  The gecko package?  We don't
> have a hamachi.zip anymore...

Argh, right.  Yes, the gecko package would be preferable; it's trivial to create the gaia package locally if needed.
Flags: needinfo?(jgriffin)
or make it 'gecko_url' / 'device_zip_url' depending on which files we upload?
(In reply to Chris AtLee [:catlee] from comment #5)
> or make it 'gecko_url' / 'device_zip_url' depending on which files we upload?

This is getting dangerously close to revamping all the upload configs to

a) specify what each filename/glob means, a la buildprops and/or artifact manifest
b) package the gecko/gaia stuff in b2g_build.py (or build.sh) instead of ril.py.  Potentially abandon ril.py if we no longer care about commercial ril; potentially de-scope to just repacking with ril.
c) list-of-dicts so we can also specify renames and whether the file to be uploaded needs to be created from other files on disk (e.g. zipfile)

I think the above 3 things are good to do, and probably deserve their own config.json version bump.
Do we want this build_url sooner, or more future-proof?  If the latter, I'll do the above 3, but that's not a small ask.  I also have Flame builds and Gaia Try at the highest priority, and TRIBE next week, so it'll take a while.  If we want it sooner, I can hardcode the gecko glob for all devices.
I'd rather have the more future-proof version, even if it means it won't be ready for a while.
Changing summary to match.
Summary: Add B2G device build url's to builds-4hr.js → revamp b2g upload configs
Whiteboard: [see comment 6]
Brainstorming:

So maybe we'll keep the top level config items 'zip_files', 'upload_files', and 'public_upload_files'.  The latter two could be in a shared dict if we specified locations and behavior inside the config, but I'm not sure that's best atm.

zip files first
"zip_files": [{
    "filename": "zipfile1.zip"
    "contents": [
        # strings are copied directly in
        # tuples are renamed/moved
    ]
},
    "filename": "zipfile2.tar.bz2",
    "compression_type": "bz2",  # defaults to zip?
    "contents: [...]
}],
"upload_files": [{
   "source": "zipfile1.zip",
   "artifact_name": "package-zip",  # allow for list?
   "artifact_url_names": ["package-zip_url", "build_url"],  # optional, defaults to "%s_url" % artifact_name.  Think this might be a list since we have legacy names, and may want to also support non-legacy names
   "target": "zipfile.zip"  # optional rename. obviously dumb for the zipfile, but might be useful for build.sh-created files
}, {
   ...
}],
"public_upload_files": [{
   # same as upload_files, i think
]}
Blocks: 991836
I will be adding gecko+gaia packages into the upload, and turning off the ril repacks as part of this bug.
Depends on: 1016979
Looking:  comment 6, comment 9, comment 10 appear to be the scope here:

* manifest-y upload configs
* multiple zips in upload configs
** gecko+gaia zip in upload configs
** kill ril repacks.  if people want the ril, they can get it elsewhere
* set build_url and/or device_zip_url properties

Explicitly out of scope:
* s3
* build.sh or build system changes

Potentially in-scope:
* flash scripts per device, if we don't get them into build.sh before then
Blocks: 1022909
This bug is different from bug 986112 in the sense that this is the build side and has nothing to do with blobber.

The files that bug 986112 is talking about are things like screenshots, emulator logs, structured logs and all.

I'm happy to aim to match a similar format as the one described in comment 9.
I'm probably not going to get to this any time soon.
Assignee: aki → nobody
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.