revamp b2g upload configs

RESOLVED WONTFIX

Status

Release Engineering
General Automation
RESOLVED WONTFIX
3 years ago
a year ago

People

(Reporter: jgriffin, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [see comment 6])

(Reporter)

Description

3 years ago
We're currently working on adding the ability to perform bisection against on-device test runs, but we lack a way to find the B2G device build for a given gecko commit.

There are many different ways we could expose this; the simplest is probably to add the build_url's to http://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz.  That file already contains emulator build url's, but not the url's for B2G device builds like the hamachi.

Once we have that, we could either query Treeherder for this data (if it exposes an API for that), or we could stand up a barebones service that would perform the mapping.
Many of our device builds don't actually upload their results anywhere ATM. See bug 966412.
(Reporter)

Comment 2

3 years ago
(In reply to Chris AtLee [:catlee] from comment #1)
> Many of our device builds don't actually upload their results anywhere ATM.
> See bug 966412.

I know; I think hamachi builds are the most relevant right now, though, and those do get uploaded.  We'd have to work with releng to tweak which builds get uploaded over time as our automation targets change.

Updated

3 years ago
Assignee: nobody → aki

Comment 3

3 years ago
Which package would we make the build_url ?  The gecko package?  We don't have a hamachi.zip anymore...
Flags: needinfo?(jgriffin)
(Reporter)

Comment 4

3 years ago
(In reply to Aki Sasaki [:aki] from comment #3)
> Which package would we make the build_url ?  The gecko package?  We don't
> have a hamachi.zip anymore...

Argh, right.  Yes, the gecko package would be preferable; it's trivial to create the gaia package locally if needed.
Flags: needinfo?(jgriffin)
or make it 'gecko_url' / 'device_zip_url' depending on which files we upload?

Comment 6

3 years ago
(In reply to Chris AtLee [:catlee] from comment #5)
> or make it 'gecko_url' / 'device_zip_url' depending on which files we upload?

This is getting dangerously close to revamping all the upload configs to

a) specify what each filename/glob means, a la buildprops and/or artifact manifest
b) package the gecko/gaia stuff in b2g_build.py (or build.sh) instead of ril.py.  Potentially abandon ril.py if we no longer care about commercial ril; potentially de-scope to just repacking with ril.
c) list-of-dicts so we can also specify renames and whether the file to be uploaded needs to be created from other files on disk (e.g. zipfile)

I think the above 3 things are good to do, and probably deserve their own config.json version bump.
Do we want this build_url sooner, or more future-proof?  If the latter, I'll do the above 3, but that's not a small ask.  I also have Flame builds and Gaia Try at the highest priority, and TRIBE next week, so it'll take a while.  If we want it sooner, I can hardcode the gecko glob for all devices.
(Reporter)

Comment 7

3 years ago
I'd rather have the more future-proof version, even if it means it won't be ready for a while.

Comment 8

3 years ago
Changing summary to match.
Summary: Add B2G device build url's to builds-4hr.js → revamp b2g upload configs
Whiteboard: [see comment 6]

Comment 9

3 years ago
Brainstorming:

So maybe we'll keep the top level config items 'zip_files', 'upload_files', and 'public_upload_files'.  The latter two could be in a shared dict if we specified locations and behavior inside the config, but I'm not sure that's best atm.

zip files first
"zip_files": [{
    "filename": "zipfile1.zip"
    "contents": [
        # strings are copied directly in
        # tuples are renamed/moved
    ]
},
    "filename": "zipfile2.tar.bz2",
    "compression_type": "bz2",  # defaults to zip?
    "contents: [...]
}],
"upload_files": [{
   "source": "zipfile1.zip",
   "artifact_name": "package-zip",  # allow for list?
   "artifact_url_names": ["package-zip_url", "build_url"],  # optional, defaults to "%s_url" % artifact_name.  Think this might be a list since we have legacy names, and may want to also support non-legacy names
   "target": "zipfile.zip"  # optional rename. obviously dumb for the zipfile, but might be useful for build.sh-created files
}, {
   ...
}],
"public_upload_files": [{
   # same as upload_files, i think
]}

Updated

3 years ago
Blocks: 991836

Comment 10

3 years ago
I will be adding gecko+gaia packages into the upload, and turning off the ril repacks as part of this bug.

Updated

3 years ago
Depends on: 1016979

Comment 11

3 years ago
Looking:  comment 6, comment 9, comment 10 appear to be the scope here:

* manifest-y upload configs
* multiple zips in upload configs
** gecko+gaia zip in upload configs
** kill ril repacks.  if people want the ril, they can get it elsewhere
* set build_url and/or device_zip_url properties

Explicitly out of scope:
* s3
* build.sh or build system changes

Potentially in-scope:
* flash scripts per device, if we don't get them into build.sh before then

Updated

3 years ago
Blocks: 1022909

Comment 12

3 years ago
This bug is different from bug 986112 in the sense that this is the build side and has nothing to do with blobber.

The files that bug 986112 is talking about are things like screenshots, emulator logs, structured logs and all.

I'm happy to aim to match a similar format as the one described in comment 9.

Comment 13

3 years ago
I'm probably not going to get to this any time soon.
Assignee: aki → nobody
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.