Open Bug 1500132 Opened 7 years ago Updated 3 years ago

Upload generated files even if build failed

Categories

(Firefox Build System :: General, enhancement)

Product:
Firefox Build System
Component:
General
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: nika, Unassigned)

Details

Currently we only upload generated files if the build succeeded as an aritfact. This is unfortunate when running into a build error in generated source files, as it makes it more frustrating to obtain the failing sources. It would be nice to attempt to upload these artifacts unconditionally.
This is already the case -- all Taskcluster workers upload all artifacts that are available, regardless of the result of the command. If the worker instance crashes, artifacts aren't uploaded, and that can make for a frustrating time debugging. But there's very little we can do about that, other than try to turn the crash into a failure. Typically we've seen this with massive OOM's or kernel bugs. Is that what is going on here? The other situation where one might see this occurring is when the task only creates the artifacts on-disk at successful completion. In that case, when the task fails, there's nothing for the Taskcluster worker to upload, and the fix is to adjust the task definition. Some context would help figure out what the issue is..
Flags: needinfo?(nika)
The file 'public/build/target.generated-files.tar.gz' (or similar) appears to be created & uploaded only if the build completes. e.g. succeeding build-win32: https://tools.taskcluster.net/groups/HRSHG1UfRQqG7KQKCezfhQ/tasks/HMJw64nWSfOUNYM0Ge1fVQ/runs/0/artifacts failing build-win32: https://tools.taskcluster.net/groups/CKqg6Iu-ThmGZRCu9ksTtg/tasks/bSjUXgi4ReqIq9An6etJsQ/runs/1/artifacts Both of these builds passed the `export` phase, but only one of them finished building. Generated files should ideally be available as soon as `export` is completed (or even during, if possible). I think this is probably a task definition issue, and based on your comment I'm guessing this isn't the right component to file that in, although I'm not sure which one is.
Flags: needinfo?(nika)
I don't see any reference to `generated-files.tar.gz` in the failing task (bSjUXgi4ReqIq9An6etJsQ), so yes, it looks like this is an issue with the build process. Perhaps that file isn't created in the export phase? I'm not sure what component that fits under, either.
Product: Taskcluster → Firefox Build System
the `package-generated-sources` logic gets triggered as an automation tier in CI: https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/build/moz-automation.mk#33 It generates that archive into $objdir/dist, and then relies on the `upload` tier to copy it over to the artifacts dir. By definition all of those automation tiers run after the build is finished. We don't currently have an easy way to wire things like that up inside the build itself, although we could probably do it in a hacky way. It'd also probably be fine to just generate the archive directly into $UPLOAD_DIR instead of dist so that it gets uploaded regardless of whether the build gets far enough to run the `upload` tier.
But if the build fails during automation/build, which is the most common cause of failures, whether the generated-sources file is created directly in the UPLOAD_DIR or not doesn't change anything.
Right, I meant "if we hacked things up so that we generated the archive post-export, we could also put it in $UPLOAD_DIR".
Keywords: in-triage
Keywords: in-triage
Discussed in triage. We'd take a patch but this isn't something the build team is going to work on.

Per conversation with :kmoir, I'm going through untriaged bugs in her components and marking the ones which look to be enhancements/tasks with the enhancement severity to get them out of the triage queue.

If this incorrect, please remove the tag.

Severity: normal → enhancement
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.