Closed Bug 1311966 Opened 3 years ago Closed 3 years ago

generic worker discards artifacts on task failure

Categories

(Taskcluster :: Workers, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla53

People

(Reporter: grenade, Assigned: grenade)

References

Details

Attachments

(1 file)

we currently have no good way to publish artifacts when a task fails in g-w. this makes it difficult to debug failures.

g-w expects to find all public artifacts in a physical folder under the task directory called "public". since tests log and create artifacts to other folders, we attempt to work around the g-w limitation by copying artifacts to the public folder after the test completes with these commands:

 `xcopy build\blobber_upload_dir public\test_info /e /i`
 `copy /y logs\*.* public\logs\`

however, if a test task command returns a non-zero exit code (failure), then subsequent commands (to copy artifacts) do not get run so the artifacts are lost.

i believe docker-worker contains a mechanism for mapping physical artifact paths to virtual ones so that this issue is avoided. we will need something similar to avoid discarding artifacts when tests fail.
Yeah, this indeed is a limitation of the generic worker, which sucks.

I guess there is no chance the task itself can write to public\logs instead of build\blobber_upload_dir ?

Could this be fixed via a mozconfig change, or is the build\blobber_upload_dir hardcoded somewhere and changing it would affect non-windows builds?
Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

https://reviewboard.mozilla.org/r/93150/#review93104

Looks great!

Shall we wait for the OCC changes to land, so we can run this with a try push before landing?
Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

https://reviewboard.mozilla.org/r/93150/#review93544

Awesome stuff! Love it.

Let's create a bug so that at a higher level, the artifact name, and the filesystem path are tracked, and then in the g-w specific implementation, it builds the symlinks as required, so that when we swap it out so that g-w handles an artifact name being different from the filesystem relative path natively, it will just be localised to the g-w implementation code section.

But I see no reason this needs to be done immediately, so let's go with this, and have a separate bug for a refactor.

Thanks Rob!
Attachment #8810814 - Flags: review?(pmoore) → review+
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7535abf9e298
symlink rather than copy artifacts; r=pmoore
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/7535abf9e298
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
See Also: → 1352457
Note - this has now been implemented in the worker directly, so the symlink was no longer needed. In bug 1349980 this got fixed on the side during the refactoring of the in-tree task generation.
See Also: → 1349980
Component: Generic-Worker → Workers
You need to log in before you can comment on or make changes to this bug.