1311966 - generic worker discards artifacts on task failure

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

we currently have no good way to publish artifacts when a task fails in g-w. this makes it difficult to debug failures.

g-w expects to find all public artifacts in a physical folder under the task directory called "public". since tests log and create artifacts to other folders, we attempt to work around the g-w limitation by copying artifacts to the public folder after the test completes with these commands:

 `xcopy build\blobber_upload_dir public\test_info /e /i`
 `copy /y logs\*.* public\logs\`

however, if a test task command returns a non-zero exit code (failure), then subsequent commands (to copy artifacts) do not get run so the artifacts are lost.

i believe docker-worker contains a mechanism for mapping physical artifact paths to virtual ones so that this issue is avoided. we will need something similar to avoid discarding artifacts when tests fail.

Comment 1

[Pete Moore [:pmoore][:pete]]

Yeah, this indeed is a limitation of the generic worker, which sucks.

I guess there is no chance the task itself can write to public\logs instead of build\blobber_upload_dir ?

Could this be fixed via a mozconfig change, or is the build\blobber_upload_dir hardcoded somewhere and changing it would affect non-windows builds?

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

https://github.com/mozilla-releng/OpenCloudConfig/commit/b3ce5b3dc1d3c302993816960a5281ad53fcb6e5

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: Bug 1311966 - symlink rather than copy artifacts;

Review commit: https://reviewboard.mozilla.org/r/93150/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/93150/

Comment 4

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

https://reviewboard.mozilla.org/r/93150/#review93104

Looks great!

Shall we wait for the OCC changes to land, so we can run this with a try push before landing?

Comment 5

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/93150/diff/1-2/

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/93150/diff/2-3/

Comment 7

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/93150/diff/3-4/

Comment 8

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/93150/diff/4-5/

Comment 9

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/93150/diff/5-6/

Comment 10

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8810814 [details]
Bug 1311966 - symlink rather than copy artifacts;

https://reviewboard.mozilla.org/r/93150/#review93544

Awesome stuff! Love it.

Let's create a bug so that at a higher level, the artifact name, and the filesystem path are tracked, and then in the g-w specific implementation, it builds the symlinks as required, so that when we swap it out so that g-w handles an artifact name being different from the filesystem relative path natively, it will just be localised to the g-w implementation code section.

But I see no reason this needs to be done immediately, so let's go with this, and have a separate bug for a refactor.

Thanks Rob!

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7535abf9e298
symlink rather than copy artifacts; r=pmoore

Comment 12

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/7535abf9e298

Comment 13

[areya]

https://hg.mozilla.org/mozilla-central/rev/7535abf9e298

Comment 14

[Pete Moore [:pmoore][:pete]]

Note - this has now been implemented in the worker directly, so the symlink was no longer needed. In bug 1349980 this got fixed on the side during the refactoring of the in-tree task generation.