Artifacts are broken in tc opt/dbg linux64 builds (tests.zip)

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: mphillips, Assigned: mrrrgn)

Tracking

unspecified

Firefox Tracking Flags

(firefox41 fixed)

Details

Attachments

(4 attachments, 2 obsolete attachments)

(Reporter)

Description

3 years ago
The jobs complete successfully; but the artifacts don't appear to upload.
Blocks: 1171033
(Assignee)

Updated

3 years ago
Assignee: nobody → winter2718
(Assignee)

Comment 1

3 years ago
Here, this job from try has successfully uploaded logs: https://tools.taskcluster.net/task-inspector/#HOQ8F7GARZ2m0Rcy7OFLVg/0 

But they aren't accessible from the treeherder link. Given this, the scope of this ticket may just involve 1.) figuring out why the th link is broken and 2.) ensuring that we're uploading all the things that need uploading (we aren't).
(Assignee)

Updated

3 years ago
Summary: Artifacts are broken in tc opt/dbg linux64 builds → Artifacts are broken in tc opt/dbg linux64 builds (tests.zip)
I noticed the tests location is 'public/build/target.tests.zip', which is probably copied from the b2g desktop config? Anyway, for b2g desktop, this script gets run [1].

For Firefox desktop does the mozharness build script handle packaging the tests? If so it might just be a matter of figuring out where it saves the test package and changing [2] accordingly.

[1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/build-b2g-desktop.sh#22
[2] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/opt_linux64.yml#54
(Assignee)

Comment 3

3 years ago
Created attachment 8616905 [details] [diff] [review]
uploads.diff

Build with this patch applied: https://tools.taskcluster.net/task-inspector/#CCJHSxbxSouwLZE_mZBddA/0
Attachment #8616905 - Flags: review?(dustin)
Comment on attachment 8616905 [details] [diff] [review]
uploads.diff

Review of attachment 8616905 [details] [diff] [review]:
-----------------------------------------------------------------

Please also remove the non-working artifact code at the beginning of the file:

https://dxr.mozilla.org/mozilla-central/source/testing/docker/desktop-build/bin/build.sh?from=testing/docker/desktop-build/bin/build.sh#90
# the TC docker worker looks for artifacts to upload in $HOME/artifacts, but
# mach wants to put them in $WORKSPACE/build/upload; symlinking lets everyone
# win!
rm -rf $WORKSPACE/build/upload
mkdir -p $HOME/artifacts
ln -s $HOME/artifacts $WORKSPACE/build/upload

I'm interested in why that doesn't work.  I suspect because we're disabling the upload step?

::: testing/docker/desktop-build/bin/build.sh
@@ +178,5 @@
>    --no-action=generate-build-stats \
>    --branch=${MH_BRANCH} \
>    --build-pool=${MH_BUILD_POOL}
> +
> +# upload auxillory files

typo

@@ +188,5 @@
> +done
> +
> +# Discard version numbers from packaged files, they just make it hard to write
> +# the right filename in the task payload where artifacts are declared
> +target_uploads="x-test.linux-x86_64.tar.bz2 linux-x86_64.tar.bz2 linux-x86_64.json tests.zip crashreporter-symbols.zip"

unused variable
Attachment #8616905 - Flags: review?(dustin)
(Assignee)

Comment 5

3 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> Comment on attachment 8616905 [details] [diff] [review]
> uploads.diff
> 
> Review of attachment 8616905 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Please also remove the non-working artifact code at the beginning of the
> file:
> 
> https://dxr.mozilla.org/mozilla-central/source/testing/docker/desktop-build/
> bin/build.sh?from=testing/docker/desktop-build/bin/build.sh#90
> # the TC docker worker looks for artifacts to upload in $HOME/artifacts, but
> # mach wants to put them in $WORKSPACE/build/upload; symlinking lets everyone
> # win!
> rm -rf $WORKSPACE/build/upload
> mkdir -p $HOME/artifacts
> ln -s $HOME/artifacts $WORKSPACE/build/upload
> 
> I'm interested in why that doesn't work.  I suspect because we're disabling
> the upload step?
> 
> ::: testing/docker/desktop-build/bin/build.sh
> @@ +178,5 @@
> >    --no-action=generate-build-stats \
> >    --branch=${MH_BRANCH} \
> >    --build-pool=${MH_BUILD_POOL}
> > +
> > +# upload auxillory files
> 
> typo
> 
> @@ +188,5 @@
> > +done
> > +
> > +# Discard version numbers from packaged files, they just make it hard to write
> > +# the right filename in the task payload where artifacts are declared
> > +target_uploads="x-test.linux-x86_64.tar.bz2 linux-x86_64.tar.bz2 linux-x86_64.json tests.zip crashreporter-symbols.zip"
> 
> unused variable

Derp, fixing typos. The only thing that ends up in uploads are the log files. We should look at fixing this in Mozharness: creating a bug for that.
(Assignee)

Comment 6

3 years ago
Created attachment 8617689 [details] [diff] [review]
uploads.diff

fixed typo and unused variable.
Attachment #8616905 - Attachment is obsolete: true
Attachment #8617689 - Flags: review?(dustin)
(Assignee)

Comment 7

3 years ago
Created attachment 8620413 [details] [diff] [review]
uploads.diff
Attachment #8617689 - Attachment is obsolete: true
Attachment #8617689 - Flags: review?(dustin)
Attachment #8620413 - Flags: review?(ahalberstadt)
Comment on attachment 8620413 [details] [diff] [review]
uploads.diff

Review of attachment 8620413 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, looks good to me. I haven't tested locally, but here's a try run along with my mochitest patch:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8dfd0f75f9ea&exclusion_profile=false

Hopefully it gets a little further now!

p.s the commit message is missing the bug number in case you didn't notice.
Attachment #8620413 - Flags: review?(ahalberstadt) → review+
Hm, looks like the test jobs are still failing with the same error. They seem to be looking for the tests.zip in:
https://queue.taskcluster.net/v1/task/f6Fiu5PCSeucmwvFVgWG7A/artifacts/public/build/target.tests.zip

Is that where you'd expect it to be? Or maybe the test jobs just need to be configured to look for it elsewhere?
(Assignee)

Comment 10

3 years ago
(In reply to Andrew Halberstadt [:ahal] from comment #9)
> Hm, looks like the test jobs are still failing with the same error. They
> seem to be looking for the tests.zip in:
> https://queue.taskcluster.net/v1/task/f6Fiu5PCSeucmwvFVgWG7A/artifacts/
> public/build/target.tests.zip
> 
> Is that where you'd expect it to be? Or maybe the test jobs just need to be
> configured to look for it elsewhere?

There still needs to be another patch to configure this (it uploads files based on ENV variables passed in). I can make sure they end up in artifacts/build -- though that's surely configurable.
(Assignee)

Comment 12

3 years ago
Created attachment 8620633 [details] [diff] [review]
dist.uploads.diff
Attachment #8620633 - Flags: review?(ahalberstadt)
(Assignee)

Comment 13

3 years ago
Created attachment 8620926 [details] [diff] [review]
versionbump.diff
Attachment #8620926 - Flags: review?(ahalberstadt)
Comment on attachment 8620633 [details] [diff] [review]
dist.uploads.diff

Review of attachment 8620633 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, but there are some json files that should also get uploaded. Namely the build json, mozinfo.json and test_packages.json. E.g:
https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1434022422/firefox-41.0a1.en-US.linux-i686.mozinfo.json

I'm not sure that they'll block getting tests running, but there are a smattering of external tools that use them.
Attachment #8620633 - Flags: review?(ahalberstadt) → review+
Attachment #8620926 - Flags: review?(ahalberstadt) → review+
https://hg.mozilla.org/mozilla-central/rev/15ff3cdc68bd
Status: NEW → RESOLVED
Last Resolved: 3 years ago
status-firefox41: --- → fixed
Resolution: --- → FIXED
(Assignee)

Comment 17

3 years ago
(In reply to Andrew Halberstadt [:ahal] from comment #14)
> Comment on attachment 8620633 [details] [diff] [review]
> dist.uploads.diff
> 
> Review of attachment 8620633 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks good, but there are some json files that should also get uploaded.
> Namely the build json, mozinfo.json and test_packages.json. E.g:
> https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> inbound-linux/1434022422/firefox-41.0a1.en-US.linux-i686.mozinfo.json
> 
> I'm not sure that they'll block getting tests running, but there are a
> smattering of external tools that use them.

The patches are in inbound now, so the target.tests.zip file should start showing up after some time passes. The other file is noted, if/when we hook up the tool that needs it, should be easy enough to add now (just by modifying an environment variable).
status-firefox41: fixed → ---
(Assignee)

Updated

3 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [leave-open]
(Assignee)

Comment 18

3 years ago
Created attachment 8621668 [details] [diff] [review]
artifactsdir.diff

Solves the missing directory problem. Here is a run with the resulting container: https://tools.taskcluster.net/task-inspector/#Q1edCfEQSBOJLUqNPlyRSg/
Attachment #8621668 - Flags: review?(ahalberstadt)
Comment on attachment 8621668 [details] [diff] [review]
artifactsdir.diff

Review of attachment 8621668 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks! The tests look in $HOME/artifacts/build, but I'm sure they can be configured to look in $HOME/artifacts.
Attachment #8621668 - Flags: review?(ahalberstadt) → review+
Whiteboard: [leave-open]
https://hg.mozilla.org/mozilla-central/rev/5f740da4a57d
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
status-firefox41: --- → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.