1172107 - Artifacts are broken in tc opt/dbg linux64 builds (tests.zip)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[mphillips]

The jobs complete successfully; but the artifacts don't appear to upload.

Comment 1

[Morgan Phillips [:mrrrgn]]

Here, this job from try has successfully uploaded logs: https://tools.taskcluster.net/task-inspector/#HOQ8F7GARZ2m0Rcy7OFLVg/0 

But they aren't accessible from the treeherder link. Given this, the scope of this ticket may just involve 1.) figuring out why the th link is broken and 2.) ensuring that we're uploading all the things that need uploading (we aren't).

Comment 2

[Andrew Halberstadt [:ahal]]

I noticed the tests location is 'public/build/target.tests.zip', which is probably copied from the b2g desktop config? Anyway, for b2g desktop, this script gets run [1].

For Firefox desktop does the mozharness build script handle packaging the tests? If so it might just be a matter of figuring out where it saves the test package and changing [2] accordingly.

[1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/build-b2g-desktop.sh#22
[2] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/opt_linux64.yml#54

Comment 3

[Morgan Phillips [:mrrrgn]]

Attachment: uploads.diff

Build with this patch applied: https://tools.taskcluster.net/task-inspector/#CCJHSxbxSouwLZE_mZBddA/0

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8616905 [details] [diff] [review]
uploads.diff

Review of attachment 8616905 [details] [diff] [review]:
-----------------------------------------------------------------

Please also remove the non-working artifact code at the beginning of the file:

https://dxr.mozilla.org/mozilla-central/source/testing/docker/desktop-build/bin/build.sh?from=testing/docker/desktop-build/bin/build.sh#90
# the TC docker worker looks for artifacts to upload in $HOME/artifacts, but
# mach wants to put them in $WORKSPACE/build/upload; symlinking lets everyone
# win!
rm -rf $WORKSPACE/build/upload
mkdir -p $HOME/artifacts
ln -s $HOME/artifacts $WORKSPACE/build/upload

I'm interested in why that doesn't work.  I suspect because we're disabling the upload step?

::: testing/docker/desktop-build/bin/build.sh
@@ +178,5 @@
>    --no-action=generate-build-stats \
>    --branch=${MH_BRANCH} \
>    --build-pool=${MH_BUILD_POOL}
> +
> +# upload auxillory files

typo

@@ +188,5 @@
> +done
> +
> +# Discard version numbers from packaged files, they just make it hard to write
> +# the right filename in the task payload where artifacts are declared
> +target_uploads="x-test.linux-x86_64.tar.bz2 linux-x86_64.tar.bz2 linux-x86_64.json tests.zip crashreporter-symbols.zip"

unused variable

Comment 5

[Morgan Phillips [:mrrrgn]]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> Comment on attachment 8616905 [details] [diff] [review]
> uploads.diff
> 
> Review of attachment 8616905 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Please also remove the non-working artifact code at the beginning of the
> file:
> 
> https://dxr.mozilla.org/mozilla-central/source/testing/docker/desktop-build/
> bin/build.sh?from=testing/docker/desktop-build/bin/build.sh#90
> # the TC docker worker looks for artifacts to upload in $HOME/artifacts, but
> # mach wants to put them in $WORKSPACE/build/upload; symlinking lets everyone
> # win!
> rm -rf $WORKSPACE/build/upload
> mkdir -p $HOME/artifacts
> ln -s $HOME/artifacts $WORKSPACE/build/upload
> 
> I'm interested in why that doesn't work.  I suspect because we're disabling
> the upload step?
> 
> ::: testing/docker/desktop-build/bin/build.sh
> @@ +178,5 @@
> >    --no-action=generate-build-stats \
> >    --branch=${MH_BRANCH} \
> >    --build-pool=${MH_BUILD_POOL}
> > +
> > +# upload auxillory files
> 
> typo
> 
> @@ +188,5 @@
> > +done
> > +
> > +# Discard version numbers from packaged files, they just make it hard to write
> > +# the right filename in the task payload where artifacts are declared
> > +target_uploads="x-test.linux-x86_64.tar.bz2 linux-x86_64.tar.bz2 linux-x86_64.json tests.zip crashreporter-symbols.zip"
> 
> unused variable

Derp, fixing typos. The only thing that ends up in uploads are the log files. We should look at fixing this in Mozharness: creating a bug for that.

Comment 6

[Morgan Phillips [:mrrrgn]]

Attachment: uploads.diff

fixed typo and unused variable.

Comment 7

[Morgan Phillips [:mrrrgn]]

Attachment: uploads.diff

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8620413 [details] [diff] [review]
uploads.diff

Review of attachment 8620413 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, looks good to me. I haven't tested locally, but here's a try run along with my mochitest patch:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8dfd0f75f9ea&exclusion_profile=false

Hopefully it gets a little further now!

p.s the commit message is missing the bug number in case you didn't notice.

Comment 9

[Andrew Halberstadt [:ahal]]

Hm, looks like the test jobs are still failing with the same error. They seem to be looking for the tests.zip in:
https://queue.taskcluster.net/v1/task/f6Fiu5PCSeucmwvFVgWG7A/artifacts/public/build/target.tests.zip

Is that where you'd expect it to be? Or maybe the test jobs just need to be configured to look for it elsewhere?

Comment 10

[Morgan Phillips [:mrrrgn]]

(In reply to Andrew Halberstadt [:ahal] from comment #9)
> Hm, looks like the test jobs are still failing with the same error. They
> seem to be looking for the tests.zip in:
> https://queue.taskcluster.net/v1/task/f6Fiu5PCSeucmwvFVgWG7A/artifacts/
> public/build/target.tests.zip
> 
> Is that where you'd expect it to be? Or maybe the test jobs just need to be
> configured to look for it elsewhere?

There still needs to be another patch to configure this (it uploads files based on ENV variables passed in). I can make sure they end up in artifacts/build -- though that's surely configurable.

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/15ff3cdc68bd

Comment 12

[Morgan Phillips [:mrrrgn]]

Attachment: dist.uploads.diff

Comment 13

[Morgan Phillips [:mrrrgn]]

Attachment: versionbump.diff

Comment 14

[Andrew Halberstadt [:ahal]]

Comment on attachment 8620633 [details] [diff] [review]
dist.uploads.diff

Review of attachment 8620633 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, but there are some json files that should also get uploaded. Namely the build json, mozinfo.json and test_packages.json. E.g:
https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1434022422/firefox-41.0a1.en-US.linux-i686.mozinfo.json

I'm not sure that they'll block getting tests running, but there are a smattering of external tools that use them.

Comment 15

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/15ff3cdc68bd

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6df4c4d057bc
https://hg.mozilla.org/integration/mozilla-inbound/rev/98f7d540d99f

Comment 17

[Morgan Phillips [:mrrrgn]]

(In reply to Andrew Halberstadt [:ahal] from comment #14)
> Comment on attachment 8620633 [details] [diff] [review]
> dist.uploads.diff
> 
> Review of attachment 8620633 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks good, but there are some json files that should also get uploaded.
> Namely the build json, mozinfo.json and test_packages.json. E.g:
> https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> inbound-linux/1434022422/firefox-41.0a1.en-US.linux-i686.mozinfo.json
> 
> I'm not sure that they'll block getting tests running, but there are a
> smattering of external tools that use them.

The patches are in inbound now, so the target.tests.zip file should start showing up after some time passes. The other file is noted, if/when we hook up the tool that needs it, should be easy enough to add now (just by modifying an environment variable).

Comment 18

[Morgan Phillips [:mrrrgn]]

Attachment: artifactsdir.diff

Solves the missing directory problem. Here is a run with the resulting container: https://tools.taskcluster.net/task-inspector/#Q1edCfEQSBOJLUqNPlyRSg/

Comment 19

[Andrew Halberstadt [:ahal]]

Comment on attachment 8621668 [details] [diff] [review]
artifactsdir.diff

Review of attachment 8621668 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks! The tests look in $HOME/artifacts/build, but I'm sure they can be configured to look in $HOME/artifacts.

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6df4c4d057bc
https://hg.mozilla.org/mozilla-central/rev/98f7d540d99f

Comment 21

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5f740da4a57d

Comment 22

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/5f740da4a57d