1437577 - Allow uploading symbols from try builds to symbol server

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[(not currently active) Ted Mielczarek]

In bug 1422096 (and the dependency tree), peterbe implemented support in the symbol server for a longstanding wish of mine: the ability to upload symbols from try builds. This should allow developers doing try builds to upload symbols to the symbol server so they can easily debug the resulting builds. (Currently the process to do so involves some painful manual steps.)

Try symbols go to a separate prefix in the s3 symbols bucket with a shorter expiration policy (28 days, which matches the lifetime of Taskcluster artifacts on try) and they don't get served in requests to the symbol server at `https://symbols.mozilla.org/`. You have to explicitly add `https://symbols.mozilla.org/try` to your symbol path to get symbols from try builds, since anyone with try server access will be able to upload files there and Microsoft's debuggers can execute arbitrary code from PDB files. This was the best compromise we could come up with currently, hopefully it's workable.

This bug covers fixing the in-tree task configuration to allow this to work. When I fixed bug 1422740 I made try builds upload to our staging instance of Tecken (so that I could test things):
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/toolkit/crashreporter/tools/upload_symbols.py#96

The simple fix here would be to remove that block (so that try uploads go to symbols.mozilla.org), and also to generate a new token on symbols.mo that only has the "Upload Try Symbols Files" permission and store that token in the `project/releng/gecko/build/level-1/gecko-symbol-upload` secret in Taskcluster:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/ci/upload-symbols/kind.yml#44
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/toolkit/crashreporter/tools/upload_symbols.py#45

I'm not sure if that actually fixes real-world use cases though, in that doing that means you will be able to run the existing in-tree symbol upload tasks, which are only scheduled to run after *nightly* build jobs. You can run nightly build jobs on try, and the symbol uploads will work (I tested the existing code on try by doing just that), but I don't think it's very common for developers to generate nightly builds on try.

We might want to also add some machinery that allows developers to upload symbols from *all* the builds in their try push, which probably involves an extra argument to `mach try` (maybe something like `--with-symbols`?), and then some taskgraph hacking to cause that to schedule symbol upload tasks after every build in the build graph. I don't think that should be very hard. Right now the upload-symbols task kind explicitly lists all the jobs it will run after:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/ci/upload-symbols/kind.yml#15

The very similar upload-generated-sources task uses `only-for-attributes: -nightly`:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/ci/upload-generated-sources/kind.yml#15

We could presumably either add a new attribute like `upload_symbols` to build tasks, or we could simply put the logic in the `upload_symbols` task transform:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/taskgraph/transforms/upload_symbols.py

Comment 1

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ad9b0cef0a5266065cc1a97e6447fcbb36b58e17

Comment 2

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=ad9b0cef0a5266065cc1a97e6447fcbb36b58e17

So I pushed this to try as `./mach try -b do -p linux64,win64 -u none`, and then used Treeherder's "Add new jobs" feature to trigger the symbol-upload tasks for the builds I did after-the-fact, and they worked perfectly. You can see the win64-debug symbol upload, for example, here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f4c3adc2da94c2699ef338880c9a09032b3800b&selectedJob=173328919

I downloaded the win64-debug build:
https://queue.taskcluster.net/v1/task/W5mwVik4T8yNOb4CVNLwug/runs/0/artifacts/public/build/target.zip

Ran it in WinDBG on my Windows machine, set my symbol path to include `SRV*https://symbols.mozilla.org/try` and everything worked. Hooray!

Oh, I also had aki update the `project/releng/gecko/build/level-1/gecko-symbol-upload` secret with a new token I generated from my account on symbols.mo which has only the "Upload Try Symbols" permission, so any symbols uploaded with that token get stored separately on the symbol server.

Comment 3

[(not currently active) Ted Mielczarek]

Attachment: bug 1437577 - Upload symbols from try server builds to symbols.mozilla.org instead of symbols.stage.mozaws.net.

The symbol server recently gained the ability to upload symbols from try
to a different prefix in the symbols bucket. If we change the token stored
in the Taskcluster secret `project/releng/gecko/build/level-1/gecko-symbol-upload`
to one that has only "upload try symbols" permissions then we can upload
symbols from try server builds directly to symbols.mo.

Review commit: https://reviewboard.mozilla.org/r/236180/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/236180/

Comment 4

[(not currently active) Ted Mielczarek]

Attachment: bug 1437577 - preserve treeherder.platform from job into task.

Tasks like upload-symbols that have a dependency on a build job want to
copy the treeherder.platform from the build job but it gets lost in
the task transform currently. This simply copies it into
an extra.treeherder-platform key to make life easier.

Review commit: https://reviewboard.mozilla.org/r/236182/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/236182/

Comment 5

[(not currently active) Ted Mielczarek]

Attachment: bug 1437577 - allow uploading symbols from any build type on try.

Previously we would only generate upload-symbols tasks for nightly builds,
since we only want to upload symbols for builds we ship to users.

On try, developers may want to use the symbol server but very rarely want
to do nightly builds, so allow uploading symbols from any build type there.

Review commit: https://reviewboard.mozilla.org/r/236184/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/236184/

Comment 6

[(not currently active) Ted Mielczarek]

Here's a diff of the full taskgraph with the second patch here applied (diffed against a tree with the patch from bug 
1353296):
https://gist.github.com/luser/3d20de933a4eb631b0bc7f5075f05a74

Here's a diff of the full taskgraph between the second and third patches here:
https://gist.github.com/luser/88386fa4d77bbbb5782e1ebc589fb3fa

There are a lot of new tasks there because we have a lot of build variants, as it turns out. Everything that's not a nightly is set to `run-on-projects: try`. I verified with aki's taskgraph-diff scripts that there are no differences in the target tasks for inbound/central nightlies/beta/release with this patch applied:

$ ls -l /tmp/output/bug1437577/total 70856
-rw-r--r-- 1 luser luser 14654088 Apr 12 12:07 beta-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 beta-parameters.json.diff
-rw-r--r-- 1 luser luser  2553727 Apr 12 12:07 central-nightly-android-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 central-nightly-android-parameters.json.diff
-rw-r--r-- 1 luser luser 28326169 Apr 12 12:07 central-nightly-desktop-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 central-nightly-desktop-parameters.json.diff
-rw-r--r-- 1 luser luser 15554618 Apr 12 12:07 inbound-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 inbound-parameters.json.diff
-rw-r--r-- 1 luser luser 11459643 Apr 12 12:07 release-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 release-parameters.json.diff

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8967508 [details]
bug 1437577 - preserve treeherder.platform from job into task.

https://reviewboard.mozilla.org/r/236182/#review242226

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8967509 [details]
bug 1437577 - allow uploading symbols from any build type on try.

https://reviewboard.mozilla.org/r/236184/#review242228

::: taskcluster/ci/upload-symbols/kind.yml:41
(Diff revision 1)
>          by-build-platform:
>              .*devedition.*: ['mozilla-beta', 'maple']
> -            default: ['all']
> +            # Only upload symbols for nightlies on most branches.
> +            .*(?<!-devedition)-nightly: ['all']
> +            # Allow symbol upload for any build type on try.
> +            default: ['try']

Is this just "allow", or will those uploads start occurring automatically for try builds that don't use try-select?

Comment 9

[Peter Bengtsson [:peterbe]]

Comment on attachment 8967507 [details]
bug 1437577 - Upload symbols from try server builds to symbols.mozilla.org instead of symbols.stage.mozaws.net.

https://reviewboard.mozilla.org/r/236180/#review242262

+1

Comment 10

[Peter Bengtsson [:peterbe]]

Comment on attachment 8967507 [details]
bug 1437577 - Upload symbols from try server builds to symbols.mozilla.org instead of symbols.stage.mozaws.net.

https://reviewboard.mozilla.org/r/236180/#review242264

oops

Comment 11

[(not currently active) Ted Mielczarek]

Comment on attachment 8967509 [details]
bug 1437577 - allow uploading symbols from any build type on try.

https://reviewboard.mozilla.org/r/236184/#review242228

> Is this just "allow", or will those uploads start occurring automatically for try builds that don't use try-select?

I don't pretend to understand all the intricacies of try scheduling (there are a lot of ways it can work) but they won't get scheduled if you use trychooser syntax (I tested that with my push here) and they don't show up by default in `mach try fuzzy` because that uses a profile based on central (you have to `mach try fuzzy --full`. I can sanity check that triggering a build via `mach try fuzzy` doesn't schedule them by default.

Comment 12

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=15acb889510bb07ed3d29ea802997f11796dad9f

Comment 13

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #11)
> > Is this just "allow", or will those uploads start occurring automatically for try builds that don't use try-select?
> 
> I don't pretend to understand all the intricacies of try scheduling (there
> are a lot of ways it can work) but they won't get scheduled if you use
> trychooser syntax (I tested that with my push here) and they don't show up
> by default in `mach try fuzzy` because that uses a profile based on central
> (you have to `mach try fuzzy --full`. I can sanity check that triggering a
> build via `mach try fuzzy` doesn't schedule them by default.

I lost track of this in the midst of traveling for a work week and some other things, so I've rebased atop latest mozilla-central and done a try push for sanity's sake and if that looks good I will land these patches.

Comment 14

[(not currently active) Ted Mielczarek]

Looking at the task graph of that try push it looks sensible, so I'm going to land this.

Comment 15

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8483a61d426f758f013e78eb1addc667c4b8c673
bug 1437577 - Upload symbols from try server builds to symbols.mozilla.org instead of symbols.stage.mozaws.net. r=peterbe

https://hg.mozilla.org/integration/mozilla-inbound/rev/01511447fa7ed82228fb8ffaa3b1850bb7429a8c
bug 1437577 - preserve treeherder.platform from job into task. r=dustin

https://hg.mozilla.org/integration/mozilla-inbound/rev/a0b66c435f62a6c3f3fd68c9e4050477a778bd86
bug 1437577 - allow uploading symbols from any build type on try. r=dustin

Comment 16

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/8483a61d426f
https://hg.mozilla.org/mozilla-central/rev/01511447fa7e
https://hg.mozilla.org/mozilla-central/rev/a0b66c435f62