Closed Bug 1437577 Opened 4 years ago Closed 4 years ago

Allow uploading symbols from try builds to symbol server

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox62 --- fixed

People

(Reporter: ted, Assigned: ted)

References

Details

Attachments

(3 files)

In bug 1422096 (and the dependency tree), peterbe implemented support in the symbol server for a longstanding wish of mine: the ability to upload symbols from try builds. This should allow developers doing try builds to upload symbols to the symbol server so they can easily debug the resulting builds. (Currently the process to do so involves some painful manual steps.)

Try symbols go to a separate prefix in the s3 symbols bucket with a shorter expiration policy (28 days, which matches the lifetime of Taskcluster artifacts on try) and they don't get served in requests to the symbol server at `https://symbols.mozilla.org/`. You have to explicitly add `https://symbols.mozilla.org/try` to your symbol path to get symbols from try builds, since anyone with try server access will be able to upload files there and Microsoft's debuggers can execute arbitrary code from PDB files. This was the best compromise we could come up with currently, hopefully it's workable.

This bug covers fixing the in-tree task configuration to allow this to work. When I fixed bug 1422740 I made try builds upload to our staging instance of Tecken (so that I could test things):
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/toolkit/crashreporter/tools/upload_symbols.py#96

The simple fix here would be to remove that block (so that try uploads go to symbols.mozilla.org), and also to generate a new token on symbols.mo that only has the "Upload Try Symbols Files" permission and store that token in the `project/releng/gecko/build/level-1/gecko-symbol-upload` secret in Taskcluster:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/ci/upload-symbols/kind.yml#44
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/toolkit/crashreporter/tools/upload_symbols.py#45

I'm not sure if that actually fixes real-world use cases though, in that doing that means you will be able to run the existing in-tree symbol upload tasks, which are only scheduled to run after *nightly* build jobs. You can run nightly build jobs on try, and the symbol uploads will work (I tested the existing code on try by doing just that), but I don't think it's very common for developers to generate nightly builds on try.

We might want to also add some machinery that allows developers to upload symbols from *all* the builds in their try push, which probably involves an extra argument to `mach try` (maybe something like `--with-symbols`?), and then some taskgraph hacking to cause that to schedule symbol upload tasks after every build in the build graph. I don't think that should be very hard. Right now the upload-symbols task kind explicitly lists all the jobs it will run after:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/ci/upload-symbols/kind.yml#15

The very similar upload-generated-sources task uses `only-for-attributes: -nightly`:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/ci/upload-generated-sources/kind.yml#15

We could presumably either add a new attribute like `upload_symbols` to build tasks, or we could simply put the logic in the `upload_symbols` task transform:
https://dxr.mozilla.org/mozilla-central/rev/9a0655ea8ae02f4d96bf23a607a94641f1c57f1b/taskcluster/taskgraph/transforms/upload_symbols.py
See Also: → 1351014
Product: Core → Firefox Build System
Depends on: 1353296
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=ad9b0cef0a5266065cc1a97e6447fcbb36b58e17

So I pushed this to try as `./mach try -b do -p linux64,win64 -u none`, and then used Treeherder's "Add new jobs" feature to trigger the symbol-upload tasks for the builds I did after-the-fact, and they worked perfectly. You can see the win64-debug symbol upload, for example, here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f4c3adc2da94c2699ef338880c9a09032b3800b&selectedJob=173328919

I downloaded the win64-debug build:
https://queue.taskcluster.net/v1/task/W5mwVik4T8yNOb4CVNLwug/runs/0/artifacts/public/build/target.zip

Ran it in WinDBG on my Windows machine, set my symbol path to include `SRV*https://symbols.mozilla.org/try` and everything worked. Hooray!

Oh, I also had aki update the `project/releng/gecko/build/level-1/gecko-symbol-upload` secret with a new token I generated from my account on symbols.mo which has only the "Upload Try Symbols" permission, so any symbols uploaded with that token get stored separately on the symbol server.
Assignee: nobody → ted
Component: General → Task Configuration
Here's a diff of the full taskgraph with the second patch here applied (diffed against a tree with the patch from bug 
1353296):
https://gist.github.com/luser/3d20de933a4eb631b0bc7f5075f05a74

Here's a diff of the full taskgraph between the second and third patches here:
https://gist.github.com/luser/88386fa4d77bbbb5782e1ebc589fb3fa

There are a lot of new tasks there because we have a lot of build variants, as it turns out. Everything that's not a nightly is set to `run-on-projects: try`. I verified with aki's taskgraph-diff scripts that there are no differences in the target tasks for inbound/central nightlies/beta/release with this patch applied:

$ ls -l /tmp/output/bug1437577/total 70856
-rw-r--r-- 1 luser luser 14654088 Apr 12 12:07 beta-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 beta-parameters.json.diff
-rw-r--r-- 1 luser luser  2553727 Apr 12 12:07 central-nightly-android-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 central-nightly-android-parameters.json.diff
-rw-r--r-- 1 luser luser 28326169 Apr 12 12:07 central-nightly-desktop-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 central-nightly-desktop-parameters.json.diff
-rw-r--r-- 1 luser luser 15554618 Apr 12 12:07 inbound-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 inbound-parameters.json.diff
-rw-r--r-- 1 luser luser 11459643 Apr 12 12:07 release-parameters.json
-rw-r--r-- 1 luser luser        0 Apr 12 12:08 release-parameters.json.diff
Comment on attachment 8967508 [details]
bug 1437577 - preserve treeherder.platform from job into task.

https://reviewboard.mozilla.org/r/236182/#review242226
Attachment #8967508 - Flags: review?(dustin) → review+
Comment on attachment 8967509 [details]
bug 1437577 - allow uploading symbols from any build type on try.

https://reviewboard.mozilla.org/r/236184/#review242228

::: taskcluster/ci/upload-symbols/kind.yml:41
(Diff revision 1)
>          by-build-platform:
>              .*devedition.*: ['mozilla-beta', 'maple']
> -            default: ['all']
> +            # Only upload symbols for nightlies on most branches.
> +            .*(?<!-devedition)-nightly: ['all']
> +            # Allow symbol upload for any build type on try.
> +            default: ['try']

Is this just "allow", or will those uploads start occurring automatically for try builds that don't use try-select?
Attachment #8967509 - Flags: review?(dustin) → review+
Comment on attachment 8967507 [details]
bug 1437577 - Upload symbols from try server builds to symbols.mozilla.org instead of symbols.stage.mozaws.net.

https://reviewboard.mozilla.org/r/236180/#review242262

+1
Comment on attachment 8967507 [details]
bug 1437577 - Upload symbols from try server builds to symbols.mozilla.org instead of symbols.stage.mozaws.net.

https://reviewboard.mozilla.org/r/236180/#review242264

oops
Attachment #8967507 - Flags: review?(peterbe) → review+
Comment on attachment 8967509 [details]
bug 1437577 - allow uploading symbols from any build type on try.

https://reviewboard.mozilla.org/r/236184/#review242228

> Is this just "allow", or will those uploads start occurring automatically for try builds that don't use try-select?

I don't pretend to understand all the intricacies of try scheduling (there are a lot of ways it can work) but they won't get scheduled if you use trychooser syntax (I tested that with my push here) and they don't show up by default in `mach try fuzzy` because that uses a profile based on central (you have to `mach try fuzzy --full`. I can sanity check that triggering a build via `mach try fuzzy` doesn't schedule them by default.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #11)
> > Is this just "allow", or will those uploads start occurring automatically for try builds that don't use try-select?
> 
> I don't pretend to understand all the intricacies of try scheduling (there
> are a lot of ways it can work) but they won't get scheduled if you use
> trychooser syntax (I tested that with my push here) and they don't show up
> by default in `mach try fuzzy` because that uses a profile based on central
> (you have to `mach try fuzzy --full`. I can sanity check that triggering a
> build via `mach try fuzzy` doesn't schedule them by default.

I lost track of this in the midst of traveling for a work week and some other things, so I've rebased atop latest mozilla-central and done a try push for sanity's sake and if that looks good I will land these patches.
Looking at the task graph of that try push it looks sensible, so I'm going to land this.
https://hg.mozilla.org/integration/mozilla-inbound/rev/8483a61d426f758f013e78eb1addc667c4b8c673
bug 1437577 - Upload symbols from try server builds to symbols.mozilla.org instead of symbols.stage.mozaws.net. r=peterbe

https://hg.mozilla.org/integration/mozilla-inbound/rev/01511447fa7ed82228fb8ffaa3b1850bb7429a8c
bug 1437577 - preserve treeherder.platform from job into task. r=dustin

https://hg.mozilla.org/integration/mozilla-inbound/rev/a0b66c435f62a6c3f3fd68c9e4050477a778bd86
bug 1437577 - allow uploading symbols from any build type on try. r=dustin
Depends on: 1460243
You need to log in before you can comment on or make changes to this bug.