Closed
Bug 1259832
Opened 8 years ago
Closed 7 years ago
Save generated files so Socorro can link to them
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(firefox57 fixed)
RESOLVED
FIXED
mozilla57
Tracking | Status | |
---|---|---|
firefox57 | --- | fixed |
People
(Reporter: blassey, Assigned: ted)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
Attachments
(4 files)
A lot of the crashes I've looked at recently have been occurring in generated code in the object directory. For example, this crash report [https://crash-stats.mozilla.com/report/index/e56984e3-1431-4858-bf44-612e52160312] unhelpfully points to https://hg.mozilla.org/releases/mozilla-beta/annotate/fb3494d06dfb/obj-firefox/ipc/ipdl/PHttpChannelChild.cpp#l1080. It would be great if we could save these generated files somewhere and link to that instead.
Comment 1•8 years ago
|
||
The "obvious" suggestion would be for dxr to build all the revisions we use for release-type builds, and soccoro to point there. Erik, how possible would that be on dxr side?
Flags: needinfo?(erik)
Assignee | ||
Comment 2•8 years ago
|
||
I've wanted to do this for a while, I've just never come up with a good plan. We'd mostly just need to get the list of generated files from the build system and upload them somewhere, and confer some of that information to the symbolstore script so that it can make the FILE lines have enough information to generate the links.
Reporter | ||
Comment 3•8 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2) > I've wanted to do this for a while, I've just never come up with a good > plan. We'd mostly just need to get the list of generated files from the > build system and upload them somewhere, and confer some of that information > to the symbolstore script so that it can make the FILE lines have enough > information to generate the links. in symbolstore script it should just be anything with obj-firefox in the path
Comment 4•8 years ago
|
||
Mike, we do index generated files, e.g. https://dxr.mozilla.org/mozilla-central/source/obj-x86_64-unknown-linux-gnu/ipc/ipdl/PHttpChannelChild.cpp. And we just added Aurora, Beta, Release, and ESR45. Does that suffice?
Flags: needinfo?(erik)
Comment 5•8 years ago
|
||
(In reply to Erik Rose [:erik][:erikrose] from comment #4) > Mike, we do index generated files, e.g. > https://dxr.mozilla.org/mozilla-central/source/obj-x86_64-unknown-linux-gnu/ > ipc/ipdl/PHttpChannelChild.cpp. And we just added Aurora, Beta, Release, and > ESR45. Does that suffice? That's a good start. Ideally, we should have the exact revisions shipped to users indexed. Can we ensure that? Now, an interesting additional question is: do we have build-time generated code that varies per platform?
Flags: needinfo?(erik)
Comment 6•8 years ago
|
||
We can set up a DXR tree for each release tag. It should cost us around 14GB of elasticsearch space per tag, but the CPU impact should be negligible, since the code doesn't change. I expect the tags could eventually rotate out after a year or so--sane? How does all that sound to you? We do index only x86_64 Linux right now, as (1) our analysis shims are tied to clang and (2) we haven't figured out how to merge indices across multiple builds yet. If that's going to be a showstopper, please say so.
Flags: needinfo?(erik)
Assignee | ||
Comment 7•8 years ago
|
||
(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #3) > in symbolstore script it should just be anything with obj-firefox in the path I don't think that's *quite* sufficient, unfortunately. And yeah, I'm not sure that we don't have generated code that varies by platform.
Reporter | ||
Comment 8•8 years ago
|
||
it seems to me that dxr might be the wrong direction then. I'm thinking the right thing to do is to markup the produced source and save it as a build artifact in ftp.
Comment 10•7 years ago
|
||
Note not having this is particularly annoying for Stylo because we have lots of code generated for property parsing, computation, animation, etc. Many of our crash reports end in functions inside generated files.
Assignee | ||
Comment 11•7 years ago
|
||
(In reply to Xidorn Quan [:xidorn] UTC+10 from comment #10) > Note not having this is particularly annoying for Stylo because we have lots > of code generated for property parsing, computation, animation, etc. Many of > our crash reports end in functions inside generated files. Can you show me an example of these generated files? I'm working on a patch in bug 1384568 and I want to make sure it captures this use case.
Assignee | ||
Comment 12•7 years ago
|
||
dmajor pointed out properties.rs, which is generated from this stylo build script: https://dxr.mozilla.org/mozilla-central/source/servo/components/style/build.rs using this Python script: https://dxr.mozilla.org/mozilla-central/source/servo/components/style/properties/build.py Unfortunately the Gecko build system doesn't know anything about what cargo build scripts do internally. :-(
Comment 13•7 years ago
|
||
I think Gecko build system can just try to keep all files in objdir/toolkit/library/<target-triplet>/{debug,release}/build/*/out. That is the place build scripts put generated files by default.
Assignee | ||
Comment 14•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5944a5aed51eb35771fec8874c6f4ae32c696ea
Assignee | ||
Comment 15•7 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #14) > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=a5944a5aed51eb35771fec8874c6f4ae32c696ea This try push implements one of the two parts I'm intending to use to fix this. It produces a `generated-sources.tar.gz` artifact from builds. The second part will involve adding a new task to fetch that artifact and upload the individual files within to a new S3 bucket.
Assignee | ||
Comment 16•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c72e11f1e5fc9390d448544150123b9518120fc0
Assignee | ||
Comment 17•7 years ago
|
||
OK, I think I have this working, I just need to try actually executing the task. I took the parameters.yml from a m-c build that included nightlies: https://queue.taskcluster.net/v1/task/au1EPpAFR6-Fl1ftKeLWSw/runs/0/artifacts/public/parameters.yml and ran `./mach taskgraph tasks --json` with it, and then applied my patch and ran it again. This is the diff between the two outputs: https://gist.github.com/anonymous/10a95d1ee5e8370fb97f963ae234b8d7 I also ran the upload_generated_sources.py script locally with an S3 bucket I set up for testing and it was able to upload the files successfully. Once I get builds for the try push in the previous comment I'm going to manually create a task using the task-creator and the task definitions from running `mach taskgraph tasks` locally, hand-editing the task definition to point at the artifacts from my try push, so I can verify that the upload works when fetching AWS creds from the secrets service.
Assignee | ||
Comment 18•7 years ago
|
||
Looking at my try push I realize that: + "task-reference": "public/build/target.generated-sources.tar.gz@<build>" is incorrect, the first patch produces target.generated-files.tar.gz.
Assignee | ||
Comment 19•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=359935d962e788a401e912d454dfcdacb7ea1c38
Assignee | ||
Comment 20•7 years ago
|
||
My `upload_generated_sources.py` script didn't actually work--I forgot to make it activate a virtualenv since in this context it will be running without an existing objdir. I fixed that (and tested by running it locally without an existing objdir) and did another try push.
Assignee | ||
Comment 21•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=81c69a731873bf311f912ebec0fc22c1fc783c5d
Assignee | ||
Comment 22•7 years ago
|
||
...so I also was missing the `--bucket` in the task commandline, and I had the TC secrets URL wrong in the script. It's a wonder I ever write code that works.
Assignee | ||
Updated•7 years ago
|
Assignee: nobody → ted
Assignee | ||
Comment 23•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2fcb61e8de4cbcfd53cf852324beefc8ab9bea83
Assignee | ||
Comment 24•7 years ago
|
||
Hooray, this seems to have succeeded: https://tools.taskcluster.net/groups/d6n5BLL2QoOZtRn2OZNQlA/tasks/d6n5BLL2QoOZtRn2OZNQlA/details
Assignee | ||
Comment 25•7 years ago
|
||
Pulling an arbitrary file out of the log shows that it's available: https://gecko-generated-sources.s3.amazonaws.com/318c1e8eb64b13914272e71f5c084728437cbe739235e89199ff606667d7f6a80e6e65e1da4a226a24737b4c03cdba307fc820c1a0222e929eb4906aba073e48/dom/bindings/AudioNodeBinding.cpp Fetching it in a CORS scenario seems to work fine as well: https://codepen.io/tedmielczarek/pen/mMWgNw
Assignee | ||
Comment 26•7 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #16) > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=c72e11f1e5fc9390d448544150123b9518120fc0 One thing I forgot to mention--this push broke artifact builds because they're trying to run the package-generated-sources build step. I need to make them skip that.
Assignee | ||
Comment 27•7 years ago
|
||
I put together a tiny webapp with flask+pygments to fetch source files from S3 and render them, and am running an instance on Heroku: https://github.com/luser/source-highlight Seems to do a reasonable job: https://source-highlight.herokuapp.com/https://gecko-generated-sources.s3.amazonaws.com/318c1e8eb64b13914272e71f5c084728437cbe739235e89199ff606667d7f6a80e6e65e1da4a226a24737b4c03cdba307fc820c1a0222e929eb4906aba073e48/dom/bindings/AudioNodeBinding.cpp#l-309 https://source-highlight.herokuapp.com/https://gecko-generated-sources.s3.amazonaws.com/83f488c3b911586a0f8658ce5d35980ab60caa4bc5d60d3abe176ce7a4f01f4db9c67174f94fd183f04418b9ef15357aab37a8a3f79fe8a80ddc612aab724866/toolkit/library/i686-unknown-linux-gnu/release/build/webrender-a0ef86f41f0206ac/out/shaders.rs
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 32•7 years ago
|
||
OK! I think this is all good to go. I triggered a try push on that review request just to make sure I didn't break anything with my recent changes, and I'll manually trigger an upload task when I get a build out of it. I filed a pair of followups: * bug 1389217 to implement the actual links in Socorro, which should be trivial * bug 1389225 to make source server indexing work for these files, so Windows debuggers can fetch them
Assignee | ||
Comment 33•7 years ago
|
||
The builds on that try push failed because I shadowed a variable in the code I added in symbolstore.py which only broke one thing: the printing of symbol file names. That broke the functional test.
Assignee | ||
Comment 34•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0eeae57188be1f6f13c609be31a899e4bf7b3994
Comment 35•7 years ago
|
||
mozreview-review |
Comment on attachment 8895974 [details] bug 1259832 - add a post-build task to upload generated source files. https://reviewboard.mozilla.org/r/167244/#review172774 I made some changes in bug 1382729 to, among other things, use $MOZ_SCM_LEVEL here. Given the backouts I saw on that bug, I think it's wise to adopt those changes here, too, before landing. I can set up some buckets and secrets for that purpose.
Attachment #8895974 -
Flags: review?(dustin) → review+
Comment 36•7 years ago
|
||
buckets / users / secrets created
Comment 37•7 years ago
|
||
mozreview-review |
Comment on attachment 8895972 [details] bug 1259832 - move artifact build automation step overrides to a common mozconfig. https://reviewboard.mozilla.org/r/167240/#review172844
Attachment #8895972 -
Flags: review?(cmanchester) → review+
Comment 38•7 years ago
|
||
mozreview-review |
Comment on attachment 8895975 [details] bug 1259832 - replace generated source file names in symbol files. https://reviewboard.mozilla.org/r/167246/#review172846
Attachment #8895975 -
Flags: review?(cmanchester) → review+
Assignee | ||
Comment 39•7 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] from comment #35) > Comment on attachment 8895974 [details] > bug 1259832 - add a post-build task to upload generated source files. > > https://reviewboard.mozilla.org/r/167244/#review172774 > > I made some changes in bug 1382729 to, among other things, use > $MOZ_SCM_LEVEL here. Given the backouts I saw on that bug, I think it's > wise to adopt those changes here, too, before landing. > > I can set up some buckets and secrets for that purpose. I agree. (In reply to Dustin J. Mitchell [:dustin] from comment #36) > buckets / users / secrets created Thanks!
Assignee | ||
Comment 40•7 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #34) > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=0eeae57188be1f6f13c609be31a899e4bf7b3994 This was mostly green, but the Windows builds failed my new test in unit-symbolstore.py because I used `os.path.join` and wound up with unexpected backslashes. I'll change that to use `mozpath.join` instead.
Comment 41•7 years ago
|
||
New buckets are gecko-generated-sources-l1 and gecko-generated-sources-l2. The l3 bucket hasn't changed - no suffix.
Comment 42•7 years ago
|
||
mozreview-review |
Comment on attachment 8895973 [details] bug 1259832 - package generated sources and upload them along with other build artifacts. https://reviewboard.mozilla.org/r/167242/#review174104 ::: python/mozbuild/mozpack/generated_sources.py:1 (Diff revision 1) > +# This Source Code Form is subject to the terms of the Mozilla Public This file should not be in `mozpack`, since `mozpack` is somewhat generic. ::: python/mozbuild/mozpack/generated_sources.py:7 (Diff revision 1) > +# License, v. 2.0. If a copy of the MPL was not distributed with this > +# file, You can obtain one at http://mozilla.org/MPL/2.0/. > + > +import json > +import os > +import os.path Nit: no need to `import os.path` when there is an `import os`. ::: python/mozbuild/mozpack/generated_sources.py:21 (Diff revision 1) > + in this objdir, where `file` is either an absolute path to the file or > + a `mozpack.File` instance. > + ''' > + # First, get the list of generated sources produced by the build backend. > + f = os.path.join(buildconfig.topobjdir, 'generated-sources.json') > + data = json.load(open(f, 'rb')) Nit: always use context manager form of `open()` to ensure file descriptors are closed immediately. ::: python/mozbuild/mozpack/generated_sources.py:27 (Diff revision 1) > + for p, f in finder.find('**/*.h'): > + yield os.path.join(base, p), f I'm really not a fan of doing a filesystem walk here. That I/O can be expensive. Can we not derive the list of files from moz.build metadata? I'd rather we generate the list of relevant files at backend generation time and then consume that list from this script. That being said, since the filesystem walking seems to be limited to specific sub-directories, I'm willing to turn a blind eye to I/O perf concerns.
Attachment #8895973 -
Flags: review?(gps) → review-
Assignee | ||
Comment 43•7 years ago
|
||
mozreview-review-reply |
Comment on attachment 8895973 [details] bug 1259832 - package generated sources and upload them along with other build artifacts. https://reviewboard.mozilla.org/r/167242/#review174104 > This file should not be in `mozpack`, since `mozpack` is somewhat generic. OK, I originally only had the `get_generated_files` function here, which seemed fairly generic, but I stuffed a few other things in here in the interim so I can put this elsewhere. > I'm really not a fan of doing a filesystem walk here. That I/O can be expensive. Can we not derive the list of files from moz.build metadata? I'd rather we generate the list of relevant files at backend generation time and then consume that list from this script. > > That being said, since the filesystem walking seems to be limited to specific sub-directories, I'm willing to turn a blind eye to I/O perf concerns. Per bug 1384568 comment 1, the moz.build emitter doesn't have any visibility into these specific cases. I tried to limit it to the couple of very specific directories where we had generated source files that aren't captured in generated-sources.json. The ipdlheaders one shouldn't be too bad, I think we effectively want to capture every file in that directory. The Rust sources one isn't great, but the directory tree isn't very deep from the point where we're walking, and stylo builds have some important generated sources in there that we really do want to capture. I'd love to have a more declarative solution here, but unfortunately we just don't have it right now.
Assignee | ||
Comment 44•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=82c5b0feee7e33d4503f9451a6f78a438837ebc6
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 49•7 years ago
|
||
Comment on attachment 8895974 [details] bug 1259832 - add a post-build task to upload generated source files. dustin: if you could give this a quick second look I'd appreciate it. I changed it to use the bucket for the SCM level of the task. I just pushed to try and requested Nightly builds, so we'll see if that actually works.
Attachment #8895974 -
Flags: review+ → review?(dustin)
Assignee | ||
Comment 50•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=63df6882e0552ede644082e6a3646ee2025a2f44
Assignee | ||
Comment 51•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8705abc19ed7b7df7a45bb67e5724c13d8d23ea4
Assignee | ||
Comment 52•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3d4b988694b1bc314f98f0e839efef14a0349a33
Comment 53•7 years ago
|
||
mozreview-review |
Comment on attachment 8895973 [details] bug 1259832 - package generated sources and upload them along with other build artifacts. https://reviewboard.mozilla.org/r/167242/#review174246
Attachment #8895973 -
Flags: review?(gps) → review+
Assignee | ||
Comment 54•7 years ago
|
||
dustin: did a project/releng/gecko/build/level-1/gecko-generated-sources-upload secret actually get created? I don't see it in the secrets manager, and my Ugs task on that last try push 404ed on it: https://treeherder.mozilla.org/logviewer.html#?job_id=123391630&repo=try&lineNumber=81 I see project/releng/gecko/build/level-{2,3}/gecko-generated-sources-upload in the secrets manager.
Flags: needinfo?(dustin)
Comment 55•7 years ago
|
||
Sorry, I think I didn't set the expiration on the secret correctly and it expired. I re-created it.
Flags: needinfo?(dustin)
Comment 56•7 years ago
|
||
mozreview-review |
Comment on attachment 8895974 [details] bug 1259832 - add a post-build task to upload generated source files. https://reviewboard.mozilla.org/r/167244/#review174514
Attachment #8895974 -
Flags: review?(dustin) → review+
Assignee | ||
Comment 57•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dece32af44a89517fc24f633652f89121b1b72c8
Assignee | ||
Comment 58•7 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #57) > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=dece32af44a89517fc24f633652f89121b1b72c8 I did in fact get green Ugs tasks on this try push! (The Win32 build timed out, but it also took 49 minutes(!) to clone the repo, so that's not my fault.) However, looking at the log from the Win64 Ugs task I noticed: [task 2017-08-16T18:55:05.434551Z] INFO - Thread-1 - Uploading "6b2e6cfe5e37b0e2446a577fea1df1922d4fb7345e2716f5c994de35f3d5562b6792a16aedcb14d33659f9c68b5713f335e03d7865e0c519cd421e604cf6a9f2/toolkit/library\x86_64-pc-windows-msvc\release\build\style-b085cfb8bb9e43c9/out/properties.rs" (423127 bytes) and sure enough, downloading the target.generated-files.tar.gz and looking at it: $ tar tzf target.generated-files.tar.gz | grep properties\.rs toolkit/library\\x86_64-pc-windows-msvc\\release\\build\\style-b085cfb8bb9e43c9/out/gecko_properties.rs toolkit/library\\x86_64-pc-windows-msvc\\release\\build\\style-b085cfb8bb9e43c9/out/properties.rs I messed up the path separators in the archive generation. :-( Fixing that shouldn't be terrible, I will fix that and do a Windows-only try push to sanity check.
Assignee | ||
Comment 59•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=78344e2435cc2379d4b202b71851d1112236066b
Assignee | ||
Comment 60•7 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #59) > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=78344e2435cc2379d4b202b71851d1112236066b That looks better: [task 2017-08-16T23:31:22.397418Z] INFO - Thread-1 - Uploading "7ccab0fcaada36876ba2d7bf686c8b77907ab791fea2e3e9c87347d1e828ec147b44a24b42b8af3abca072f03b635194b73a24f6b11f3c3a552e5be4782514b4/toolkit/library/i686-pc-windows-msvc/release/build/style-853b9d8a7b7bd19c/out/gecko/bindings.rs" (12783 bytes)
Assignee | ||
Comment 61•7 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/a02695cbf5762eb0eb7087239319807eb447ca1e bug 1259832 - move artifact build automation step overrides to a common mozconfig. r=chmanchester https://hg.mozilla.org/integration/mozilla-inbound/rev/14d18d7cf454c4c3d0f6d49d1d01660e06e4be4b bug 1259832 - package generated sources and upload them along with other build artifacts. r=gps https://hg.mozilla.org/integration/mozilla-inbound/rev/73bf88110b3821d62a3d393e85b56896a12f2930 bug 1259832 - add a post-build task to upload generated source files. r=dustin https://hg.mozilla.org/integration/mozilla-inbound/rev/7781a37a4db0378b06ef3ad377e18be37e63ad83 bug 1259832 - replace generated source file names in symbol files. r=chmanchester
Backed these out for apparently scheduling nightlies on inbound for every push and platform https://hg.mozilla.org/integration/mozilla-inbound/rev/82d8999c065b
Flags: needinfo?(ted)
Something in this or the other bug you pushed busted android checkstyle,findbugs,lint,"test" jobs like https://treeherder.mozilla.org/logviewer.html#?job_id=123725305&repo=mozilla-inbound
Assignee | ||
Comment 64•7 years ago
|
||
(In reply to Wes Kocher (:KWierso) from comment #63) > Something in this or the other bug you pushed busted android > checkstyle,findbugs,lint,"test" jobs like > https://treeherder.mozilla.org/logviewer.html#?job_id=123725305&repo=mozilla- > inbound Thanks. Looks like I need to skip the new automation build step here as well: https://dxr.mozilla.org/mozilla-central/source/mobile/android/config/mozconfigs/android-api-15-frontend/nightly?q=path%3Amozconfigs%2Fandroid-api-15-frontend%2Fnightly&redirect_type=single#1 Not sure why I didn't notice this on try.
Flags: needinfo?(ted)
Assignee | ||
Comment 65•7 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0cb71d746a3997138e0f516c6e63a049fe0071df
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 70•7 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #64) > Not sure why I didn't notice this on try. I'm going to go with "because pushing to try with -b do -p all does not schedule those jobs". :-(
Assignee | ||
Comment 71•7 years ago
|
||
mozreview-review |
Comment on attachment 8895974 [details] bug 1259832 - add a post-build task to upload generated source files. https://reviewboard.mozilla.org/r/167244/#review175082 ::: taskcluster/ci/upload-generated-sources/kind.yml:20 (Diff revisions 2 - 3) > only-for-attributes: > - nightly > > job-template: > description: Upload generated source files from build > + attributes: This was the fix to prevent these tasks from being scheduled on inbound, and thus scheduling nightly builds for their own use. ::: taskcluster/taskgraph/transforms/upload_generated_sources.py:32 (Diff revisions 2 - 3) > # Label the job to match the build task it's uploading from. > job['label'] = dep_task.label.replace("build-", "upload-generated-sources-") > - # Copy over some bits of treeherder metdata from the build task. > + # Copy over some bits of metdata from the build task. > dep_th = dep_task.task['extra']['treeherder'] > + job.setdefault('attributes', {}) > + job['attributes']['build_platform'] = dep_task.attributes.get('build_platform') This fix was necessary to get these jobs scheduled from the actual nightly build cron job scheduler. In prior testing I had forced the tasks to run on try (which worked), and verified that they showed up in the output of `mach taskgraph tasks`, which lists *all* tasks, but neither of those were sufficient to verify that they would get scheduled along with actual nightly builds in automation. I needed to use `mach taskgraph target` with a parameters.yml from a nightly cron decision task to verify this.
Assignee | ||
Comment 72•7 years ago
|
||
mozreview-review |
Comment on attachment 8895973 [details] bug 1259832 - package generated sources and upload them along with other build artifacts. https://reviewboard.mozilla.org/r/167242/#review175086 I had to add `MOZ_AUTOMATION_PACKAGE_GENERATED_SOURCES=0` to some mobile mozconfigs. For mobile artifact builds I simply included the mozconfig.artifact.automation I added in the earlier patch (those changes could have gone into the earlier patch, but here they are). For the couple of other one-off build types I simply included it inline.
Assignee | ||
Comment 73•7 years ago
|
||
Comment on attachment 8895973 [details] bug 1259832 - package generated sources and upload them along with other build artifacts. Pretty minimal additional changes here to just skip the new automation build step in some Android builds that I missed because they don't run on try.
Attachment #8895973 -
Flags: review+ → review?(gps)
Assignee | ||
Comment 74•7 years ago
|
||
Comment on attachment 8895974 [details] bug 1259832 - add a post-build task to upload generated source files. dustin: I made two changes to this patch from the previous patch, I noted them both in mozreview.
Attachment #8895974 -
Flags: review+ → review?(dustin)
Comment 75•7 years ago
|
||
mozreview-review |
Comment on attachment 8895973 [details] bug 1259832 - package generated sources and upload them along with other build artifacts. https://reviewboard.mozilla.org/r/167242/#review175356
Attachment #8895973 -
Flags: review?(gps) → review+
Comment 76•7 years ago
|
||
mozreview-review |
Comment on attachment 8895974 [details] bug 1259832 - add a post-build task to upload generated source files. https://reviewboard.mozilla.org/r/167244/#review175502
Attachment #8895974 -
Flags: review?(dustin) → review+
Assignee | ||
Comment 77•7 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/7fffe26efe40f211687e6cefe8b166e35835ef3a bug 1259832 - move artifact build automation step overrides to a common mozconfig. r=chmanchester https://hg.mozilla.org/integration/mozilla-inbound/rev/1f1dd3ab877d9733604aee3a2bcb66eccf70eb9e bug 1259832 - package generated sources and upload them along with other build artifacts. r=gps https://hg.mozilla.org/integration/mozilla-inbound/rev/4a6df9405fb5b6a89373e71e4118fc03712c9458 bug 1259832 - add a post-build task to upload generated source files. r=dustin https://hg.mozilla.org/integration/mozilla-inbound/rev/3cd6728702e32f02a1201c4d83aa60676fc902d7 bug 1259832 - replace generated source file names in symbol files. r=chmanchester
Assignee | ||
Comment 78•7 years ago
|
||
This doesn't look like it's doing anything dumb on inbound, which is good. We'll have to wait for it to get merged to central and for nightlies to be built with it included to see if it actually does what I want. Given the success of my recent try pushes (most recently in comment 65) I have high confidence that that part will work.
Comment 79•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/7fffe26efe40 https://hg.mozilla.org/mozilla-central/rev/1f1dd3ab877d https://hg.mozilla.org/mozilla-central/rev/4a6df9405fb5 https://hg.mozilla.org/mozilla-central/rev/3cd6728702e3
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox57:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Assignee | ||
Comment 80•7 years ago
|
||
This is mostly-working, in that there were Ugs tasks scheduled for the nightly builds yesterday, like: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=nightly&selectedJob=124377496 However, having these frames show up in crash stacks breaks Socorro's report/index page (oops), which peterbe is fixing in bug 1389217. Also, after looking at the logs/symbols from those nightlies, I don't think this is working properly for Windows builds right now. My best guess is that it's an issue with backslashes. I'll file a followup to fix that, and I've already done a local build to try it out on Windows.
Comment 81•7 years ago
|
||
Pushed by mozilla@jorgk.com: https://hg.mozilla.org/comm-central/rev/a0f8d0262f8b Keep build files in sync (Port bug 1259832: package generated sources and upload them along with other build artifacts). rs=bustage-fix
Updated•6 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•