Closed Bug 1403548 Opened 2 years ago Closed 2 years ago

Updates for ASan reporter Nightly build

Categories

(Release Engineering :: Custom Release Requests, defect, major)

x86_64
Linux
defect
Not set
major

Tracking

(firefox61 fixed)

RESOLVED FIXED
Tracking Status
firefox61 --- fixed

People

(Reporter: decoder, Assigned: catlee)

References

(Blocks 1 open bug)

Details

(Keywords: sec-want)

Attachments

(7 files)

As discussed during the last All Hands, we need updates for a special kind of AddressSanitizer build that we now have in TC under gecko.v2.mozilla-central.latest.firefox.linux64-asan-reporter-opt. This is for the ASan Nightly project that we started in Q3 already.

We need those updates only for Linux, 64-bit, english language.

I'm still working on the server-side proxy parts in parallel but we should be able to wrap this up in Q4.

Chris, can you help me find someone to look at this? Thanks!
Flags: needinfo?(catlee)
We're not going to be able to spend time on this in Q4 unfortunately. However, we would be happy to give you some pointers for where to get started.

Off the top of my head:

- We need to change what the build identifies itself as to the update server. Perhaps changing the build platform identifier would work?

- Need to add the build & update generation + signing logic to the nightly task generation. The entry point for this code is here: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py#371
Flags: needinfo?(catlee)
Depends on: 1428357
Bulk change of QA Contact to :jlund, per https://bugzilla.mozilla.org/show_bug.cgi?id=1428483
QA Contact: catlee → jlund
Attachment #8946855 - Flags: review?(bhearsum)
Comment on attachment 8946855 [details] [diff] [review]
add linux64-asan to update_platform_map

Review of attachment 8946855 [details] [diff] [review]:
-----------------------------------------------------------------

another option is to call these 'linux64-asan-reporter', since that's what the build actually is
Comment on attachment 8946855 [details] [diff] [review]
add linux64-asan to update_platform_map

Don't forget to update balrogworker checkouts!
Attachment #8946855 - Flags: review?(bhearsum) → review+
Attachment #8946855 - Flags: checked-in+
Testing on oak:
https://www.torontopubliclibrary.ca/detail.jsp?Entt=RDMEVT312807&R=EVT312807

Looks like the builds aren't producing crash symbols, causing beetmover to fail.
(In reply to Chris AtLee [:catlee] from comment #7)
> Testing on oak:
> https://www.torontopubliclibrary.ca/detail.jsp?Entt=RDMEVT312807&R=EVT312807
> 
> Looks like the builds aren't producing crash symbols, causing beetmover to
> fail.

ASan builds don't use separate crash symbols and have the crash reporter disabled.
Chris, how do we proceed here?
Flags: needinfo?(catlee)
I just synced up with Julien, and he recommends using a distinct channel for updates to prevent possibly updating regular nightly users to these asan builds. I'll start updating my patches to append -asan to the update channel, and also we need to bump the limit for mar filesize signing.
Flags: needinfo?(catlee)
Depends on: 1438967
(In reply to Chris AtLee [:catlee] from comment #7)
> Testing on oak:
> https://www.torontopubliclibrary.ca/detail.jsp?Entt=RDMEVT312807&R=EVT312807

Opps, copy/paste fail. The builds are here: https://treeherder.mozilla.org/#/jobs?repo=oak&filter-searchStr=nightly

Builds look like they're being produced, signed and published. However, it looks like they're conflicting with the generic linux64 builds.
Attachment #8962689 - Flags: checked-in+
Attached patch tools-asan.diffSplinter Review
Attachment #8964623 - Flags: review?(bhearsum)
Attachment #8964623 - Flags: review?(bhearsum) → review+
Attachment #8964623 - Flags: checked-in+
and with that, it looks like updates are working on the 'oak' branch.

you should be able to download https://queue.taskcluster.net/v1/task/JJkERZ5YSGukhQLFjmr1Dg/runs/0/artifacts/public/build/target.tar.bz2 and have it update to the latest nightly builds on oak.
The latest builds on 'oak' have the update channel set to 'nightly-oak' instead of 'nightly-oak-asan'. This shouldn't impact testing.
Assignee: nobody → catlee
Comment on attachment 8969279 [details]
Bug 1403548: Enable nightly asan reportor builds and updates

https://reviewboard.mozilla.org/r/238006/#review243688

I lean toward asking you to hold off landing on central based trees until Bug 1440546 lands to avoid bitrotting that one. But overall looks good.

::: taskcluster/taskgraph/transforms/beetmover.py:149
(Diff revision 1)
>      ],
>      'linux-devedition-nightly': _DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_EN_US + [
>          "host/bin/mar",
>          "host/bin/mbsdiff",
>      ],
> +    'linux64-asan-reporter-nightly':

:shudder: but I guess this is the best we can do right now.

Maybe an inline comment though?
Attachment #8969279 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8969279 [details]
Bug 1403548: Enable nightly asan reportor builds and updates

https://reviewboard.mozilla.org/r/238006/#review243688

good call. I'll make sure to wait for that to land first :)
Comment on attachment 8969279 [details]
Bug 1403548: Enable nightly asan reportor builds and updates

https://reviewboard.mozilla.org/r/238006/#review243688

> :shudder: but I guess this is the best we can do right now.
> 
> Maybe an inline comment though?

fixed in a the followup commit
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a63caceffcee
Enable nightly asan reportor builds and updates r=Callek
Backed out changeset a63caceffcee (bug 1403548) for liniting failure on builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_repackage.py. CLOSED TREE 

Log:
[task 2018-04-25T20:51:10.853Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/beetmover_repackage.py:133:5 | continuation line missing indentation or outdented (E122)
[taskcluster 2018-04-25 20:51:11.379Z] === Task Finished ===
[taskcluster 2018-04-25 20:51:11.380Z] Unsuccessful task run with exit code: 1 completed in 284.875 seconds

https://treeherder.mozilla.org/logviewer.html#?job_id=175593482&repo=autoland&lineNumber=255

Push with the failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a63caceffcee5cbf5f6a3043b70f142bbbeab12c

Backout:
https://hg.mozilla.org/integration/autoland/rev/2df9e20ea6c0aee71514c217c8efb95c2f90de5d
Flags: needinfo?(catlee)
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7f1909255cc9
Enable nightly asan reportor builds and updates r=Callek
https://hg.mozilla.org/mozilla-central/rev/7f1909255cc9
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/343c66d9abb2
filter out asan on beta+release. r=catlee
Flags: needinfo?(catlee)
You need to log in before you can comment on or make changes to this bug.