Bug 863846 (asan-maintenance)

Maintain Firefox ASan build process and compatibility

NEW
Assigned to

Status

mozilla.org
Security Assurance: Applications
5 years ago
2 years ago

People

(Reporter: decoder, Assigned: decoder)

Tracking

(Depends on: 8 bugs)

Details

(Assignee)

Description

5 years ago
This bug is meant for tracking all efforts related to keeping our ASan builds working and compatible with latest Firefox and ASan features.
(Assignee)

Updated

5 years ago
Alias: asan-maintenance
(Assignee)

Updated

5 years ago
Depends on: 864008
(Assignee)

Updated

5 years ago
Depends on: 864023
(Assignee)

Updated

5 years ago
Depends on: 857189
(Assignee)

Updated

5 years ago
Depends on: 844088
(Assignee)

Comment 1

5 years ago
Also this bug will be tracking test failures (either due to incompatibilities or bugs) since we want the tests to be working and green.
(Assignee)

Updated

5 years ago
Depends on: 843923
(Assignee)

Updated

5 years ago
Depends on: 787715
(Assignee)

Updated

5 years ago
Depends on: 750932
(Assignee)

Updated

5 years ago
Depends on: 860867
Summary: Maintain ASan build process and compatibility → Maintain Firefox ASan build process and compatibility
(Assignee)

Updated

5 years ago
Depends on: 865921
(Assignee)

Updated

5 years ago
Depends on: 866525
(Assignee)

Updated

5 years ago
Depends on: 868481
(Assignee)

Updated

5 years ago
Depends on: 872565
(Assignee)

Updated

5 years ago
Depends on: 872577
(Assignee)

Updated

5 years ago
Depends on: 874486
(Assignee)

Updated

5 years ago
Depends on: 874527
(Assignee)

Updated

5 years ago
Depends on: 876717
(Assignee)

Updated

5 years ago
Depends on: 880193
(Assignee)

Updated

5 years ago
Depends on: 880239
(Assignee)

Updated

5 years ago
Depends on: 886842
(Assignee)

Updated

5 years ago
Depends on: 892017
(Assignee)

Updated

5 years ago
Depends on: 895845
(Assignee)

Updated

5 years ago
Depends on: 873649
(Assignee)

Updated

5 years ago
Depends on: 898230

Comment 2

5 years ago
Hi,
What are these builds specifically for?
If we find a regression, how hard is it to reproduce it locally and do bisections?

The reason I ask is because I see these jobs broken and hidden on tbpl.
This is very costly and I wonder if the per-checkin approach was appropriate for this specific type of jobs. There are other options we can consider.

If we're still trying to get them green; would it make sense to only run them on the try server? or a project branch?
(Assignee)

Comment 3

5 years ago
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #2)
> Hi,
> What are these builds specifically for?

These builds are for security testing.

> If we find a regression, how hard is it to reproduce it locally and do
> bisections?

Locally reproducing these issues is easy most of the time, bisecting the issue is usually a bit harder.

> 
> The reason I ask is because I see these jobs broken and hidden on tbpl.
> This is very costly and I wonder if the per-checkin approach was appropriate
> for this specific type of jobs. There are other options we can consider.

What other options would these be? We eventually want these to be even per checkin on mozilla-inbound, and on other branches. Also we're working on running all our regressions tests for these. Furthermore, these builds are also actively used by other people.

> 
> If we're still trying to get them green; would it make sense to only run
> them on the try server? or a project branch?

No, we need these to find security regressions on mozilla-central. And the fact that they're orange right now doesn't matter at all, it's a silly make check failure due to the Clang update, but the failure is not affecting the functionality of the builds.
(Assignee)

Comment 4

5 years ago
Just a correction, seems like the builds are really busted right now, I'll look into that (I was working on the orange due to the Clang update in the last few days).
(Assignee)

Updated

5 years ago
Depends on: 898484
(Assignee)

Comment 5

5 years ago
Filed bug 898484 for the build bustage and investigating :)

Comment 6

5 years ago
(In reply to Christian Holler (:decoder) from comment #3) 
> > The reason I ask is because I see these jobs broken and hidden on tbpl.
> > This is very costly and I wonder if the per-checkin approach was appropriate
> > for this specific type of jobs. There are other options we can consider.
> 
> What other options would these be? We eventually want these to be even per
> checkin on mozilla-inbound, and on other branches. Also we're working on
> running all our regressions tests for these. Furthermore, these builds are
> also actively used by other people.
> 

Are we expecting people to backout changes if they break these builds on tbpl? (I'm trying to determine if it will be a tier-1 build or not)
Are they going to be visible?

If we had these builds only to be triggered during PGO builds, would that be sufficient?
Sheriffs do not merge from m-i to m-c without making sure that all jobs on a PGO changeset come out green (assuming that they're visible).
A breakage would only happen in between two PGO changesets.
We could also add mechanisms to allow us to trigger the build on non-PGO-requested changesets (to be able to do regression hunting).

I'm trying to determine what different levels of support releng shoud have and if this type of build would have to explicitly per-checking & visible or something else.

Thanks for your info!
(Assignee)

Comment 7

5 years ago
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #6)
> (In reply to Christian Holler (:decoder) from comment #3) 
> > > The reason I ask is because I see these jobs broken and hidden on tbpl.
> > > This is very costly and I wonder if the per-checkin approach was appropriate
> > > for this specific type of jobs. There are other options we can consider.
> > 
> > What other options would these be? We eventually want these to be even per
> > checkin on mozilla-inbound, and on other branches. Also we're working on
> > running all our regressions tests for these. Furthermore, these builds are
> > also actively used by other people.
> > 
> 
> Are we expecting people to backout changes if they break these builds on
> tbpl? (I'm trying to determine if it will be a tier-1 build or not)
> Are they going to be visible?

We're strongly aiming for that, yes.

> 
> If we had these builds only to be triggered during PGO builds, would that be
> sufficient?
> Sheriffs do not merge from m-i to m-c without making sure that all jobs on a
> PGO changeset come out green (assuming that they're visible).
> A breakage would only happen in between two PGO changesets.
> We could also add mechanisms to allow us to trigger the build on
> non-PGO-requested changesets (to be able to do regression hunting).

I'm not sure about that, might be sufficient right now. But later when we want to be on m-i I assume it's not.

> 
> I'm trying to determine what different levels of support releng shoud have
> and if this type of build would have to explicitly per-checking & visible or
> something else.

The ultimate goal is to have build + tests visible such that any regressions are directly detected and backed out. Of course that's still a lot of work to go.
(Assignee)

Updated

5 years ago
Depends on: 899504
(Assignee)

Updated

5 years ago
Depends on: 899492
(Assignee)

Updated

5 years ago
Depends on: 899542
(Assignee)

Updated

5 years ago
Depends on: 899802
(Assignee)

Updated

5 years ago
Depends on: 902132
(Assignee)

Updated

5 years ago
Depends on: 902157
(Assignee)

Updated

5 years ago
Depends on: 905636
(Assignee)

Updated

5 years ago
Depends on: 906100
(Assignee)

Updated

5 years ago
Depends on: 914174
(Assignee)

Updated

5 years ago
Depends on: 917242
(Assignee)

Updated

5 years ago
Depends on: 919145
(Assignee)

Updated

5 years ago
Depends on: 919154
(Assignee)

Updated

5 years ago
Depends on: 919438
(Assignee)

Updated

5 years ago
Depends on: 920055
(Assignee)

Updated

4 years ago
Depends on: 923916
(Assignee)

Updated

4 years ago
Depends on: 923486
(Assignee)

Updated

4 years ago
Depends on: 847350
(Assignee)

Updated

4 years ago
Depends on: 925873
(Assignee)

Updated

4 years ago
Depends on: 929024
Depends on: 930584

Updated

4 years ago
Depends on: 928808
(Assignee)

Updated

4 years ago
Depends on: 932159
(Assignee)

Updated

4 years ago
Depends on: 934715

Updated

4 years ago
Depends on: 935795

Updated

4 years ago
Depends on: 937582
(Assignee)

Updated

4 years ago
Depends on: 939513
(Assignee)

Updated

4 years ago
Depends on: 948171
(Assignee)

Updated

4 years ago
Depends on: 961394
(Assignee)

Updated

4 years ago
Depends on: 987688
(Assignee)

Updated

4 years ago
Depends on: 989823
(Assignee)

Updated

4 years ago
Depends on: 990266

Updated

4 years ago
Depends on: 664901
(Assignee)

Updated

4 years ago
Depends on: 1026162
(Assignee)

Updated

4 years ago
Depends on: 982693
(Assignee)

Updated

4 years ago
Depends on: 1051190
(Assignee)

Updated

4 years ago
Depends on: 1052439
(Assignee)

Updated

4 years ago
Depends on: 1057104
(Assignee)

Updated

4 years ago
Depends on: 1063944

Updated

3 years ago
Depends on: 1057551
(Assignee)

Updated

3 years ago
Depends on: 1146895
No longer depends on: 923916
(Assignee)

Updated

3 years ago
Depends on: 1170177

Updated

3 years ago
Depends on: 1182378

Updated

3 years ago
Depends on: 1182382
Depends on: 1187101
(Assignee)

Updated

2 years ago
Depends on: 1252072
(Assignee)

Updated

2 years ago
Depends on: 1267650
(Assignee)

Updated

2 years ago
Depends on: 1253299
You need to log in before you can comment on or make changes to this bug.