Open Bug 863846 (asan-maintenance) Opened 7 years ago Updated 3 months ago

Maintain Firefox ASan build process and compatibility

Categories

(mozilla.org :: Security Assurance: Applications, task)

All
Linux
task
Not set

Tracking

(Not tracked)

People

(Reporter: decoder, Assigned: decoder)

References

(Depends on 9 open bugs)

Details

This bug is meant for tracking all efforts related to keeping our ASan builds working and compatible with latest Firefox and ASan features.
Alias: asan-maintenance
Depends on: 864008
Depends on: 864023
Depends on: 857189
Depends on: 844088
Also this bug will be tracking test failures (either due to incompatibilities or bugs) since we want the tests to be working and green.
Depends on: 843923
Depends on: 787715
Depends on: 750932
Depends on: 860867
Summary: Maintain ASan build process and compatibility → Maintain Firefox ASan build process and compatibility
Depends on: 865921
Depends on: 866525
Depends on: 868481
Depends on: 872565
Depends on: 872577
Depends on: 874486
Depends on: 874527
Depends on: 876717
Depends on: 880193
Depends on: 880239
Depends on: 886842
Depends on: 892017
Depends on: 895845
Depends on: 873649
Depends on: 898230
Hi,
What are these builds specifically for?
If we find a regression, how hard is it to reproduce it locally and do bisections?

The reason I ask is because I see these jobs broken and hidden on tbpl.
This is very costly and I wonder if the per-checkin approach was appropriate for this specific type of jobs. There are other options we can consider.

If we're still trying to get them green; would it make sense to only run them on the try server? or a project branch?
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #2)
> Hi,
> What are these builds specifically for?

These builds are for security testing.

> If we find a regression, how hard is it to reproduce it locally and do
> bisections?

Locally reproducing these issues is easy most of the time, bisecting the issue is usually a bit harder.

> 
> The reason I ask is because I see these jobs broken and hidden on tbpl.
> This is very costly and I wonder if the per-checkin approach was appropriate
> for this specific type of jobs. There are other options we can consider.

What other options would these be? We eventually want these to be even per checkin on mozilla-inbound, and on other branches. Also we're working on running all our regressions tests for these. Furthermore, these builds are also actively used by other people.

> 
> If we're still trying to get them green; would it make sense to only run
> them on the try server? or a project branch?

No, we need these to find security regressions on mozilla-central. And the fact that they're orange right now doesn't matter at all, it's a silly make check failure due to the Clang update, but the failure is not affecting the functionality of the builds.
Just a correction, seems like the builds are really busted right now, I'll look into that (I was working on the orange due to the Clang update in the last few days).
Depends on: 898484
Filed bug 898484 for the build bustage and investigating :)
(In reply to Christian Holler (:decoder) from comment #3) 
> > The reason I ask is because I see these jobs broken and hidden on tbpl.
> > This is very costly and I wonder if the per-checkin approach was appropriate
> > for this specific type of jobs. There are other options we can consider.
> 
> What other options would these be? We eventually want these to be even per
> checkin on mozilla-inbound, and on other branches. Also we're working on
> running all our regressions tests for these. Furthermore, these builds are
> also actively used by other people.
> 

Are we expecting people to backout changes if they break these builds on tbpl? (I'm trying to determine if it will be a tier-1 build or not)
Are they going to be visible?

If we had these builds only to be triggered during PGO builds, would that be sufficient?
Sheriffs do not merge from m-i to m-c without making sure that all jobs on a PGO changeset come out green (assuming that they're visible).
A breakage would only happen in between two PGO changesets.
We could also add mechanisms to allow us to trigger the build on non-PGO-requested changesets (to be able to do regression hunting).

I'm trying to determine what different levels of support releng shoud have and if this type of build would have to explicitly per-checking & visible or something else.

Thanks for your info!
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #6)
> (In reply to Christian Holler (:decoder) from comment #3) 
> > > The reason I ask is because I see these jobs broken and hidden on tbpl.
> > > This is very costly and I wonder if the per-checkin approach was appropriate
> > > for this specific type of jobs. There are other options we can consider.
> > 
> > What other options would these be? We eventually want these to be even per
> > checkin on mozilla-inbound, and on other branches. Also we're working on
> > running all our regressions tests for these. Furthermore, these builds are
> > also actively used by other people.
> > 
> 
> Are we expecting people to backout changes if they break these builds on
> tbpl? (I'm trying to determine if it will be a tier-1 build or not)
> Are they going to be visible?

We're strongly aiming for that, yes.

> 
> If we had these builds only to be triggered during PGO builds, would that be
> sufficient?
> Sheriffs do not merge from m-i to m-c without making sure that all jobs on a
> PGO changeset come out green (assuming that they're visible).
> A breakage would only happen in between two PGO changesets.
> We could also add mechanisms to allow us to trigger the build on
> non-PGO-requested changesets (to be able to do regression hunting).

I'm not sure about that, might be sufficient right now. But later when we want to be on m-i I assume it's not.

> 
> I'm trying to determine what different levels of support releng shoud have
> and if this type of build would have to explicitly per-checking & visible or
> something else.

The ultimate goal is to have build + tests visible such that any regressions are directly detected and backed out. Of course that's still a lot of work to go.
Depends on: 899504
Depends on: 899492
Depends on: 899542
Depends on: 899802
Depends on: 902132
Depends on: 902157
Depends on: 905636
Depends on: 906100
Depends on: 914174
Depends on: 917242
Depends on: 919145
Depends on: 919154
Depends on: 919438
Depends on: 920055
Depends on: 923916
Depends on: 923486
Depends on: 847350
Depends on: 925873
Depends on: 929024
Depends on: 930584
Depends on: 928808
Depends on: 932159
Depends on: 934715
Depends on: 935795
Depends on: 937582
Depends on: 939513
Depends on: 948171
Depends on: 961394
Depends on: 987688
Depends on: 989823
Depends on: 990266
Depends on: 664901
Depends on: asan-macbuilds
Depends on: 982693
Depends on: 1051190
Depends on: 1052439
Depends on: 1057104
Depends on: 1063944
Depends on: 1057551
Depends on: 1146895
No longer depends on: 923916
Depends on: 1170177
Depends on: 1182378
Depends on: 1182382
Depends on: 1187101
Depends on: 1252072
Depends on: 1267650
Depends on: 1253299
Depends on: 1587774
Depends on: 1592259
You need to log in before you can comment on or make changes to this bug.