Open Bug 863846 (asan-maintenance) Opened 11 years ago Updated 6 months ago

[meta] Maintain Firefox ASan build process and compatibility

Categories

(Core :: Sanitizers, task)

Product:
Core
Component:
Sanitizers
Platform:
All
Linux
Type:
task

Tracking

()

People

(Reporter: decoder, Unassigned)

References

(Depends on 65 open bugs)

Details

(Keywords: meta) 

This bug is meant for tracking all efforts related to keeping our ASan builds working and compatible with latest Firefox and ASan features.
Alias: asan-maintenance
Depends on: 864008
Depends on: 864023
Depends on: 857189
Depends on: 844088
Also this bug will be tracking test failures (either due to incompatibilities or bugs) since we want the tests to be working and green.
Depends on: 843923
Depends on: 787715
Depends on: 750932
Depends on: 860867
Summary: Maintain ASan build process and compatibility → Maintain Firefox ASan build process and compatibility
Depends on: 865921
Depends on: 866525
Depends on: 872565
Depends on: 872577
Depends on: 874486
Depends on: 874527
Depends on: 876717
Depends on: 880193
Depends on: 880239
Depends on: 886842
Depends on: 892017
Depends on: 895845
Depends on: 873649
Depends on: 898230
Hi,
What are these builds specifically for?
If we find a regression, how hard is it to reproduce it locally and do bisections?

The reason I ask is because I see these jobs broken and hidden on tbpl.
This is very costly and I wonder if the per-checkin approach was appropriate for this specific type of jobs. There are other options we can consider.

If we're still trying to get them green; would it make sense to only run them on the try server? or a project branch?
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #2)
> Hi,
> What are these builds specifically for?

These builds are for security testing.

> If we find a regression, how hard is it to reproduce it locally and do
> bisections?

Locally reproducing these issues is easy most of the time, bisecting the issue is usually a bit harder.

> 
> The reason I ask is because I see these jobs broken and hidden on tbpl.
> This is very costly and I wonder if the per-checkin approach was appropriate
> for this specific type of jobs. There are other options we can consider.

What other options would these be? We eventually want these to be even per checkin on mozilla-inbound, and on other branches. Also we're working on running all our regressions tests for these. Furthermore, these builds are also actively used by other people.

> 
> If we're still trying to get them green; would it make sense to only run
> them on the try server? or a project branch?

No, we need these to find security regressions on mozilla-central. And the fact that they're orange right now doesn't matter at all, it's a silly make check failure due to the Clang update, but the failure is not affecting the functionality of the builds.
Just a correction, seems like the builds are really busted right now, I'll look into that (I was working on the orange due to the Clang update in the last few days).
Depends on: 898484
Filed bug 898484 for the build bustage and investigating :)
(In reply to Christian Holler (:decoder) from comment #3) 
> > The reason I ask is because I see these jobs broken and hidden on tbpl.
> > This is very costly and I wonder if the per-checkin approach was appropriate
> > for this specific type of jobs. There are other options we can consider.
> 
> What other options would these be? We eventually want these to be even per
> checkin on mozilla-inbound, and on other branches. Also we're working on
> running all our regressions tests for these. Furthermore, these builds are
> also actively used by other people.
> 

Are we expecting people to backout changes if they break these builds on tbpl? (I'm trying to determine if it will be a tier-1 build or not)
Are they going to be visible?

If we had these builds only to be triggered during PGO builds, would that be sufficient?
Sheriffs do not merge from m-i to m-c without making sure that all jobs on a PGO changeset come out green (assuming that they're visible).
A breakage would only happen in between two PGO changesets.
We could also add mechanisms to allow us to trigger the build on non-PGO-requested changesets (to be able to do regression hunting).

I'm trying to determine what different levels of support releng shoud have and if this type of build would have to explicitly per-checking & visible or something else.

Thanks for your info!
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #6)
> (In reply to Christian Holler (:decoder) from comment #3) 
> > > The reason I ask is because I see these jobs broken and hidden on tbpl.
> > > This is very costly and I wonder if the per-checkin approach was appropriate
> > > for this specific type of jobs. There are other options we can consider.
> > 
> > What other options would these be? We eventually want these to be even per
> > checkin on mozilla-inbound, and on other branches. Also we're working on
> > running all our regressions tests for these. Furthermore, these builds are
> > also actively used by other people.
> > 
> 
> Are we expecting people to backout changes if they break these builds on
> tbpl? (I'm trying to determine if it will be a tier-1 build or not)
> Are they going to be visible?

We're strongly aiming for that, yes.

> 
> If we had these builds only to be triggered during PGO builds, would that be
> sufficient?
> Sheriffs do not merge from m-i to m-c without making sure that all jobs on a
> PGO changeset come out green (assuming that they're visible).
> A breakage would only happen in between two PGO changesets.
> We could also add mechanisms to allow us to trigger the build on
> non-PGO-requested changesets (to be able to do regression hunting).

I'm not sure about that, might be sufficient right now. But later when we want to be on m-i I assume it's not.

> 
> I'm trying to determine what different levels of support releng shoud have
> and if this type of build would have to explicitly per-checking & visible or
> something else.

The ultimate goal is to have build + tests visible such that any regressions are directly detected and backed out. Of course that's still a lot of work to go.
Depends on: 899504
Depends on: 899492
Depends on: 899542
Depends on: 899802
Depends on: 902132
Depends on: 902157
Depends on: 905636
Depends on: 906100
Depends on: 914174
Depends on: 917242
Depends on: 919145
Depends on: 919154
Depends on: 919438
Depends on: 920055
Depends on: 923916
Depends on: 923486
Depends on: 847350
Depends on: 925873
Depends on: 929024
Depends on: 930584
Depends on: 928808
Depends on: 932159
Depends on: 934715
Depends on: 935795
Depends on: 937582
Depends on: 939513
Depends on: 948171
Depends on: 961394
Depends on: 987688
Depends on: 989823
Depends on: 990266
Depends on: 664901
Depends on: asan-macbuilds
Depends on: 982693
Depends on: 1051190
Depends on: 1052439
Depends on: 1057104
Depends on: 1063944
Depends on: 1057551
Depends on: 1146895
No longer depends on: 923916
Depends on: 1170177
Depends on: 1182378
Depends on: 1182382
Depends on: 1187101
Depends on: 1252072
Depends on: 1267650
Depends on: 1253299
Depends on: 1587774
Depends on: 1592259
Depends on: 1617233
Depends on: 1612963
Depends on: 1611389
Depends on: 1599684
Depends on: 1303021
Depends on: 990096
Depends on: 1494639
Depends on: 1506500
Depends on: 1500310
Depends on: 1385822
Depends on: 1492069
Depends on: 1352507
Depends on: 1425780
Depends on: 1511412
Depends on: 1357012
Depends on: 1486415
Depends on: winasan
Depends on: 1552911
Depends on: 1594147
Depends on: 1487660
Depends on: 1378374
Depends on: 1187533
Depends on: 1587248
Depends on: 1557739
Depends on: 1542321
Depends on: 1308556
Depends on: 1422389
Depends on: 1555085
Depends on: 1540378
Depends on: 1481385
Depends on: 1279108
Depends on: 1555786
Depends on: 1516834
Depends on: 1436759
Depends on: 1191492
Depends on: 1365894
Depends on: 1475514
Depends on: 1388113
Depends on: 1064117
Depends on: 1488803
Depends on: 1480965
Depends on: 992968
Depends on: 1278080
Depends on: 1270752
Depends on: 1401459
Depends on: 1489287
Depends on: 1253883
Depends on: 1609054
Depends on: 1407375
Depends on: 1557208
Depends on: 1539159
Depends on: 1402871
Depends on: 1375842
Depends on: 1576187
Depends on: 1560207
Depends on: 1419608
Depends on: 1471953
Depends on: 1487085
Depends on: 779025
Depends on: 1506163
Depends on: 1229825
Depends on: 1387799
Depends on: 1547784
Depends on: 1477493
Depends on: 1366654
Depends on: 1513065
Depends on: 1279096
Depends on: 1493710
Depends on: 1437481
Depends on: 1021612
Depends on: 1594136
Depends on: 1452627
Depends on: 1568475
Depends on: 1397702
Component: Security Assurance: Applications → Sanitizers
Product: mozilla.org → Core
Version: other → unspecified
Assignee: choller → nobody
Keywords: meta
Summary: Maintain Firefox ASan build process and compatibility → [meta] Maintain Firefox ASan build process and compatibility
No longer depends on: CVE-2020-6807
See Also: → asan-nightly-project
Depends on: 1749192
Depends on: 1752538

I don't know how useful it is to just make every intermittent failure in an ASan build blocking this bug. Lots of these crashes are just null derefs or other things that would show up in other builds, too.

Yes, please only add ASan specific issues to this tracking bug (e.g. do not add SEGV / access violation bugs).

Depends on: 1763002
Depends on: 1791335
Severity: normal → S3
Depends on: 1796753
No longer depends on: 1796753
No longer depends on: 1728042
Depends on: 1191514, 1788083
You need to log in before you can comment on or make changes to this bug.