1660340 - Update to clang 11.0.0 rc2

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, task)

Description

[(Away)]

Next week is the beginning of Nightly 82, and clang 11.0.0 rc2 was tagged today. After the multiple backouts of bug 1616692, we plan to skip clang 10 and go straight to clang 11. In order to maximize the amount of testing time on Nightly, we'll start with rc2 and pick up the final tag of clang 11 when it becomes available.

Comment 1

[(Away)]

Attachment: Switch builds to clang 11.0.0 rc2

This changes most of our automation builds to clang 11.0.0 rc2.

Not included:

• code coverage builds, per bug 1660341
• mingw builds, which have traditionally been on their own update cadence, and in this case are blocked anyway by bug 1658632

This will leave some unused clang-9 task definitions. I intend to clean them up, but at a later date. For now I want to focus on making sure this update sticks, since patches like this have a tendency to bounce.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/edb1d37f48f4
Switch builds to clang 11.0.0 rc2 r=froydnj

Comment 3

[Razvan Maries]

Backed out for perma failures.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=browsertime%2Candroid%2C7.0%2Cmotog5%2Cshippable%2Copt&revision=edb1d37f48f49f9884607092f0d7b851fea0d088&selectedTaskRun=GCmOz4_RQ_2_A3fzbJG9ZQ.0

Log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314259758&repo=autoland&lineNumber=1863

Backout: https://hg.mozilla.org/integration/autoland/rev/75cb26180d625a9a4fbb75789bddd920c0344859

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d445d03a6ca0
Switch builds to clang 11.0.0 rc2 r=froydnj

Comment 5

[Razvan Maries]

Relanded: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=8fcc41684a9fc0b1efff4a6ba45b4f8254214da5

Greg, Nathan, could you take a look at: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=GCmOz4_RQ_2_A3fzbJG9ZQ.0&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=browsertime%2Candroid%2C7.0%2Cmotog5%2Cshippable%2Copt&revision=edb1d37f48f49f9884607092f0d7b851fea0d088 ?

Comment 6

[Mike Hommey [:glandium]]

There's a crash in the logcat output:

08-28 08:37:08.009  8236  8259 W google-breakpad: ExceptionHandler::GenerateDump cloned child 
08-28 08:37:08.009  8236  8259 W google-breakpad: 8419
08-28 08:37:08.009  8236  8259 W google-breakpad: 
08-28 08:37:08.009  8236  8259 W google-breakpad: ExceptionHandler::SendContinueSignalToChild sent continue signal to child
08-28 08:37:08.010  8419  8259 W google-breakpad: ExceptionHandler::WaitForContinueSignal waiting for continue signal...

But we don't get more than that. No minidump, no stack in the logs.

Comment 7

[Greg Mierzwinski [:sparky]]

There's no crash reporting available on browsertime yet because of some issues in geckodriver.

I ran the geckoview example locally on the MotoG5 and as soon as we try to load a page it crashes. I tested the app itself without a harness testing it and as soon as I open it, it crashes before I can even input a url.

I received a notification on the phone telling me to report the the crash to Mozilla, is there somewhere we can find these crash reports after reporting?

Comment 8

[Greg Mierzwinski [:sparky]]

This should be the crash report for this failure: https://crash-stats.mozilla.org/signature/?product=GeckoViewExample&signature=libfreebl3.so%400x5389e%20%7C%20libfreebl3.so%400xc91b%20%7C%20libfreebl3.so%400x3b24d%20%7C%20libsoftokn3.so%400x4243&version=&date=%3C2020-08-28T13%3A10%3A29%2B00%3A00&date=%3E%3D2020-08-21T13%3A10%3A29%2B00%3A00

Comment 9

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/d445d03a6ca0

Comment 10

[(Away)]

Unfortunately I don't even know how to get started investigating that failure since I have very little experience with Android builds. As far as I can tell, we don't even have symbols for that crash report? Whatever Mike and Greg say, I'll trust them.

Comment 11

[Greg Mierzwinski [:sparky]]

:dmajor, do you know anyone that can debug that crash report or debug geckoview_example locally? The problem isn't related to the harness because I can trigger the startup crash by manually opening geckoview.

This is a serious issue and all perf tests are busted on G5 right now because of this: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=browsertime%2Candroid%2C7.0%2Cmotog5%2Cshippable%2Copt&revision=56166cae2e26429f4786ad1013adae78189a12e8&selectedTaskRun=QSU3V3n8T9K5Q-jHONBmJQ.0

Comment 12

[(Away)]

(In reply to Greg Mierzwinski [:sparky] from comment #11)

:dmajor, do you know anyone that can debug that crash report or debug geckoview_example locally? The problem isn't related to the harness because I can trigger the startup crash by manually opening geckoview.

This is a serious issue and all perf tests are busted on G5 right now because of this: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=browsertime%2Candroid%2C7.0%2Cmotog5%2Cshippable%2Copt&revision=56166cae2e26429f4786ad1013adae78189a12e8&selectedTaskRun=QSU3V3n8T9K5Q-jHONBmJQ.0

froydnj, nalexander: is this something that one of you has experience with?

Comment 13

[Greg Mierzwinski [:sparky]]

Hi :snorp, can you help us out here? There's a startup crash in geckoview that's causing all of our perf tests to fail.

Comment 14

[James Willcox (:snorp) ([e-mail]) (he/him)]

Greg, GVE will report crashes if you click the notification that appears when the crash occurs. It will then print the crash ID to logcat. If you grep for "sent crash report" you should find it. Then we can look up the crash-stats link and see what's going on.

Comment 15

[Greg Mierzwinski [:sparky]]

:snorp here's the crash that I'm getting: https://crash-stats.mozilla.org/report/index/6a9aeadf-bd0e-43e1-88ea-4ff4a0200828

Comment 16

[James Willcox (:snorp) ([e-mail]) (he/him)]

It looks like it's a problem only on 32bit ARM. I installed such a build on my Pixel 4 and immediately reproduced the crash: https://crash-stats.mozilla.org/report/index/ffcd0667-12af-4ade-8915-486220200828

This appears to be some NEON going wrong in NSS somehow.

Comment 17

[James Willcox (:snorp) ([e-mail]) (he/him)]

Given that we get SIGBUS, I would guess there is some kind of alignment problem. Perhaps we were just getting lucky with the prior toolchain?

:m_kato you've done a bunch of ARM stuff in NSS, any ideas?

Comment 18

[Mike Hommey [:glandium]]

See bug 1661810.

Comment 19

[Narcis Beleuzu [:NarcisB]]

Backed out for causing Btime failures on Android 7.0

Backout link: https://hg.mozilla.org/integration/autoland/rev/592e556af3ecee3700c80dac778585e93cbe755f
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314317947&repo=autoland&lineNumber=1864

Comment 20

[Intermittent Failures Robot]

27 failures in 787 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 27

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16-shippable: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1660340&startday=2020-08-28&endday=2020-08-28&tree=all

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merged: https://hg.mozilla.org/mozilla-central/rev/592e556af3ec

Comment 22

[Intermittent Failures Robot]

28 failures in 4046 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #29 most frequent failure this week.

Repository breakdown:

• autoland: 28

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16-shippable: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1660340&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/57b1de9d90dc
Switch builds to clang 11.0.0 rc2 r=froydnj

Comment 24

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/57b1de9d90dc

Comment 25

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #4)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d445d03a6ca0
Switch builds to clang 11.0.0 rc2 r=froydnj

== Change summary for alert #26815 (as of Sun, 30 Aug 2020 17:15:54 GMT) ==

Regressions:

9% build times windows2012-64-shippable opt nightly taskcluster-c5d.4xlarge 2,084.19 -> 2,270.97
8% build times windows2012-64-shippable opt nightly taskcluster-c5.4xlarge 2,126.13 -> 2,290.79
0.46% installer size osx-shippable opt nightly 75,996,904.81 -> 76,345,195.00

Improvements:

1% installer size osx-cross opt 75,296,820.08 -> 74,567,956.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26815

Comment 26

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Narcis Beleuzu [:NarcisB] from comment #19)

Backed out for causing Btime failures on Android 7.0

Backout link: https://hg.mozilla.org/integration/autoland/rev/592e556af3ecee3700c80dac778585e93cbe755f
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314317947&repo=autoland&lineNumber=1864

== Change summary for alert #26822 (as of Mon, 31 Aug 2020 05:21:17 GMT) ==

Regressions:

1% installer size osx-cross opt 74,600,928.67 -> 75,361,047.17

Improvements:

9% build times windows2012-64-shippable opt nightly taskcluster-c5.4xlarge 2,290.79 -> 2,090.83

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26822

Comment 27

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I did a backfill but I'm 99% that the regression is caused by this bug. I am waiting for the test to trigger the alert so I can file the regreesion bug, but please be aware.
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2614662,1,2&timerange=1209600&zoom=1598808958916,1598967830474,74301150.08049797,75650004.23706341

Comment 28

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Alexandru Ionescu - PTO retuning Sept 14th (needinfo me) [:alexandrui] from comment #27)

I did a backfill but I'm 99% that the regression is caused by this bug. I am waiting for the test to trigger the alert so I can file the regreesion bug, but please be aware.
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,2614662,1,2&timerange=1209600&zoom=1598808958916,1598967830474,74301150.08049797,75650004.23706341

I actually had to link the graph with a regression, not an improvement.
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=QGX-HzCxTsCirN5PeiPktQ.0&searchStr=Windows%2C2012%2Cx64%2Cshippable%2Copt%2CProfile-guided%2Coptimization%2Cbuilds%2Cbuild-win64-shippable%2Fopt%2CB&tochange=3379062235deff8142bd12ee5da16782912102f5&fromchange=c8320f7940b28a128275198c307b41bba9ec26b9

Comment 29

[(Away)]

I'm having trouble understanding the last couple of comments. Comment 27 seems to be highlighting an improvement in installer size. Comment 28 is pointing to a push range that doesn't show this landing. If there is something that you would like me to look at, please let me know what regression you had in mind and why the link demonstrates it. Thanks!

Comment 30

[Florin Strugariu [:Bebe]]

the improvment was caused by the backout of Bug 1661786 as you can see from this graph:
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,1921077,1,2&timerange=1209600&zoom=1598257383000,1599465765369,1735.8558003107707,2689.278694709142

there are other regresssions in the bugs attached to the regression list: 1663138, 1663139