1513173 - 4.59 - 45.84% build times (windows2012-32, windows2012-64) regression on push 6a4b98caa59b556990fbfc2c06fde08e851ef9fe (Fri Dec 7 2018)

Status: VERIFIED FIXED

(Core :: CSS Parsing and Computation, defect, P3)

Description

[Florin Strugariu [:Bebe]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=6a4b98caa59b556990fbfc2c06fde08e851ef9fe

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 46%  build times windows2012-32 debug rusttests taskcluster-c5.4xlarge     501.30 -> 731.09
 45%  build times windows2012-32 debug rusttests taskcluster-c4.4xlarge     627.79 -> 909.67
 45%  build times windows2012-64 debug rusttests taskcluster-c4.4xlarge     644.65 -> 932.90
 40%  build times windows2012-64 debug rusttests taskcluster-c5.4xlarge     513.46 -> 719.96
  7%  build times windows2012-64 debug plain taskcluster-c5.4xlarge         1,847.86 -> 1,978.31
  5%  build times windows2012-64 opt plain taskcluster-c4.4xlarge           2,748.28 -> 2,884.17
  5%  build times windows2012-64 debug plain taskcluster-c4.4xlarge         2,339.83 -> 2,447.25


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=18085

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Comment 1

[Emilio Cobos Álvarez (:emilio)]

Do we track build times on Linux? Have they not regressed? I don't know why this would be an issue only on Windows.

Comment 2

[Emilio Cobos Álvarez (:emilio)]

Off-hand, looks like either Darling got slower, or our derive code got slower. I suspect there's a bit of both.

Before, we won't allocate so many strings on the heap. Looks like all the calls to ident.as_ref() which became &ident.to_string() may be a problem (both in our source and Darling's), but I haven't profiled yet.

Bastian, is there anything that I may have missed here? Perhaps we should file an upstream issue about this.

Comment 3

[Florin Strugariu [:Bebe]]

Linux looks OK no changes:

https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-inbound,1807990,1,2&series=mozilla-inbound,1808013,1,2&series=mozilla-inbound,1625247,1,2&series=mozilla-inbound,1625324,1,2

https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-inbound,1807880,1,2&series=mozilla-inbound,1625806,1,2&series=mozilla-inbound,1807857,1,2&series=mozilla-inbound,1625820,1,2

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #1)
> Do we track build times on Linux? Have they not regressed? I don't know why
> this would be an issue only on Windows.

Actually, there are build time regressions on Linux also, ranging from 5 to 13% on a quick glance.

Perfherder didn't catch them because there are too few builds.

Comment 5

[Bastien Orivel [:eijebong]]

Opened https://github.com/TedDriggs/darling/pull/45 it'll hopefully improve things a bit

Comment 6

[Bastien Orivel [:eijebong]]

Attachment: 0001-Bug-1513173-Update-darling-to-0.8.1-and-revendor-rus.patch

Comment 7

[Emilio Cobos Álvarez (:emilio)]

Comment on attachment 9030813 [details] [diff] [review]
0001-Bug-1513173-Update-darling-to-0.8.1-and-revendor-rus.patch

Review of attachment 9030813 [details] [diff] [review]:
-----------------------------------------------------------------

Let's see if this helps, thanks!

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5f4d93cef87c
Update darling to 0.8.1 and revendor rust dependencies. r=emilio

Comment 9

[Bastien Orivel [:eijebong]]

I also filed https://github.com/servo/servo/pull/22439 which might help

Comment 10

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/5f4d93cef87c

Comment 11

[Ionuț Goldan [:igoldan]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #7)
> Comment on attachment 9030813 [details] [diff] [review]
> 0001-Bug-1513173-Update-darling-to-0.8.1-and-revendor-rus.patch
> 
> Review of attachment 9030813 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Let's see if this helps, thanks!

I checked all the regressions. I also looked for a machine that compiled Firefox using this new change [1].
I didn't see any improvement. Thus, I have to re open this.

[1] https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-inbound,1682421,1,2&highlightedRevisions=5f4d93cef87c

Comment 12

[Tim Spurway [:tspurway]]

Bastien - can you take another look at this

Comment 13

[Bastien Orivel [:eijebong]]

Sure. Apparently though last times the perf numbers I found were not right. If I want to push to try to see if some changes impact the build times, what are the parameters I should pass to the try run and how can I find the results afterwards ?

Comment 14

[Emilio Cobos Álvarez (:emilio)]

Do you know how to do that :igoldan / :Bebe?

Comment 15

[Ionuț Goldan [:igoldan]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #14)
> Do you know how to do that :igoldan / :Bebe?

First, you need to update to a clean baseline (latest central for example). You then  run | ./mach try -b do -p linux64,win32,win64 -u none -t none |.

After that, you apply your changes on top of that baseline and run the same mach command again.
Then you go to Treeherder to get the revisions of those 2 Try pushes and input them here [1].
You'll be landed on a table-like page which does perf comparisons. It defaults on Talos measurements, but you have to switch to build_metrics measurements. The build time differences should be obvious there.

If you're having difficulties, don't hesitate to needinfo? me today or any of :davehunt, :jmaher or :rwood in the days that follow.

[1] https://treeherder.mozilla.org/perf.html#/comparechooser

Comment 16

[Bastien Orivel [:eijebong]]

So I did that, https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=cb3ded6edca930bfda5cfe605cb8013f702fb192&newProject=try&newRevision=a9f75fcf891d61ebaebca1f8353696973f2ca390&framework=2 which shows flaky numbers... Comparing the try push with my patch with central though gives great numbers https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=a9f75fcf891d61ebaebca1f8353696973f2ca390&framework=2&showOnlyComparable=1&selectedTimeRange=172800 which one should I believe ?

Comment 17

[Ionuț Goldan [:igoldan]]

(In reply to Bastien Orivel [:eijebong] from comment #16)
> So I did that,
> https://treeherder.mozilla.org/perf.html#/
> compare?originalProject=try&originalRevision=cb3ded6edca930bfda5cfe605cb8013f
> 702fb192&newProject=try&newRevision=a9f75fcf891d61ebaebca1f8353696973f2ca390&
> framework=2 which shows flaky numbers... Comparing the try push with my
> patch with central though gives great numbers
> https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-
> central&newProject=try&newRevision=a9f75fcf891d61ebaebca1f8353696973f2ca390&f
> ramework=2&showOnlyComparable=1&selectedTimeRange=172800 which one should I
> believe ?

I retriggered all the jobs 4 extra times, hoping to get better results. Maybe a high load on Try caused a spike?
:jmaher what do you think?

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

just looking at data provided on mozilla-central vs try, I see sccache metrics on try and just plain build times on central.  It is also hard because we don't seem to be able to force the build machine type, so getting multiple data points is difficult.

Maybe :ted can shed light on build times on try vs m-c and if my theory on sccache is affecting try only.

Comment 19

[(not currently active) Ted Mielczarek]

If you're worried about sccache impacting the results you can compare the "plain" builds--those don't use sccache, so they produce more stable numbers. With other builds you can see the effect of sccache, the first build will be slower and then retriggers will be faster because they have lots of cache hits.

Comparing with the same instance types is definitely important and we don't have any way to ensure that. That might be a useful thing to file a Taskcluster bug on?

If you can retrigger enough plain builds to get comparable numbers you should be able to look at those without worrying about the effects of caching.

If these changes caused a slowdown in Rust compilation it's not surprising that the rusttest builds would show the biggest regression--they build almost nothing *but* Rust code so any regression there would be larger percentage-wise.

One thing that's possible is that these changes made sccache less effective, like adding crates that sccache can't cache for some reason. Unfortunately due to bug 1507827 I can't currently compare sccache stats on rusttest builds.

Comment 20

[Ionuț Goldan [:igoldan]]

(In reply to Bastien Orivel [:eijebong] from comment #16)

So I did that,
https://treeherder.mozilla.org/perf.html#/
compare?originalProject=try&originalRevision=cb3ded6edca930bfda5cfe605cb8013f
702fb192&newProject=try&newRevision=a9f75fcf891d61ebaebca1f8353696973f2ca390&
framework=2 which shows flaky numbers... Comparing the try push with my
patch with central though gives great numbers
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-
central&newProject=try&newRevision=a9f75fcf891d61ebaebca1f8353696973f2ca390&f
ramework=2&showOnlyComparable=1&selectedTimeRange=172800 which one should I
believe ?

Given the fact your Try change brought back the improvements without leveraging any sccache, as explained in comment 18 and comment 19, I say trust the m-c/try comparison.

Comment 21

[Bastien Orivel [:eijebong]]

Attachment: 0001-Bug-1513173-Part-1-Update-all-the-syn-0.15-related-d.patch

Comment 22

[Bastien Orivel [:eijebong]]

Attachment: 0002-Bug-1513173-Part-2-Revendor-rust-dependencies.patch

Comment 23

[Emilio Cobos Álvarez (:emilio)]

Comment on attachment 9034996 [details] [diff] [review]
0001-Bug-1513173-Part-1-Update-all-the-syn-0.15-related-d.patch

Review of attachment 9034996 [details] [diff] [review]:
-----------------------------------------------------------------

This is fine regardless, but please mention in the commit message how does this fix the regression. r=me with that.

Comment 24

[Emilio Cobos Álvarez (:emilio)]

From #servo:

14:41 <emilio> eijebong: hey, mind elaborating on the commit message why does that fix the regression?
14:44 <eijebong> emilio: Huuu honestly I have no idea I just tried that because it seemed like the next logical thing to do. The numbers I was told to trust are all super green witht that soo...
14:44 <eijebong> So i can elaborate on why it's there but that's about it

I guess that's fine. I see a couple of parsing changes in the revendor patch that could affect perf.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/11bcaa1ae024
Part 2: Revendor rust dependencies r=emilio
https://hg.mozilla.org/integration/mozilla-inbound/rev/55d05fe0a947
Part 1: Update all the syn 0.15 related dependencies r=emilio

Comment 26

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/11bcaa1ae024
https://hg.mozilla.org/mozilla-central/rev/55d05fe0a947

Comment 27

[Ryan VanderMeulen [:RyanVM]]

Florin, can you confirm that the regression is fixed now?

Comment 28

[Ionuț Goldan [:igoldan]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #27)

Florin, can you confirm that the regression is fixed now?

The patch from comment 25 didn't fix the regression. We only won back a small amount of build time on the rusttests scenario.
As for the plain scenario, I didn't see any kind of improvement.

Comment 29

[Ionuț Goldan [:igoldan]]

One of the perf improvements after push from comment 25:

== Change summary for alert #18639 (as of Tue, 08 Jan 2019 16:44:38 GMT) ==

Improvements:

5% build times windows2012-32 debug rusttests taskcluster-c4.4xlarge 910.29 -> 860.87

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=18639

Comment 30

[Ionuț Goldan [:igoldan]]

ni? :davehunt to assess investigation difficulties over build_metrics alerts.

Comment 31

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #30)

ni? :davehunt to assess investigation difficulties over build_metrics alerts.

I've raised bug 1521515 as suggested in comment 19.

Comment 32

[Liz Henry (:lizzard) (relman/hg->git project)]

Is there anything else that needs to happen in this bug or is that contingent on bug 1521515?

Comment 33

[Ionuț Goldan [:igoldan]]

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #32)

Is there anything else that needs to happen in this bug or is that contingent on bug 1521515?

Most of the regressions here haven't been fixed. :svoisen could we make another attempt on fixing this bug? The regressions are pretty big.

Comment 34

[Sean Voisen (:svoisen)]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #33)

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #32)

Is there anything else that needs to happen in this bug or is that contingent on bug 1521515?

Most of the regressions here haven't been fixed. :svoisen could we make another attempt on fixing this bug? The regressions are pretty big.

I'm OK with making another attempt, just not sure what else can be done here.

emilio or bastien: Do you have other ideas on how to fix this regression?

Comment 35

[Emilio Cobos Álvarez (:emilio)]

Not really, other than hoping that the rust libraries fix this upstream. Bastian may know better though.

Comment 36

[Ionuț Goldan [:igoldan]]

Bastien, can you propose any other solutions to this?

Comment 37

[Bastien Orivel [:eijebong]]

Sorry, been quite busy with work and life these past weeks, will try to have a look at this this week end if I don't end up sleeping through it ;)

Comment 38

[Bastien Orivel [:eijebong]]

By looking at https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=mozilla-inbound,1808013,1 I see that most of the perf was fixed at the beginning of february (or is it something unrelated ?).

Anyway, I looked at some diffs again and except updating all the syn related crates again I don't have any idea what to do.

Comment 39

[Ionuț Goldan [:igoldan]]

(In reply to Bastien Orivel [:eijebong] from comment #38)

By looking at https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=mozilla-inbound,1808013,1 I see that most of the perf was fixed at the beginning of february (or is it something unrelated ?).

Indeed, most of them got fixed. There's only one exception, on windows2012-64 debug rusttests, which was slightly fixed.

Comment 40

[Ionuț Goldan [:igoldan]]

(In reply to Bastien Orivel [:eijebong] from comment #38)

Anyway, I looked at some diffs again and except updating all the syn related crates again I don't have any idea what to do.

Can you please do this update?

Comment 41

[Bastien Orivel [:eijebong]]

I'l currently pretty busy but I'll try do to it this weekend

Comment 42

[Ionuț Goldan [:igoldan]]

(In reply to Bastien Orivel [:eijebong] from comment #41)

I'l currently pretty busy but I'll try do to it this weekend

Have you managed update the syn related crates?

Comment 43

[Bastien Orivel [:eijebong]]

Attachment: 0001-Bug-1513173-Part-1-Update-syn-0.15-related-dependenc.patch

Comment 44

[Bastien Orivel [:eijebong]]

Attachment: 0002-Bug-1513173-Part-2-Revendor-dependencies.patch

Comment 45

[Emilio Cobos Álvarez (:emilio)]

LGTM, thanks Bastien.

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/82ddefcec03f
Part 1: Update syn 0.15 related dependencies. r=emilio
https://hg.mozilla.org/integration/mozilla-inbound/rev/7f34457bee51
Part 2: Revendor dependencies. r=emilio

Comment 47

[Daniel Varga [:dvarga]]

Backed out 2 changesets (Bug 1513173) for build bustage error: Could not compile `style' on a CLOSED TREE

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=7f34457bee51919780c547c4427deb22b4bd36cc

Backout link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=33d7b9832cc40165c4636a0afb0dd477dcdff270

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237857346&repo=mozilla-inbound&lineNumber=21842

14:49:04 INFO - Running z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name style 'servo\components\style\lib.rs' --color never --crate-type lib --emit=dep-info,link -C opt-level=1 -C panic=abort -C debuginfo=2 -C debug-assertions=on --cfg 'feature="bindgen"' --cfg 'feature="fallible"' --cfg 'feature="gecko"' --cfg 'feature="gecko_debug"' --cfg 'feature="nsstring"' --cfg 'feature="regex"' --cfg 'feature="style_traits"' --cfg 'feature="toml"' --cfg 'feature="use_bindgen"' -C metadata=0049bc38d66e108f -C extra-filename=-0049bc38d66e108f --out-dir 'z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps' --target aarch64-pc-windows-msvc -C 'linker=z:/build/build/src/build/cargo-linker.bat' -L 'dependency=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps' -L 'dependency=z:/build/build/src/obj-firefox\debug\deps' --extern 'app_units=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libapp_units-758f71177c3f219a.rlib' --extern 'arrayvec=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libarrayvec-5ff12b8467994e4b.rlib' --extern 'atomic_refcell=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libatomic_refcell-c49645a3acd260c4.rlib' --extern 'bitflags=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libbitflags-76cc32b886a8eb42.rlib' --extern 'byteorder=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libbyteorder-31aa52d699306620.rlib' --extern 'cssparser=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libcssparser-4283ed5388ca39d0.rlib' --extern 'derive_more=z:/build/build/src/obj-firefox\debug\deps\derive_more-0ea02c406c167607.dll' --extern 'euclid=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libeuclid-d2bf5597858aff96.rlib' --extern 'fallible=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libfallible-afe5594c39b05aac.rlib' --extern 'fxhash=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libfxhash-c910f9ef68925545.rlib' --extern 'hashglobe=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libhashglobe-351033b494fada89.rlib' --extern 'indexmap=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libindexmap-1fe562ff8901e429.rlib' --extern 'itertools=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libitertools-ddd0ed4ddd1dbe0a.rlib' --extern 'itoa=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libitoa-54cdc17f53b029a0.rlib' --extern 'lazy_static=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\liblazy_static-a096049b29f746e8.rlib' --extern 'log=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\liblog-bc4fc5ff638af530.rlib' --extern 'malloc_size_of=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libmalloc_size_of-817b46d18b5857e4.rlib' --extern 'malloc_size_of_derive=z:/build/build/src/obj-firefox\debug\deps\malloc_size_of_derive-d677afe941f803cb.dll' --extern 'matches=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libmatches-788303291339cd74.rlib' --extern 'debug_unreachable=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libdebug_unreachable-18bb755479040dbd.rlib' --extern 'nsstring=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnsstring-53dd42da07a490b5.rlib' --extern 'num_derive=z:/build/build/src/obj-firefox\debug\deps\num_derive-7a71ec9971846d02.dll' --extern 'num_integer=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnum_integer-33e2d2b9ed0883ce.rlib' --extern 'num_traits=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnum_traits-812ea2f683e9e0df.rlib' --extern 'num_cpus=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnum_cpus-a87e4c81de158417.rlib' --extern 'ordered_float=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libordered_float-8bbd2b09c39be596.rlib' --extern 'owning_ref=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libowning_ref-f7e2d7445cf7d2ac.rlib' --extern 'parking_lot=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libparking_lot-66549b11043832ef.rlib' --extern 'precomputed_hash=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libprecomputed_hash-6f51913a5008e4aa.rlib' --extern 'rayon=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\librayon-fe8e1df3916c00e3.rlib' --extern 'selectors=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libselectors-56877b06ec743a7f.rlib' --extern 'servo_arc=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libservo_arc-c38cbf0fd06d7603.rlib' --extern 'smallbitvec=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libsmallbitvec-c883a27205aaea53.rlib' --extern 'smallvec=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libsmallvec-a1bbe4ca8c4ae49b.rlib' --extern 'style_derive=z:/build/build/src/obj-firefox\debug\deps\style_derive-c7548e8786e02f01.dll' --extern 'style_traits=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libstyle_traits-fa965013ff719e95.rlib' --extern 'thin_slice=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libthin_slice-6674ed01b155a348.rlib' --extern 'time=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libtime-a15c680c9e12b471.rlib' --extern 'to_shmem=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libto_shmem-25cbe9bd6b1293fa.rlib' --extern 'to_shmem_derive=z:/build/build/src/obj-firefox\debug\deps\to_shmem_derive-a18ad809685359d6.dll' --extern 'uluru=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libuluru-109a08f98890a096.rlib' --extern 'unicode_bidi=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libunicode_bidi-469c7387d3c9e3bb.rlib' --extern 'unicode_segmentation=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libunicode_segmentation-d09f702b12c945b3.rlib' --extern 'void=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libvoid-faa70a7aa07462e9.rlib' -C opt-level=2 -C debuginfo=2 -Dwarnings
14:49:04 INFO - LLVM ERROR: SEH unwind data splitting not yet implemented
14:49:04 ERROR - error: Could not compile style.
14:49:04 INFO - Caused by:
14:49:04 INFO - process didn't exit successfully: z:/build/build/src/sccache2/sccache.exe 'z:/build/build/src/rustc/bin/rustc.exe' --crate-name style 'servo\components\style\lib.rs' --color never --crate-type lib --emit=dep-info,link -C opt-level=1 -C panic=abort -C debuginfo=2 -C debug-assertions=on --cfg 'feature="bindgen"' --cfg 'feature="fallible"' --cfg 'feature="gecko"' --cfg 'feature="gecko_debug"' --cfg 'feature="nsstring"' --cfg 'feature="regex"' --cfg 'feature="style_traits"' --cfg 'feature="toml"' --cfg 'feature="use_bindgen"' -C metadata=0049bc38d66e108f -C extra-filename=-0049bc38d66e108f --out-dir 'z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps' --target aarch64-pc-windows-msvc -C 'linker=z:/build/build/src/build/cargo-linker.bat' -L 'dependency=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps' -L 'dependency=z:/build/build/src/obj-firefox\debug\deps' --extern 'app_units=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libapp_units-758f71177c3f219a.rlib' --extern 'arrayvec=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libarrayvec-5ff12b8467994e4b.rlib' --extern 'atomic_refcell=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libatomic_refcell-c49645a3acd260c4.rlib' --extern 'bitflags=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libbitflags-76cc32b886a8eb42.rlib' --extern 'byteorder=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libbyteorder-31aa52d699306620.rlib' --extern 'cssparser=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libcssparser-4283ed5388ca39d0.rlib' --extern 'derive_more=z:/build/build/src/obj-firefox\debug\deps\derive_more-0ea02c406c167607.dll' --extern 'euclid=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libeuclid-d2bf5597858aff96.rlib' --extern 'fallible=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libfallible-afe5594c39b05aac.rlib' --extern 'fxhash=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libfxhash-c910f9ef68925545.rlib' --extern 'hashglobe=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libhashglobe-351033b494fada89.rlib' --extern 'indexmap=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libindexmap-1fe562ff8901e429.rlib' --extern 'itertools=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libitertools-ddd0ed4ddd1dbe0a.rlib' --extern 'itoa=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libitoa-54cdc17f53b029a0.rlib' --extern 'lazy_static=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\liblazy_static-a096049b29f746e8.rlib' --extern 'log=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\liblog-bc4fc5ff638af530.rlib' --extern 'malloc_size_of=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libmalloc_size_of-817b46d18b5857e4.rlib' --extern 'malloc_size_of_derive=z:/build/build/src/obj-firefox\debug\deps\malloc_size_of_derive-d677afe941f803cb.dll' --extern 'matches=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libmatches-788303291339cd74.rlib' --extern 'debug_unreachable=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libdebug_unreachable-18bb755479040dbd.rlib' --extern 'nsstring=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnsstring-53dd42da07a490b5.rlib' --extern 'num_derive=z:/build/build/src/obj-firefox\debug\deps\num_derive-7a71ec9971846d02.dll' --extern 'num_integer=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnum_integer-33e2d2b9ed0883ce.rlib' --extern 'num_traits=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnum_traits-812ea2f683e9e0df.rlib' --extern 'num_cpus=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libnum_cpus-a87e4c81de158417.rlib' --extern 'ordered_float=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libordered_float-8bbd2b09c39be596.rlib' --extern 'owning_ref=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libowning_ref-f7e2d7445cf7d2ac.rlib' --extern 'parking_lot=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libparking_lot-66549b11043832ef.rlib' --extern 'precomputed_hash=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libprecomputed_hash-6f51913a5008e4aa.rlib' --extern 'rayon=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\librayon-fe8e1df3916c00e3.rlib' --extern 'selectors=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libselectors-56877b06ec743a7f.rlib' --extern 'servo_arc=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libservo_arc-c38cbf0fd06d7603.rlib' --extern 'smallbitvec=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libsmallbitvec-c883a27205aaea53.rlib' --extern 'smallvec=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libsmallvec-a1bbe4ca8c4ae49b.rlib' --extern 'style_derive=z:/build/build/src/obj-firefox\debug\deps\style_derive-c7548e8786e02f01.dll' --extern 'style_traits=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libstyle_traits-fa965013ff719e95.rlib' --extern 'thin_slice=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libthin_slice-6674ed01b155a348.rlib' --extern 'time=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libtime-a15c680c9e12b471.rlib' --extern 'to_shmem=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libto_shmem-25cbe9bd6b1293fa.rlib' --extern 'to_shmem_derive=z:/build/build/src/obj-firefox\debug\deps\to_shmem_derive-a18ad809685359d6.dll' --extern 'uluru=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libuluru-109a08f98890a096.rlib' --extern 'unicode_bidi=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libunicode_bidi-469c7387d3c9e3bb.rlib' --extern 'unicode_segmentation=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libunicode_segmentation-d09f702b12c945b3.rlib' --extern 'void=z:/build/build/src/obj-firefox\aarch64-pc-windows-msvc\debug\deps\libvoid-faa70a7aa07462e9.rlib' -C opt-level=2 -C debuginfo=2 -Dwarnings (exit code: 101)
14:49:04 INFO - z:/build/build/src/config/makefiles/rust.mk:235: recipe for target 'force-cargo-library-build' failed
14:49:04 INFO - mozmake.EXE[4]: *** [force-cargo-library-build] Error 101
14:49:04 INFO - mozmake.EXE[4]: Leaving directory 'z:/build/build/src/obj-firefox/toolkit/library/rust'
14:49:04 INFO - z:/build/build/src/config/recurse.mk:74: recipe for target 'toolkit/library/rust/target' failed
14:49:04 INFO - mozmake.EXE[3]: *** [toolkit/library/rust/target] Error 2
14:49:04 INFO - mozmake.EXE[3]: *** Waiting for unfinished jobs....
14:49:04 INFO - mozmake.EXE[4]: Entering directory 'z:/build/build/src/obj-firefox/dom/plugins/base'
14:49:04 INFO - dom/plugins/base/Unified_cpp_dom_plugins_base0.obj
14:49:04 INFO - mozmake.EXE[4]: Leaving directory 'z:/build/build/src/obj-firefox/dom/plugins/base'
14:49:04 INFO - mozmake.EXE[4]: Entering directory 'z:/build/build/src/obj-firefox/dom/svg'

Comment 48

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/34c966d332f6
Part 1: Update syn 0.15 related dependencies. r=emilio
https://hg.mozilla.org/integration/mozilla-inbound/rev/b32d33fe8eb2
Part 2: Revendor dependencies. r=emilio

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2d35edac9363
followup: Update js's cargo.lock file.

Comment 50

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Since pulsebot reports a push, I am removing the stalled keyword.

Please update the bug's regressed-by field.

Comment 51

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/34c966d332f6
https://hg.mozilla.org/mozilla-central/rev/b32d33fe8eb2
https://hg.mozilla.org/mozilla-central/rev/2d35edac9363

Comment 52

[Ionuț Goldan [:igoldan]]

All regressions seem fixed.