1436058 - Update webrender to 342bc314db94aa439b2001249c5f24ccfcbccc22

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Kartikaya Gupta (email:[e-mail])]

+++ This bug was initially created as a clone of Bug #1434723 +++

I'm filing this as a placeholder bug for the next webrender update. I may be running a cron script [1] that does try pushes with webrender update attempts, so that we can track build/test breakages introduced by webrender on a rolling basis. This bug will hold the try push links as well as dependencies filed for those breakages, so that we have a better idea going into the update of what needs fixing. I might abort the cron job because once things get too far out of sync it's hard to fully automate fixing all the breakages.

When we are ready to actually land the update, we can rename this bug and use it for the update, and then file a new bug for the next "future update".

[1] https://github.com/staktrace/moz-scripts/blob/master/try-latest-webrender.sh

Comment 1

[Kartikaya Gupta (email:[e-mail])]

WR @ 08e49649f1fc9cacff4e10ebc390babcea752236

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a9ad3b81dd77ed55a7860af182e6a5a849769dcf
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e7faeb54681f82f3a63580308a8cea9d9411460

Green

Comment 2

[Kartikaya Gupta (email:[e-mail])]

WR @ a4202af2130e999ead44e10d8eb4f323c0ddb582

https://treeherder.mozilla.org/#/jobs?repo=try&revision=eef6d3bc2aa54aa9d8f1be8dc84d5ec1934ab3eb
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f8beedef255a872e301ae21346cefb07643aaf6b

Green, but some of those intermittents are getting more frequent. I'll have to investigate

WR @ d96d02bd9b8141c2fa4a692ee0b35220be0a8873

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bd2a2ceffa4abbc8852aa19c7a3ba566b828fc72
https://treeherder.mozilla.org/#/jobs?repo=try&revision=08de16708073eca50fdf6d8b194d40cce8f43ddb

Green

Comment 3

[Kartikaya Gupta (email:[e-mail])]

WR @ 30f2cd5e648bbcadbe8200d479de3f7d76089d0b

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3be1a14608bff4f0ae6b45af286120e83d7a41df
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf51475dc0c3c3e623caed799bef4d9dc6007973

Windows build bustage and linux assertion failures, from bug 1432708. Reported there.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

WR @ a492386a97fb73ffc449dc2ee4bee9163238d083

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b518a54e92d91d24fa52740258e77782d8e73605
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6c64072fee05f10b79607668ae633b52288903f

Now we're busted everywhere. Probably from the env_logger change in a06537e1 (servo/webrender#2385).

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Oops, mid-aired.

Comment 6

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #4)
> Now we're busted everywhere. Probably from the env_logger change in a06537e1
> (servo/webrender#2385).

Ah, it was the log version bump to 0.4. Apparently a lot of stuff in the log crate got renamed. I'm working through it now.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

WR @ e7c95ae9fb6b7f8d973ae5106a1ba8be6adea007 with log fixes

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e55bcc9125cc9cc60f5dd9d4de4d633544cd4bb6
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0104f57ea782fbdeefc45688e651da78c59ef677

Comment 8

[Kartikaya Gupta (email:[e-mail])]

^ Seems to be back to the way things were, i.e. windows bustage and linux debug assertion failures from bug 1432708.

Comment 9

[Kartikaya Gupta (email:[e-mail])]

WR @ 65c68cc1d1d8958c103fa3a528b05d81e27783d3

https://treeherder.mozilla.org/#/jobs?repo=try&revision=06579c71a9ca29a41336ef79e08cd80f11ce00c9
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c66db8bc6c90600cc41f54c48cab47c8df26cadd

Same as before

Comment 10

[Kartikaya Gupta (email:[e-mail])]

WR @ 65c68cc1d1d8958c103fa3a528b05d81e27783d3 with the updated patch from bug 1432708

https://treeherder.mozilla.org/#/jobs?repo=try&revision=16522f15dc603cf33295af84acfb2f15004e8c62

Still seeing failures here

WR @ d51d6ec0def765055af6a1e2b2bc8b9bd6e89baf

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d3438b000238893e1567219f6fa314d303506451
https://treeherder.mozilla.org/#/jobs?repo=try&revision=20aae923643f0ea862be94980527f628f6fa8b7d

Ditto

Comment 11

[Kartikaya Gupta (email:[e-mail])]

WR @ 7184d6b6445ba5119acf18cd38c14815f0e8fb16 with a better job of applying the necessary gecko-side patches

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9420c5e883214d72eefa055bb574a3d12e42efcb
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ace60c8b28bdb6a54ed67235ad9e9b5e8fefc441

Windows has gone more red but it might just be that we're failing more reftests than before. I'll look closer. Otherwise things are looking decent.

Comment 12

[Kartikaya Gupta (email:[e-mail])]

WR @ 7184d6b6445ba5119acf18cd38c14815f0e8fb16

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf1f0e821ba910be62e3443687e6e2a7cf288915
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f08626e59d54e2f149ef450fdd1e75bf70bc7403
https://treeherder.mozilla.org/#/jobs?repo=try&revision=94de55fce94e0d6a2f31973397ecaa0eecfdf699
https://treeherder.mozilla.org/#/jobs?repo=try&revision=326b5e9cbc8efa50c62ea087086e3bf42e51ace6
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c9513c4c50d498c748f33de00434dc3b84d8e85a
https://treeherder.mozilla.org/#/jobs?repo=try&revision=147affc252e08cb44f36bf8eebafc5e848c0eb8e
https://treeherder.mozilla.org/#/jobs?repo=try&revision=98d356d779693e0be37d30a1d552f0a27ab7affa
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0136dc220ea9c498760df4090ab5f47f83f1a5d5
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ad75ddc115a75f102afbba19651409672b6e8c0
https://treeherder.mozilla.org/#/jobs?repo=try&revision=485f8a02c2df32fbc66a13725b0b267a8299df94
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ff0d17d232f7df879cf6929cabdbdb651bae46e1
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bd0c24f524d0a0389294b3b0a57de5b006f2c2a5

Same as before (Linux green, some Windows reftests red)

WR @ d5bf40389e54ee96fc7e313fbe1a590f1464b3a4

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3ec0fcc24ac51bc3e7f3592f51f8d4500a5cada
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2ecef6debfbd65ffe3fbb0289c14922809cfab3b

Ditto

WR @ eceed87325ef1dbbd220a4ccbb9e7860f2679428

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5f4754657c4cfc7c1c0e797992a57c06d6a4e3a
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e81014896d5dc6cf83c694f6790d0b957ee9c63e

Two reftests are orange now - one can be fuzzed I think but the other one is all wrong. Probably regression from servo/webrender#2408 since that's the only thing in the range.

Comment 13

[Kartikaya Gupta (email:[e-mail])]

So the windows redness seems to be a real issue. Back at d96d02bd9b8141c2fa4a692ee0b35220be0a8873 (comment 2) things were good, in that the windows opt reftests run ran 15720 reftests and 270 of them were unexpected in some way (233 unexpected-fail, 37 unexpected-pass). However as of 7184d6b6445ba5119acf18cd38c14815f0e8fb16 (comment 11) the windows opt reftest run only gets through around 594 reftests (of which 255 are unexpected-fail) before the logfile hits the size limit and the job is aborted. That's what makes it show up as red. So the windows opt run is now failing a *lot* more reftests than it used to, which I consider a regression.

The regression range right now is d96d02bd9b8141c2fa4a692ee0b35220be0a8873..7184d6b6445ba5119acf18cd38c14815f0e8fb16 which is pretty wide but thankfully all the bustage was on the gecko side and so I can do additional try pushes to narrow this down. That's what I'm working on now.


I also kicked off another try push to verify the linux reftest failures came from servo/webrender#2408 and not some random gecko change.

Comment 14

[Kartikaya Gupta (email:[e-mail])]

Also for the windows reftest failures, it looks like in a lot of cases the wrong page is getting rendered. This might either be an existing race condition that got aggravated, or could be something like a bug in the texture cache where we're pulling the wrong textures. I downloaded one of those builds and ran it locally and for the most part it seems to work ok. However I did notice at least one problem, on about:newtab the firefox logo on the top-left was getting replaced by a tiled version of the refresh icon from the URL bar. So that might also point to a texture cache problem.

Comment 15

[Kartikaya Gupta (email:[e-mail])]

Another problem I noticed with the windows build was that loading layout/reftests/reftest-sanity/647192-1.html shows red for the scrollbars, and it doesn't do that in a current nightly on the same profile or on OS X. This reftest is one of the ones failing in the try push, I picked it at random to see if it would repro locally and it did.

Comment 16

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #13)
> I also kicked off another try push to verify the linux reftest failures came
> from servo/webrender#2408 and not some random gecko change.

This is verified now, left a comment at https://github.com/servo/webrender/pull/2408#issuecomment-364930396

Comment 17

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #13)
> The regression range right now is
> d96d02bd9b8141c2fa4a692ee0b35220be0a8873..
> 7184d6b6445ba5119acf18cd38c14815f0e8fb16 which is pretty wide but thankfully
> all the bustage was on the gecko side and so I can do additional try pushes
> to narrow this down. That's what I'm working on now.

In WR-chronological order:

WR @ 30f2cd5e648bbcadbe8200d479de3f7d76089d0b
https://treeherder.mozilla.org/#/jobs?repo=try&revision=36e1f1cd1c046d07d52b4f129b337788c81f486e
==> good

WR @ 7ae075a3238915530144727896e5d275d4057f34
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3f73d0d02e5fa26bb2e4734c4a85fa8662d03de3
==> good

WR @ b7771429fd8afdea91a0e5d4657bcbaee994909d
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ca5fd387d2ab74a445a9f91ed982f3b9e8dfb7a0
==> good

WR @ a492386a97fb73ffc449dc2ee4bee9163238d083
https://treeherder.mozilla.org/#/jobs?repo=try&revision=daa19eb9727c5011333d391c9555bd4749ab6dd4
==> good

WR @ 342bc314db94aa439b2001249c5f24ccfcbccc22
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bfbd4dad817e82c997ee99aea0d152f938f1ebef
==> good

WR @ 65c68cc1d1d8958c103fa3a528b05d81e27783d3
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7c21bcbd670096d2ef119605b5c51ca6b72f7968
==> bad

WR @ d51d6ec0def765055af6a1e2b2bc8b9bd6e89baf
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2e71f7fa91c2a39d6939f7269bcabf9b9f8b0abb
==> pending

So the regression range is narrowed to 342bc314db94aa439b2001249c5f24ccfcbccc22..65c68cc1d1d8958c103fa3a528b05d81e27783d3:
* 65c68cc1 Auto merge of #2399 - glennw:image-brush, r=kvark
* e0aebb58 Add brush_image shader.

So from servo/webrender#2399 - I'll leave a comment there.

Comment 18

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1436058 - Update webrender to 342bc314db94aa439b2001249c5f24ccfcbccc22.

Review commit: https://reviewboard.mozilla.org/r/219524/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/219524/

Comment 19

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1436058 - Update Cargo lockfiles and re-vendor rust dependencies.

Review commit: https://reviewboard.mozilla.org/r/219526/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/219526/

Comment 20

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1436058 - Update for log 0.4 bump in WR PR 2385.

Review commit: https://reviewboard.mozilla.org/r/219528/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/219528/

Comment 21

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1436058 - Remove async image pipelines when the renderer notifies pipelines removed.

Patch originally attached to - and reviewed by sotaro on - bug 1432708.

Review commit: https://reviewboard.mozilla.org/r/219530/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/219530/

Comment 22

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1436058 - Don't remove a WebRender pipeline if we re-created one with the same id.

Patch originally attached to - and reviewed by sotaro on - bug 1432708.

Review commit: https://reviewboard.mozilla.org/r/219532/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/219532/

Comment 23

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 8950271 [details]
Bug 1436058 - Remove async image pipelines when the renderer notifies pipelines removed.

https://reviewboard.mozilla.org/r/219530/#review225254

Comment 24

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 8950272 [details]
Bug 1436058 - Don't remove a WebRender pipeline if we re-created one with the same id.

https://reviewboard.mozilla.org/r/219532/#review225256

Comment 25

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8950268 [details]
Bug 1436058 - Update webrender to 342bc314db94aa439b2001249c5f24ccfcbccc22.

https://reviewboard.mozilla.org/r/219524/#review225258

Comment 26

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8950269 [details]
Bug 1436058 - Update Cargo lockfiles and re-vendor rust dependencies.

https://reviewboard.mozilla.org/r/219526/#review225262

Comment 27

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8950270 [details]
Bug 1436058 - Update for log 0.4 bump in WR PR 2385.

https://reviewboard.mozilla.org/r/219528/#review225266

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a4dc76286299
Update webrender to 342bc314db94aa439b2001249c5f24ccfcbccc22. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/9cca7938b734
Update Cargo lockfiles and re-vendor rust dependencies. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/cacc7bcb3220
Update for log 0.4 bump in WR PR 2385. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/a408b8f2f7ab
Remove async image pipelines when the renderer notifies pipelines removed. r=kats
https://hg.mozilla.org/integration/autoland/rev/f53e340b051c
Don't remove a WebRender pipeline if we re-created one with the same id. r=kats

Comment 29

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/a4dc76286299
https://hg.mozilla.org/mozilla-central/rev/9cca7938b734
https://hg.mozilla.org/mozilla-central/rev/cacc7bcb3220
https://hg.mozilla.org/mozilla-central/rev/a408b8f2f7ab
https://hg.mozilla.org/mozilla-central/rev/f53e340b051c

Comment 30

[Glenn Watson [:gw]]

There is a fix / workaround for the mipmap issue here - https://github.com/servo/webrender/pull/2409.

I have a local Windows build with this changeset, but I'm currently unable to reproduce any issues. All of the reftests I've run on Windows appear to run as expected (the sanity, flexbox and layout bugs directory, at least).

Comment 31

[Glenn Watson [:gw]]

I have a possible theory on the Windows bustage.

If I run the flexbox/ reftests on both Linux and Windows the memory usage is relatively stable.

However, when I run the image/test/reftest reftests, there appears to be a significant memory leak. On Windows, the memory usage grows by ~100MB/sec until the process seems to go idle / timeout at ~4 GB allocated. Since the default Windows build is 32 bit, that would seem like an OOM is probably occurring somewhere. On Linux, I see the same memory growth, but since it's a 64-bit build, it doesn't have the same address space issues.

Is it possible that this could be the cause of the Windows issues on CI? There are a couple of patches in this changeset that deal with managing lifetime of image and pipeline resources, could they be related?

Comment 32

[Kartikaya Gupta (email:[e-mail])]

(Replied on bug 1437572)