1628642 - Intermittent toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html | Correctly intercepted page content - got "Start Middle\n", expected "Start Middle\n End"

Status: RESOLVED FIXED

(WebExtensions :: General, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=296919334&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Zt4Z1hxxQYK5xIH0SmYNyw/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Zt4Z1hxxQYK5xIH0SmYNyw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2020-04-09T09:24:34.911Z] 09:24:34     INFO - TEST-START | toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html
[task 2020-04-09T09:24:34.996Z] 09:24:34     INFO - GECKO(2265) | ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-04-09T09:24:35.010Z] 09:24:35     INFO - TEST-INFO | started process screencapture
[task 2020-04-09T09:24:35.177Z] 09:24:35     INFO - TEST-INFO | screencapture: exit 0
[task 2020-04-09T09:24:35.177Z] 09:24:35     INFO - Buffered messages logged at 09:24:34
[task 2020-04-09T09:24:35.178Z] 09:24:35     INFO - add_task | Entering test 
[task 2020-04-09T09:24:35.178Z] 09:24:35     INFO - Extension loaded
[task 2020-04-09T09:24:35.178Z] 09:24:35     INFO - Extension loaded
[task 2020-04-09T09:24:35.178Z] 09:24:35     INFO - Buffered messages finished
[task 2020-04-09T09:24:35.178Z] 09:24:35     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html | Correctly intercepted page content - got "Start Middle\n", expected "Start Middle\n End"
[task 2020-04-09T09:24:35.178Z] 09:24:35     INFO -     SimpleTest.is@SimpleTest/SimpleTest.js:383:14
[task 2020-04-09T09:24:35.178Z] 09:24:35     INFO -     @toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html:82:14
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO -     async*nextTick/<@SimpleTest/SimpleTest.js:2090:34
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO -     nextTick@SimpleTest/SimpleTest.js:2115:11
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO -     setTimeout handler*SimpleTest_setTimeoutShim@SimpleTest/SimpleTest.js:788:41
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO -     add_task@SimpleTest/SimpleTest.js:2045:17
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO -     @toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html:18:9
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO - add_task | Leaving test 
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO - GECKO(2265) | MEMORY STAT | vsize 6723MB | residentFast 92MB | heapAllocated 12MB
[task 2020-04-09T09:24:35.179Z] 09:24:35     INFO - TEST-OK | toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html | took 118ms

Comment 1

[Bogdan Tara[:bogdan_tara | bogdant]]

This escalated to Tier 1:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&searchStr=linux%2C18.04%2Cx64%2Copt%2Cmochitests%2Ctest-linux1804-64%2Fopt-mochitest-plain-e10s-5%2Cm%285%29&revision=33d2485721c6ffd20a952d698754041a14be11b3&selectedJob=296930131

Comment 2

[Bogdan Tara[:bogdan_tara | bogdant]]

It started to pop up on Windows too.
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=296958731&repo=autoland&lineNumber=9742

Comment 3

[Intermittent Failures Robot]

52 failures in 1016 pushes (0.051 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 31
• mozilla-central: 13
• try: 8

Platform breakdown:

• linux1804-64: 13
• android-em-7-0-x86_64: 6
• macosx1014-64: 2
• linux1804-64-shippable: 4
• macosx1014-64-shippable: 1
• windows7-32: 1
• linux1804-64-qr: 8
• linux1804-64-tsan: 3
• linux1804-64-devedition: 1
• windows10-64-shippable: 3
• linux1804-64-shippable-qr: 4
• windows10-64: 1
• linux1804-64-asan: 2
• windows10-64-shippable-qr: 2
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-04-09&endday=2020-04-09&tree=all

Comment 4

[Intermittent Failures Robot]

39 failures in 571 pushes (0.068 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 13
• try: 5

Platform breakdown:

• windows7-32-shippable: 5
• windows10-64-shippable: 3
• windows10-64-shippable-qr: 3
• linux1804-64-shippable: 2
• windows10-64-qr: 1
• linux1804-64: 9
• linux1804-64-shippable-qr: 4
• android-em-7-0-x86_64: 5
• linux1804-64-asan: 3
• linux1804-64-qr: 1
• macosx1014-64: 1
• linux1804-64-tsan: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-04-10&endday=2020-04-10&tree=all

Comment 5

[Shane Caraveo (:mixedpuppy)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=27de46f16eff3f8d03eba9fc39e5db12a69068e3

Comment 6

[Shane Caraveo (:mixedpuppy)]

The test run in comment 5 is to see if this is a timing issue in the test itself.

Comment 7

[Intermittent Failures Robot]

37 failures in 231 pushes (0.16 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 9
• try: 5

Platform breakdown:

• linux1804-64-shippable-qr: 8
• linux1804-64-shippable: 4
• linux1804-64: 8
• linux1804-64-qr: 8
• linux1804-64-asan: 3
• linux1804-64-devedition: 1
• windows10-64: 1
• macosx1014-64-shippable: 1
• linux1804-64-tsan: 2
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-04-11&endday=2020-04-11&tree=all

Comment 8

[Intermittent Failures Robot]

155 failures in 4823 pushes (0.032 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 92
• mozilla-central: 42
• try: 21

Platform breakdown:

• linux1804-64: 34
• android-em-7-0-x86_64: 12
• macosx1014-64: 3
• linux1804-64-shippable: 11
• macosx1014-64-shippable: 3
• windows7-32: 1
• linux1804-64-qr: 26
• linux1804-64-tsan: 9
• linux1804-64-devedition: 2
• windows10-64-shippable: 7
• linux1804-64-shippable-qr: 18
• windows10-64: 3
• linux1804-64-asan: 9
• windows10-64-shippable-qr: 6
• windows7-32-shippable: 10
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 9

[Natalia Csoregi [:nataliaCs]]

The failure appeared: https://treeherder.mozilla.org/#/jobs?repo=try&revision=27de46f16eff3f8d03eba9fc39e5db12a69068e3&selectedJob=297251812&group_state=expanded

Shane, should we disable the test for the time being?
Thank you.

Comment 10

[Matt Woodrow (:mattwoodrow)]

Pernosco link: https://pernos.co/debug/igUdXYLFWmZBXz2g0FTaAw/index.html

I'll try to debug this tomorrow.

Comment 11

[Matt Woodrow (:mattwoodrow)]

It looks like the issue happens when the second extension runs first.

The second extension is emitting data in two chunks (one with the page content, one with the extra appended data).

The first extension handles ondata once, prepending its content and then disconnecting. It doesn't wait for the second ondata to forward that through.

I think the first extension should do its prepending in onstart, directly copy all data in ondata, and then disconnect in onstop.

Comment 12

[Matt Woodrow (:mattwoodrow)]

Actually, that seems like it should behave fine, since the first extension is disconnecting.

It looks like we get to StreamFilterParent::DoSendData with aData=" End" (the second chunk of data from second extension being received by the filter parent for first extension). At this point mState==Disconnecting, so we don't call SendData, and just drop the data.

It seems like a pretty bad problem if disconnecting can race with later packets of the response and cause them to be dropped.

Comment 13

[Matt Woodrow (:mattwoodrow)]

I'm not entirely sure how to fix this.

We could post the data back to the IOThread to call BufferData with it, but it's possible in the mean time for another OnDataAvaible call to have arrived, seen mState==Disconnecting and also called BufferData ahead of us (and thus in the wrong order).

I suspect we need to route all data through the actor thread (which is the only thread where mState is non-racy) so that we can resolve these ordering races.

Any thoughts kmag?

Comment 14

[Intermittent Failures Robot]

67 failures in 500 pushes (0.134 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 39
• mozilla-central: 14
• try: 14

Platform breakdown:

• linux1804-64-asan: 5
• linux1804-64-qr: 24
• macosx1014-64: 2
• linux1804-64-shippable-qr: 9
• windows10-64-shippable-qr: 1
• linux1804-64-shippable: 8
• macosx1014-64-shippable: 2
• linux1804-64: 5
• windows10-64-shippable: 2
• linux1804-64-tsan: 1
• windows7-32: 2
• android-em-7-0-x86_64: 4
• windows10-64: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-04-13&endday=2020-04-13&tree=all

Comment 15

[Cosmin Sabou [:CosminS]]

Attachment: Bug 1628642 - Disable test_ext_streamfilter_multiple.html on all platforms for frequent failures. r=aryx

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7b84e0046f62
Disable test_ext_streamfilter_multiple.html on all platforms for frequent failures. r=aryx

Comment 17

[Shane Caraveo (:mixedpuppy)]

Given that this seems like a high probability of data loss or corruption and that it is not so uncommon to have a couple extensions doing filters, I think this bug needs to be a priority. I'm not certain how best to handle this issue, but it's probably also going to be related to bug 1629299.

Comment 18

[Matt Woodrow (:mattwoodrow)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #17)

Given that this seems like a high probability of data loss or corruption and that it is not so uncommon to have a couple extensions doing filters, I think this bug needs to be a priority. I'm not certain how best to handle this issue, but it's probably also going to be related to bug 1629299.

I think it could happen even with a single extension, it just needs a network response delivered in multiple OnDataAvailable calls, and an extension that uses disconnect() from the not-last one.

As for fixing this, we have mBufferedData and the two-stage+round-trip disconnect to prevent races due to the asynchronous connection between StreamFilterParent/Child. Unfortunately, we also have a second async step from IO thread to actor thread, which isn't handled.

One solution could be to extend the round-trip disconnect synchronization to go through the IO thread (and also the main thread somehow).

Another would be to unconditionally forward OnDataAvailable to the actor thread, and only make decisions on that thread. This might add some more overhead, but seems simpler to get right.

Either way, I think we want to more clearly mark which thread each member variable is valid on, and try to racy accesses.

Comment 19

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/7b84e0046f62

Comment 20

[Intermittent Failures Robot]

47 failures in 1019 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 24
• try: 15
• mozilla-central: 8

Platform breakdown:

• linux1804-64-qr: 20
• android-em-7-0-x86_64: 3
• linux1804-64-shippable: 4
• linux1804-64: 8
• linux1804-64-shippable-qr: 4
• linux1804-64-tsan: 1
• linux1804-64-asan: 5
• macosx1014-64-shippable: 1
• windows10-64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-04-14&endday=2020-04-14&tree=all

Comment 21

[Intermittent Failures Robot]

119 failures in 4528 pushes (0.026 failures/push) were associated with this bug in the last 7 days.

This is the #5 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 63
• mozilla-central: 22
• try: 34

Platform breakdown:

• linux1804-64-asan: 10
• linux1804-64-qr: 45
• macosx1014-64: 2
• linux1804-64-shippable-qr: 13
• windows10-64-shippable-qr: 2
• linux1804-64-shippable: 13
• macosx1014-64-shippable: 3
• linux1804-64: 15
• windows10-64-shippable: 2
• linux1804-64-tsan: 2
• windows7-32: 2
• android-em-7-0-x86_64: 8
• windows10-64: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 22

[Kris Maglione [:kmag]]

I think we need to check for the Disconnecting state in the DoSendData callback, and buffer the data rather than dropping it if we are disconnecting.

Comment 23

[Shane Caraveo (:mixedpuppy)]

@CosminS There were a few race fixes a month or so after this test was disabled, can we reenable the test for a bit and see if we get the intermittent back?

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ef0e4ae63cfd
Unskip test_ext_streamfilter_multiple.html to see if the failures are still present. a=testonly

Comment 25

[Cosmin Sabou [:CosminS]]

Just pushed a testonly change and will circle back next week to see if the failures are still around.

Comment 26

[Shane Caraveo (:mixedpuppy)]

ni? me if they come back.

Comment 27

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/ef0e4ae63cfd

Comment 28

[Intermittent Failures Robot]

3 failures in 4230 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• linux1804-64-asan: 2
• android-em-7-0-x86_64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 29

[Cosmin Sabou [:CosminS]]

Shane, there are 22 failures since the test was re-enabled, most on linux1804-64-asan: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-10-21&endday=2020-10-28&tree=trunk&bug=1628642
Recent log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=320049341&repo=autoland&lineNumber=4756

[task 2020-10-28T19:06:25.301Z] 19:06:25     INFO - TEST-START | toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html
[task 2020-10-28T19:06:25.760Z] 19:06:25     INFO - GECKO(3196) | ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-10-28T19:06:25.788Z] 19:06:25     INFO - TEST-INFO | started process screentopng
[task 2020-10-28T19:06:26.108Z] 19:06:26     INFO - TEST-INFO | screentopng: exit 0
[task 2020-10-28T19:06:26.109Z] 19:06:26     INFO - Buffered messages logged at 19:06:25
[task 2020-10-28T19:06:26.109Z] 19:06:26     INFO - add_task | Entering test 
[task 2020-10-28T19:06:26.109Z] 19:06:26     INFO - Extension loaded
[task 2020-10-28T19:06:26.109Z] 19:06:26     INFO - Extension loaded
[task 2020-10-28T19:06:26.109Z] 19:06:26     INFO - Buffered messages finished
[task 2020-10-28T19:06:26.110Z] 19:06:26     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html | Correctly intercepted page content - got "Start Middle\n", expected "Start Middle\n End"
[task 2020-10-28T19:06:26.110Z] 19:06:26     INFO -     SimpleTest.is@SimpleTest/SimpleTest.js:500:14
[task 2020-10-28T19:06:26.110Z] 19:06:26     INFO -     @toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html:82:14
[task 2020-10-28T19:06:26.111Z] 19:06:26     INFO - add_task | Leaving test 
[task 2020-10-28T19:06:26.112Z] 19:06:26     INFO - GECKO(3196) | MEMORY STAT | vsize 20974599MB | residentFast 861MB
[task 2020-10-28T19:06:26.112Z] 19:06:26     INFO - TEST-OK | toolkit/components/extensions/test/mochitest/test_ext_streamfilter_multiple.html | took 580ms
[task 2020-10-28T19:06:26.113Z] 19:06:26     INFO - TEST-START | toolkit/components/extensions/test/mochitest/test_ext_streamfilter_processswitch.html

Could you have a look over them? Thank you.

Comment 30

[Intermittent Failures Robot]

34 failures in 4275 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #33 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 30
• mozilla-central: 4

Platform breakdown:

• linux1804-64-qr: 4
• linux1804-64-asan: 15
• windows7-32: 8
• linux1804-64-shippable: 3
• linux1804-64: 2
• macosx1014-64: 1
• android-em-7-0-x86_64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 31

[Shane Caraveo (:mixedpuppy)]

Ok, this will require some more investigation. I'm wondering if disabling it on linux and win7 (or maybe make it osx-only) would be sufficient to bring the numbers down low enough to be acceptable, that way at least the test is running.

Comment 32

[Cosmin Sabou [:CosminS]]

Attachment: Bug 1628642 - Disable test_ext_streamfilter_multiple.html on linux and !debug all platforms. r=mixedpuppy,#intermittent-reviewers

Comment 33

[Cosmin Sabou [:CosminS]]

Most failures are on opt builds and linux so made a disable patch for that to bring the failures down.
https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2020-10-08&endday=2020-11-07&tree=trunk&bug=1628642

Comment 34

[Intermittent Failures Robot]

39 failures in 4128 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #33 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 13
• autoland: 25
• try: 1

Platform breakdown:

• macosx1014-64: 3
• windows7-32-shippable: 1
• linux1804-64-asan: 18
• android-em-7-0-x86_64-shippable-qr: 1
• linux1804-64-shippable-qr: 1
• linux1804-64-shippable: 3
• windows10-64-shippable: 2
• linux1804-64-qr: 3
• windows7-32: 2
• macosx1014-64-qr: 1
• windows10-64-shippable-qr: 1
• android-em-7-0-x86_64: 1
• android-em-7-0-x86_64-qr: 1
• linux1804-64-tsan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628642&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/32fe9aa2c5c4
Disable test_ext_streamfilter_multiple.html on linux and !debug all platforms. r=mixedpuppy,#intermittent-reviewers

Comment 36

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/32fe9aa2c5c4

Comment 37

[Intermittent Failures Robot]

12 failures in 4040 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 8

Platform breakdown:

• windows7-32-shippable: 2
• linux1804-64-shippable: 2
• linux1804-64-asan: 5
• windows7-32: 2
• android-em-7-0-x86_64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2020-11-09&endday=2020-11-15&tree=all

Comment 38

[Intermittent Failures Robot]

3 failures in 4810 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 1

Platform breakdown:

• macosx1014-64-qr: 2
• android-em-7-0-x86_64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2020-11-16&endday=2020-11-22&tree=all

Comment 39

[Intermittent Failures Robot]

2 failures in 3712 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• macosx1014-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2020-11-30&endday=2020-12-06&tree=all

Comment 40

[Intermittent Failures Robot]

3 failures in 4060 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• android-em-7-0-x86_64-qr: 1
• macosx1014-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 41

[Intermittent Failures Robot]

2 failures in 3421 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform breakdown:

• macosx1014-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2020-12-14&endday=2020-12-20&tree=all

Comment 42

[Intermittent Failures Robot]

1 failures in 1450 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2020-12-21&endday=2020-12-27&tree=all

Comment 43

[Intermittent Failures Robot]

4 failures in 525 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 1
• macosx1014-64-qr: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2020-12-28&endday=2021-01-03&tree=all

Comment 44

[Intermittent Failures Robot]

2 failures in 2956 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform breakdown:

• macosx1014-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-01-04&endday=2021-01-10&tree=all

Comment 45

[Intermittent Failures Robot]

2 failures in 3810 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows7-32: 1
• macosx1014-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-01-11&endday=2021-01-17&tree=all

Comment 46

[Intermittent Failures Robot]

5 failures in 3488 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 3

Platform and build breakdown:

• macosx1014-64-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-01-18&endday=2021-01-24&tree=all

Comment 47

[Intermittent Failures Robot]

4 failures in 3891 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform and build breakdown:

• macosx1014-64-qr: 2
  • debug: 2
• android-em-7-0-x86_64-qr: 1
  • debug: 1
• android-em-7-0-x86_64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-01-25&endday=2021-01-31&tree=all

Comment 48

[Intermittent Failures Robot]

3 failures in 3966 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-02-01&endday=2021-02-07&tree=all

Comment 49

[Intermittent Failures Robot]

3 failures in 4322 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• macosx1014-64-qr: 2
  • debug: 2
• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 50

[Intermittent Failures Robot]

4 failures in 4327 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64: 1
  • debug: 1
• macosx1014-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 51

[Intermittent Failures Robot]

3 failures in 3683 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform and build breakdown:

• macosx1014-64-qr: 2
  • debug: 2
• android-em-7-0-x86_64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 52

[Intermittent Failures Robot]

7 failures in 4622 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 3
  • debug: 3
• android-em-7-0-x86_64: 1
  • debug: 1
• macosx1014-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 53

[Intermittent Failures Robot]

8 failures in 4257 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 6

Platform and build breakdown:

• macosx1014-64-qr: 7
  • debug: 7
• windows7-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 54

[Intermittent Failures Robot]

4 failures in 3967 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64-qr: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-03-22&endday=2021-03-28&tree=all

Comment 55

[Intermittent Failures Robot]

5 failures in 3125 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5

Platform and build breakdown:

• macosx1014-64-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-03-29&endday=2021-04-04&tree=all

Comment 56

[Intermittent Failures Robot]

3 failures in 3854 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• macosx1014-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 57

[Intermittent Failures Robot]

1 failures in 3224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 58

[Intermittent Failures Robot]

1 failures in 3061 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-05-17&endday=2021-05-23&tree=all

Comment 59

[Intermittent Failures Robot]

1 failures in 3502 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-06-07&endday=2021-06-13&tree=all

Comment 60

[Intermittent Failures Robot]

1 failures in 2940 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-07-19&endday=2021-07-25&tree=all

Comment 61

[Intermittent Failures Robot]

1 failures in 3266 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-09-06&endday=2021-09-12&tree=all

Comment 62

[Intermittent Failures Robot]

1 failures in 3315 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-10-11&endday=2021-10-17&tree=all

Comment 63

[Intermittent Failures Robot]

1 failures in 3186 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-10-18&endday=2021-10-24&tree=all

Comment 64

[Intermittent Failures Robot]

1 failures in 2845 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-10-25&endday=2021-10-31&tree=all

Comment 65

[Intermittent Failures Robot]

1 failures in 3030 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-11-01&endday=2021-11-07&tree=all

Comment 66

[Intermittent Failures Robot]

1 failures in 2723 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-11-08&endday=2021-11-14&tree=all

Comment 67

[Intermittent Failures Robot]

1 failures in 3170 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-11-29&endday=2021-12-05&tree=all

Comment 68

[Intermittent Failures Robot]

2 failures in 3263 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-12-13&endday=2021-12-19&tree=all

Comment 69

[Intermittent Failures Robot]

1 failures in 1011 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2021-12-27&endday=2022-01-02&tree=all

Comment 70

[Intermittent Failures Robot]

1 failures in 3060 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-01-10&endday=2022-01-16&tree=all

Comment 71

[Intermittent Failures Robot]

1 failures in 3444 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-01-17&endday=2022-01-23&tree=all

Comment 72

[Intermittent Failures Robot]

1 failures in 3304 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-03-14&endday=2022-03-20&tree=all

Comment 73

[Intermittent Failures Robot]

1 failures in 3323 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-03-28&endday=2022-04-03&tree=all

Comment 74

[Intermittent Failures Robot]

1 failures in 3317 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-05-09&endday=2022-05-15&tree=all

Comment 75

[Intermittent Failures Robot]

1 failures in 3336 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 76

[Intermittent Failures Robot]

3 failures in 3170 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1
• windows10-64-2004-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-06-13&endday=2022-06-19&tree=all

Comment 77

[Intermittent Failures Robot]

1 failures in 3075 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1628642&startday=2022-06-20&endday=2022-06-26&tree=all

Comment 79

[Rob Wu [:robwu]]

I was looking into this, and found why this is happening. In short, data is dropped due to a race condition.

I added some logging (C++ side patch in bug 1765991, plus enabling logging in the test itself with https://hg-edge.mozilla.org/try/rev/4b27c4f993461b5e2ac0a0fd5547d5cbbb231c5c). The relevant log extract is below. I'll attach the patch which has more explanations.

With logging: https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=YdQgBGHYQdCQf4ulapfTTQ.0&revision=e65ceeddcef65d516eaa15c13a011e2f1de42d91

HTML5 Parser]: D/StreamFilterParent StreamFilterParent::OnDataAvailable @p1      -- "End" data
Socket Thread]: D/StreamFilterParent StreamFilterParent::Destroy @p2
Socket Thread]: D/StreamFilterParent StreamFilterParent::RecvWrite @p1 mState=2  -- "Start "
HTML5 Parser]: D/StreamFilterParent StreamFilterParent::Write @p1 mState=2       -- "Start"
Socket Thread]: D/StreamFilterParent StreamFilterParent::RecvWrite @p1 mState=2  -- "Middle"
Socket Thread]: D/StreamFilterParent StreamFilterParent::RecvDisconnect @p1 mState=2
Socket Thread]: D/StreamFilterParent StreamFilterParent::DoSendData @p1 mState=5 -- "End"  mState != 2 - dropped!

Comment 80

[Rob Wu [:robwu]]

Attachment: Bug 1628642 - Avoid data loss in DoSendData when Disconnecting

When state is Disconnecting, OnDataAvailable (on the IO thread) should
buffer data. Otherwise it dispatches DoSendData to the actor thread,
which supposedly sends data to the StreamFilterChild.

But because the Disconnecting state is set from the actor, it is
possible for a race condition to happen: If the Disconnecting state is
entered between the dispatch and the actual running of DoSendData(),
then before this patch the data would be dropped.

This patch fixes the issue by putting the data back at the front of the
queue, where it logically should have been.

Comment 82

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S4. However, the following bug duplicate has higher severity:

• Bug 1644621: S3

:robwu, could you consider increasing the severity of this bug to S3?

For more information, please visit BugBot documentation.

Comment 83

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/25caad538a99
Avoid data loss in DoSendData when Disconnecting r=valentin,jesup

Comment 84

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/25caad538a99