1626278 - [wfh] Session statistics disappear from about:webrtc when exiting video-call with close tab

Status: RESOLVED FIXED

(Core :: WebRTC: Networking, defect, P3)

Description

[Peter Magyari (Desktop QA)]

Affected Versions
Nightly 76.0a1 (2020-03-31)
Beta 75.0b11 (2020-03-28)

Affected Platforms
All

Steps to reproduce

1. Launch Firefox
2. Navigate to https://whereby.com and log in
3. Open about:webrtc in a new tab
4. Session Statistics section shows a log looking like:
   "[ 6442450952 ] https://meet.google.com/_meet/uhz-ivbg-apr?hs=49 Mon Mar 30 2020 17:38:17 GMT+0300 (Eastern European Summer Time)"
5. Exit the call using "Close Tab"
6. Refresh the tab with about:webrtc

Expected Result
The logs are kept in about:webrtc and the string "(closed)" appears in the log title after leaving the call.

Actual Result
The logs are lost.

Note
Happens only when you exit the call using close tab, the logs are kept if you exit the call by pressing "leave".

Comment 1

[Byron Campen [:bwc]]

Hmm. Maybe a teardown race. Will have to look into it.

Comment 2

[Daniel Bodea [:danibodea]]

Attachment: aboutWebrtc_lost_logs.html

Apparently this also occurs on the Zoom Web app.
These are the logs of a session with 3 users, 2 active logs displayed during the call and one log displayed after the call was closed.

Comment 3

[Daniel Bodea [:danibodea]]

Attachment: aboutWebrtc_lost_logs.html

Description above. [Nevermind this]

Comment 4

[Firefox Bug Husbandry Bot]

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is P3 (Backlog,) indicating it has been triaged, the bug's Severity is being updated to S3 (normal.)

Comment 5

[Daniel Bodea [:danibodea]]

Session statistics logs are also lost when closing the call by the web-app's "Leave"/"Close call" buttons.

This issue was seen to reproduce intermittently with Jitsi, Facebook and Whereby on Mac OS 10.12.6. More often with Whereby then with the others.

Comment 6

[Daniel Bodea [:danibodea]]

In the case of Google Meet, Vonage, Amazon Chime on Ubuntu 20, the logs get deleted when closing the tab altogether.

Comment 9

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Use NS_INLINE_DECL_THREADSAFE_REFCOUNTING_WITH_DESTROY to ensure that MediaTransportHandler stays around long enough to finish pending stats queries. r?mjf

Comment 10

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7cffa459725cae7089d56c972b5cde4b4442b519

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7b4dc0692c0a1bba47c93acbb370bb6463b497e6

Comment 11

[Byron Campen [:bwc]]

Still trying to work the kinks out of this...

https://treeherder.mozilla.org/#/jobs?repo=try&revision=36518cc7a936a9394b48a995bf478e4af4ee22c3

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5391143efcf40c2f0d250357ddc49c7368f3105d

Comment 12

[Byron Campen [:bwc]]

Ok, so the problem here is that the ICE ctx is being torn down before the final stats query completes sometimes. However, delaying the teardown of the ICE ctx until the stats query finishes sometimes results in STS shutting down before the ICE ctx is destroyed, leading to assertions.

Comment 13

[Byron Campen [:bwc]]

Another problem is that sometimes closing the window causes the content process to be torn down before the final stats query completes. We might be able to compensate for this by getting rid of PeerConnectionCtx::mStatsForClosedPeerConnections, and instead relying on PeerConnectioonCtx::mLastReports, with a final update when a pc is closed.

Comment 14

[Byron Campen [:bwc]]

It also seems that the only long-term storage we do of these stats is in PeerConnectionCtx, which will disappear when its content process disappears. We really need to be storing this stuff in the parent process, presumably in WebrtcGlobalInformation.

Comment 15

[Byron Campen [:bwc]]

Looking at WebrtcGlobalInformation, I think things like SetDebugLevel and SetAecDebug aren't working at all. Those appear to only take effect on the process on which that API is invoked, but that API is only invoked on the parent process; there is no fan-out to the content processes like there is for GetAllStats/GetLogging.

Comment 16

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1b303e5a186b064cc65116304555816c95df391f

Comment 17

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d2b592423b488fb62eddc3435c93336f4e4718f

Comment 18

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Simplify/promisify PWebrtcGlobal. r?ng

Depends on D87180

Comment 19

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Add a stats stash to WebrtcGlobalInformation, to avoid stats from being lost when a content process goes away. r?ng

Depends on D88496

Comment 20

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Filter stats for closed PeerConnections the same way we do for live ones. r?ng

Depends on D88497

Comment 21

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c5663fa313a21750a2fd7ba0c74cb90a147dd1f0

Comment 22

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8915ceaa7ae0d1eadd3c656451672f9ee26eabd1

Comment 23

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5d86ebc01106077e45b26036aac51a7cdcbeb65b

Comment 24

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=34d57344c8e1cfcd0f5de632c06c861c9210b44b

Comment 25

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3b37e7f55bad6610a340f3cc07525db67d5fbc5f

Comment 26

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=952b572643b61d2e06d8b8077133e5448885efbb

Comment 27

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Stash for ICE logs, and move log filtering into a single place. r?ng

Depends on D88498

Comment 28

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5cd266e7a37dac9f13388a75974d9ffd88e1afb6

Comment 29

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f849a189588d1baa69a81320f177ace6e43dc87b

Comment 30

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=73158334d55efda7078465d4c2bf707afabca95c

Comment 31

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5bcf2dae2788e0e4a40944a8efa202569413679a

Comment 32

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=29627b0e9818e0fbd07e9ad15bb8f2fa2fb7a877

Comment 33

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1e52b3a647ec401cd58dabcead54b9db2286f3ed

Comment 34

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=191c1b87bc9f6ad772419d7b6a960d45712347f7

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b648d7eac3d599a79a6a09b1296ede536172c910

Try looks good.

Comment 35

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Move sigslot stuff to STS, and ensure we disconnect before releasing our ref to the MediaTransportHandler. r?mjf

Depends on D89677

Comment 36

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4708d51506931441f8ca4674f8599a99d09dc30

Comment 37

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9d279ec59db3349af32b69f004d7b95d5e0f93e

Comment 38

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Implement MozPromise::AllSettled, based on JS Promise API. r?jya

Depends on D87180

Comment 39

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Add a browser-chrome test for the WebrtcGlobalInformation interface. r?ng

Comment 40

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Test case for MozPromise::AllSettled. r?jya

Depends on D91467

Comment 41

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Ensure that pcid is unique. r?mjf

This fixes a failure in browser_WebrtcGlobalInformation.js on windows opt.

Depends on D90626

Comment 42

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Remove transports from main, just like we do under normal circumstances. r?mjf

Depends on D93261

Comment 43

[Byron Campen [:bwc]]

Try looks good:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b2a6570c1224e147fd0ae33b58497f303cde322

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0af67ce15ed1452718d9e70dad7b9d0c55273b76

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab2ad6bb968c63082552775e7fac7ac890a610dc

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5f9e4f116124bc6071d1ffdfec36933eaee00ef2

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/82ed327e71ed
Add a browser-chrome test for the WebrtcGlobalInformation interface. r=ng
https://hg.mozilla.org/integration/autoland/rev/3239d49be3ee
Use NS_INLINE_DECL_THREADSAFE_REFCOUNTING_WITH_DESTROY to ensure that MediaTransportHandler stays around long enough to finish pending stats queries. r=mjf
https://hg.mozilla.org/integration/autoland/rev/25c03ac56306
Implement MozPromise::AllSettled, based on JS Promise API. r=jya
https://hg.mozilla.org/integration/autoland/rev/5870625073f6
Test case for MozPromise::AllSettled. r=jya
https://hg.mozilla.org/integration/autoland/rev/387a53fd83f3
Simplify/promisify PWebrtcGlobal. r=ng
https://hg.mozilla.org/integration/autoland/rev/9b612ea670d4
Add a stats stash to WebrtcGlobalInformation, to avoid stats from being lost when a content process goes away. r=ng
https://hg.mozilla.org/integration/autoland/rev/5f453c8df70b
Filter stats for closed PeerConnections the same way we do for live ones. r=ng
https://hg.mozilla.org/integration/autoland/rev/afdb75a17ac9
Stash for ICE logs, and move log filtering into a single place. r=ng
https://hg.mozilla.org/integration/autoland/rev/cd2e50c8af34
Move sigslot stuff to STS, and ensure we disconnect before releasing our ref to the MediaTransportHandler. r=mjf
https://hg.mozilla.org/integration/autoland/rev/59da0d11510e
Ensure that pcid is unique. r=mjf
https://hg.mozilla.org/integration/autoland/rev/59228ee9d9e0
Remove transports from main, just like we do under normal circumstances. r=mjf

Comment 45

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 11 changesets (bug 1626278) for leaks and WebrtcGlobalInformation related failures.

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2Cx64%2Cdebug%2Cbuild-linux64-base-toolchains%2Fdebug%2Cbb&fromchange=59228ee9d9e0388c2c33fa9e674ba8cdfcdb5eca&tochange=aab39b45b9d9e9a71a0aa99bf32a04b7d5c2dacb&selectedTaskRun=dTFknuerQHq53oGatwNhAw.0

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2C18.04%2Cx64%2Cdebug%2Cmochitests%2Cwith%2Cnetworking%2Con%2Csocket%2Cprocess%2Cwithout%2Ce10s%2Ctest-linux1804-64%2Fdebug-mochitest-chrome-spi-nw-1proc%2Cc3&fromchange=e820249c4b87c91b6d39e2dd76eb6653ab50d7ce&tochange=aab39b45b9d9e9a71a0aa99bf32a04b7d5c2dacb&selectedTaskRun=bW99--hzS_KwHhz85BnVSg.0

Backout link: https://hg.mozilla.org/integration/autoland/rev/aab39b45b9d9e9a71a0aa99bf32a04b7d5c2dacb

Failures logs:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318525928&repo=autoland&lineNumber=75617

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318526974&repo=autoland&lineNumber=122759

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318523579&repo=autoland&lineNumber=15503

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318536746&repo=autoland&lineNumber=117226

Comment 46

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #44)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/82ed327e71ed
Add a browser-chrome test for the WebrtcGlobalInformation interface. r=ng
...

== Change summary for alert #27214 (as of Wed, 14 Oct 2020 03:02:20 GMT) ==

Improvements:

2% compiler_metrics num_static_constructors windows2012-64 debug 163.00 -> 159.33

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=27214

Comment 47

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Bogdan Tara[:bogdan_tara | bogdant] from comment #45)

Backed out 11 changesets (bug 1626278) for leaks and WebrtcGlobalInformation related failures.
...
Backout link: https://hg.mozilla.org/integration/autoland/rev/aab39b45b9d9e9a71a0aa99bf32a04b7d5c2dacb

== Change summary for alert #27211 (as of Wed, 14 Oct 2020 00:30:14 GMT) ==

Regressions:

2% compiler_metrics num_static_constructors windows2012-32 debug 158.33 -> 162.00
2% compiler_metrics num_static_constructors windows2012-64 debug 159.33 -> 163.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=27211

Comment 48

[Byron Campen [:bwc]]

Ah. Presentation tests. Looking into it.

Comment 49

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Shutdown MediaTransportHandlerSTS on STS shutdown. Also add some logging. r?mjf,valentin

Depends on D93262

Comment 50

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f57fa13df7e
Add a browser-chrome test for the WebrtcGlobalInformation interface. r=ng
https://hg.mozilla.org/integration/autoland/rev/5c0c76832287
Use NS_INLINE_DECL_THREADSAFE_REFCOUNTING_WITH_DESTROY to ensure that MediaTransportHandler stays around long enough to finish pending stats queries. r=mjf
https://hg.mozilla.org/integration/autoland/rev/8432bc08676e
Implement MozPromise::AllSettled, based on JS Promise API. r=jya
https://hg.mozilla.org/integration/autoland/rev/991bc13ae061
Test case for MozPromise::AllSettled. r=jya
https://hg.mozilla.org/integration/autoland/rev/90cf750aba51
Simplify/promisify PWebrtcGlobal. r=ng
https://hg.mozilla.org/integration/autoland/rev/1cf19408d522
Add a stats stash to WebrtcGlobalInformation, to avoid stats from being lost when a content process goes away. r=ng
https://hg.mozilla.org/integration/autoland/rev/5861d6340ee8
Filter stats for closed PeerConnections the same way we do for live ones. r=ng
https://hg.mozilla.org/integration/autoland/rev/d91d9eb46983
Stash for ICE logs, and move log filtering into a single place. r=ng
https://hg.mozilla.org/integration/autoland/rev/1227d9db743d
Move sigslot stuff to STS, and ensure we disconnect before releasing our ref to the MediaTransportHandler. r=mjf
https://hg.mozilla.org/integration/autoland/rev/15f89d2c5981
Ensure that pcid is unique. r=mjf
https://hg.mozilla.org/integration/autoland/rev/cc2023032191
Remove transports from main, just like we do under normal circumstances. r=mjf
https://hg.mozilla.org/integration/autoland/rev/eeed6ebfc444
Shutdown MediaTransportHandlerSTS on profile-change-net-teardown. Also add some logging. r=mjf

Comment 51

[Narcis Beleuzu [:NarcisB]]

Backed out for mochitest leakchecks

Backout link: https://hg.mozilla.org/integration/autoland/rev/41459f9dfbe958dfc4f5e33f0da22e900ab59312
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319049077&repo=autoland&lineNumber=4515

Comment 52

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Pulsebot from comment #50)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f57fa13df7e
Add a browser-chrome test for the WebrtcGlobalInformation interface. r=ng

== Change summary for alert #27268 (as of Tue, 20 Oct 2020 06:18:43 GMT) ==

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
2%	compiler_metrics	num_static_constructors	windows2012-64		161.00 -> 157.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=27268

Comment 53

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Narcis Beleuzu [:NarcisB] from comment #51)

Backed out for mochitest leakchecks

Backout link: https://hg.mozilla.org/integration/autoland/rev/41459f9dfbe958dfc4f5e33f0da22e900ab59312
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319049077&repo=autoland&lineNumber=4515

== Change summary for alert #27264 (as of Mon, 19 Oct 2020 19:44:38 GMT) ==

Regressions:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
3%	compiler_metrics	num_static_constructors	windows2012-32		157.00 -> 161.00
3%	compiler_metrics	num_static_constructors	windows2012-64		157.00 -> 161.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=27264

Comment 54

[Byron Campen [:bwc]]

Huh. Another case where STS is being shut down before XPCOM shutdown. I think at this point we need a way to register for all/any STS shutdown events, regardless of why they are happening, driven by nsSocketTransportService::Shutdown. I can implement this, but I'd like some feedback from the necko team. What do you think?

Comment 55

[Valentin Gosu [:valentin] (he/him)]

(In reply to Byron Campen [:bwc] from comment #54)

Huh. Another case where STS is being shut down before XPCOM shutdown. I think at this point we need a way to register for all/any STS shutdown events, regardless of why they are happening, driven by nsSocketTransportService::Shutdown. I can implement this, but I'd like some feedback from the necko team. What do you think?

Yes, I think sounds OK.

Comment 56

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/jobs?repo=try&revision=e259185f7e106bbbd637b2bff4b05013b2261372

https://treeherder.mozilla.org/jobs?repo=try&revision=b834be46218646a304491b75bd4323b111c2dd97

https://treeherder.mozilla.org/#/jobs?repo=try&revision=92ede659de2d82fc667e2e7c01171f45d6988b79

Comment 57

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4ff4e961e3ab77d1257b89aa9d95b94fe7d0693f

https://treeherder.mozilla.org/#/jobs?repo=try&revision=36cfef756eb2dc51ed76c76ce274ec70dc41aec8

https://treeherder.mozilla.org/#/jobs?repo=try&revision=eab826ce3ae8b3a795f99c8225ffe50af622696b

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2b2100c4a24210c8b70df5441bf87c319b7b7126

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4fa3a777a94ce677a8018dd3e40e344a1cd743ab

Comment 58

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Stop requiring additional dispatches to STS for ICE ctx teardown. r?mjf

Also some improvements in discipline wrt bare pointers.

Depends on D93655

Comment 59

[Byron Campen [:bwc]]

Attachment: Bug 1626278: Make this test-case function dispatch to STS like it is supposed to. r?mjf

Depends on D97016

Comment 60

[Byron Campen [:bwc]]

https://treeherder.mozilla.org/jobs?repo=try&revision=8af3d6e5f88b52cb8041eb4a27590c54ae1f1160

https://treeherder.mozilla.org/jobs?repo=try&revision=70af97a4ff4a45b8b411821efe6cb6f788eed6f4

https://treeherder.mozilla.org/jobs?repo=try&revision=d6301973b3c8fa35cb67f013bb4d1eefb0c3640a

https://treeherder.mozilla.org/jobs?repo=try&revision=1e8ef552f0358b8aaeb214d43c8b5f80dca35dd2

Seeing some known oranges, will do some pushes on base revision to verify I'm not making them worse.

Comment 61

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f2a432dd0bf0
Add a browser-chrome test for the WebrtcGlobalInformation interface. r=ng
https://hg.mozilla.org/integration/autoland/rev/8c7dd3e44dba
Use NS_INLINE_DECL_THREADSAFE_REFCOUNTING_WITH_DESTROY to ensure that MediaTransportHandler stays around long enough to finish pending stats queries. r=mjf
https://hg.mozilla.org/integration/autoland/rev/340bc094a9fa
Implement MozPromise::AllSettled, based on JS Promise API. r=jya
https://hg.mozilla.org/integration/autoland/rev/6a691d46d210
Test case for MozPromise::AllSettled. r=jya
https://hg.mozilla.org/integration/autoland/rev/d7eaecb56f9d
Simplify/promisify PWebrtcGlobal. r=ng
https://hg.mozilla.org/integration/autoland/rev/8b4f6eae0d07
Add a stats stash to WebrtcGlobalInformation, to avoid stats from being lost when a content process goes away. r=ng
https://hg.mozilla.org/integration/autoland/rev/4b2382b08333
Filter stats for closed PeerConnections the same way we do for live ones. r=ng
https://hg.mozilla.org/integration/autoland/rev/2bb01ffd3a40
Stash for ICE logs, and move log filtering into a single place. r=ng
https://hg.mozilla.org/integration/autoland/rev/6b53a7cecd9b
Move sigslot stuff to STS, and ensure we disconnect before releasing our ref to the MediaTransportHandler. r=mjf
https://hg.mozilla.org/integration/autoland/rev/81cd310b0622
Ensure that pcid is unique. r=mjf
https://hg.mozilla.org/integration/autoland/rev/0316785c107e
Remove transports from main, just like we do under normal circumstances. r=mjf
https://hg.mozilla.org/integration/autoland/rev/3b87a745032e
Shutdown MediaTransportHandlerSTS on STS shutdown. Also add some logging. r=mjf,valentin
https://hg.mozilla.org/integration/autoland/rev/9903d462188b
Stop requiring additional dispatches to STS for ICE ctx teardown. r=mjf
https://hg.mozilla.org/integration/autoland/rev/575b67d9c9b6
Make this test-case function dispatch to STS like it is supposed to. r=mjf

Comment 62

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/f2a432dd0bf0
https://hg.mozilla.org/mozilla-central/rev/8c7dd3e44dba
https://hg.mozilla.org/mozilla-central/rev/340bc094a9fa
https://hg.mozilla.org/mozilla-central/rev/6a691d46d210
https://hg.mozilla.org/mozilla-central/rev/d7eaecb56f9d
https://hg.mozilla.org/mozilla-central/rev/8b4f6eae0d07
https://hg.mozilla.org/mozilla-central/rev/4b2382b08333
https://hg.mozilla.org/mozilla-central/rev/2bb01ffd3a40
https://hg.mozilla.org/mozilla-central/rev/6b53a7cecd9b
https://hg.mozilla.org/mozilla-central/rev/81cd310b0622
https://hg.mozilla.org/mozilla-central/rev/0316785c107e
https://hg.mozilla.org/mozilla-central/rev/3b87a745032e
https://hg.mozilla.org/mozilla-central/rev/9903d462188b
https://hg.mozilla.org/mozilla-central/rev/575b67d9c9b6