1544244 - High frequency security/manager/ssl/tests/unit/test_toolkit_securityreporter.js | xpcshell return code: -11 when Gecko 68 merges to Beta on 2019-05-06

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P1)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=240158462&repo=try

https://queue.taskcluster.net/v1/task/QnUzJi5_QeSMXY3r75YDcw/runs/0/artifacts/public/logs/live_backing.log

Central as Beta sim

[task 2019-04-13T13:59:36.780Z] 13:59:36 INFO - TEST-START | services/sync/tests/unit/test_clients_engine.js
[task 2019-04-13T13:59:42.802Z] 13:59:42 INFO - TEST-PASS | services/sync/tests/unit/test_clients_engine.js | took 6025ms
[task 2019-04-13T13:59:42.810Z] 13:59:42 INFO - Retrying tests that failed when run in parallel.
[task 2019-04-13T13:59:42.819Z] 13:59:42 INFO - TEST-START | security/manager/ssl/tests/unit/test_toolkit_securityreporter.js
[task 2019-04-13T13:59:43.122Z] 13:59:43 WARNING - TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_toolkit_securityreporter.js | xpcshell return code: -11
[task 2019-04-13T13:59:43.124Z] 13:59:43 INFO - TEST-INFO took 303ms
[task 2019-04-13T13:59:43.124Z] 13:59:43 INFO - >>>>>>>
[task 2019-04-13T13:59:43.124Z] 13:59:43 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2019-04-13T13:59:43.124Z] 13:59:43 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2019-04-13T13:59:43.125Z] 13:59:43 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2019-04-13T13:59:43.125Z] 13:59:43 INFO - running event loop
[task 2019-04-13T13:59:43.125Z] 13:59:43 INFO - "CONSOLE_MESSAGE: (info) No chrome package registered for chrome://branding/locale/brand.properties"
[task 2019-04-13T13:59:43.125Z] 13:59:43 INFO - security/manager/ssl/tests/unit/test_toolkit_securityreporter.js | Starting
[task 2019-04-13T13:59:43.125Z] 13:59:43 INFO - (xpcshell/head.js) | test pending (2)
[task 2019-04-13T13:59:43.125Z] 13:59:43 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_toolkit_securityreporter.js | - Binary util BadCertServer should exist - true == true
[task 2019-04-13T13:59:43.127Z] 13:59:43 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_toolkit_securityreporter.js | - certificate folder (bad_certs) should exist - true == true
[task 2019-04-13T13:59:43.127Z] 13:59:43 INFO - (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2019-04-13T13:59:43.127Z] 13:59:43 INFO - PID 13031 | sending 'GET / HTTP/1.0
[task 2019-04-13T13:59:43.128Z] 13:59:43 INFO - PID 13031 | '
[task 2019-04-13T13:59:43.130Z] 13:59:43 INFO - (xpcshell/head.js) | test pending (2)
[task 2019-04-13T13:59:43.130Z] 13:59:43 INFO - (xpcshell/head.js) | test finished (2)
[task 2019-04-13T13:59:43.130Z] 13:59:43 INFO - PID 13031 | HTTP/1.0 200 OK
[task 2019-04-13T13:59:43.131Z] 13:59:43 INFO - PID 13031 | content-type: text/plain
[task 2019-04-13T13:59:43.132Z] 13:59:43 INFO - PID 13031 | connection: close
[task 2019-04-13T13:59:43.133Z] 13:59:43 INFO - PID 13031 | server: httpd.js
[task 2019-04-13T13:59:43.134Z] 13:59:43 INFO - PID 13031 | date: Sat, 13 Apr 2019 13:59:42 GMT
[task 2019-04-13T13:59:43.135Z] 13:59:43 INFO - PID 13031 | content-length: 3
[task 2019-04-13T13:59:43.136Z] 13:59:43 INFO - (xpcshell/head.js) | test run_next_test 1 pending (2)
[task 2019-04-13T13:59:43.137Z] 13:59:43 INFO - (xpcshell/head.js) | test finished (2)
[task 2019-04-13T13:59:43.138Z] 13:59:43 INFO - security/manager/ssl/tests/unit/test_toolkit_securityreporter.js | Starting
[task 2019-04-13T13:59:43.139Z] 13:59:43 INFO - (xpcshell/head.js) | test pending (2)
[task 2019-04-13T13:59:43.139Z] 13:59:43 INFO - (xpcshell/head.js) | test run_next_test 1 finished (2)
[task 2019-04-13T13:59:43.140Z] 13:59:43 INFO - PID 13031 | found pre-defined host 'expired.example.com'
[task 2019-04-13T13:59:43.141Z] 13:59:43 INFO - PID 13031 | found pre-defined host 'expired.example.com'
[task 2019-04-13T13:59:43.141Z] 13:59:43 INFO - PID 13031 | PR_Recv failed: SSL_ERROR_EXPIRED_CERT_ALERT
[task 2019-04-13T13:59:43.142Z] 13:59:43 INFO - "handling expired.example.com"
[task 2019-04-13T13:59:43.143Z] 13:59:43 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_toolkit_securityreporter.js | - Actual and expected connection result should match - 2153390069 == 2153390069

Comment 1

[Intermittent Failures Robot]

2 failures in 3025 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2

Platform breakdown:

• linux64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-04-08&endday=2019-04-14&tree=all

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p2 as the bug is tracked by a release manager for the current nightly.
See How Do You Triage for more information

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: Bug 1544244 - fix cookie-checking in test_toolkit_securityreporter.js r?mgoodwin

Before this patch, test_toolkit_securityreporter.js would unconditionally try to
get the value of the "Cookie" header. If a header isn't available, httpd.js
apparently throws an exception. Interestingly, exceptions thrown in httpd.js
path handlers aren't reported to the test framework, so we weren't aware of
this. Additionally, the test didn't have any way of waiting until the security
report it was expecting had arrived, so it just continued on and "succeeded".
This patch addresses these issues by using "throws" to make sure no cookies are
available and by inserting extra add_test/run_next_test pairs when the test is
waiting for a report.

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

This is a bit speculative, so I'll mark this leave-open for now.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2091fc58b131
fix cookie-checking in test_toolkit_securityreporter.js r=mgoodwin

Comment 6

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/2091fc58b131

Comment 7

[Andreea Pavel [:apavel]]

This still occurs in recent beta sim: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=5934df7d1b1ee6c3f80480a58d3224c8b37aa0e4&selectedJob=241702461

Comment 8

[Intermittent Failures Robot]

1 failures in 2673 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• linux32-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 9

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Thanks for checking - I'm still looking into this.

Comment 10

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1544244 - run test_toolkit_securityreporter.js sequentially (it uses hard-coded ports) r?mgoodwin

test_toolkit_securityreporter.js uses BadCertServer, which uses hard-coded
ports, so it needs to run sequentially.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/216356ce131f
run test_toolkit_securityreporter.js sequentially (it uses hard-coded ports) r=mgoodwin

Comment 12

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/216356ce131f

Comment 13

[Intermittent Failures Robot]

5 failures in 3031 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 5

Platform breakdown:

• linux64-shippable-qr: 2
• linux32-devedition: 1
• linux64-devedition: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-04-22&endday=2019-04-28&tree=all

Comment 14

[Andreea Pavel [:apavel]]

No occurences in the most recent beta sims:

Early https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=31cce5e216c71f1c626133d23e2bbe068d181d9c

Late: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=728cf9e84573847c36a4db12e992e07cef317eeb

Comment 15

[Andreea Pavel [:apavel]]

1 occurrence in today's beta sims, late beta: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=6e23ba002889a39c3cefe62b43991a1f52429f60&selectedJob=243911429

Comment 16

[Patricia Lawless]

Hi Dana, are there any updates for this one? (tracking for 68) Thanks!

Comment 17

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

No updates, really - I haven't had time to look into this further (but it looks like it may be slightly less frequent based on comment 14 and comment 15?)

Comment 18

[Intermittent Failures Robot]

9 failures in 4705 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 9

Platform breakdown:

• linux64-shippable-qr: 4
• linux64-shippable: 1
• android-em-7-0-x86_64: 2
• linux64-devedition: 1
• linux32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-05-06&endday=2019-05-12&tree=all

Comment 19

[Intermittent Failures Robot]

10 failures in 4658 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 10

Platform breakdown:

• android-em-7-0-x86_64: 2
• linux32-shippable: 3
• linux32-devedition: 2
• linux64-devedition: 1
• linux64-shippable-qr: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 20

[Alexandru Michis [:malexandru]]

1 occurence in today's central as late beta sim: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=47ef2c13fdfec55fb5609c938bea79e55f13abaa&selectedJob=247760013

Comment 21

[Intermittent Failures Robot]

2 failures in 4730 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2

Platform breakdown:

• android-em-7-0-x86_64: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 22

[Intermittent Failures Robot]

3 failures in 4337 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3

Platform breakdown:

• linux64-devedition: 2
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-05-27&endday=2019-06-02&tree=all

Comment 23

[Intermittent Failures Robot]

2 failures in 5197 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2

Platform breakdown:

• android-em-7-0-x86_64: 1
• linux64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-06-03&endday=2019-06-09&tree=all

Comment 24

[Intermittent Failures Robot]

5 failures in 3501 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 3
• mozilla-release: 2

Platform breakdown:

• linux32-shippable: 3
• android-em-7-0-x86_64: 1
• linux64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 25

[Intermittent Failures Robot]

6 failures in 4710 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 6

Platform breakdown:

• linux64: 2
• linux32-shippable: 3
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 26

[Ryan VanderMeulen [:RyanVM]]

This is looking pretty frequent on 68 along with bug 1563652. Can you please take another look, Dana?

Comment 27

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Let's see how this does: https://treeherder.mozilla.org/#/jobs?repo=try&revision=adea49567a76dfb02a721fc45e7020a4c789411d

Comment 28

[Intermittent Failures Robot]

7 failures in 4298 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-esr68: 3
• mozilla-release: 3

Platform breakdown:

• linux64: 1
• linux32-shippable: 1
• linux64-shippable: 4
• linux64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 29

[Intermittent Failures Robot]

4 failures in 4608 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 2
• mozilla-release: 2

Platform breakdown:

• linux64-shippable: 3
• linux32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 30

[Intermittent Failures Robot]

6 failures in 4439 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 4
• mozilla-esr68: 2

Platform breakdown:

• linux32-shippable: 3
• linux64-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 31

[Intermittent Failures Robot]

6 failures in 4666 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2
• mozilla-beta: 2
• mozilla-esr68: 2

Platform breakdown:

• linux64-shippable: 1
• linux64: 1
• linux64-shippable-qr: 1
• linux32: 1
• linux32-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 32

[Intermittent Failures Robot]

7 failures in 4839 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 7

Platform breakdown:

• linux64: 4
• linux32: 1
• linux64-shippable: 1
• linux32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 33

[Intermittent Failures Robot]

6 failures in 5073 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 6

Platform breakdown:

• linux64: 2
• linux64-shippable-qr: 3
• linux64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 34

[Intermittent Failures Robot]

11 failures in 5047 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 5
• mozilla-release: 5
• try: 1

Platform breakdown:

• linux64: 9
• linux64-qr: 1
• linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 35

[Intermittent Failures Robot]

7 failures in 4408 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2
• mozilla-esr68: 5

Platform breakdown:

• linux64: 5
• linux32-shippable: 1
• linux64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-09-02&endday=2019-09-08&tree=all

Comment 36

[Intermittent Failures Robot]

6 failures in 4637 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 4
• try: 1
• mozilla-release: 1

Platform breakdown:

• linux64: 2
• linux32: 2
• linux32-shippable: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-09-09&endday=2019-09-15&tree=all

Comment 37

[Intermittent Failures Robot]

8 failures in 4350 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 5
• mozilla-release: 3

Platform breakdown:

• linux64: 5
• linux64-qr: 1
• linux64-shippable-qr: 1
• linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 38

[Intermittent Failures Robot]

15 failures in 4530 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 4
• mozilla-esr68: 11

Platform breakdown:

• linux64-qr: 2
• linux64: 8
• linux32: 1
• linux32-shippable: 3
• linux64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 39

[Intermittent Failures Robot]

12 failures in 4292 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 7
• mozilla-release: 5

Platform breakdown:

• linux64: 5
• linux64-qr: 2
• linux64-shippable: 1
• linux32-shippable: 2
• linux32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-09-30&endday=2019-10-06&tree=all

Comment 40

[Intermittent Failures Robot]

8 failures in 3801 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 4
• mozilla-esr68: 4

Platform breakdown:

• linux64: 5
• linux64-shippable-qr: 2
• linux32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 41

[Intermittent Failures Robot]

3 failures in 3964 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 3

Platform breakdown:

• linux32-shippable: 1
• linux64: 1
• linux64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 42

[Intermittent Failures Robot]

7 failures in 4182 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 6
• try: 1

Platform breakdown:

• linux64: 5
• linux32: 1
• linux32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-10-21&endday=2019-10-27&tree=all

Comment 43

[Intermittent Failures Robot]

13 failures in 4314 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 10
• mozilla-release: 3

Platform breakdown:

• linux32: 1
• linux64: 9
• linux32-shippable: 2
• linux64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-10-28&endday=2019-11-03&tree=all

Comment 44

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Any chance you can look at this once more, Dana? Thank you in advance.

Comment 45

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1544244 - disable test_toolkit_securityreporter.js because TLS error reports are disabled by default and it intermittently fails r?kjacobs

Comment 46

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

We can disable this - the feature it tests is disabled by default, and will probably be removed anyway.

Comment 47

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/36c6363aa297
disable test_toolkit_securityreporter.js because TLS error reports are disabled by default and it intermittently fails r=kjacobs

Comment 48

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/36c6363aa297

Comment 49

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/3672dfac6087

Comment 50

[Intermittent Failures Robot]

3 failures in 4850 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 2
• mozilla-release: 1

Platform breakdown:

• linux32: 1
• linux64-shippable-qr: 1
• linux32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1544244&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 51

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/5ab6f6b30152