Intermittent browser/base/content/test/general/browser_addCertException.js | Uncaught exception - TypeError: content.document.getElementById(...) is null

RESOLVED DUPLICATE of bug 1303298

Status

()

Firefox
General
P3
normal
RESOLVED DUPLICATE of bug 1303298
2 years ago
a year ago

People

(Reporter: Treeherder Bug Filer, Assigned: johannh)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [stockwell fixed])

Comment 1

2 years ago
Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.
Priority: -- → P3

Comment 2

2 years ago
26 failures in 289 pushes (0.09 failures/push) were associated with this bug in the last 7 days. 

This is the #15 most frequent failure this week. 

Repository breakdown:
* mozilla-inbound: 11
* autoland: 6
* graphics: 4
* mozilla-central: 2
* mozilla-aurora: 2
* try: 1

Platform breakdown:
* linux64: 20
* linux32: 4
* osx-10-10: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2016-12-05&endday=2016-12-11&tree=all

Comment 3

a year ago
38 failures in 526 pushes (0.072 failures/push) were associated with this bug in the last 7 days. 

This is the #15 most frequent failure this week. 

Repository breakdown:
* autoland: 14
* mozilla-inbound: 11
* mozilla-aurora: 5
* try: 4
* mozilla-central: 4

Platform breakdown:
* linux64: 26
* linux32: 9
* windows7-32-vm: 2
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2016-12-12&endday=2016-12-18&tree=all
this is showing up ~5 times/day, primarily on linux debug.

With the screenshot we can see that there is a problem loading a page:
https://public-artifacts.taskcluster.net/U8A4o6JORjWK0tB9VE_a4Q/0/public/test_info//mozilla-test-fail-screenshot_kOQ0ZO.png

running locally, I don't see this, instead I see a warning but the page continues to load.

Here is the log for this test case:
 [task 2016-12-20T01:41:00.903353Z] 01:41:00     INFO - TEST-START | browser/base/content/test/general/browser_addCertException.js

 [task 2016-12-20T01:41:00.940705Z] 01:41:00     INFO - ++DOCSHELL 0x7f373b41e000 == 13 [pid = 1273] [id = {4ffaf0ed-65f7-4728-aa7b-80982f7cfb85}]

 [task 2016-12-20T01:41:00.941088Z] 01:41:00     INFO - ++DOMWINDOW == 100 (0x7f373b41e800) [pid = 1273] [serial = 163] [outer = (nil)]

 [task 2016-12-20T01:41:01.102708Z] 01:41:01     INFO - ++DOMWINDOW == 101 (0x7f373b74a000) [pid = 1273] [serial = 164] [outer = 0x7f373b41e800]

 [task 2016-12-20T01:41:01.333366Z] 01:41:01     INFO - ++DOMWINDOW == 102 (0x7f373b75b800) [pid = 1273] [serial = 165] [outer = 0x7f373b41e800]

 [task 2016-12-20T01:41:02.738104Z] 01:41:02     INFO - ++DOMWINDOW == 103 (0x7f373badc800) [pid = 1273] [serial = 166] [outer = 0x7f373b41e800]

 [task 2016-12-20T01:41:03.931953Z] 01:41:03     INFO - [1273] WARNING: attempt to modify an immutable nsStandardURL: file /home/worker/workspace/build/src/netwerk/base/nsStandardURL.cpp, line 1688

 [task 2016-12-20T01:41:04.041904Z] 01:41:04     INFO - --DOMWINDOW == 102 (0x7f373fc8c800) [pid = 1273] [serial = 115] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.045512Z] 01:41:04     INFO - --DOMWINDOW == 101 (0x7f374be21000) [pid = 1273] [serial = 118] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.047994Z] 01:41:04     INFO - --DOMWINDOW == 100 (0x7f373b744000) [pid = 1273] [serial = 124] [outer = (nil)] [url = about:addons]

 [task 2016-12-20T01:41:04.054906Z] 01:41:04     INFO - --DOMWINDOW == 99 (0x7f373fb61000) [pid = 1273] [serial = 64] [outer = (nil)] [url = about:blank]

 [task 2016-12-20T01:41:04.057068Z] 01:41:04     INFO - --DOMWINDOW == 98 (0x7f373badb000) [pid = 1273] [serial = 127] [outer = (nil)] [url = about:blank]

 [task 2016-12-20T01:41:04.060555Z] 01:41:04     INFO - --DOMWINDOW == 97 (0x7f3762152800) [pid = 1273] [serial = 133] [outer = (nil)] [url = about:blank]

 [task 2016-12-20T01:41:04.064734Z] 01:41:04     INFO - --DOMWINDOW == 96 (0x7f373a210000) [pid = 1273] [serial = 141] [outer = (nil)] [url = about:newtab]

 [task 2016-12-20T01:41:04.069049Z] 01:41:04     INFO - --DOMWINDOW == 95 (0x7f374aded800) [pid = 1273] [serial = 138] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.070979Z] 01:41:04     INFO - --DOMWINDOW == 94 (0x7f373fb9b800) [pid = 1273] [serial = 132] [outer = (nil)] [url = about:blank]

 [task 2016-12-20T01:41:04.072876Z] 01:41:04     INFO - --DOMWINDOW == 93 (0x7f373ef8e000) [pid = 1273] [serial = 131] [outer = (nil)] [url = about:blank]

 [task 2016-12-20T01:41:04.074929Z] 01:41:04     INFO - --DOMWINDOW == 92 (0x7f373f1ab000) [pid = 1273] [serial = 146] [outer = (nil)] [url = https://tls1.example.com/]

 [task 2016-12-20T01:41:04.077034Z] 01:41:04     INFO - --DOMWINDOW == 91 (0x7f375fc11800) [pid = 1273] [serial = 152] [outer = (nil)] [url = about:support]

 [task 2016-12-20T01:41:04.081452Z] 01:41:04     INFO - --DOMWINDOW == 90 (0x7f373a304000) [pid = 1273] [serial = 144] [outer = (nil)] [url = about:blank]

 [task 2016-12-20T01:41:04.084220Z] 01:41:04     INFO - --DOMWINDOW == 89 (0x7f374350e800) [pid = 1273] [serial = 106] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.086179Z] 01:41:04     INFO - --DOMWINDOW == 88 (0x7f373baeb800) [pid = 1273] [serial = 102] [outer = (nil)] [url = https://example.com/browser/browser/base/content/test/general/test_bug959531.html]

 [task 2016-12-20T01:41:04.088779Z] 01:41:04     INFO - --DOMWINDOW == 87 (0x7f3740fd6800) [pid = 1273] [serial = 98] [outer = (nil)] [url = http://mochi.test:8888/browser/browser/base/content/test/general/print_postdata.sjs]

 [task 2016-12-20T01:41:04.091695Z] 01:41:04     INFO - --DOMWINDOW == 86 (0x7f3746674800) [pid = 1273] [serial = 109] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.095769Z] 01:41:04     INFO - --DOMWINDOW == 85 (0x7f375c6d2000) [pid = 1273] [serial = 121] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.100122Z] 01:41:04     INFO - --DOMWINDOW == 84 (0x7f374e17a000) [pid = 1273] [serial = 89] [outer = (nil)] [url = about:rights]

 [task 2016-12-20T01:41:04.102122Z] 01:41:04     INFO - --DOMWINDOW == 83 (0x7f373fc8f800) [pid = 1273] [serial = 85] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.104487Z] 01:41:04     INFO - --DOMWINDOW == 82 (0x7f373ef84800) [pid = 1273] [serial = 129] [outer = (nil)] [url = chrome://browser/content/browser.xul]

 [task 2016-12-20T01:41:04.107027Z] 01:41:04     INFO - --DOMWINDOW == 81 (0x7f375bd87000) [pid = 1273] [serial = 112] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.110776Z] 01:41:04     INFO - --DOMWINDOW == 80 (0x7f3740e23000) [pid = 1273] [serial = 94] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.112803Z] 01:41:04     INFO - --DOMWINDOW == 79 (0x7f3761f8d000) [pid = 1273] [serial = 81] [outer = (nil)] [url = about:home]

 [task 2016-12-20T01:41:04.346140Z] 01:41:04     INFO - *************************

 [task 2016-12-20T01:41:04.346580Z] 01:41:04     INFO - A coding exception was thrown and uncaught in a Task.

 [task 2016-12-20T01:41:04.349407Z] 01:41:04     INFO - Full message: TypeError: content.document.getElementById(...) is null

 [task 2016-12-20T01:41:04.354373Z] 01:41:04     INFO - Full stack: @chrome://mochikit/content/tests/BrowserTestUtils/content-task.js line 52 > eval:4:5

 [task 2016-12-20T01:41:04.354667Z] 01:41:04     INFO - @chrome://mochikit/content/tests/BrowserTestUtils/content-task.js:54:5

 [task 2016-12-20T01:41:04.354849Z] 01:41:04     INFO - *************************

 [task 2016-12-20T01:41:04.398516Z] 01:41:04     INFO - ++DOMWINDOW == 80 (0x7f373c125000) [pid = 1273] [serial = 167] [outer = 0x7f373b41e800]

 [task 2016-12-20T01:41:04.447428Z] 01:41:04     INFO - TEST-INFO | started process screentopng

 [task 2016-12-20T01:41:07.291405Z] 01:41:07     INFO - TEST-INFO | screentopng: exit 0

 [task 2016-12-20T01:41:07.294660Z] 01:41:07     INFO - Buffered messages logged at 01:41:00

 [task 2016-12-20T01:41:07.297126Z] 01:41:07     INFO - Entering test bound 

 [task 2016-12-20T01:41:07.299442Z] 01:41:07     INFO - Buffered messages logged at 01:41:02

 [task 2016-12-20T01:41:07.303774Z] 01:41:07     INFO - Console message: [JavaScript Error: "expired.example.com:443 uses an invalid security certificate.

 [task 2016-12-20T01:41:07.305474Z] 01:41:07     INFO - 

 [task 2016-12-20T01:41:07.307197Z] 01:41:07     INFO - The certificate expired on 01/06/2010 05:20 AM. The current time is 12/20/2016 01:41 AM.

 [task 2016-12-20T01:41:07.308805Z] 01:41:07     INFO - 

 [task 2016-12-20T01:41:07.311526Z] 01:41:07     INFO - Error code: <a id="errorCode" title="SEC_ERROR_EXPIRED_CERTIFICATE">SEC_ERROR_EXPIRED_CERTIFICATE</a>

 [task 2016-12-20T01:41:07.312976Z] 01:41:07     INFO - "]

 [task 2016-12-20T01:41:07.315047Z] 01:41:07     INFO - Buffered messages logged at 01:41:04

 [task 2016-12-20T01:41:07.317437Z] 01:41:07     INFO - Console message: [JavaScript Error: "expired.example.com:443 uses an invalid security certificate.

 [task 2016-12-20T01:41:07.327429Z] 01:41:07     INFO - 

 [task 2016-12-20T01:41:07.329201Z] 01:41:07     INFO - The certificate expired on 01/06/2010 05:20 AM. The current time is 12/20/2016 01:41 AM.

 [task 2016-12-20T01:41:07.332780Z] 01:41:07     INFO - 

 [task 2016-12-20T01:41:07.334386Z] 01:41:07     INFO - Error code: <a id="errorCode" title="SEC_ERROR_EXPIRED_CERTIFICATE">SEC_ERROR_EXPIRED_CERTIFICATE</a>

 [task 2016-12-20T01:41:07.335833Z] 01:41:07     INFO - "]

 [task 2016-12-20T01:41:07.337850Z] 01:41:07     INFO - Buffered messages finished

 [task 2016-12-20T01:41:07.341074Z] 01:41:07     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/general/browser_addCertException.js | Uncaught exception - TypeError: content.document.getElementById(...) is null

 [task 2016-12-20T01:41:07.342968Z] 01:41:07     INFO - Leaving test bound 

 [task 2016-12-20T01:41:07.345258Z] 01:41:07     INFO - [1273] WARNING: attempt to modify an immutable nsStandardURL: file /home/worker/workspace/build/src/netwerk/base/nsStandardURL.cpp, line 1688

 [task 2016-12-20T01:41:07.347565Z] 01:41:07     INFO - MEMORY STAT | vsize 1343MB | residentFast 310MB | heapAllocated 118MB

 [task 2016-12-20T01:41:07.355923Z] 01:41:07     INFO - TEST-OK | browser/base/content/test/general/browser_addCertException.js | took 4368ms

 [task 2016-12-20T01:41:07.357996Z] 01:41:07     INFO - Not taking screenshot here: see the one that was previously logged

 [task 2016-12-20T01:41:07.364459Z] 01:41:07     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/general/browser_addCertException.js | Found an unexpected tab at the end of test run: https://expired.example.com/ - 




Is it possible that we are failing to add the necessary certificate exception?  Maybe there is an issue with ssltunnel?  This should be hackable via a one click loaner.

:Gijs, I see you modified this test in the past, could you help find someone to shed more light on this?
Flags: needinfo?(gijskruitbosch+bugs)
possibly related to bug 1303298 ?
(In reply to Joel Maher ( :jmaher) from comment #4)
> this is showing up ~5 times/day, primarily on linux debug.
> 
> With the screenshot we can see that there is a problem loading a page:
> https://public-artifacts.taskcluster.net/U8A4o6JORjWK0tB9VE_a4Q/0/public/
> test_info//mozilla-test-fail-screenshot_kOQ0ZO.png

That's an error page title. It's the wrong error page title (cf. https://expired.badssl.com/ says "Insecure Connection").

Nihanth or Johann might have a better idea of what's wrong here given they've looked at this stuff before.
 
> Is it possible that we are failing to add the necessary certificate
> exception?  Maybe there is an issue with ssltunnel?  This should be hackable
> via a one click loaner.

I don't know. It feels like a timeout. Also note that all of these look different. These are some of the top screenshots from the previous bot comment listing failures:

http://mozilla-releng-blobs.s3.amazonaws.com/blobs/autoland/sha512/f18997d0b3720b388ff90bd74666a3f662a3ff21f189527b3164a1bf67fe5eb88a4dae467e505aba45c64646a885562e94375d6748dbaa2ce5bb847903b70ecd

https://public-artifacts.taskcluster.net/GBNEX_t3TEqrSR76-6GG8Q/0/public/test_info//mozilla-test-fail-screenshot_om0tvw.png

https://public-artifacts.taskcluster.net/TpKYxaA-QeOHjY9Q5b497A/0/public/test_info//mozilla-test-fail-screenshot_PMfCuR.png

https://public-artifacts.taskcluster.net/AxPJjiI6QGuVsHSwrJ3rbQ/0/public/test_info//mozilla-test-fail-screenshot_blKPxI.png

https://public-artifacts.taskcluster.net/OJLzfZ5lTwCFqOwksr6YTg/0/public/test_info//mozilla-test-fail-screenshot_5Ko_e1.png

https://public-artifacts.taskcluster.net/QyEFWzOjQX6tevqKOq5dsQ/0/public/test_info//mozilla-test-fail-screenshot_RX6Imt.png

Looks to me like we sometimes also fail when the page *has* loaded (or maybe it loads inbetween us failing and taking a screenshot) and sometimes we fail while trying to load the dummy page (which isn't a cert error). So really not sure what is going on.
Flags: needinfo?(nhnt11)
Flags: needinfo?(jhofmann)
Flags: needinfo?(gijskruitbosch+bugs)
See Also: → bug 1303298
(Assignee)

Comment 7

a year ago
Yes this is definitely the same thing as bug 1303298 and it's getting stranger. In that bug we added a timeout to check if we're actually on the about:certerror page. It seems like the test never fails at the timeout even though the page clearly hasn't loaded (yet?), since the getElementById call fails. I initially checked for the presence of a DOM classname attribute instead of the url string, maybe we should do that again to be sure. I'll continue work in the other bug.

I can't explain the failures for the cases where the page seemingly has loaded other than that the page was not loaded at the time the getElementById function failed.
Flags: needinfo?(nhnt11)
Flags: needinfo?(jhofmann)
(Assignee)

Comment 8

a year ago
Let me add that we should not disable these two tests, there are not too many tests that affect the cert error page and this is UI that is crucial to user security. Also, the failures might reveal some larger issue we haven't considered yet.
thanks Johann, and I appreciate the information that these are some of the few tests which test cert error page.  What can we do to investigate this more?

As both of these issues are pretty frequent, I suspect there is a common root cause to fix or greatly reduce these- any information would be helpful.
(In reply to Johann Hofmann [:johannh] from comment #8)
> Let me add that we should not disable these two tests, there are not too
> many tests that affect the cert error page and this is UI that is crucial to
> user security. Also, the failures might reveal some larger issue we haven't
> considered yet.

Given that this is on Linux, have you tried using rr in chaos mode to reproduce this (with a debug build)? That might be the quickest way to reproduce this locally and get some more info. Alternatively, as jmaher noted, a one-click loaner might help give insight in what's happening on the machine when this fails?

We could also just land a logging patch specifically for this test (and enable logs even on successful runs using requestCompleteLog).

Johann, can you drive investigating this? :-)
Flags: needinfo?(jhofmann)
One more thought: is it possible that we're waiting for a network stop (for the error page to have loaded) and that's being triggered by the initial about:blank load in a tab?
35 failures in 609 pushes (0.057 failures/push) were associated with this bug in the last 7 days. 

This is the #18 most frequent failure this week. 

Repository breakdown:
* autoland: 11
* mozilla-inbound: 10
* mozilla-aurora: 9
* mozilla-central: 4
* try: 1

Platform breakdown:
* linux64: 22
* linux32: 11
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2016-12-19&endday=2016-12-25&tree=all
(Assignee)

Comment 13

a year ago
> Johann, can you drive investigating this? :-)

Yes this is totally on my list
Assignee: nobody → jhofmann
Status: NEW → ASSIGNED
Flags: needinfo?(jhofmann)
12 failures in 305 pushes (0.039 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 5
* autoland: 3
* mozilla-central: 2
* mozilla-aurora: 2

Platform breakdown:
* linux64: 8
* linux32: 3
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2016-12-26&endday=2017-01-01&tree=all
29 failures in 563 pushes (0.052 failures/push) were associated with this bug in the last 7 days. 

This is the #48 most frequent failure this week. 

Repository breakdown:
* autoland: 11
* mozilla-inbound: 9
* mozilla-aurora: 4
* try: 3
* mozilla-central: 2

Platform breakdown:
* linux64: 23
* linux32: 5
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-02&endday=2017-01-08&tree=all
17 failures in 128 pushes (0.133 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 7
* autoland: 5
* mozilla-central: 2
* mozilla-aurora: 2
* graphics: 1

Platform breakdown:
* linux64: 11
* linux32: 5
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-10&endday=2017-01-10&tree=all
17 failures in 155 pushes (0.11 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 6
* autoland: 6
* try: 2
* mozilla-central: 2
* mozilla-aurora: 1

Platform breakdown:
* linux32: 10
* linux64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-11&endday=2017-01-11&tree=all
74 failures in 722 pushes (0.102 failures/push) were associated with this bug in the last 7 days. 

This is the #23 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 28
* mozilla-inbound: 27
* mozilla-central: 7
* mozilla-aurora: 7
* try: 4
* graphics: 1

Platform breakdown:
* linux64: 42
* linux32: 31
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-09&endday=2017-01-15&tree=all
17 failures in 96 pushes (0.177 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 7
* autoland: 7
* mozilla-aurora: 2
* graphics: 1

Platform breakdown:
* linux64: 11
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-16&endday=2017-01-16&tree=all
Could this be CPOW related? You could try running with MOZ_DEBUG_DEAD_CPOWS=1 and see if it reproduces more easily.
58 failures in 690 pushes (0.084 failures/push) were associated with this bug in the last 7 days. 

This is the #23 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 25
* mozilla-inbound: 19
* try: 5
* mozilla-aurora: 4
* graphics: 4
* mozilla-central: 1

Platform breakdown:
* linux64: 30
* linux32: 24
* osx-10-10: 2
* windows8-64: 1
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-16&endday=2017-01-22&tree=all
22 failures in 152 pushes (0.145 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 10
* mozilla-inbound: 9
* mozilla-beta: 2
* try: 1

Platform breakdown:
* linux64: 19
* linux32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-24&endday=2017-01-24&tree=all
24 failures in 153 pushes (0.157 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 11
* autoland: 8
* mozilla-beta: 2
* try: 1
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* linux64: 24

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-25&endday=2017-01-25&tree=all
following up here, this has been pretty frequent on linux64 debug the last month.  sometimes there are other bugs that account for the work on intermittents, not sure if that is the case here.

Please do let me know if there is more information needed or something I could do to help narrow this down.
Seems like this is identical to bug 1303298. If the recently landed patch for that bug sticks, it should get rid of this intermittent as well.
Flags: needinfo?(jhofmann)
(Assignee)

Comment 26

a year ago
What Nihanth said. Let's wait if bug 1303298 stays closed. In fact let me dupe this bug in favor of 1303298, I previously wasn't sure about it but I think we can safely say it's the same issue now.
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
Flags: needinfo?(jhofmann)
Resolution: --- → DUPLICATE
Duplicate of bug: 1303298
79 failures in 749 pushes (0.105 failures/push) were associated with this bug in the last 7 days. 

This is the #20 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* mozilla-inbound: 36
* autoland: 26
* try: 7
* mozilla-beta: 6
* graphics: 2
* mozilla-central: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 68
* linux32: 10
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292960&startday=2017-01-23&endday=2017-01-29&tree=all
Whiteboard: [stockwell fixed]
You need to log in before you can comment on or make changes to this bug.