Intermittent TEST-UNEXPECTED-TIMEOUT | dom/canvas/test/webgl-mochitest/test_capture.html | application timed out after 330 seconds with no output

REOPENED
Unassigned

Status

()

Core
Canvas: WebGL
REOPENED
11 months ago
a month ago

People

(Reporter: Treeherder Bug Filer, Unassigned)

Tracking

({intermittent-failure, leave-open})

unspecified
intermittent-failure, leave-open
Points:
---

Firefox Tracking Flags

(firefox58 disabled, firefox59 disabled)

Details

(Whiteboard: [stockwell disabled])

Attachments

(1 attachment)

Comment 1

10 months ago
6 failures in 901 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32-stylo: 3
* windows7-32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-08-07&endday=2017-08-13&tree=all

Comment 2

10 months ago
8 failures in 949 pushes (0.008 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-inbound: 2
* mozilla-central: 2

Platform breakdown:
* windows7-32: 7
* windows7-32-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-08-14&endday=2017-08-20&tree=all

Comment 3

10 months ago
9 failures in 908 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 6
* try: 1
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 8
* windows7-32-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-08-21&endday=2017-08-27&tree=all

Comment 4

10 months ago
10 failures in 939 pushes (0.011 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 6
* mozilla-inbound: 2
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 7
* windows7-32-stylo: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 5

9 months ago
16 failures in 924 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 11
* mozilla-inbound: 3
* mozilla-central: 2

Platform breakdown:
* windows7-32: 10
* windows7-32-stylo-disabled: 5
* windows7-32-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 6

9 months ago
3 failures in 1032 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2
* try: 1

Platform breakdown:
* windows7-32-stylo-disabled: 2
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 7

9 months ago
12 failures in 943 pushes (0.013 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 7
* mozilla-central: 2
* autoland: 2
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 7
* windows7-32-stylo-disabled: 4
* windows7-32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 8

9 months ago
26 failures in 885 pushes (0.029 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 9
* mozilla-central: 7
* autoland: 5
* mozilla-beta: 4
* oak: 1

Platform breakdown:
* windows7-32: 15
* windows7-32-stylo-disabled: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 9

9 months ago
40 failures in 824 pushes (0.049 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 21
* mozilla-inbound: 12
* mozilla-beta: 7

Platform breakdown:
* windows7-32: 33
* windows7-32-stylo-disabled: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-10-02&endday=2017-10-08&tree=all
this failure has increased in the last week- all on win7, mostly on debug.

here is a recent log:
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=135515211&lineNumber=10105

and the related screenshot:
https://public-artifacts.taskcluster.net/JjWCu2nkSYaTTfOThsk54Q/0/public/test_info/mozilla-test-fail-screenshot_c7wuuc.png

here is data from the log:
13:44:31     INFO -  670 INFO None671 INFO TEST-START | dom/canvas/test/webgl-mochitest/test_capture.html
13:44:31     INFO -  GECKO(216) | [216, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file z:/build/build/src/dom/fetch/FetchDriver.cpp, line 412
13:44:31     INFO -  GECKO(216) | ++DOMWINDOW == 64 (0D8C8C00) [pid = 216] [serial = 64] [outer = 268E0C00]
13:44:31     INFO -  GECKO(216) | [216, Main Thread] WARNING: 'NS_FAILED(rr->RetargetDeliveryTo(sts))', file z:/build/build/src/dom/fetch/FetchDriver.cpp, line 661
13:44:31     INFO -  GECKO(216) | [216, Main Thread] WARNING: stylo: Web Components not supported yet: file z:/build/build/src/dom/base/nsDocument.cpp, line 6445
13:44:31     INFO -  GECKO(216) | [216, Main Thread] WARNING: stylo: Web Components not supported yet: file z:/build/build/src/dom/base/nsDocument.cpp, line 6445
13:44:31     INFO -  GECKO(216) | Hit MOZ_CRASH() at z:/build/build/src/memory/build/mozjemalloc.cpp:1445
13:50:01     INFO -  Buffered messages finished
13:50:01  WARNING -  TEST-UNEXPECTED-TIMEOUT | dom/canvas/test/webgl-mochitest/test_capture.html | application timed out after 330 seconds with no output
13:50:01     INFO -  672 ERROR Force-terminating active process(es).
13:50:01     INFO -  Determining child pids from psutil
13:50:01     INFO -  Found child pids: []
13:50:01     INFO -  Killing process: 216
13:50:01     INFO -  TEST-INFO | started process screenshot
13:50:01     INFO -  TEST-INFO | screenshot: exit 0
13:50:02     INFO -  TEST-INFO | Main app process: exit 1
13:50:02     INFO -  Buffered messages finished
13:50:02    ERROR -  673 ERROR TEST-UNEXPECTED-FAIL | dom/canvas/test/webgl-mochitest/test_capture.html | application terminated with exit code 1


not sure if the MOZ_CRASH() message above is useful?

:milan, can you help get this prioritized and in the queue of someone who can fix it when needed?
Flags: needinfo?(milan)
Whiteboard: [stockwell needswork]
We have the file access denied error in FetchDriver::HttpFetch - is this OK?
Flags: needinfo?(milan) → needinfo?(overholt)
(In reply to Milan Sreckovic [:milan] from comment #11)
> We have the file access denied error in FetchDriver::HttpFetch - is this OK?

File not found is the actual error.
(In reply to Milan Sreckovic [:milan] from comment #12)
> (In reply to Milan Sreckovic [:milan] from comment #11)
> > We have the file access denied error in FetchDriver::HttpFetch - is this OK?
> 
> File not found is the actual error.

Hmm, I don't see that in the logs. Can you point me to it? I also don't see any fetch calls in http://searchfox.org/mozilla-central/source/dom/canvas/test/webgl-mochitest/test_capture.html but maybe it's in one of the helper utilities?
Flags: needinfo?(overholt) → needinfo?(milan)
I was looking at:
13:44:31     INFO -  GECKO(216) | [216, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file z:/build/build/src/dom/fetch/FetchDriver.cpp, line 412

inside of FetchDriver::HttpFetch and 0x80520012 appears to be file not found.  However, I don't know this code either way, this was a shot in the dark.
Flags: needinfo?(milan)
I see now (I totally didn't look in the bug for that; sorry).

Ben knows about FetchDriver; maybe he can opine whether or not that file not found error is pertinent here (see comment 14).
Flags: needinfo?(bkelly)

Comment 16

8 months ago
I don't see this test (or any test in that directory) using fetch() or a service worker, etc.  I'm not sure why this is even running for this test.

Can you add this code:

  if (NS_FAILED(rv)) {
    nsAutoCString spec;
    if (uri) {
      uri->GetSpec(spec);
    }
    printf_stderr("### ### FetchDriver failed AsyncOpen() for url %s\n", spec.get());
  }

Between the AsyncOpen2() and the NS_ENSURE_SUCCESS() here:

https://searchfox.org/mozilla-central/source/dom/fetch/FetchDriver.cpp#411

Once we know the URL we might have a better idea of what is failing there.
Flags: needinfo?(bkelly)

Comment 17

8 months ago
33 failures in 947 pushes (0.035 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 17
* mozilla-inbound: 14
* mozilla-central: 2

Platform breakdown:
* windows7-32: 29
* windows7-32-stylo-disabled: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-10-09&endday=2017-10-15&tree=all

Comment 18

8 months ago
44 failures in 864 pushes (0.051 failures/push) were associated with this bug in the last 7 days. 

This is the #26 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 27
* mozilla-inbound: 13
* mozilla-central: 2
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 35
* windows7-32-stylo-disabled: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-10-16&endday=2017-10-22&tree=all
Andrew, are you going to follow-up on Comment 16?
Flags: needinfo?(overholt)
Oh, comment 16 was intended for Milan (or someone on the graphics team).
Flags: needinfo?(overholt) → needinfo?(milan)
Created attachment 8922014 [details] [diff] [review]
Diagnostic patch for tests only, not to land
Assignee: nobody → milan
Flags: needinfo?(milan)
:philor, looks like I have a wrong try syntax to get to the minimal number of tests that reproduce this.  What should I use?
Flags: needinfo?(philringnalda)
My choice probably would have been "try: -b d -p win32 -u mochitest-gl -t none" because debug appears to be the most frequent platform, it's a webgl mochitest so you don't need all the rest, and you only need the [Windows 7]-style filtering when there's something that would otherwise run that you don't want. The only other win32 platform is WinXP, which doesn't run unless you ask for it specifically, so there's no need to put yourself in the position of having to figure out how taskcluster has broken try syntax this time and having to file a bug on them.

But once you have a push, even if it doesn't run what you want, there's (generally) no more need to worry about try syntax, because you can instead do what I did, click the downward-pointing triangle to the right of the push, choose "Add new jobs", select the Win7 PGO gl8, "Trigger new jobs", wait for it to run to be sure test_capture.html is still running in that chunk, then retrigger. Zero failures across 25 runs, so I gambled that nobody would have broken choosing tests for builds which haven't run, and chose the Win7 debug gl8 and triggered that, and now we're waiting for a debug build to run to see whether it will actually run the test, whether test_capture.html will still be in gl8 on it as well, and whether it will take fewer retriggers to hit something on debug.
Flags: needinfo?(philringnalda)
Oh, a) you probably didn't actually need a test failure, you just needed the line in https://public-artifacts.taskcluster.net/J-rwcrkuQUS5Y70IRWSfKQ/0/public/logs/live_backing.log that says "### ### FetchDriver failed AsyncOpen() for url resource://app/update.locale", and b) hmm, zero failures across 25 runs on PGO and 100 runs on debug, when this test has a greater than 10% failure rate (6 of the last 50 that ran on autoland), are you sure we can't just land that diagnostic patch, to give it a little pause to think instead of hanging, crashing with one of ten different signatures, or blowing up because program linking failed?

Comment 26

8 months ago
34 failures in 912 pushes (0.037 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 15
* autoland: 15
* mozilla-central: 2
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 29
* windows7-32-stylo-disabled: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-10-23&endday=2017-10-29&tree=all
Failures on this bug started to increase from Nov 3. In the last week it has 56 failures on Windows 7, pgo, debug, opt build types. 

Failing test suites:opt-mochitest-webgl-e10s, debug-mochitest-webgl.

Link to a recent log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=142106093&lineNumber=8648

Part of a recent log:
05:39:36     INFO -  [1684, Main Thread] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/quota/ActorsParent.cpp, line 2827
8646
05:39:36     INFO -  [1684, IPDL Background] WARNING: '!quotaManager', file z:/build/build/src/dom/quota/ActorsParent.cpp, line 6462
8647
05:43:55     INFO -  Buffered messages finished
8648
05:43:55  WARNING -  TEST-UNEXPECTED-TIMEOUT | dom/canvas/test/webgl-mochitest/test_capture.html | application timed out after 370 seconds with no output
8649
05:43:55     INFO -  672 ERROR Force-terminating active process(es).
8650
05:43:55     INFO -  Determining child pids from psutil
8651
05:43:55     INFO -  Found child pids: []
8652
05:43:55     INFO -  Killing process: 216
8653
05:43:55     INFO -  TEST-INFO | started process screenshot
8654
05:43:55     INFO -  TEST-INFO | screenshot: exit 0
8655
05:43:55     INFO -  TEST-INFO | Main app process: exit 1
8656
05:43:55     INFO -  Buffered messages finished
8657
05:43:55    ERROR -  673 ERROR TEST-UNEXPECTED-FAIL | dom/canvas/test/webgl-mochitest/test_capture.html | application terminated with exit code 1
8658
05:43:55     INFO -  runtests.py | Application ran for: 0:10:19.203000
8659
05:43:55     INFO -  zombiecheck | Reading PID log: c:\users\genericworker\appdata\local\temp\tmpqh8y61pidlog
8660
05:43:55     INFO -  mozcrash Copy/paste: Z:\task_1509772767\build\win32-minidump_stackwalk.exe c:\users\genericworker\appdata\local\temp\tmpzycbns.mozrunner\minidumps\23d52c9a-d660-4495-9f9c-a5c1f9a75e1c.dmp Z:\task_1509772767\build\symbols
8661
05:44:17     INFO -  mozcrash Saved minidump as Z:\task_1509772767\build\blobber_upload_dir\23d52c9a-d660-4495-9f9c-a5c1f9a75e1c.dmp   

There are these related bugs:

- 1412383 it was initially created as a clone of the current bug, 
- 1412554 it is investigating a possible OOM failure which is similar to 1412383 (above bug). 

:milan do you have any updates on this?
Flags: needinfo?(milan)

Comment 28

8 months ago
58 failures in 857 pushes (0.068 failures/push) were associated with this bug in the last 7 days. 

This is the #30 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 32
* mozilla-inbound: 20
* mozilla-central: 6

Platform breakdown:
* windows7-32: 30
* windows7-32-stylo-disabled: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-10-30&endday=2017-11-05&tree=all
Priority: P5 → --
Whiteboard: [stockwell needswork] → [stockwell needswork:owner]
Ben, can we ignore "### ### FetchDriver failed AsyncOpen() for url resource://app/update.locale" or is that a pointer in the right direction?
Flags: needinfo?(milan) → needinfo?(bkelly)

Comment 30

8 months ago
This suggests to me that the FetchDriver warning is unrelated to the test timeout, but I don't know for sure.  I don't really know what the test is doing or if the thing doing app/update.locale could be interfering with it.
Flags: needinfo?(bkelly)

Comment 31

8 months ago
19 failures in 138 pushes (0.138 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 12
* mozilla-inbound: 5
* mozilla-central: 2

Platform breakdown:
* windows7-32: 10
* windows7-32-stylo-disabled: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-11-06&endday=2017-11-06&tree=all

Comment 32

8 months ago
25 failures in 174 pushes (0.144 failures/push) were associated with this bug yesterday.    

** This test has failed more than 200 times in the last 30 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* autoland: 16
* mozilla-inbound: 8
* mozilla-central: 1

Platform breakdown:
* windows7-32-stylo-disabled: 14
* windows7-32: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-11-07&endday=2017-11-07&tree=all
Whiteboard: [stockwell needswork:owner] → [stockwell disable-recommended]

Updated

7 months ago
See Also: → bug 1412477

Comment 33

7 months ago
93 failures in 849 pushes (0.11 failures/push) were associated with this bug in the last 7 days. 

This is the #13 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* autoland: 55
* mozilla-inbound: 30
* mozilla-central: 8

Platform breakdown:
* windows7-32: 47
* windows7-32-stylo-disabled: 46

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 34

7 months ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a4188f697b6a
Disable TEST-UNEXPECTED-TIMEOUT | dom/canvas/test/webgl-mochitest/test_capture.html on windows/debug for frequent failures. r=me, a=testonly
this has been disabled, please remember to enable it while working on a patch
Whiteboard: [stockwell disable-recommended] → [stockwell disabled]

Comment 36

7 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/a4188f697b6a
Status: NEW → RESOLVED
Last Resolved: 7 months ago
status-firefox59: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
https://hg.mozilla.org/releases/mozilla-beta/rev/ff2c59d08d4e
status-firefox58: --- → disabled
status-firefox59: fixed → disabled
Target Milestone: mozilla59 → ---

Comment 38

7 months ago
14 failures in 762 pushes (0.018 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 6
* autoland: 6
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 11
* windows7-32-stylo-disabled: 2
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 39

7 months ago
4 failures in 744 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* mozilla-release: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 3
* windows7-32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-11-20&endday=2017-11-26&tree=all

Comment 40

7 months ago
6 failures in 792 pushes (0.008 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 3
* autoland: 2
* mozilla-release: 1

Platform breakdown:
* windows7-32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 41

6 months ago
2 failures in 889 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 42

6 months ago
1 failures in 423 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-12-11&endday=2017-12-17&tree=all

Comment 43

6 months ago
4 failures in 590 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 3
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 44

6 months ago
1 failures in 147 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 1

Platform breakdown:
* windows7-32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2017-12-25&endday=2017-12-31&tree=all

Comment 45

5 months ago
10 failures in 462 pushes (0.022 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 8
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2018-01-01&endday=2018-01-07&tree=all

Comment 46

5 months ago
1 failures in 788 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 1

Platform breakdown:
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388624&startday=2018-01-08&endday=2018-01-14&tree=all
Assignee: milaninbugzilla → nobody
You need to log in before you can comment on or make changes to this bug.