Closed Bug 1286871 Opened 8 years ago Closed 8 years ago

-6.0% performance regression in Firefox Nightly in Unity3D Robot Lab and AngryBots on July 8th (in emunittests suite)

Categories

(Core :: Graphics: CanvasWebGL, defect)

50 Branch
x86_64
All
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla52
Tracking Status
firefox49 --- unaffected
firefox50 --- fixed
firefox51 --- fixed
firefox52 --- fixed

People

(Reporter: jujjyl, Assigned: mtseng)

References

Details

(Keywords: perf, regression, Whiteboard: [gfx-noted])

Attachments

(2 files, 3 obsolete files)

STR:

1. Visit https://s3.amazonaws.com/mozilla-games/emunittest-0.4/2015-08-28-emunittest_0.4-RobotLab-u5.1.3f1_hg-e1.34.6-release-prof/index.html?playback&novsync

2. After the page loads, the benchmark runs for ~30 seconds and prints something like

TEST PASSED. Timescore: 27340.17. (lower is better)

Observed:

Running the test five times in Nightlies from different dates, and averaging, the recent Firefox Nightlies give the following results:

Firefox Nightly 64-bit 50.0a1 (2016-07-06):
25156.22 ms
25159.36 ms
25918.28 ms
26549.68 ms
26851.22 ms

Avg. 25926.95 ms (good)

Firefox Nightly 64-bit 50.0a1 (2016-07-07):
25079.33 ms
25144.56 ms
25580.70 ms
25652.69 ms
26368.44 ms

Avg. 25565.14 ms (good)

Firefox Nightly 64-bit 50.0a1 (2016-07-08):
26100.04 ms
26734.92 ms
27304.35 ms
27568.13 ms
27760.80 ms

Avg. 27093.65 ms (bad)

Firefox Nightly 64-bit 50.0a1 (2016-07-13):
26657.94 ms
26914.51 ms
26935.91 ms
27251.02 ms
27990.52 ms

Avg. 27149.98 ms (bad)

The same effect is seen on Firefox Nightly in both Windows 10 Pro and OS X 10.11.5.

Here is a graphed view from emunittest harness: http://clb.demon.fi:6932/results?uuid=53974141-edac-48d8-a822-0362e5ae106c&test=robotlab

The regression is also visible in AngryBots demo, another Unity3D title (same renderer): http://clb.demon.fi:6932/results?uuid=53974141-edac-48d8-a822-0362e5ae106c&test=angrybots

Mozregression finds the following:

22:54.79 INFO: Got as far as we can go bisecting nightlies...
22:54.79 INFO: Last good revision: 63cc31d6cc1c8089590461016ce0b4a2fb77ecbc (2016-07-07)
22:54.79 INFO: First bad revision: 45682df2d2d45e5a8385fd842579e661a4b60bc5 (2016-07-08)
22:54.79 INFO: Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=63cc31d6cc1c8089590461016ce0b4a2fb77ecbc&tochange=45682df2d2d45e5a8385fd842579e661a4b60bc5

However I was not able to get the pushlog any narrower via mozregression, because it looks like once mozregression navigates from mozilla-central nightlies to taskcluster (printing 22:54.80 INFO: Switching bisection method to taskcluster), the builds that are received from taskcluster are likely built with different flags, which have debugging or less optimizations enabled(?), since the results are in 30000+ msecs range and not comparable.

Marking this tentatively WebGL, but only because the demo is rendering heavy and I see Jeff Gilbert's WebGL commmits in the push range :). No idea if this is would definitely be WebGL specific.
Is it possible to get a tighter regressionwindow via triaging manual builds? That would be very helpful in  separating if this would be WebGL or something else (there are JS core commits as well in the range).
Hi Jukka, what constitutes the difference between a good and bad score when testing?
Flags: needinfo?(jujjyl)
It is difficult to say in absolute numbers what a good and a bad score is, since it depends on hardware. I would suggest running Nightly on 2016-07-06, and establish that this gives a consistent same good score like shown above, and then run on Nightly 2016-07-08, which should carry the performance loss.

Note that Firefox Stable channel does not give scores that would be comparable to Firefox Nightly in absolute values, since Firefox Nightly is expected to be a tiny bit slower than Stable due to different optimization build flags, so FF Stable can't be used to find a good baseline.

For reference, the raw numbers posted in comment 0 are from the following system:

Windows 10
3.0 GHz Intel 8-Core i7-5960X, 32GB of RAM
SLI 2x ASUS NVIDIA GeForce GTX 980 Ti, 4+4GB of VRAM, driver version 368.81

I have been running this test on a Linux box as well with a very weak Intel integrated GPU on a Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz (Mac Mini 2011 running Ubuntu Linux), and there the performance regression was not visible. That might be due to the extremely weak GPU (Intel HD 4000 iirc).

The performance drop was also observed on

Mac Pro (Late 2013)
OS X El Capitan 10.11.5
3.5GHz 6-Core Intel Xeon E5
32GB 1866MHz DDR3 ECC
2x AMD FirePro D500 3072MB

Which is in this graph: http://clb.demon.fi:6932/results?uuid=53974141-edac-48d8-a822-0362e5ae106c&test=robotlab . Note how the scores between the dates 8th of May and 6th of July are very level, between 2800ms and 29400ms (ignore the spikes, which are noisy results of conflicting system load), and scores after 7th of July are each consistently above 30000 msecs.
Flags: needinfo?(jujjyl)
Version: Trunk → 50 Branch
(In reply to Jukka Jylänki from comment #0)
> ...
> 
> Marking this tentatively WebGL, but only because the demo is rendering heavy
> and I see Jeff Gilbert's WebGL commmits in the push range :). No idea if
> this is would definitely be WebGL specific.

I could be convinced I'm seeing this slowdown (more like 2.5% rather than 6% though) as a result bug 1285047 patch "initialize all 3D texture data".  It'd be good to understand if that's expected.
Flags: needinfo?(jgilbert)
When I go all the way to the end of the range from comment 0, I do get to ~5% slowdown.  Let me see what else in that range could have contributed.
I could also be convinced that some of that slowdown is coming from bug 1282150, but the deviation between the runs is too large to be really confident.
Flags: needinfo?(bzbarsky)
(In reply to Milan Sreckovic [:milan] from comment #4)
> (In reply to Jukka Jylänki from comment #0)
> > ...
> > 
> > Marking this tentatively WebGL, but only because the demo is rendering heavy
> > and I see Jeff Gilbert's WebGL commmits in the push range :). No idea if
> > this is would definitely be WebGL specific.
> 
> I could be convinced I'm seeing this slowdown (more like 2.5% rather than 6%
> though) as a result bug 1285047 patch "initialize all 3D texture data". 
> It'd be good to understand if that's expected.

There should be *a* slowdown if they're using 3d textures from webgl2, but only if they're causing us to reallocate every frame, other otherwise often. The overhead of initializing 3d textures should be once per texture.
Flags: needinfo?(jgilbert)
> I could also be convinced that some of that slowdown is coming from bug 1282150

That would be pretty odd.  I don't think there were any real behavior changes in canvas/webgl code in there.  There was the renaming of RootingCxForThread() to RootingCx(), but the actual behavior did not change.
Flags: needinfo?(bzbarsky)
Right - I didn't know if the test would show asm.js type of slowdowns, so, JS changes, etc.

On the other hand, with the latest local build, I see even more of a slowdown :)

The numbers are all quite high (bad) though, probably because nightly build by default doesn't do enough optimizations.  For example, I was getting ~52k at the start of the range, ~54k after bug 1285047, ~56k after bug 1282150, and ~60k with the latest local build.

Jukka, can you do anything with this information?
(ni?(Jukka) for comment 9)

Do we care about fixing this in 50 now that it's going to Aurora?
Flags: needinfo?(milan)
Flags: needinfo?(jujjyl)
Profiling a good vs bad build should show if it's a particular bit of the WebGL that's being hit a lot, and we can drill down from there.

Aurora is still relatively permeable to us if we figure this out soon.
Thanks Milan for taking a look.

With this kind of performance bug where there is some amount of noise, I realize we need a good easy method of automating multiple runs of a test and computing the averages to reduce noise, in a way that is easy for anyone to test&verify in a more unattended manner. I am currently developing this as a priority item to help triage these kind of items in the suite.

It looks like this regression could be OS X specific, or at least on my Windows 10 GTX 980 Ti computer, I am not seeing such a large bump after accounting for noise, but on OS X the behavior is pronounced.

Andrew, I don't know an answer to that yet. It probably depends on where the root issue is and how difficult it will be to fix. If the fix is manageable, then uplifting to Aurora would be nice. I'll post back here once I've gotten a good noisefree solution for triaging this.
Flags: needinfo?(milan)
Flags: needinfo?(jujjyl)
Bisected this now by manually building commits from gecko-dev github repo, and it looks like the offending commit is like Milan estimated in comment 4:

commit 9c70251dd9a4139bb8d5cadcffb50f8035c3c905
Author: Jeff Gilbert <jgilbert@mozilla.com>
Date:   Wed Jul 6 15:08:20 2016 -0700

    Bug 1285047 - Initialize all 3D texture data. - r=mtseng
    
I updated emunittest to log results from multiple sequential runs. Here are the results from two test batches, where a clean Firefox profile was created in the beginning, and the test was run six times. For each case, the bottommost line shows the averaged result over 4 runs, with two most outlier results discarded from the average:

First batch run (bad):

 Run Date         | Test Name                       | Total time (lower is better) |   FPS | CPU Time
-----------------------------------------------------------------------------------------------------
 2016-08-04 12:00 | Robot Lab-2015-08-28            |           27725ms (no vsync) | 90.23 |  20226ms
 2016-08-04 12:00 | Robot Lab-2015-08-28            |           24658ms (no vsync) | 92.04 |  19637ms
 2016-08-04 12:01 | Robot Lab-2015-08-28            |           25851ms (no vsync) | 87.48 |  20643ms
 2016-08-04 12:01 | Robot Lab-2015-08-28            |           25271ms (no vsync) | 89.69 |  20096ms
 2016-08-04 12:02 | Robot Lab-2015-08-28            |           25269ms (no vsync) | 89.66 |  19970ms
 2016-08-04 12:02 | Robot Lab-2015-08-28            |           25770ms (no vsync) | 87.64 |  20388ms
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |                      25540ms | 89.30 |  20170ms
                  |                                 |                              |       |         
 Run Date         | Test Name                       | CPU Idle | Page load time | # of janked frames
----------------------------------------------------------------------------------------------------
 2016-08-04 12:00 | Robot Lab-2015-08-28            |    8.74% |      5378.09ms |                  3
 2016-08-04 12:00 | Robot Lab-2015-08-28            |    9.63% |      2740.95ms |                  8
 2016-08-04 12:01 | Robot Lab-2015-08-28            |    9.71% |      2813.66ms |                  9
 2016-08-04 12:01 | Robot Lab-2015-08-28            |    9.88% |      2798.15ms |                  4
 2016-08-04 12:02 | Robot Lab-2015-08-28            |   10.48% |      2793.25ms |                  8
 2016-08-04 12:02 | Robot Lab-2015-08-28            |   10.66% |      2788.70ms |                  8
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |    9.92% |      2798.44ms |                7.0
                  |                                 |          |                |                   

Second batch run (bad):

 Run Date         | Test Name                       | Total time (lower is better) |   FPS | CPU Time
-----------------------------------------------------------------------------------------------------
 2016-08-04 12:04 | Robot Lab-2015-08-28            |           28611ms (no vsync) | 86.75 |  21081ms
 2016-08-04 12:04 | Robot Lab-2015-08-28            |           25218ms (no vsync) | 89.74 |  20173ms
 2016-08-04 12:04 | Robot Lab-2015-08-28            |           25590ms (no vsync) | 88.41 |  20407ms
 2016-08-04 12:05 | Robot Lab-2015-08-28            |           26417ms (no vsync) | 85.36 |  21107ms
 2016-08-04 12:05 | Robot Lab-2015-08-28            |           25691ms (no vsync) | 88.00 |  20346ms
 2016-08-04 12:06 | Robot Lab-2015-08-28            |           26251ms (no vsync) | 85.78 |  20890ms
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |                      25987ms | 87.24 |  20681ms
                  |                                 |                              |       |         
 Run Date         | Test Name                       | CPU Idle | Page load time | # of janked frames
----------------------------------------------------------------------------------------------------
 2016-08-04 12:04 | Robot Lab-2015-08-28            |    8.56% |      5371.64ms |                  4
 2016-08-04 12:04 | Robot Lab-2015-08-28            |    9.48% |      2743.68ms |                  6
 2016-08-04 12:04 | Robot Lab-2015-08-28            |    9.79% |      2780.45ms |                  9
 2016-08-04 12:05 | Robot Lab-2015-08-28            |    9.91% |      2803.41ms |                  4
 2016-08-04 12:05 | Robot Lab-2015-08-28            |   10.48% |      2780.64ms |                  7
 2016-08-04 12:06 | Robot Lab-2015-08-28            |   10.40% |      2775.17ms |                  6
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |    9.90% |      2784.92ms |                5.8
                  |                                 |          |                |                   


The previous commit to that is

commit bf05496a833cfc1b4fb168b94e75fc93086111f2
Author: Benjamin Smedberg <benjamin@smedbergs.us>
Date:   Thu Jul 7 12:14:25 2016 -0400

    Bug 1282866 - remove widget/qt and other supporting QT code, r=dougt.  This patch does not remove all of the checks for MOZ_WIDGET_QT (which are dead code), but that will be a followup mentored bug.

which gives the following results:

First batch run (good):

 Run Date         | Test Name                       | Total time (lower is better) |   FPS | CPU Time
-----------------------------------------------------------------------------------------------------
 2016-08-04 10:38 | Robot Lab-2015-08-28            |           26515ms (no vsync) | 95.49 |  19095ms
 2016-08-04 10:38 | Robot Lab-2015-08-28            |           23895ms (no vsync) | 95.64 |  18902ms
 2016-08-04 10:39 | Robot Lab-2015-08-28            |           23427ms (no vsync) | 97.67 |  18419ms
 2016-08-04 10:39 | Robot Lab-2015-08-28            |           23534ms (no vsync) | 97.23 |  18423ms
 2016-08-04 10:39 | Robot Lab-2015-08-28            |           23897ms (no vsync) | 95.56 |  18679ms
 2016-08-04 10:40 | Robot Lab-2015-08-28            |           23959ms (no vsync) | 95.29 |  18666ms
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |                      23821ms | 95.98 |  18668ms
                  |                                 |                              |       |         
 Run Date         | Test Name                       | CPU Idle | Page load time | # of janked frames
----------------------------------------------------------------------------------------------------
 2016-08-04 10:38 | Robot Lab-2015-08-28            |    8.83% |      5388.52ms |                  3
 2016-08-04 10:38 | Robot Lab-2015-08-28            |    9.62% |      2790.79ms |                  6
 2016-08-04 10:39 | Robot Lab-2015-08-28            |   10.05% |      2770.41ms |                  8
 2016-08-04 10:39 | Robot Lab-2015-08-28            |   10.44% |      2782.40ms |                  7
 2016-08-04 10:39 | Robot Lab-2015-08-28            |   10.75% |      2787.48ms |                  7
 2016-08-04 10:40 | Robot Lab-2015-08-28            |   11.06% |      2789.92ms |                  5
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |   10.22% |      2787.65ms |                6.3
                  |                                 |          |                |                   

Second batch run (good):

 Run Date         | Test Name                       | Total time (lower is better) |   FPS | CPU Time
-----------------------------------------------------------------------------------------------------
 2016-08-04 10:41 | Robot Lab-2015-08-28            |           26448ms (no vsync) | 96.62 |  18867ms
 2016-08-04 10:41 | Robot Lab-2015-08-28            |           24018ms (no vsync) | 94.95 |  19043ms
 2016-08-04 10:42 | Robot Lab-2015-08-28            |           23334ms (no vsync) | 98.12 |  18347ms
 2016-08-04 10:42 | Robot Lab-2015-08-28            |           23749ms (no vsync) | 96.10 |  18660ms
 2016-08-04 10:42 | Robot Lab-2015-08-28            |           23639ms (no vsync) | 96.47 |  18525ms
 2016-08-04 10:43 | Robot Lab-2015-08-28            |           23826ms (no vsync) | 95.80 |  18563ms
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |                      23808ms | 96.25 |  18654ms
                  |                                 |                              |       |         
 Run Date         | Test Name                       | CPU Idle | Page load time | # of janked frames
----------------------------------------------------------------------------------------------------
 2016-08-04 10:41 | Robot Lab-2015-08-28            |    8.85% |      5572.51ms |                  3
 2016-08-04 10:41 | Robot Lab-2015-08-28            |    9.60% |      2766.84ms |                  6
 2016-08-04 10:42 | Robot Lab-2015-08-28            |    9.99% |      2777.44ms |                  7
 2016-08-04 10:42 | Robot Lab-2015-08-28            |   10.34% |      2765.23ms |                  5
 2016-08-04 10:42 | Robot Lab-2015-08-28            |   10.65% |      2746.81ms |                  7
 2016-08-04 10:43 | Robot Lab-2015-08-28            |   11.08% |      2791.83ms |                  6
 (4 samples)      | Robot Lab-2015-08-28 (Averaged) |   10.14% |      2775.33ms |                6.0
                  |                                 |          |                |                   

Jeff, would you be able to give this a peek? To test, visit

http://s3.amazonaws.com/mozilla-games/emunittest-public/index.html?selectedTests=robotlab&autorun&novsync&numtimes=6

and wait for the test to run six times. After the test has finished for the first time, you'll need to click on the "always accept popups" banner button since each test opens up in a new tab. After all the tests have finished, the result sheet like shown above is printed at the very bottom of the site.
Flags: needinfo?(jgilbert)
I cannot look at this right now.
Milan may be able to assign this.
Flags: needinfo?(jgilbert) → needinfo?(milan)
Morris, you reviewed the patch for bug 1285047, so you would have some inside knowledge.  This is considered a regression against 50, which is already in aurora, so some immediate attention would be good.
Flags: needinfo?(mtseng)
Flags: needinfo?(milan)
Flags: needinfo?(howareyou322)
I was debugging the Robot Lab demo today, and I don't see it to use 3D textures anywhere, or it doesn't at least call any of the WebGL entry points gl.texStorage3D, gl.texImage3D or gl.texSubImage3D that I was able to tell. The other regressed test case, AngryBots, should not use 3D textures either I believe. That makes the performance impact a bit peculiar (although there is naturally the chance that I might have done a mistake in debugging when looking at this). I wonder if there is any chance that a perf impact could be caused by that commit even if 3D textures are not in use?
Assignee: nobody → mtseng
Flags: needinfo?(howareyou322)
Keywords: perf
I'll check this.
Flags: needinfo?(mtseng)
We call MakeContextCurrent unconditionally now. Add a check to bail out function if we have nothing to clear.
Attachment #8778823 - Flags: review?(jgilbert)
Jukka, could you try the patch at comment 18 to see if that solve the issue? Thanks.
Flags: needinfo?(jujjyl)
Attachment #8778823 - Flags: review?(jgilbert) → review+
Attached file results.txt
Thanks Morris. Tried out the patch on latest Nightly, with two batches of runs and 6 times run on each batch. Without the patch, it looks like the averaged results are

26084ms
26360ms

and with the patch, it gives

26176ms
26072ms

(lower is better) So the patch doesn't seem to explain the performance difference. Looking at best case times, there might be a small improvement.

Attached the full run log of the tests.

Btw Morris, you can run the test yourself in the URL above in comment 13:

http://s3.amazonaws.com/mozilla-games/emunittest-public/index.html?selectedTests=robotlab&autorun&novsync&numtimes=6

and then scroll down to the very bottom to see the results once the tests have finished.
Flags: needinfo?(jujjyl)
After more benchmark, I found why this performance regression happened. This function is in hot path which means it will be called many many times. Thus, any variable initialize like vector init in this case would result in consuming more cpu time. I added an early return more aggresive just like the version before regression. Jeff, could you review the patch again? Thanks. Jukka, I have tested it in my local and it seems works. Can you double confirm the result? Thanks.
Attachment #8781072 - Flags: review?(jgilbert)
Attachment #8778823 - Attachment is obsolete: true
Ni Jukka for comment 21.
Flags: needinfo?(jujjyl)
Comment on attachment 8781072 [details] [diff] [review]
Bail out if we don't need initialize any framebuffer attachments.

Feels like lambda is an overkill here.
Comment on attachment 8781072 [details] [diff] [review]
Bail out if we don't need initialize any framebuffer attachments.

Review of attachment 8781072 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/canvas/WebGLFramebuffer.cpp
@@ +1058,5 @@
>      // Cool! We've checked out ok. Just need to initialize.
>  
>      //////
>      // Check if we need to initialize anything
> +    const auto fnCheckNeedInit = [&]() {

Replicating the logic like this is going to be prone to bugs due to divergent code.

Instead, if the initializer for std::vector is taking too much time, replace it with a UniquePtr<std::vector>, and initialize it as needed.
Attachment #8781072 - Flags: review?(jgilbert) → review-
Store std::vector in UniquePtr to avoid vector construction time.
Attachment #8781486 - Flags: review?(jgilbert)
Attachment #8781072 - Attachment is obsolete: true
Comment on attachment 8781486 [details] [diff] [review]
Using UniquePtr to store std::vector.

Review of attachment 8781486 [details] [diff] [review]:
-----------------------------------------------------------------

Let's take this only if my recommendation doesn't pan out.

::: dom/canvas/WebGLFramebuffer.cpp
@@ +1070,5 @@
>          if (info.mDepth == 1)
>              return false;
>  
> +        if (!tex3DToClear)
> +            tex3DToClear.reset(new std::vector<WebGLFBAttachPoint*>());

{} for even one-line non-control-flow statements.

@@ +1084,2 @@
>  
> +    UniquePtr<std::vector<WebGLFBAttachPoint*>> attachmentsToClear;

You know, these probably aren't necessary. What's probably causing the problem is not std::vector declarations, but rather using the fill constructor and reserve().

Can you try just doing these two ops lazily? The std::vector ctor should be very very minimal. (on par with the UniquePtr ctor)
Attachment #8781486 - Flags: review?(jgilbert) → review+
Your suggestion works. So I updated my patch with your suggestion. Thanks.
Attachment #8781486 - Attachment is obsolete: true
Flags: needinfo?(jujjyl)
Comment on attachment 8788086 [details]
Bug 1286871 - Only reserve std::vector when necessary.

https://reviewboard.mozilla.org/r/76678/#review75244
Attachment #8788086 - Flags: review?(jgilbert) → review+
Morris, is this ready to land?
Flags: needinfo?(mtseng)
Jeff is refactoring code here in bug 1303879. So let's wait 1303879 landing.
Flags: needinfo?(mtseng)
Can we measure it now that bug 1303879 has landed on nightly?  We've asked for the aurora approval on that bug, but it's a big change and I don't think we should uplift to 50.  We probably still want this simple patch as a solution to uplift straight to beta.
Flags: needinfo?(mtseng)
(In reply to Milan Sreckovic [:milan] from comment #32)
> Can we measure it now that bug 1303879 has landed on nightly?  We've asked
> for the aurora approval on that bug, but it's a big change and I don't think
> we should uplift to 50.  We probably still want this simple patch as a
> solution to uplift straight to beta.

This makes sense. I'll do benchmark today.
Flags: needinfo?(mtseng)
The benchmark shows that bug 1303879 fix the performance issue. However, I'll request uplift the simple patch to beta.
Comment on attachment 8788086 [details]
Bug 1286871 - Only reserve std::vector when necessary.

Approval Request Comment
[Feature/regressing bug #]: bug 1285047
[User impact if declined]: Slightly performance drop (about 6%)
[Describe test coverage new/current, TreeHerder]: Tested on locally
[Risks and why]: Low, just defer some resource allocation request.
[String/UUID change made/needed]: N/A
Attachment #8788086 - Flags: approval-mozilla-beta?
Comment on attachment 8788086 [details]
Bug 1286871 - Only reserve std::vector when necessary.

Perf improvement validated on Nightly, Beta50+
Attachment #8788086 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Depends on: 1303879
Target Milestone: --- → mozilla52
Closing this as fixed by bug 1303879.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: