Closed Bug 973149 Opened 6 years ago Closed 4 years ago

Animations cause CPU Kernel mode overhead

Categories

(Core :: Graphics, defect)

27 Branch
x86
Windows XP
defect
Not set

Tracking

()

RESOLVED WORKSFORME

People

(Reporter: jigebren-bug, Unassigned)

References

Details

(Keywords: perf)

Attachments

(3 files)

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:26.0) Gecko/20100101 Firefox/26.0 (Beta/Release)
Build ID: 20131205075310

Steps to reproduce:

As from Firefox 27, animations (ie. an image on a webpage or an animated icon in the toolbar) create an excessive CPU Kernel mode usage on my computer:

Windows XP SP3, Pentium 4 2.4GHz, Nvidia GeForce FX 5500

Summary:
Firefox recently became very sluggish on my computer with no apparent reason. After having unsuccessfully disabled extensions, started form a new profile, and even reinstalled Windows, it came to my mind it was likely rather close after updating to 27 (I'm afraid I have rather stopped paying attention to Firefox updates since the rapid-release cycle move).  I tried downgrading to 25 and it was all for the best. After some investigations I noticed the weird CPU usage, which was unusually high and mostly in Kernel mode, and that it has likely something to do with animations.
I tried 28, and even a nightly 30.0a1, the issue was still there (the latest 27.0.1 release did not fix it either). I'm currently back to 26 with updates disabled to be able to keep using Firefox.

Steps to Reproduce:
On my computer, loading the page below is enough to be sure whether the issue is here or not:
https://en.wikipedia.org/wiki/File:Rotating_earth_%28large%29.gif
- Firefox 26: the CPU usage is at most 30%, Kernel usage is roughly half this value.
- Firefox 27: the CPU usage it at least 75%, and mostly for the Kernel.
I'm afraid though this page may not be complex enough to introduce a noticeable difference on a more modern computer...

Anyway, I've installed MozRegression to find a better regression window than 26<->27 and I came down to this:

Last good revision: 8f8a683dfc42 (2013-09-20)
First bad revision: a2c31dc69ab3 (2013-09-21)
Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8f8a683dfc42&tochange=a2c31dc69ab3
> Anyway, I've installed MozRegression to find a better regression window 

Thank you very much for doing that!

Initially guessing one of the graphics changes there is responsible, at least assuming that graphics driver CPU usage would get counted as part of the kernel.

That would be possibly one of:

http://hg.mozilla.org/mozilla-central/rev/063627f4d8c2
http://hg.mozilla.org/mozilla-central/rev/a2bd329f6ba0
http://hg.mozilla.org/mozilla-central/rev/663104548b52
http://hg.mozilla.org/mozilla-central/rev/1c153d5c8bdd

with that third one being pretty Windows-specific...
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(bas)
Component: General → Graphics
This is the "graphics" section from "about:support" for Firefox 26 on my computer.
You can notice the line:
      "AzureContentBackend": "none"

For Firefox 27 this line is on my computer:
      "AzureContentBackend": "cairo"
I have some interesting info about this issue: I just compared the result from "about:support" for Firefox 26 and 27, and as you can see in my previous attachment there's a difference at the line AzureContentBackend. v26 uses "none" when v27 uses "cairo".

So I tried to prevent this use of cairo in Firefox 27:
In "about:config" I've modified the entry "gfx.content.azure.backends" from "direct2d,cairo" to simply "direct2d". And after restarting Firefox 27, the issue was no more!

It should be noticed (still in the previous attachment) that apparently my graphics card is too old to support Direct2D acceleration. It would explain why this issue doesn't affect much more computers (ie. the cairo fallback is never used when Direct2D is supported already).


Well, that's all I can tell, but at least now we know that: this issue comes from the possible use of cairo in AzureContentBackend, which for some reasons was prevented in v26 and allowed in v27.
This issue likely affects only rather old GFX cards / drivers.
There's currently a workaround for FF 27 and later.
Symptoms of 100% cpu with very slow page loading (~1 minute plus) started immediately after upgrading from v26 to v27.  Safe mode made no difference, nor did v27.0.1.  Seems to effect all pages, even access to local network router config pages were very slow, due to high cpu. 

Changing   direct2d,cairo   to just    direct2d   as documented above, fixed it.  Pages load swiftly now.  

Win XP SP3, AMD Sempron 2Ghz.  Nvidia Geforce FX5200 - with a driver from 2006 as that is more stable than recent ones.  

Hope this helps you narrow it down.
Further to my posting of Feb 21, v27.0.1 continued to work fine with the removal of cairo as above.

I then tried updating to V28 and page loads slowed right down again:  39 secs for front page of Wikipedia, longer for more complex pages up to 96 secs.  Removal of cairo does not fix it, nor running with add-ons disabled.

I have also have a dual boot XP/Ubuntu machine which showed the same slowdown upgrading to V28 on XP, but v28 works fine on Ubuntu - this is a Pentium 4 with truly ancient Nvidia graphics card.

Have had to go back to 27.0.1 on XP as cannot wait for over minute to get to a page.

Other browsers like IE and Opera work fine.
Could someone check whether the bug still exists when they backout http://hg.mozilla.org/mozilla-central/rev/663104548b52? It does seem rather suspect, and I think in XP GDI still runs in the kernel.
Flags: needinfo?(bas)
Windows XP x86
Firefox v29.0.1

Workarounds stopped working starting with version 28. 2014-06-10 still no fix. Pages load slowly.

It appears that about:config was crippled.
"gfx.content.azure.enabled" does nothing, dummy option. This is not the only option.
For example, another option is "browser.fixup.alternate.enabled", it doesn't work, Fx adds www and com when you quickly copy/paste a word into address bar, and when you clean "browser.fixup.alternate.prefix" and "browser.fixup.alternate.suffix", Fx still adds www

I wonder why developers did that?
I see no edit option for comments, hmm..

I just wanted to add, my graphics card is nVidia GeForce 4 MX 440 (on AGP8x) driver (forceware) version: 93.71 (I think that is the latest official driver).

Firefox worked fine up to Fx v26, starting with Fx v27 pages started loading very slowly, fixed using gfx.content.azure.enabled solution (by removing cairo), starting with Fx v28 old fixes doesn't work, no new fixes up to this day, June 10th (Fx v29).
I'm just adding the link to the discussion on the Support Forum, which obviously attracted more attention / vote from users than this bug report on bugzilla:
https://support.mozilla.org/en-US/questions/985969
    70 replies    562 have this problem    24483 views (on 10th June)

To summarize the current state:
  (details can be found in this post:
   https://support.mozilla.org/fr/questions/985969?page=3#answer-546599)
- Firefox 26 worked just fine.
- 27 introduced this cpu overhead with animated images. A workaround was found though (ie. disabling cairo).
- 28 and later removed the possibility to use the workaround. So now, affected users have no choice but to stick with 27 or older.

So far it appears that only older computers using Windows XP are affected (I also only remember nVidia graphic cards being mentioned).

For info I just tried with latest official Fx 29.0.1 on win XP, the issue is still here.
If I dual boot on Linux (Mint 17 MATE), there's no issue with 29 (note that I'm currently not using the nVidia drivers on Linux).

@Bas Schouten
To whom was your question above addressed? I mean, do we have to compile Fx to check that?
It is not just Windows XP. Users have the same problem on Windows 7 and Windows 8.1. 

The problem was so serious that I downgraded all installations to FF24 esr.
Played around with Fx v30.0, problem is still here.

Dear Developers, by ignoring this issue, you are losing your user base. I use too many Fx unique addons to give up on you easily, but most people don't even care about addons, they will just switch to Chrome or something..


(In reply to jigebren from comment #9)
> I'm just adding the link to the discussion on the Support Forum, which
> obviously attracted more attention / vote from users than this bug report on
> bugzilla:
> https://support.mozilla.org/en-US/questions/985969
>     70 replies    562 have this problem    24483 views (on 10th June)

I believe, jigebren, that's because, first of all, people are too lazy to create a new account to report a bug (I for one was).
And most importantly, people don't see a readily available link to this bugzilla bug report page. I think user "hj21c1yg" (or moderator) should edit his front page post and in bold upper-case text write something along the lines:
"Starting with version 28, there are no known workarounds. IF YOU HAVE THIS PROBLEM, GO TO THIS BUGZILLA PAGE, SIGN UP AND VOTE".

To be honest, it's a shame Support Forum is basically useless. So many people had reported they have this issues, and no one from development team seems to care. :\
Oh well..
Just wanted to point out that in my Feb posting I mention that even accessing the config pages of my local routers was just as slowed down as pages out on the internet.  The former are boringly static and do not contain any animations - contrary to the title of this bug report.  

I do not get any slowdown problem on Win7 now I've tried on yet another pc.

I did take the trouble to create an account to find / update this bug report, but it would have been easier to switch to another browser as I have a couple of others on each machine.
@asusmember
When a page is loaded, there's an animation of the icon in the tab, and I've already noticed that this single animation was enough to slowdown Fx (by opening several tabs and rolling the mouse wheel to make the animated tab visible / invisible, while checking the cpu load). So I presume there could be slowdown even on a static page loading.

@Mac & others
It would be useful to know the gfx card and driver version of people that have this issue with Win 7 or 8. You can easily get that from the "about:support" page. For example on my computer:
    "adapterDescription": "NVIDIA GeForce FX 5500",
    "adapterVendorID": "0x10de",
    "adapterDeviceID": "0x0326",
    "adapterRAM": "Unknown",
    "adapterDrivers": "nv4_disp",
    "driverVersion": "6.14.11.7516",
    "driverDate": "5-2-2008",
Agreed.  I had not noticed the tiny circling arrow in the tab, which means there is always an animation present regardless of web page content.

Here is my graphics card and driver version on WinXP, in case it might help.  The driver is deliberately old, as newer ones provoked Blue Screen Stop codes on my set up - so I rolled back a couple of years ago to get stability.

Adapter Description	NVIDIA GeForce FX 5200
Adapter Drivers	nv4_disp
Adapter RAM	Unknown
Vendor ID	0x10de
Device ID	0x0322
Driver Date	10-22-2006
Driver Version	6.14.10.9371


Thanks for giving this bug more publicity.
So, what do you think guys, will this bug ever be fixed?

We all know Cairo is causing this problem. All developers have to do is uncripple about:config backend options and make 'em functional again, simple as that.
The older graphics backend that setting gfx.content.azure.enabled to false enabled isn't functional anymore, so it's not just a matter of making the option work again (I don't know if the old backend is completely gone yet, but they've been switching over to Azure / Moz2D for quite a while now).
(In reply to Emanuel Hoogeveen [:ehoogeveen] from comment #16)
> The older graphics backend that setting gfx.content.azure.enabled to false
> enabled isn't functional anymore, so it's not just a matter of making the
> option work again (I don't know if the old backend is completely gone yet,
> but they've been switching over to Azure / Moz2D for quite a while now).

I meant that we know what causing it, maybe they should just restore the old backend for those who need it?
I found a workaround

1. In Firefox go to "about:support" address.

2. When Troubleshooting page loads, look for "Application Basics" table, and in that table look for "Profile Folder" and hit "Show Folder" button near it. Your profile folder will open.

3. In the opened "*.default" folder create a new folder and name it "chrome" (remove inverted commas).

4. Open your newly created folder and minimize it for now.

5. Open Microsoft Notepad (windows logo+R type notepad). To the new empty document copy-paste the following line:
".tab-throbber { display: none !important }" (remove inverted commas)
Click "File" - "Save as", name your document "userChrome.css" (remove inverted commas) and save it in your earlier created "chrome" folder.

6. Restart Firefox (close it, wait a minute, and then open it up again).

7. Enjoy.

Now pages load almost instantly, and no annoying throbber (activity animation). After page loads favicon appears as usual.
(In reply to Bas Schouten (:bas.schouten) from comment #6)
> Could someone check whether the bug still exists when they backout
> http://hg.mozilla.org/mozilla-central/rev/663104548b52? It does seem rather
> suspect, and I think in XP GDI still runs in the kernel.

Just wondering as his bug is not making any progress,did you get an answer to your question. 

I am also wondering if the needs info flag may be delaying triage and progress of this bug. 

This issue is getting quite a lot of interest on the official support forum. If not the most viewed it is certainly one of the highest voted and viewed active questions on sumo
 77 replies
576 have this problem
26843 views 
https://support.mozilla.org/en-US/questions/985969
Flags: needinfo?(bas)
(In reply to John Hesling [:John99] from comment #19)
> (In reply to Bas Schouten (:bas.schouten) from comment #6)
> > Could someone check whether the bug still exists when they backout
> > http://hg.mozilla.org/mozilla-central/rev/663104548b52? It does seem rather
> > suspect, and I think in XP GDI still runs in the kernel.
> 
> Just wondering as his bug is not making any progress,did you get an answer
> to your question. 
> 
> I am also wondering if the needs info flag may be delaying triage and
> progress of this bug. 
> 
> This issue is getting quite a lot of interest on the official support forum.
> If not the most viewed it is certainly one of the highest voted and viewed
> active questions on sumo
>  77 replies
> 576 have this problem
> 26843 views 
> https://support.mozilla.org/en-US/questions/985969

I never got an answer to that question, sadly.
Flags: needinfo?(bas)
Jeff, do you agree this might well be caused by http://hg.mozilla.org/mozilla-central/rev/663104548b52?
Flags: needinfo?(jmuizelaar)
(In reply to Bas Schouten (:bas.schouten) from comment #21)
> Jeff, do you agree this might well be caused by
> http://hg.mozilla.org/mozilla-central/rev/663104548b52?

It could be yes.
Flags: needinfo?(jmuizelaar)
(In reply to Bas Schouten (:bas.schouten) from comment #20)
> I never got an answer to that question, sadly.

In fact asked you (comment #20) whether we had to be able to compile Fx to test that, but I guess you overlooked it...

If you can produce a Win32 exe file without this Rev 663104548b52, I'll gladly test it and report (you can directly email to me whatever file has to be tested).
BTW, if that's of any help, I remember I once did some tests using Very Sleepy and Fx 28 to try to find where it hangs. It appeared that the most cpu consuming calls when displaying an animated GIF are:
  mozilla::image::FrameBlender::DrawFrameTo, 18.4% ,xul [...\src\frameblender.cpp, Ln:453]
  NtGdiBitBlt, 18.4%, GDI32
  NtGdiAlphaBlend, 7.9%, GDI32
  NtGdiCreateCompatibleBitmap, 5.8% ,GDI32

I think (though I can no longer guarantee for this part) that I also tried with Fx 27 and the "gfx.content.azure.backends" workaround enabled, just to compare. And in this case the only noticeable call was:
  NtUserWaitMessage, 3.6%, USER32
(EDIT for comment #23)
> In fact asked you (comment #20) ...
Sorry, you have to read: In fact I asked you (comment #9) ...
(In reply to jigebren from comment #23)
> (In reply to Bas Schouten (:bas.schouten) from comment #20)
> > I never got an answer to that question, sadly.
> 
> In fact asked you (comment #20) whether we had to be able to compile Fx to
> test that, but I guess you overlooked it...
> 
> If you can produce a Win32 exe file without this Rev 663104548b52, I'll
> gladly test it and report (you can directly email to me whatever file has to
> be tested).

There should be a build to within a few hours:

https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/Unknown-cf3a0df1b90d/
Flags: needinfo?(matt.woodrow)
(In reply to Matt Woodrow (:mattwoodrow) from comment #26)
> There should be a build to within a few hours:
> 
> https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/Unknown-
> cf3a0df1b90d/

I think you meant: https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mwoodrow@mozilla.com-cf3a0df1b90d/try-win32/ :)
I also assumed it was the "mwoodrow@mozilla.com-cf3a0df1b90d" build, and I can confirm that this build fixes the issue.

I was a bit surprised that the displayed version was still around 27, anyway I ensured the "gfx.content.azure.backends" workaround was actually disabled (I have reset the profile to default) for my test.
(In reply to jigebren from comment #28)
> I also assumed it was the "mwoodrow@mozilla.com-cf3a0df1b90d" build, and I
> can confirm that this build fixes the issue.
> 
> I was a bit surprised that the displayed version was still around 27, anyway
> I ensured the "gfx.content.azure.backends" workaround was actually disabled
> (I have reset the profile to default) for my test.

Oops, it built against the revision it was added, not against the latest one.

Trying again: https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mwoodrow@mozilla.com-6979a1cb3dda/try-win32
(In reply to Matt Woodrow (:mattwoodrow) from comment #29)
> Trying again:
> https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mwoodrow@mozilla.com-6979a1cb3dda/try-win32

Ok, this build doesn't have the bug either. It apparently hangs sometime on page loading with CPU at 100%, but it worked after a simple refresh. And when it works, it's seems it can be even better than the Fx 24 ESR I'm currently using (see results herebelow).

To avoid drawing quick conclusion about the Rev 663104548b52, I also tried to download a slightly older nightly build to compare with. I used this one (ie. just one day before): http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-24-03-02-00-mozilla-central/firefox-33.0a1.en-US.win32.zip
And to my surprise the bug was apparently already gone in this one (note that I have already tried some Nightly builds before with no success).

So for completeness' sake I had to compare with a version that actually has the bug: I used the last official (Fx 30).

Here's the CPU usage I observed when displaying 2 animated GIF (http://www.changethethought.com/tag/animated-gifs/)
  Fx 24 ESR                      : CPU 100% (up to 60% for the kernel)
  Fx 30 (official)               : CPU 100% (~70% for the kernel)
  Fx 33.0a1 (nightly/2014-06-24) : CPU 100% (~30% for the kernel)
  Fx 33.0a1 (mwoodrow build)     : CPU  70% (less than 10% for the kernel)


I also tried with the image I use as a reference from the beginning of this bug report (https://en.wikipedia.org/wiki/File:Rotating_earth_%28large%29.gif)
  Fx 24 ESR                      : CPU  45% (20% kernel)
  Fx 30 (official)               : CPU 100% (75% for the kernel)
  Fx 33.0a1 (nightly/2014-06-24) : CPU  45% (almost nothing for the kernel)
  Fx 33.0a1 (mwoodrow build)     : CPU  25% (almost nothing for the kernel)

My conclusion at this point: either the current nightly (33.0a1) has somehow fixed the bug already, or another point was improved and made this bug unnoticeable (note the lower cpu kernel usage for example). Anyway, we can see that removing Rev 663104548b52 is definitively worth it.
Bas, looks like it was that patch. Think we should just back it out?
Flags: needinfo?(bas)
(In reply to Matt Woodrow (:mattwoodrow) from comment #31)
> Bas, looks like it was that patch. Think we should just back it out?

I'd let Jeff provide feedback on this.

Also remember that depending on his exact configuration, on nightly he might not be using GDI at all, in theory. But the backout still seems to make everything better so I guess he must be.
Flags: needinfo?(bas) → needinfo?(jmuizelaar)
Attached file profiler.zip
I did a memory profiler of the nightly version: 
Windows XP 32 bit
Page: [https://upload.wikimedia.org/wikipedia/commons/2/2c/Rotating_earth_%28large%29.gif]

and screencasted this as well: http://screencast.com/t/WieiUOMpS
Attachment #8450531 - Flags: review+
Blocks: 911393
Flags: needinfo?(jmuizelaar)
Any progress with this bug during the last month? On Czech support forum, we have a couple of concerned users.
(In reply to Jeff Muizelaar [:jrmuizel] from comment #36)
> Can someone who can reproduce this issue get a profile of the problem

I switched for ESR 24 to latest official FF (31.0), installed the Gecko Profiler add-on and took a profile while displaying this page: https://en.wikipedia.org/wiki/File:Rotating_earth_%28large%29.gif

The attached profile was saved from Cleopatra, and I confirm the bug was here (CPU at 75%). If this is not enough, or if you need another version (ie. latest build) to be profiled, let me know.
Benoit, can you look at this profile and see if this is something we don't know about?
Flags: needinfo?(bgirard)
Sorry for the delay,

I just looked at the profile and there's massive gaps in the sampling. This usually means that the firefox process is starved. It could be CPU or thrashing.

If it's something caused by firefox it's not captured in the profile.

Can you confirm that you have sufficient physical memory left. Is Firefox usually all CPU time of all available cores?
Flags: needinfo?(bgirard)
It would also be useful to have a profile from a version of firefox that does not show the problem.
(In reply to Benoit Girard (:BenWa) from comment #39)
> Can you confirm that you have sufficient physical memory left. Is Firefox
> usually all CPU time of all available cores?
I have 1 GB of memory on this computer, not much but should be enough. And only one core. During the record of the profile the CPU was not at 100%.

(In reply to Jeff Muizelaar [:jrmuizel] from comment #41)
> It would also be useful to have a profile from a version of firefox that
> does not show the problem.
I tried but with the Nightly 27 and older the Gecko Profiler gives me the following message: "Error in worker: Exception: Error: No threads in the profile. Make sure that you specified valid thread names when profiling.[object Object] (http://people.mozilla.org/~bgirard/cleopatra/js/parserWorker.js:921)"

I've left the Profiler options untouched so far, don't know if there's something I could try here. I also can't try in 24 ESR since the Profiler is not compatible...
I recorded a new profile with previous official FF 31.0, maybe this one will be better... I ensured that Firefox was the only application running, only one tab was opened (https://en.wikipedia.org/wiki/File:Rotating_earth_%28large%29.gif), I used a clean profile (no addon except Gecko Profiler, no options modified). The CPU was less than 50% when recording (the bug was here though).
http://people.mozilla.org/~bgirard/cleopatra/#report=f7c487a9f96b6b68b0e639e41fc529dab05a073e
I dont know if this is the same issue but here is my report.

My previous best config was to keep layers.acceleration enabled but direct2d disabled.  As direct2d caused some crashes and kept my gpu from staying in idle mode.  Also direct2d was slower for some reason.

I then updated to esrv31 as I noticed this was available even tho esrv24 wasnt EOL and played around.

I was noticing some improvements but also some serious regressions, the 2 serious regressions are.

Firefox randomly spiking its cpu usage every 5-8 seconds, I did a lot of testing on this and cannot pin it to a extension/addon or page, it simply happens on esrv31 but does not on esrv24.

The second issue I think may possible be related to this bug report.  The 2nd issue is the firefox gui seems massively resource intensive compared to previous versions.  If I run in my previous optimised config, now the tab closing/opening animation is jerky or even skipping the animation altogether due to resource starvation of some sort, I then notice later animations in web pages were also struggling, e.g. opening and closing tweets on twitter has an animation which on v31 was jerky on v24 smooth.  I can only get it smooth on v31 by enabling direct2d, so now on v31 direct2d is faster than not direct2d.  That to me is good and bad, I think its good that they fixed whatever was slowing down direct2d in earlier versions, but now is an obvious regression in software mode.  I have up until this point assumed its all to do with the new GUI layout, but this v27 thing might be the cause, what I havent done is tested intermediate firefox versions between v24 and v31.

Firefox is quite possibly the most resource intensive app on my system, it stresses my cpu more than most of my games.  This isnt a weak rig.

Windows 7 SP1 x64
Haswell 4670k overclocked to 4.3ghz with idle low clocks disabled.
16 gig ram
OS drive is SSD.
Browser disk cache files using ramdisk.

Right now I am in direct2d mode as with it off its simply too jerky when the browser has to do anything remotely demanding, I am having to use nvidia inspector to keep my GPU in 2d clocks mode.  This doesnt slow down the browser, it up's to 3d clocks for what seems to be no justifiable reason when direct2d mode is enabled.
jigebren, 
do you still see this problem?

FWIW, this is the only bug to be marked a regression of bug 911393
Flags: needinfo?(jigebren-bug)
Keywords: perf
@Wayne
Just tested with the Rotating Earth Gif image and latest official Firefox (44.0.2) and as far as I can see the issue is definitively gone.

Displaying the Wikipedia page mentioned above my cpu is at 20% average, less than 5% for the kernel usage, so the situation seems even better than with Fx 26.
Flags: needinfo?(jigebren-bug)
(In reply to jigebren from comment #46)
> @Wayne
> Just tested with the Rotating Earth Gif image and latest official Firefox
> (44.0.2) and as far as I can see the issue is definitively gone.

Thank you jigebren. I am closing this bug report based on your testing.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.