Open Bug 1516491 Opened 5 years ago Updated 2 years ago

Since FF 64.0, image saving got extremely slow when saving to a folder containing thousands of images (7500+)

Categories

(Core :: Widget: Gtk, defect)

64 Branch
defect

Tracking

()

UNCONFIRMED

People

(Reporter: gnu4ever, Unassigned)

References

Details

(Keywords: regression)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

From a web site hosting pictures, I downloaded some of them, to my usual directory storing more than 7500 pictures.


Actual results:

Firefox 65.0 started to get slower and slower as I was trying to download several pictures. The "Save as" window was taking SECONDS to display after 1 or 2 pictures were downloaded ; the same abnormal delay for displaying the contextual menu (right click on a picture in order to select option "Save picture as..." takes more and more time as I download more and more pictures).


Expected results:

Firefox 65.0 should react very quickly to each of my demands for download, as it used to UP TO release 64.0.
I don't know what happens with release 65.0, but downloading pictures turns into a nightmare each time :(
I'm CERTAIN that the fact my directory stores several thousands pictures is causing trouble, since downloading pictures to an empty test directory behaves normally again - as it used to behave with Firefox 64.0 when I was saving pictures to my 7500+ pictures directory.
Summary: Since FF 65.0, image saving got extremely slow when saving to a folder containing thousands of images (7500+) → Since FF 64.0, image saving got extremely slow when saving to a folder containing thousands of images (7500+)
Version: 65 Branch → 64 Branch
It look like you can reliable reproduce the problem ?
Could run our mozregression tool to find the change in Firefox that caused this ?
- https://mozilla.github.io/mozregression/
Flags: needinfo?(gnu4ever)
(In reply to Valdo from comment #0)
> Firefox 65.0 started to get slower and slower [...]
> 
> Expected results:
> 
> Firefox 65.0 should react very quickly to each of my demands for download,
> as it used to UP TO release 64.0 [...]

Sorry for the confusion with FF release numbers! I'm indeed talking about current public release at the time of writing, which is 64.0.

So anywhere I wrote FF 65.0 in my OP, please read 64.0 instead.

The same goes for "previous FF releases" (which worked perfectly well, even with heavily populated target directories), please substitute "up to 64.0" with "up to 63.0".
Flags: needinfo?(gnu4ever)
(In reply to Matthias Versen [:Matti] from comment #1)
> It look like you can reliable reproduce the problem ?
> Could run our mozregression tool to find the change in Firefox that caused
> this ?
> - https://mozilla.github.io/mozregression/

Thanks for the suggestion Matthias. However I'm unable to start the tool, getting the following errors:

Traceback (most recent call last):
  File "/home/travis/build/mozilla/mozregression/venv/local/lib/python2.7/site-packages/cx_Freeze/initscripts/__startup__.py", line 14, in run
  File "/home/travis/build/mozilla/mozregression/venv/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 26, in run
  File "mozregui/main.py", line 3, in <module>
  File "./mozregui/patch_requests.py", line 17, in patch
  File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 43, in <module>
  File "/usr/local/lib/python2.7/dist-packages/urllib3/__init__.py", line 8, in <module>
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 35, in <module>
  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 3, in <module>
  File "/usr/local/lib/python2.7/dist-packages/urllib3/filepost.py", line 12, in <module>
LookupError: no codec search functions registered: can't find encoding

Let me mention that "travis" is NOT my username (it's apparently hard-coded in the GNU/Linux executable I downloaded from https://github.com/mozilla/mozregression/releases -> v0.9.35). Note: I'm running Kubuntu 18.04.1. Package python-qt4 is already installed on my machine.

If the quickest workaround is to install the command line version of mozregression, tell me so, but since I'll soon be away from my PC until Jan. 1st and getting back at work right after, I won't be able to search for the faulty FF patch before Jan. 5th I'm afraid.
The commandline version of mozregression is better and you should not have problems with as Linux user.
7500 images is an edge case and difficult to reproduce for me. See you next year :-)

Valdo, have you been able to run mozregression?
Otherwise, it is going to be hard for us to act on it
thanks

Flags: needinfo?(gnu4ever)

Hi Matthias and Sylvestre (HNY!),
I'm terribly sorry for the lateness but a change IRL prevented me from spending time on the expected tests and I'm not sure at this point when I'll be able to dedicate time to this unfortunately. I fully understand my participation would be really helpful to you guys, I'd like to help, but it's currently impossible. I'll do my best to find a timeframe.
Note: I'm not sure whether I should take action on the Needinfo flag or not, that's new to me, so in case I'm responsible for this please tell me what I should do exactly :) Thanks.

Flags: needinfo?(gnu4ever)

(In reply to Valdo from comment #6)

Note: I'm not sure whether I should take action on the Needinfo flag or not, that's new to me, so in case I'm responsible for this please tell me what I should do exactly :) Thanks.

No need for explanations finally, I've just got how it works.

Thank you very much for doing that work !

That change also caused another regression with bug 1517074.
The fix from in bug 1517074 changed the filepicker back to the old Firefox version and only KDE users get the native GTK dialog that seem to have caused your problem.

Blocks: 1490186
Component: Untriaged → Widget: Gtk
Keywords: regression
Product: Firefox → Core

You're welcome!
The bad news is that the native GTK dialog behaves badly with Plasma when working on heavily populated directories :) Who's fault? GTK itself? Or some kind of conflict with Plasma? In any case I'm afraid this issue I detected won't be easily resolvable...
Do you think I should raise it to Gnome and/or KDE bug trackers?

Well, that explains some things - I'm using KDE too. I don't have a good way to debug this in detail but I think the problem is that file pickers aren't actually released when they're closed and the sort when the directory updates is very slow. So the first image saved to a directory just causes that file picker instance to update, the second causes both to update, the third causes all three to update, and so on.

This can also cause the entire Firefox UI to lock up if something else is updating a directory that a file has previously been saved to, in some cases making it completely unresponsive until the other program stops changing anything in that directory. (See bug #1378769). This is what actually tipped me off to the possible cause; I don't have a good way of working out how many Gtk file dialogs are being kept around or why, but Firefox was freezing with 100% CPU deep in the Gtk file dialog sort code even though none was open at the time. I suspect this can also cause a pretty bad heap-unclassified memory leak, possibly even into the gigabytes in my case unless something else is causing this.

bug 1517074 disabled the native dialog unless you're running under flatpak or when GTK_USE_PORTAL env variable is set:

https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/toolkit/system/gnome/nsGIOService.cpp#27

so you should not get the native file dialog in KDE by default inn Firefox 65.

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.