Closed Bug 1058177 Opened 11 years ago Closed 3 years ago

GVFS is going crazy, hogs CPU and 100% disk, trashes SSD with lots of file create/delete, indefiniftely

Categories

(Core :: Widget: Gtk, defect, P3)

Product:
Core
Component:
Widget: Gtk
Platform:
x86_64
Linux
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: BenB, Unassigned)

References

Details

(Keywords: hang, regression, Whiteboard: tpi:+)

Ben Bucksch 2014-08-25 12:31:35 PDT I just see that gvfs-metadata is *again* hogging CPU and 100% disk. If I don't kill it, this goes on infinitely. I had done "rm -rf ~/.local/share/gvfs-metadata/" just 2 hours ago, so that's definitely not working. And it's not a question of random storage corruption. I plenty of free space on all drives, so that's not it either. I can't tell you a reproduction, now would I know how to find one. I've never ever even started it. ... [reply] [-] Comment 37 Ben Bucksch 2014-08-25 12:49:47 PDT Given that the process is going wild right now, I've been looking at lsof|grep gvfs and ~/.local/share/gvfs-metadata/ . What I see there is most scary. First, in ~/.local/share/gvfs-metadata/, I see usually 3 files at a time with a UUID filename, of which 1-2 are with .log extension, the others are not. The files change every second (!), they get created and deleted immediately. No wonder we have a CPU and disk hog. Are you actively trying to kill my SSD and my filesystem, or is this create/delete cycle a bug? In fact, this happens so fast that an "ls -la" throws errors at any given time, but different ones each time, because ls and the filesystem apparently can't keep it with the extremely fast create/deletes. An example: $ ls -la ls: Zugriff auf uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-3cae41a6.log nicht möglich: Datei oder Verzeichnis nicht gefunden insgesamt 44 drwx------ 2 ben ben 4096 Aug 25 21:43 . drwx------ 28 ben ben 4096 Aug 25 20:34 .. -rw------- 1 ben ben 56 Aug 25 21:43 uuid-567fed53-db21-4c1b-bf3f-129a448a56e7 ?????????? ? ? ? ? ? uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-3cae41a6.log -rw-rw-r-- 1 ben ben 32768 Aug 25 21:43 uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-fd5950d4.log Please note the "?" of the file stats, and that it says 44 files, but only 3 show up. This is not a filesystem corruption, because the files do not stay like that. The next ls -la shows: $ ls -la insgesamt 40 drwx------ 2 ben ben 115 Aug 25 21:46 . drwx------ 28 ben ben 4096 Aug 25 20:34 .. -rw------- 1 ben ben 56 Aug 25 21:46 uuid-567fed53-db21-4c1b-bf3f-129a448a56e7 -rw-rw-r-- 1 ben ben 32768 Aug 25 21:46 uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-329ad117.log so the ? are gone. Then, 2 ls later, again I get: $ ls -la ls: Zugriff auf uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-d5f586ab.log nicht möglich: Datei oder Verzeichnis nicht gefunden ls: Zugriff auf uuid-567fed53-db21-4c1b-bf3f-129a448a56e7.DK8GNX nicht möglich: Datei oder Verzeichnis nicht gefunden insgesamt 12 drwx------ 2 ben ben 4096 Aug 25 21:47 . drwx------ 28 ben ben 4096 Aug 25 20:34 .. -rw------- 1 ben ben 56 Aug 25 21:47 uuid-567fed53-db21-4c1b-bf3f-129a448a56e7 ?????????? ? ? ? ? ? uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-d5f586ab.log ?????????? ? ? ? ? ? uuid-567fed53-db21-4c1b-bf3f-129a448a56e7.DK8GNX So, this is a constantly changing situation. This is substantiated by lsof, which shows lots of "DEL". Again, they change with each invocation: gvfsd-htt 14751 ben 9w IPv4 510436234 0t0 TCP mymachine:48423->myinternetproxy:3128 (CLOSE_WAIT) gvfsd-htt 14751 ben 10u unix 0x0000000000000000 0t0 510437000 socket gvfsd-htt 14751 ben 11w unix 0x0000000000000000 0t0 510436999 socket gvfsd-htt 14751 ben 12u IPv4 510437004 0t0 TCP mymachine:48424->myinternetproxy:3128 (CLOSE_WAIT) thunderbi 27588 ben DEL REG 8,8 1973904 /home/me/.local/share/gvfs-metadata/root thunderbi 27588 ben mem REG 8,6 94616 50734229 /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so thunderbi 27588 ben mem REG 8,6 169000 50734535 /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so thunderbi 27588 ben DEL REG 8,8 1970882 /home/me/.local/share/gvfs-metadata/root-861ac150.log thunderbi 27588 ben 97r REG 8,8 541548 1973904 /home/me/.local/share/gvfs-metadata/root (deleted) thunderbi 27588 ben 99r REG 8,8 32768 1970882 /home/me/.local/share/gvfs-metadata/root-861ac150.log (deleted) thunderbi 27705 ben mem REG 8,6 94616 50734229 /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so thunderbi 27705 ben mem REG 8,6 169000 50734535 /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so gvfsd-met 29802 ben cwd DIR 8,6 4096 128 / gvfsd-met 29802 ben rtd DIR 8,6 4096 128 / gvfsd-met 29802 ben txt REG 8,6 56640 17596969 /usr/lib/gvfs/gvfsd-metadata gvfsd-met 29802 ben mem REG 8,6 94616 50734229 /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so gvfsd-met 29802 ben mem REG 8,8 32768 68484339 /home/me/.local/share/gvfs-metadata/uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-0cf40ef4.log gvfsd-met 29802 ben DEL REG 8,8 68484330 /home/me/.local/share/gvfs-metadata/uuid-567fed53-db21-4c1b-bf3f-129a448a56e7.VKJ7MX gvfsd-met 29802 ben mem REG 8,6 26258 36002336 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache gvfsd-met 29802 ben 0u CHR 1,3 0t0 1029 /dev/null gvfsd-met 29802 ben 1u CHR 1,3 0t0 1029 /dev/null gvfsd-met 29802 ben 2u CHR 1,3 0t0 1029 /dev/null gvfsd-met 29802 ben 3u 0000 0,9 0 6826 anon_inode gvfsd-met 29802 ben 4u CHR 1,3 0t0 1029 /dev/null gvfsd-met 29802 ben 5u unix 0x0000000000000000 0t0 838838128 socket bash 29984 ben cwd DIR 8,8 4096 68484332 /home/me/.local/share/gvfs-metadata lsof 30025 ben cwd DIR 8,8 4096 68484332 /home/me/.local/share/gvfs-metadata grep 30026 ben cwd DIR 8,8 4096 68484332 /home/me/.local/share/gvfs-metadata lsof 30027 ben cwd DIR 8,8 4096 68484332 /home/me/.local/share/gvfs-metadata firefox 32745 ben DEL REG 8,8 1973904 /home/me/.local/share/gvfs-metadata/root firefox 32745 ben DEL REG 8,8 1970907 /home/me/.local/share/gvfs-metadata/uuid-567fed53-db21-4c1b-bf3f-129a448a56e7.A6L0KX firefox 32745 ben DEL REG 8,8 1970908 /home/me/.local/share/gvfs-metadata/uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-421f706a.log firefox 32745 ben DEL REG 8,8 1964811 /home/me/.local/share/gvfs-metadata/uuid-9fb1aaeb-0f31-4beb-a64e-e27dccc59897-6b0316d0.log firefox 32745 ben mem REG 8,6 94616 50734229 /usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so firefox 32745 ben mem REG 8,6 169000 50734535 /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so firefox 32745 ben DEL REG 8,8 1970882 /home/me/.local/share/gvfs-metadata/root-861ac150.log firefox 32745 ben DEL REG 8,8 1964810 /home/me/.local/share/gvfs-metadata/uuid-9fb1aaeb-0f31-4beb-a64e-e27dccc59897.VE11KX firefox 32745 ben 65r REG 8,8 5976 1964810 /home/me/.local/share/gvfs-metadata/uuid-9fb1aaeb-0f31-4beb-a64e-e27dccc59897.VE11KX (deleted) firefox 32745 ben 66r REG 8,8 32768 1964811 /home/me/.local/share/gvfs-metadata/uuid-9fb1aaeb-0f31-4beb-a64e-e27dccc59897-6b0316d0.log (deleted) firefox 32745 ben 72r REG 8,8 541548 1973904 /home/me/.local/share/gvfs-metadata/root (deleted) firefox 32745 ben 73r REG 8,8 37668 1970907 /home/me/.local/share/gvfs-metadata/uuid-567fed53-db21-4c1b-bf3f-129a448a56e7.A6L0KX (deleted) firefox 32745 ben 75r REG 8,8 32768 1970908 /home/me/.local/share/gvfs-metadata/uuid-567fed53-db21-4c1b-bf3f-129a448a56e7-421f706a.log (deleted) firefox 32745 ben 77r REG 8,8 32768 1970882 /home/me/.local/share/gvfs-metadata/root-861ac150.log (deleted) The thing that scares me most is that Firefox is the process, not gvfs. (Yet, killing gvfsd-metadata solves it - for the short term, so it's definitely related to that process.) What in the world is Firefox and Thunderbird during here? FWIW, these are self-compiled 64bit trunk builds of Firefox and Thunderbird, because I'm a Mozilla developer, but I have seen this bug since 1-2 years, so it's not related to any new Mozilla code change. I'm on Ubuntu 12.04 LTS. [reply] [-] Comment 38 Ben Bucksch 2014-08-25 12:57:05 PDT gvfsd-htt 14751 ben 9w IPv4 510436234 0t0 TCP mymachine:48423->myinternetproxy:3128 (CLOSE_WAIT) That's probably the most scary. What is gvfsd doing on my HTTP internet proxy, talking to the world? [reply] [-] Comment 39 Ben Bucksch 2014-08-25 13:03:40 PDT So, reading the Mozilla source code http://mxr.mozilla.org/comm-central/search?string=gioservice the linking with gvfs is for GIO, and the only thing it *should* be doing is default app integration, i.e. knowing which app to start for which filetype or URL. I.e. /etc/mime.types et al. Definitely nothing about generic file metadata, and nothing that should write to .local/share/gvfs-metadata/ . And MOST DEFINITELY nothing that should talk over the internet. xref https://bugzilla.gnome.org/show_bug.cgi?id=637095 xref bug 494163
I think that this is a bug in gvfs, not Mozilla. I'm filing it here, because * firefox and thunderbird are the processes that are creating/deleting all these files (but that stops immediately when I kill gvfsd-metadata), and are linking gvfs. * Mozilla should never trigger file metadata creating. We must make sure that nothing we do can ever trigger any such action from gvfs. If we can't ensure it, we need to remove gvfs. (No program has the right to index all my drives without my *explicit* permission. This can be dangerous. People regularly keep private stuff on a physical USB drive, for the explicit reason to not leave trails on the local computer - whatever reason they might have. Having metadata about files stored on a *different* drive (partition) is a privacy threat by its very nature and should never be done without explicit user approval.) Esp. these network communications over my proxy, from a daemon that's keeping metadata about all my local files, is super-scary.
Changing component because download manager is the only code I know using metadata.
Component: Widget: Gtk → Download Manager
Product: Core → Toolkit
(In reply to Karl Tomlinson (:karlt) from comment #2) > Changing component because download manager is the only code I know using > metadata. I don't think this component is actually monitored by the people who have the expertise to understand this issue. I can only say that the fact our code sets "metadata::download-uri" looks correct, and according to comment 1 the reported issue might actually be elsewhere.
Component: Download Manager → Widget: Gtk
Product: Toolkit → Core
The expertise is watching https://bugzilla.gnome.org/show_bug.cgi?id=637095#c47 and not Widget::Gtk bugs. The core bug is in gio or gvfs, but saving a data url as metadata sounds like a Download Manager bug to me.
Blocks: 797349
Component: Widget: Gtk → General
Keywords: regression
Product: Core → Toolkit
(In reply to Karl Tomlinson (:karlt) from comment #4) > The core bug is in gio or gvfs, but saving a data url as metadata sounds > like a Download Manager bug to me. Ah, that definitely sounds like one, we shouldn't save "data:" URIs (or probably any scheme that is not http/https/ftp) as metadata, on any platform. I'd appreciate if you can file a separate dependent bug for this specific issue.
Per request in last comment, I filed bug 1060067 about not saving data: URLs. This is not a complete fix, though, because I don't think I've been saving data: URLs when the bug hit me. Back to GTK, because the gvfs code is there and there's no other reasonable place for it to live. I'll try to drive the gvfs bug to be fixed, but we should do from our side what's possible to prevent this. By definition <https://bugzilla.mozilla.org/page.cgi?id=fields.html#bug_severity>, this is Critical. It's not only triggering a hang, but gvfs kills the disk, esp. SSDs.
Severity: normal → critical
Component: General → Widget: Gtk
Product: Toolkit → Core
Hi, just to add some hint to this issue, still happens on my FF 37.0.2 on Ubuntu 14.04 x64, to verify that GVFS goes mad, just try this little HTML example, it loads a dinamyc generated cangas from a web service, if you try to save the canvas as an image, GVFS goes mad! <!DOCTYPE html> <html lang="en-gb" > <head> <title>MGS Creativa - Web Development - Joomla/VirtueMart Development</title> </head> <body> <div style="width: 2219px; height: 699px; padding-bottom: 30px;"> <script src="http://cdn.tagul.com/embed/svue2u2dfq7m"></script> <!-- Please don't remove attribution to Tagul.com --> <div style="display: table; margin: 0 auto;"> <a href="http://tagul.com/">Created with Tagul.com</a> </div> </div> </body> </html>
Same problem after installing Firefox 38.0 on OpenSUSE 13.1
Priority: -- → P2
Whiteboard: tpi:+
Moving to p3 because no activity for at least 1 year(s). See https://github.com/mozilla/bug-handling/blob/master/policy/triage-bugzilla.md#how-do-you-triage for more information
Priority: P2 → P3
Moving to p3 because no activity for at least 1 year(s). See https://github.com/mozilla/bug-handling/blob/master/policy/triage-bugzilla.md#how-do-you-triage for more information

¡Hola Ben!

Is this still an issue on a current Firefox Nightly install, please?

You can get the Firefox Nightly installer from https://nightly.mozilla.org/

¡Gracias!
Alex

status-firefox73: --- → ?
status-firefox74: --- → ?
status-firefox75: --- → ?
status-firefox-esr68: --- → ?
Flags: needinfo?(ben.bucksch)

Alex, please do not randomly ask bug reporters to do things for you. If you want to help, please investigate it yourself. Unless you have specific reason to believe that the bug was fixed, you have to assume it's still there. I am not going to deliberately trigger a condition that destroys my hardware.

This was a Critical bug which has the potential to destroy the hardware (!). Do NOT close it unless you are sure the bug is fixed or the related code has been deleted.

status-firefox73: ? → ---
status-firefox74: ? → ---
status-firefox75: ? → ---
status-firefox-esr68: ? → ---
Flags: needinfo?(ben.bucksch)

¡Hola Ben!

Thanks for getting back.

I've asked as I don't currently have the configuration that you've reported.

https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/517021/comments/79 seems to indicate this or a closely related bug was fixed by https://bugzilla.gnome.org/show_bug.cgi?id=637095

If that's in fact not the case, please file a new issue over at https://gitlab.gnome.org/GNOME/gvfs/issues

I do not close out bugs unless the reporter or a 3rd party that can reproduce the bug verify them as resolved.

¡Gracias!
Alex

Flags: needinfo?(ben.bucksch)

Alex, thanks for these links. Yes, indeed, this shows that the bug was fixed in Gnome and Ubuntu. And I had verified back then that it was indeed fixed.

However, comment 7 and 8 suggest that other people also ran into this problem. Before closing this, somebody with Gnome should at least try the test case in comment 7.

Flags: needinfo?(ben.bucksch)

Hi,
I tried tried the test case in comment 7 on Ubuntu 21.04 and GNOME and wasn't able to reproduce the issue.

Cheers
Stuart

Thanks.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.