Open Bug 1803754 Opened 1 year ago Updated 2 months ago

Kubuntu 14.04 with self-built libraries: Log/console spam "failed to open /dev/dri/renderD1xx: Permission denied"

Categories

(Core :: Security: Process Sandboxing, defect, P3)

Firefox 107
x86_64
Linux
defect

Tracking

()

UNCONFIRMED

People

(Reporter: rjvbertin, Unassigned)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(1 file, 1 obsolete file)

Steps to reproduce:

Launch Firefox or Waterfox G5, open youtube.com or any other site doing similar operations.

This bug was initially reported via the site-problem reporter, as https://github.com/webcompat/web-bugs/issues/114647 . Additional details can be found there. <<<

Actual results:

Since a few updates ago I am seeing repeated instances of the following terminal output chunk:

failed to open /dev/dri/renderD128: Permission denied
failed to open /dev/dri/renderD129: Permission denied
failed to open /dev/dri/renderD130: Permission denied
failed to open /dev/dri/renderD131: Permission denied
failed to open /dev/dri/renderD132: Permission denied
failed to open /dev/dri/renderD133: Permission denied
failed to open /dev/dri/renderD134: Permission denied
failed to open /dev/dri/renderD135: Permission denied
failed to open /dev/dri/renderD136: Permission denied
failed to open /dev/dri/renderD137: Permission denied
failed to open /dev/dri/renderD138: Permission denied
failed to open /dev/dri/renderD139: Permission denied
failed to open /dev/dri/renderD140: Permission denied
failed to open /dev/dri/renderD141: Permission denied
failed to open /dev/dri/renderD142: Permission denied
failed to open /dev/dri/renderD143: Permission denied
failed to open /dev/dri/renderD144: Permission denied
failed to open /dev/dri/renderD145: Permission denied
failed to open /dev/dri/renderD146: Permission denied
failed to open /dev/dri/renderD147: Permission denied
failed to open /dev/dri/renderD148: Permission denied
failed to open /dev/dri/renderD149: Permission denied
failed to open /dev/dri/renderD150: Permission denied
failed to open /dev/dri/renderD151: Permission denied
failed to open /dev/dri/renderD152: Permission denied
failed to open /dev/dri/renderD153: Permission denied
failed to open /dev/dri/renderD154: Permission denied
failed to open /dev/dri/renderD155: Permission denied
failed to open /dev/dri/renderD156: Permission denied
failed to open /dev/dri/renderD157: Permission denied
failed to open /dev/dri/renderD158: Permission denied
failed to open /dev/dri/renderD159: Permission denied
failed to open /dev/dri/renderD160: Permission denied
failed to open /dev/dri/renderD161: Permission denied
failed to open /dev/dri/renderD162: Permission denied
failed to open /dev/dri/renderD163: Permission denied
failed to open /dev/dri/renderD164: Permission denied
failed to open /dev/dri/renderD165: Permission denied
failed to open /dev/dri/renderD166: Permission denied
failed to open /dev/dri/renderD167: Permission denied
failed to open /dev/dri/renderD168: Permission denied
failed to open /dev/dri/renderD169: Permission denied
failed to open /dev/dri/renderD170: Permission denied
failed to open /dev/dri/renderD171: Permission denied
failed to open /dev/dri/renderD172: Permission denied
failed to open /dev/dri/renderD173: Permission denied
failed to open /dev/dri/renderD174: Permission denied
failed to open /dev/dri/renderD175: Permission denied
failed to open /dev/dri/renderD176: Permission denied
failed to open /dev/dri/renderD177: Permission denied
failed to open /dev/dri/renderD178: Permission denied
failed to open /dev/dri/renderD179: Permission denied
failed to open /dev/dri/renderD180: Permission denied
failed to open /dev/dri/renderD181: Permission denied
failed to open /dev/dri/renderD182: Permission denied
failed to open /dev/dri/renderD183: Permission denied
failed to open /dev/dri/renderD184: Permission denied
failed to open /dev/dri/renderD185: Permission denied
failed to open /dev/dri/renderD186: Permission denied
failed to open /dev/dri/renderD187: Permission denied
failed to open /dev/dri/renderD188: Permission denied
failed to open /dev/dri/renderD189: Permission denied
failed to open /dev/dri/renderD190: Permission denied
failed to open /dev/dri/renderD191: Permission denied
failed to open /dev/dri/renderD128: Permission denied
failed to open /dev/dri/renderD129: Permission denied
failed to open /dev/dri/renderD130: Permission denied
failed to open /dev/dri/renderD131: Permission denied
failed to open /dev/dri/renderD132: Permission denied
failed to open /dev/dri/renderD133: Permission denied
failed to open /dev/dri/renderD134: Permission denied
failed to open /dev/dri/renderD135: Permission denied
failed to open /dev/dri/renderD136: Permission denied
failed to open /dev/dri/renderD137: Permission denied
failed to open /dev/dri/renderD138: Permission denied
failed to open /dev/dri/renderD139: Permission denied
failed to open /dev/dri/renderD140: Permission denied
failed to open /dev/dri/renderD141: Permission denied
failed to open /dev/dri/renderD142: Permission denied
failed to open /dev/dri/renderD143: Permission denied
failed to open /dev/dri/renderD144: Permission denied
failed to open /dev/dri/renderD145: Permission denied
failed to open /dev/dri/renderD146: Permission denied
failed to open /dev/dri/renderD147: Permission denied
failed to open /dev/dri/renderD148: Permission denied
failed to open /dev/dri/renderD149: Permission denied
failed to open /dev/dri/renderD150: Permission denied
failed to open /dev/dri/renderD151: Permission denied
failed to open /dev/dri/renderD152: Permission denied
failed to open /dev/dri/renderD153: Permission denied
failed to open /dev/dri/renderD154: Permission denied
failed to open /dev/dri/renderD155: Permission denied
failed to open /dev/dri/renderD156: Permission denied
failed to open /dev/dri/renderD157: Permission denied
failed to open /dev/dri/renderD158: Permission denied
failed to open /dev/dri/renderD159: Permission denied
failed to open /dev/dri/renderD160: Permission denied
failed to open /dev/dri/renderD161: Permission denied
failed to open /dev/dri/renderD162: Permission denied
failed to open /dev/dri/renderD163: Permission denied
failed to open /dev/dri/renderD164: Permission denied
failed to open /dev/dri/renderD165: Permission denied
failed to open /dev/dri/renderD166: Permission denied
failed to open /dev/dri/renderD167: Permission denied
failed to open /dev/dri/renderD168: Permission denied
failed to open /dev/dri/renderD169: Permission denied
failed to open /dev/dri/renderD170: Permission denied
failed to open /dev/dri/renderD171: Permission denied
failed to open /dev/dri/renderD172: Permission denied
failed to open /dev/dri/renderD173: Permission denied
failed to open /dev/dri/renderD174: Permission denied
failed to open /dev/dri/renderD175: Permission denied
failed to open /dev/dri/renderD176: Permission denied
failed to open /dev/dri/renderD177: Permission denied
failed to open /dev/dri/renderD178: Permission denied
failed to open /dev/dri/renderD179: Permission denied
failed to open /dev/dri/renderD180: Permission denied
failed to open /dev/dri/renderD181: Permission denied
failed to open /dev/dri/renderD182: Permission denied
failed to open /dev/dri/renderD183: Permission denied
failed to open /dev/dri/renderD184: Permission denied
failed to open /dev/dri/renderD185: Permission denied
failed to open /dev/dri/renderD186: Permission denied
failed to open /dev/dri/renderD187: Permission denied
failed to open /dev/dri/renderD188: Permission denied
failed to open /dev/dri/renderD189: Permission denied
failed to open /dev/dri/renderD190: Permission denied
failed to open /dev/dri/renderD191: Permission denied

Expected results:

None of that should appear, certainly not about device files that do not exist on my system and above all not as repeatedly as now.

My /dev/dri has

9901 0 drwxr-xr-x   2 root root        80 Oct 25 00:12 ./
1025 0 drwxr-xr-x  19 root root      4980 Nov 14 00:35 ../
9908 0 crw-rw----+  1 root video 226,   0 Oct 25 00:13 card0
9902 0 crw-rwxrw-+  1 root video 226, 128 Oct 25 00:13 renderD128

My account is member of the video group, and I set the perms on renderD128 to 0676 myself, from 0660, just to make certain that there shouldn't be a normal Unix permissions issue for any Firefox process that runs under my UID.

I saw an issue here which suggested MOZ_WAYLAND_DRM_DEVICE=/dev/dri/renderD128 might help, but it didn't (not even to get rid of the attempts to do something with non-existent devices.

(In reply to René Bertin from comment #0)

Since a few updates ago I am seeing repeated instances of the following terminal output chunk:

Thanks for the report!
Please

  • open about:support, click on "Copy text to clipboard" and paste it here. Which Linux distribution do you have?
  • try to find a regression range:
    $ pip3 install mozregression
    $ ~/.local/bin/mozregression --good 100 --bad 107 -a https://www.youtube.com/watch?v=1La4QzGeaaQ
OS: Unspecified → Linux
Attached file about:support output

I'm still rocking Kubuntu 14.04 (for the Plasma4 desktop, basically..) but with a self-built 4.14 kernel, newer XOrg, Mesa, GTk 3.24 & DRM related apps and libraries (all self-built).

I'll see if my system can handle the (I assume) considerable load of that mozregression script. Meanwhile I can already say that the issue does not exist in 98.0.r20220322144853 but does exist in 99.0.r20220411174855 . I'll constrain the regression search with --good 98 --bad 99, should make it a bit less work.

More things might break in the future. Please try using latest Ubuntu with a ported Plasma4 theme:
https://store.kde.org/p/1162362/
https://www.ubuntubuzz.com/2019/02/bring-back-kde-4-oxygen-theme-to-kde-plasma-5.html


FEATURE_FAILURE_DDX_INTEL

Firefox' hardware rendering is blocked for the deprecated X11 driver "Intel DDX".

https://packages.debian.org/en/stretch/xserver-xorg-video-intel

The use of this driver is discouraged if your hw is new enough (ca. 2007 and newer). You can try uninstalling this driver and let the server use it's builtin modesetting driver instead.

Please try switching to the build-in and recommended "modesetting" driver.
This can be done with $ sudo apt-get remove xserver-xorg-video-intel and a reboot.


media.ffmpeg.vaapi.enabled: true

This has no effect because VAAPI hardware video decoding depends on X11-EGL and hardware rendering (Mesa 22, X11 modesetting).

Blocks: wr-linux
Summary: Log/console spam "failed to open /dev/dri/renderD1xx: Permission denied" → Kubuntu 14.04 with self-built libraries: Log/console spam "failed to open /dev/dri/renderD1xx: Permission denied"

I meant the full Plasma4 desktop, not just the KDE4 Oxygen theme (which I do not use)!

Isn't there a more easily revertible way to use the modesetting driver, like via XOrg.conf? I do have (E)GL rendering but am still at Mesa 21. FFMpeg's VAAPI works just fine in VLC and QMPlay2 though.

I'm fully aware that more and more things will start failing, but I'll take that bridge when I get there. Meanwhile, the issue at hand is weird and even if there is an explanation why I'd get a permissions error there should still be no need to print out this error every time or to provoke it with non-existing devices.

Here's the bisection result

54:19.17 INFO: Last good revision: d6aabaf771d4c6ee0a63b24e79b3afe379ce7bd6
54:19.17 INFO: First bad revision: 1edd1e4056e62592fe5f457926e5012ad30003d6
54:19.17 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=d6aabaf771d4c6ee0a63b24e79b3afe379ce7bd6&tochange=1edd1e4056e62592fe5f457926e5012ad30003d6

54:19.17 DEBUG: Starting merge handling...
54:19.17 DEBUG: Using url: https://hg.mozilla.org/integration/autoland/json-pushes?changeset=1edd1e4056e62592fe5f457926e5012ad30003d6&full=1
54:19.17 DEBUG: redo: attempt 1/3
54:19.17 DEBUG: redo: retry: calling _default_get with args: ('https://hg.mozilla.org/integration/autoland/json-pushes?changeset=1edd1e4056e62592fe5f457926e5012ad30003d6&full=1',), kwargs: {}, attempt #1
54:19.19 DEBUG: urllib3.connectionpool: Resetting dropped connection: hg.mozilla.org
54:20.26 DEBUG: urllib3.connectionpool: https://hg.mozilla.org:443 "GET /integration/autoland/json-pushes?changeset=1edd1e4056e62592fe5f457926e5012ad30003d6&full=1 HTTP/1.1" 200 None
54:20.28 DEBUG: Found commit message:
Bug 1129492 - Remove X11 access from the Linux content process sandbox. r=gcp,jgilbert

Background: The X11 protocol has a very permissive security model;
clients have essentially full access to the windows of other clients,
and to global resources like input devices.  Previously, our sandbox
policy for content processes needed to allow access to the X server;
this limited its effectiveness against a dedicated attacker.

This patch turns on the `security.sandbox.content.headless` pref added
in bug 1640345, which removes the sandbox policy rules that allowed
making new X11 connections, as well as opening the Xauthority file,
reading hardware info needed by Mesa, etc.  It also runs content
processes in headless mode (whence the name) so they won't connect to a
display server at startup.

This also removes access to the Wayland compositor: the sandbox policy
never allowed that (as of when socket connections became default-deny),
but now content processes won't connect to it at startup.  Wayland is
more capability-oriented so this is less significant for security, but at
a minimum it removes unnecessary attack surface.

Note that if the `webgl.out-of-process` pref is turned off, WebGL
will break unless `security.sandbox.content.headless` is also turned
off.  (Similarly, `widget.non-native-theme.enabled` is needed to render
scrollbars and form controls in content.)  As a result, this patch
adjusts the job definitions used by CI to test in-process WebGL so that
that they will continue to work.

Differential Revision: https://phabricator.services.mozilla.com/D138613

54:20.28 DEBUG: Did not find a branch, checking all integration branches

Apparently the bisect is correct: disabling security.sandbox.content.headless stops the logspam. If I understand the commit message above correctly, disabling this sandbox feature should not matter much on a single-user system that isn't visible outside its LAN? Does running content process headless have an impact on resource use (RAM, mainly)?

1edd1e4056e62592fe5f457926e5012ad30003d6 Jed Davis — Bug 1129492 - Remove X11 access from the Linux content process sandbox. r=gcp,jgilbert
1c322856010d08a9d7e9acbf89da2ea2a57b96e7 Jed Davis — Bug 1638466 - Enable out-of-process WebGL on Linux. r=jgilbert

(In reply to René Bertin from comment #7)

Apparently the bisect is correct: disabling security.sandbox.content.headless stops the logspam. If I understand the commit message above correctly, disabling this sandbox feature should not matter much on a single-user system that isn't visible outside its LAN? Does running content process headless have an impact on resource use (RAM, mainly)?

If I understand correctly, disabling security.sandbox.content.headless could grant X11 access (input devices, window access) to hostile javascript in the content process. Personally I wouldn't change that pref, I would rather disable WebGL (webgl.disabled=true) or try something different (newer Ubuntu, force-enabling EGL, LIBGL_ALWAYS_SOFTWARE=1 environment variable).

gfx.x11-egl.force-disabled: true

Would force-enabling EGL by setting gfx.x11-egl.force-disabled to false, gfx.x11-egl.force-enabled to true and restarting Firefox fix the log spam?

Keywords: regression
Regressed by: 1129492
Hardware: Unspecified → x86_64

:gcp, since you are the author of the regressor, bug 1129492, could you take a look? Also, could you set the severity field?

For more information, please visit auto_nag documentation.

Flags: needinfo?(gpascutto)

Does running content process headless have an impact on resource use (RAM, mainly)?

Yes, I'm fairly sure this improved (lowered) memory usage.

None of that should appear, certainly not about device files that do not exist on my system and above all not as repeatedly as now.

This sounds like an issue with WebGL remoting or the policy to allow GPU access. The regressing bug is probably linked because it depends on WebGL remoting. Given the age of the software involved, and Mesa's tendency to change everything completely around here in every version, it's perhaps not entirely unexpected. "Something" in content (not actually Firefox I suspect, but some library shipped on your distribution that we depend on) is trying to access the GPU from where that is forbidding, and then trying every possible combination (which are the nonexistent devices).

Given that there's no functionality impact, and the age of everything involved, it will probably take a while before we get to this.

Severity: -- → S4
Flags: needinfo?(gpascutto)
Priority: -- → P3
Component: Graphics → Security: Process Sandboxing

That is, it makes more sense if https://hg.mozilla.org/mozilla-central/rev/1c322856010d is the actual regressor.

(In reply to Darkspirit from comment #8)

Would force-enabling EGL by setting gfx.x11-egl.force-disabled to false, gfx.x11-egl.force-enabled to true and restarting Firefox fix the log spam?

No, sadly.

Does security.sandbox.content.headless require a restart or does it affect newly created (sub)processes immediately? Doesn't appear to be the case, but that way you could turn it on when visiting sites serving up sensitive information.

BTW, I presume that the part of the commit responsible for the perms errors I'm seeing is the disallowing the reading of hardware info needed by Mesa? Maybe cutting that access off entirely was a tad overzealous

(The more fundamental question is what business javascript code [run in a browser] can have getting direct X11 access at all but that's probably a rabbit hole I should be going down into.)

As said, I'm at mesa 21.0.1 and also have libdrm 2.4.105 . Xorg is at 1.18.3 (from Xenial), libva2 at 2.6.1 and i965-va-driver at 2.1.0 . That's the previous major Mesa release, which I presume will still common in the more conservative distros. Any of those I need to look at updating, or any suggestions of another library where the actual access attempts occur?

(In reply to Gian-Carlo Pascutto [:gcp] from comment #11)

That is, it makes more sense if https://hg.mozilla.org/mozilla-central/rev/1c322856010d is the actual regressor.

Setting webgl.out-of-process to false does prevent the messages from being logged. I understand that disabling this also means you have to disable the sandboxing or else WebGL will break, but I do not see mention that webGL will be forced out-of-process if sandboxing is enabled.

do not see mention that webGL will be forced out-of-process if sandboxing is enabled.

It won't be, but it's also not expected to work:
https://bugzilla.mozilla.org/show_bug.cgi?id=1129492#c43

Am I missing something? That comment (the commit message) mentions that the sandboxings needs to be disabled if webgl is set to be in-process. The message isn't specific enough if the 2 settings need to be equal (both on or both off).
"It's also not expected to work" (if sandboxing is enabled) can also imply that webgl (probably) doesn't work at all if sandboxing is on. That too doesn't follow from the commit message...

Your understanding is correct, I was just being too terse responding. After looking more, the regressor does make sense:
https://searchfox.org/mozilla-central/rev/abcee8d2c97a5c8a1fbeaf84607ea427be72497a/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#409

That is, enabling headless is what actually removes the access to the devices inside the content sandbox. (This is actually in the commit message in comment 6 and mentioned again comment 8, which I missed because I was already mentally on the wrong track). I think the mystery is why content is trying to access the GPU in the first place. And why flipping webgl.out-of-process to false makes the messages go away isn't clear to me either, I think we'd expect the opposite.

As said, I'm at mesa 21.0.1 and also have libdrm 2.4.105 . Xorg is at 1.18.3 (from Xenial), libva2 at 2.6.1 and i965-va-driver at 2.1.0 . That's the previous major Mesa release

Highlighting this because Kubuntu 14.04 in the topic is misleading wrt Mesa versions, though you're definitely running a mixture of things that's far outside of the normal (tested) stuff :-)

FWIW, the workaround above (with webgl.out-of-process and security.sandbox.content.headless) no longer have effect.

It still beats my code would try to access inexisting files and then get a permission-denied error. Could it be that the errno is from another call and simply not reset?

(In reply to René Bertin from comment #17)

Could it be that the errno is from another call and simply not reset?

That could actually be Mesa's loader/loader.c and/or glx/dri2_glx.c, I'll be checking that.

Annoyingly I get the error with my main browser profile, but not with other profiles - or I haven't yet figured out what kind of content triggers it.

I can confirm that the error messages indeed come from Mesa itself, to be specific from a call to loader_open_render_node() which apparently thinks that I have a lot more devices than I have device files. I have tinkered a bit in the lower level loader_open_device() which tries to open the devices files (open(device_name, O_RDRW[|O_CLOEXEC])), making the function bail if the device doesn't exist (checking with stat(2)).

diff --git b/src/loader/orig.loader.c a/src/loader/loader.c
index d64bc7c..c8d1ed8 100644
--- b/src/loader/orig.loader.c
+++ a/src/loader/loader.c
@@ -83,18 +83,25 @@ int
 loader_open_device(const char *device_name)
 {
    int fd;
+   struct stat dum;
+   if (stat(device_name, &dum) == -1 && errno == ENOENT) {
+//       log_(_LOADER_WARNING, "loader_open_device() pid=%d %s doesn't exist\n", getpid(), device_name);
+        return -1;
+   }
 #ifdef O_CLOEXEC
+   errno = 0;
    fd = open(device_name, O_RDWR | O_CLOEXEC);
    if (fd == -1 && errno == EINVAL)
 #endif
    {
+      errno = 0;
       fd = open(device_name, O_RDWR);
       if (fd != -1)
          fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) | FD_CLOEXEC);
    }
    if (fd == -1 && errno == EACCES) {
-      log_(_LOADER_WARNING, "failed to open %s: %s\n",
-           device_name, strerror(errno));
+      log_(_LOADER_WARNING, "loader_open_device() pid=%d failed to open %s: %s\n",
+           getpid(), device_name, strerror(errno));
    }
    return fd;
 }

This works but for some as yet inexplicable reason 1 process still gives me a slew of "permission denied" errors for non-existing device files. From what I understand this can happen (only, for non-existing files) if there is an access problem on one of the components of the full path to the file. That shouldn't be the case here; /dev and /dev/dri both are R+X for U+G+O. The only reason I can think of is that this particular process runs chroot'ed from somewhere it cannot see /dev/dri at all - is that possible?
(I'm certain it's using my Mesa library because of the more detailed error messages that I implemented, outputting the PID.)

loader_open_device() pid=5528 failed to open /dev/dri/renderD129: Permission denied
loader_open_device() pid=5528 failed to open /dev/dri/renderD130: Permission denied
loader_open_device() pid=5528 failed to open /dev/dri/renderD131: Permission denied

(In reply to René Bertin from comment #19)

The only reason I can think of is that this particular process runs chroot'ed from somewhere it cannot see /dev/dri at all - is that possible?

I thus modified the warning message a bit more to print the current working directory (or the error when trying to get it):

   if (fd == -1 && errno == EACCES) {
      char *error = strdup(strerror(errno));
      char *cwd = getcwd(NULL,0);
      if (!cwd) {
           cwd = strdup(strerror(errno));
      }
      log_(_LOADER_WARNING, "loader_open_device() pid=%d (cwd=%s) failed to open %s: %s\n",
           getpid(), cwd, device_name, error);
      if (error) free(error);
      if (cwd) free(cwd);
   }

I see this:

loader_open_device() pid=27363 (cwd=No such file or directory) failed to open /dev/dri/renderD189: Permission denied
loader_open_device() pid=27363 (cwd=No such file or directory) failed to open /dev/dri/renderD190: Permission denied
loader_open_device() pid=27363 (cwd=No such file or directory) failed to open /dev/dri/renderD191: Permission denied
ATTENTION: default value of option mesa_glthread overridden by environment.
Sandbox: seccomp sandbox violation: pid 27430, tid 27430, syscall 79, args 140126061268992 4096 140126489411616 177 0 0.
loader_open_device() pid=27430 (cwd=Function not implemented) failed to open /dev/dri/renderD128: Permission denied
Sandbox: seccomp sandbox violation: pid 27430, tid 27430, syscall 79, args 140126061268992 4096 140126489411616 177 0 0.
loader_open_device() pid=27430 (cwd=Function not implemented) failed to open /dev/dri/renderD128: Permission denied
ATTENTION: default value of option mesa_glthread overridden by environment.
Sandbox: seccomp sandbox violation: pid 27430, tid 27430, syscall 41, args 1 526337 0 140729702037072 0 1024.
Sandbox: seccomp sandbox violation: pid 27430, tid 27430, syscall 41, args 1 526337 0 140729702037072 36 1024.
ATTENTION: default value of option mesa_glthread overridden by environment.

According to man getwd, this means that pid 27363 runs in an unlinked working directory. Not sure when that would raise ENOPERM when trying to open /dev/dri/renderDXYZ though.
Pid 27430 ("Utility Process") is more interesting: I have no documentation what causes getcwd() to raise "Function not implemented". Syscall 41 is probably something socket-related, BTW.

Attachment #9386894 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: