Closed Bug 799658 Opened 12 years ago Closed 11 years ago

Investigate sharing xpti-working-set across processes

Categories

(Core :: XPCOM, defect)

defect
Not set
normal

Tracking

()

RESOLVED WONTFIX

People

(Reporter: cjones, Assigned: sinker)

References

Details

(Whiteboard: [slim:1MB][MemShrink:P2][tech-p2])

Attachments

(1 file, 3 obsolete files)

I have very little idea of what this is.  jlebar tells me it's khuey territory.

This is costing 1MB and change USS per gecko process.

If this is a matter of mmap()ing some files with the right flags, we can definitely take it.  If it requires major restructuring of XPCOM, not doable for v1.
This is the memory used by the XPCOM typelib system.  There are two things to do here:

#1: Generate C struct representations of the typelib data and compile that into libxul.
#2: Tweak the XPT in-memory format to not have a ton of relocations (meaning not use char* everywhere) so that more memory can be shared between processes.

I have mostly complete patches for #1 sitting around.  #2 is a lot less important if we end up with some sort of zygote process that we fork, which I've heard is a possibility.  Otherwise we'll want #2 as well.
Assignee: nobody → khuey
As things are currently implemented, we couldn't get a zygote process through even early stages of XPCOM init while remaining forkable, so that's not going to happen for a while (ever?).
Ok.  We'll need part 2 then to see a significant win here.
Whiteboard: [slim:1MB]
> Whiteboard: [slim:1MB]

I don't want to mess up your whiteboard tags, but to be clear: This is 1mb per child process, whereas another bug you marked with [slim:0.7MB] is 0.7MB per device.  Big difference!
Yeah, the tags aren't meant to be that precise, they're there so I can quickly look at a bug and remember what the estimated win is.  Context is needed to interpret the numbers.

If you have an idea for an almost-as-concise-but-more-precise system, please implement it! :)
There is a working patch for part 1 in Bug 685241.  Testing the effects on a device could be useful.
Well, actually, we still need to make the edits to b2g's package manifest too.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #2)
> As things are currently implemented, we couldn't get a zygote process
> through even early stages of XPCOM init while remaining forkable, so that's
> not going to happen for a while (ever?).

We can share inner representation by putting it on a shared memory with a backing file.  By mapping the backing file at exactly the same location, we can share every bit among processes.
Attached patch prove of concept for comment 8 (obsolete) — Splinter Review
This patch also deal with #2 in comment 1.
It is actually saving ~700k/chid, not 1MB that much.
Ping on this bug?  What's the status here?
Assignee: khuey → tlee
Attachment #673176 - Attachment is obsolete: true
Attachment #679858 - Flags: review?
Attachment #679858 - Flags: review? → review?(justin.lebar+bug)
Comment on attachment 679858 [details] [diff] [review]
Load xpti to shared memory at parent

>+#ifdef XPTI_SHMEM_ENABLE

Nit: Can we s/XPTI_SHMEM_ENABLE/XPTI_SHMEM/?  ENABLE is redundant.  (If we left
ENABLE in the name, we'd probably prefer ENABLE_XPTI_SHMEM.)

>+    if (NS_SUCCEEDED(rv) &&
>+        XRE_GetProcessType() != GeckoProcessType_Default) { // child
>+        // Register XPTHeader objects from shmem and created by
>+        // chrome.  ParseManifest() still do any thing, but
>+        // ManifestXPT() stop loading xpti.

The English here needs a bit of work.  How about:

          // Register XPTHeader objects created by our parent process which
          // live in shared memory.  We still call ParseManifest() below, but
          // when it calls into ManifestXPT, we bail and do nothing.

Also, it looks like RegisterManifest() can be called more than once; surely you
don't want to run this code twice?

>+        xptiInterfaceInfoManager *iimgr =
>+            xptiInterfaceInfoManager::GetSingleton();
>+        for (int i = 0; i < gXPTIShmemHeaders->count; i++) {
>+            XPTHeader *header = gXPTIShmemHeaders->headers[i];
>+            iimgr->RegisterXPTHeader(header);
>+        }
>+    }
>+#endif
>     if (NS_SUCCEEDED(rv)) {
>         buf[len] = '\0';
>         ParseManifest(aType, aFile, buf, aChromeOnly);
>     } else if (NS_BOOTSTRAPPED_LOCATION != aType) {
>         nsCString uri;
>         aFile.GetURIString(uri);
>         LogMessage("Could not read chrome manifest '%s'.", uri.get());
>     }

>@@ -41,16 +45,19 @@ xptiInterfaceInfoManager::SizeOfIncludin
> }
> 
> // static
> int64_t
> xptiInterfaceInfoManager::GetXPTIWorkingSetSize()
> {
>     size_t n = XPT_SizeOfArena(gXPTIStructArena, XPTMallocSizeOf);
> 
>+    if (gXPTIStructArenaShmem != NULL) {
>+        n += XPT_SizeOfArena(gXPTIStructArenaShmem, XPTMallocSizeOf);
>+    }

I don't think we should count the shmem stuff in the child processes; it's free
except in terms of virtual memory.  If we do want to count it, we should count
it separately from the non-shmem data.

>+#ifdef XPTI_SHMEM_ENABLE
>+/**
>+ * This address should be selected very carefully, make sure it is
>+ * available over every process (chrome & content processes).

s/, make sure it is available over/; we must make sure it is available to/

>+ * 0x94000000 is available for GONK.  It would be better to reserve it
>+ * with prelink.

Nit: I think we call it "Gonk", not "GONK".

> xptiWorkingSet::xptiWorkingSet()
>     : mTableReentrantMonitor("xptiWorkingSet::mTableReentrantMonitor")
> {
>     MOZ_COUNT_CTOR(xptiWorkingSet);
> 
>     mIIDTable.Init(XPTI_HASHTABLE_SIZE);
>     mNameTable.Init(XPTI_HASHTABLE_SIZE);
> 
>+#ifdef XPTI_SHMEM_ENABLE
>+    // Map shared memory for XPTI
>+    bool parent = XRE_GetProcessType() == GeckoProcessType_Default;
>+    int fd_flags = parent ? (O_RDWR | O_TRUNC | O_CREAT) : O_RDONLY;
>+    int fd = open(XPTI_SHMEM_BACKING, fd_flags);
>+    XPT_ASSERT(fd >= 0);
>+    lseek(fd, XPTI_SHMEM_SIZE, SEEK_SET);
>+    write(fd, "", 1);
>+
>+    int protect = parent ? (PROT_READ | PROT_WRITE) : PROT_READ;
>+    void *xpti_shmem = mmap(XPTI_SHMEM_ADDRESS,
>+                            XPTI_SHMEM_SIZE,
>+                            protect,
>+                            MAP_FIXED | MAP_SHARED, // flags
>+                            fd,
>+                            0); // offset
>+    close(fd);

This is my biggest concern about this patch: What is stopping a malicious child
process from remapping this segment with PR_WRITE and writing garbage into it?
Indeed, couldn't any process on the device write arbitrary memory into this
segment?

We need to lock the shmem someone so that only the parent process can modify it.

I stopped reading here.  We'll definitely need someone else to look at this,
but I'm happy to have another look at it.
Attachment #679858 - Flags: review?(justin.lebar+bug) → review-
Change wording.  Remove backing file after create a read-only fd, and send fd through IPC.
Attachment #679858 - Attachment is obsolete: true
Attachment #680018 - Flags: review?(justin.lebar+bug)
> I don't think we should count the shmem stuff in the child processes; it's
> free
> except in terms of virtual memory.  If we do want to count it, we should
> count
> it separately from the non-shmem data.
The space of shmem is only counted in parent process.  gXPTIStructArenaShmem is always NULL.
Comment on attachment 680018 [details] [diff] [review]
Load xpti to shared memory at parent, v2

>diff --git a/dom/ipc/PContent.ipdl b/dom/ipc/PContent.ipdl
>--- a/dom/ipc/PContent.ipdl
>+++ b/dom/ipc/PContent.ipdl
>@@ -353,14 +353,18 @@ parent:
>         returns (bool showPassword);
> 
>     // Notify the parent of the presence or absence of private docshells
>     PrivateDocShellsExist(bool aExist);
> 
>     // Tell the parent that the child has gone idle for the first time
>     async FirstIdle();
> 
>+    // Get a named file descriptor from parent
>+    sync GetNamedFileDescriptor(nsCString aName)
>+        returns (FileDescriptor aFd);

I'd rather we had GetXPTIShmemFD() or something like that.  This loose coupling
only obfuscates what this method is used for.  It also implies that this method
has more than one use, which isn't true at this point.

>+bool
>+ContentParent::RecvGetNamedFileDescriptor(const nsCString &aName,
>+                                          FileDescriptor *aFd)
>+{
>+#ifdef ENABLE_XPTI_SHMEM
>+    if (aName == NS_LITERAL_CSTRING(XPTI_SHMEM_FD_NAME)) {
>+        *aFd = GetXPTIShmemFd();
>+    } else
>+#endif
>+    {
>+        return false;
>+    }
>+    return true;

A more canonical way of writing this in Gecko would be

>+#ifdef ENABLE_XPTI_SHMEM
>+    if (aName == NS_LITERAL_CSTRING(XPTI_SHMEM_FD_NAME)) {
>+        *aFd = GetXPTIShmemFd();
>+        return true;
>+    }
>+#endif
>+    return false;

This is because we avoid return-after-else.  This is also clearer, imo.

Note that I believe our IPC layer treats |false| as a fatal error, and will
kill the child if we return false here.  That's OK with me, but please indicate
in the IPDL file that you should not call this method speculatively!  :)

>diff --git a/dom/ipc/Makefile.in b/dom/ipc/Makefile.in
>--- a/dom/ipc/Makefile.in
>+++ b/dom/ipc/Makefile.in
>@@ -110,9 +110,13 @@ DEFINES += -DBIN_SUFFIX='"$(BIN_SUFFIX)"
> ifeq ($(MOZ_WIDGET_TOOLKIT),$(findstring $(MOZ_WIDGET_TOOLKIT),android gtk2 gonk qt))
> DEFINES += -DMOZ_ENABLE_FREETYPE
> endif
> 
> ifdef MOZ_PERMISSIONS
> DEFINES += -DMOZ_PERMISSIONS
> endif
> 
>+ifeq ($(MOZ_WIDGET_TOOLKIT),gonk)
>+DEFINES += -DENABLE_XPTI_SHMEM
>+endif

Can define this at some higher level?  That way, you don't have to repeat it
multiple times.  Also, it would be nice if we could turn this on for B2G
desktop builds, since we do a fair bit of testing there.  (If it's Linux-only
on desktop, that's fine; if it works on desktop Mac, that's even better.)

I don't know where to put this change; maybe we add something to
b2g/confvars.sh which causes autoconf to set the defines?  We can ask a build
peer (khuey, glandium, gps) for details.

>diff --git a/xpcom/reflect/xptinfo/public/xptinfo.h b/xpcom/reflect/xptinfo/public/xptinfo.h
>+#define XPTI_SHMEM_FD_NAME "XPTIShmemFd"
>+
>+int GetXPTIShmemFd();

Doesn't this need an ifdef guard?

>diff --git a/xpcom/reflect/xptinfo/src/xptiInterfaceInfoManager.cpp b/xpcom/reflect/xptinfo/src/xptiInterfaceInfoManager.cpp
>--- a/xpcom/reflect/xptinfo/src/xptiInterfaceInfoManager.cpp
>+++ b/xpcom/reflect/xptinfo/src/xptiInterfaceInfoManager.cpp
>@@ -9,16 +9,20 @@
> #include "nsDependentString.h"
> #include "nsString.h"
> #include "nsISupportsArray.h"
> #include "nsArrayEnumerator.h"
> #include "nsDirectoryService.h"
> #include "mozilla/FileUtils.h"
> #include "nsIMemoryReporter.h"
> 
>+#ifdef ENABLE_XPTI_SHMEM
>+#include "nsXULAppAPI.h"
>+#endif

I don't mind if you include this unconditionally, but if you want to leave the
condition in (that's also fine), please add a comment indicating why it's a
conditional include.

>@@ -62,16 +71,19 @@ NS_MEMORY_REPORTER_IMPLEMENT(xptiWorking
> 
> // static
> xptiInterfaceInfoManager*
> xptiInterfaceInfoManager::GetSingleton()
> {
>     if (!gInterfaceInfoManager) {
>         gInterfaceInfoManager = new xptiInterfaceInfoManager();
>         NS_ADDREF(gInterfaceInfoManager);
>+        if (XRE_GetProcessType() != GeckoProcessType_Default) {
>+            gInterfaceInfoManager->RegisterXPTHeadersShmem();
>+        }

Doesn't this need an ifdef?

>diff --git a/xpcom/reflect/xptinfo/src/xptiWorkingSet.cpp b/xpcom/reflect/xptinfo/src/xptiWorkingSet.cpp
>+#ifdef ENABLE_XPTI_SHMEM
>+    // Map shared memory for XPTI
>+    bool parent = XRE_GetProcessType() == GeckoProcessType_Default;
>+    int fd = -1;
>+    if (parent) {
>+        fd = open(XPTI_SHMEM_BACKING, O_RDWR | O_TRUNC | O_CREAT);

Would mkstemp be better here?  Also, if it was me, I'd unlink the file
immediately after checking that open() succeeded, since we don't want to leave
temp files cluttering the fs.  (You could put both calls to open() in the
parent up here, or you could dup and fcntl the inside your if (parent) branch
later on.)

>+        lseek(fd, XPTI_SHMEM_SIZE, SEEK_SET);
>+        write(fd, "", 1);
>+    } else {                    // child
>+        UpdateShmemFd();
>+        fd = mShmemFd;
>+    }
>+    XPT_ASSERT(fd >= 0);
>+
>+    int protect = parent ? (PROT_READ | PROT_WRITE) : PROT_READ;
>+    void *xpti_shmem = mmap(XPTI_SHMEM_ADDRESS,
>+                            XPTI_SHMEM_SIZE,
>+                            protect,
>+                            MAP_FIXED | MAP_SHARED, // flags
>+                            fd,
>+                            0); // offset
>+
>+    XPT_ASSERT(xpti_shmem != nullptr);
>+    gXPTIShmemHeaders = (xptiShmemHeaders *)xpti_shmem;
>+
>+    if (parent) {
>+        // Create a new read-only fd for sharing with children, but
>+        // remove it from filesystem to prevent it from being abused.
>+        mShmemFd = open(XPTI_SHMEM_BACKING, O_RDONLY);
>+        XPT_ASSERT(mShmemFd >= 0);
>+        fcntl(mShmemFd, F_SETFD, FD_CLOEXEC);
>+
>+        close(fd);
>+        unlink(XPTI_SHMEM_BACKING);
>+
>+        // Parent process need an arena for loading XPTI
>+        int header_size = (sizeof(xptiShmemHeaders) + 0xff) & ~0xff;
>+        int arena_size = XPTI_SHMEM_SIZE - header_size;
>+        gXPTIStructArenaShmem =
>+            XPT_NewArenaMem((uint8_t *)xpti_shmem + header_size,
>+                            arena_size,
>+                            sizeof(double),
>+                            "xptiWorkingSet structs (shmem)");
>+    }

This still looks suboptimal to me, primarily since we have to write to the file
system.  But I'm not the best person to speak to whether this is the right way
to share memory here.  glandium, can you help?

>+#endif // ENABLE_XPTI_SHMEM
>+
>     gXPTIStructArena = XPT_NewArena(XPTI_STRUCT_ARENA_BLOCK_SIZE, sizeof(double),
>                                     "xptiWorkingSet structs");
> }        
> 
> static PLDHashOperator
> xpti_Invalidator(const char* keyname, xptiInterfaceEntry* entry, void* arg)
> {
>     entry->LockedInvalidateInterfaceInfo();
>@@ -42,12 +121,45 @@ xptiWorkingSet::InvalidateInterfaceInfos
> xptiWorkingSet::~xptiWorkingSet()
> {
>     MOZ_COUNT_DTOR(xptiWorkingSet);
> 
>     // Only destroy the arena if we're doing leak stats. Why waste shutdown
>     // time touching pages if we don't have to?
> #ifdef NS_FREE_PERMANENT_DATA
>     XPT_DestroyArena(gXPTIStructArena);
>+#ifdef ENABLE_XPTI_SHMEM
>+    XPT_DestroyArena(gXPTIStructArenaShmem);
>+#endif
> #endif
> }
> 
>+#ifdef ENABLE_XPTI_SHMEM
>+/**
>+ * Get fd that backing shared memory for XPTI from parent process.

Retrieve /the/ fd that /backs/ shared XPTI memory in the parent process, and
store it in mShmemFd.

Maybe it should be called GetShmemFdFromParent() or something.

>+void
>+xptiWorkingSet::UpdateShmemFd()
>+{
>+    XPT_ASSERT(XRE_GetProcessType() != GeckoProcessType_Default);
>+    // Only child processes go here.
>+    mozilla::ipc::FileDescriptor fd;
>+
>+    DebugOnly<bool> r = mozilla::dom::ContentChild::GetSingleton()->
>+        SendGetNamedFileDescriptor(NS_LITERAL_CSTRING(XPTI_SHMEM_FD_NAME),
>+                                   &fd);
>+    XPT_ASSERT(r);

We customarily call this |rv|.

>+    mShmemFd = fd.PlatformHandle();
>+}

>+int
>+GetXPTIShmemFd()
>+{
>+    return xptiInterfaceInfoManager::GetSingleton()->
>+        GetWorkingSet()->GetShmemFd();

Please add a comment somewhere that this is a read-only fd in both the parent
and the child processes.

I'm not an expert on Linux shmem or on XPTI, so I feel like I've nitted on this
patch enough.  I think glandium should look at the shmem stuff.  Since has
recently looked at this code (for an alternative implementation), I'm tempted
to suggest him as a reviewer for the rest of it.  bsmedberg, is that OK with
you?
Attachment #680018 - Flags: review?(mh+mozilla)
Attachment #680018 - Flags: review?(khuey)
Attachment #680018 - Flags: review?(justin.lebar+bug)
Flags: needinfo?(benjamin)
Comment on attachment 680018 [details] [diff] [review]
Load xpti to shared memory at parent, v2

Review of attachment 680018 [details] [diff] [review]:
-----------------------------------------------------------------

::: xpcom/reflect/xptinfo/src/xptiWorkingSet.cpp
@@ +28,5 @@
> +/**
> + * This address should be selected very carefully, we must make sure
> + * it is available to every process (chrome & content processes).
> + * 0x94000000 is available for Gonk.  It would be better to reserve it
> + * with prelink.

Oh, that makes me very uncomfortable.
> #2: Tweak the XPT in-memory format to not have a ton of relocations (meaning
> not use char* everywhere) so that more memory can be shared between
> processes.

I barely understand this so forgive me if I'm talking nonsense... but all the xpti data (including strings) ends up in a single arena, and this arena's chunks are now 16 KiB, which is a multiple of 4 KiB.  So sharing doesn't seem like it would be that hard?
> Oh, that makes me very uncomfortable.

Not without reason.  But we control the full stack on these phones, so perhaps this is sane.

If there's an easy way to do something better, then that would be great, of course.
Comment on attachment 680018 [details] [diff] [review]
Load xpti to shared memory at parent, v2

Review of attachment 680018 [details] [diff] [review]:
-----------------------------------------------------------------

::: xpcom/reflect/xptinfo/src/xptiWorkingSet.cpp
@@ +31,5 @@
> + * 0x94000000 is available for Gonk.  It would be better to reserve it
> + * with prelink.
> + */
> +#ifndef XPTI_SHMEM_ADDRESS
> +#define XPTI_SHMEM_ADDRESS ((void *)0x94000000)

ewwww.

@@ +60,5 @@
> +    int fd = -1;
> +    if (parent) {
> +        fd = open(XPTI_SHMEM_BACKING, O_RDWR | O_TRUNC | O_CREAT);
> +        lseek(fd, XPTI_SHMEM_SIZE, SEEK_SET);
> +        write(fd, "", 1);

Use ftruncate().

@@ +73,5 @@
> +                            XPTI_SHMEM_SIZE,
> +                            protect,
> +                            MAP_FIXED | MAP_SHARED, // flags
> +                            fd,
> +                            0); // offset

It would be safer not to use MAP_FIXED, and to check that xpti_shmem == XPTI_SHMEM_ADDRESS. That would avoid unknowingly overlapping existing mappings.

I would be more comfortable if the parent would only use XPTI_SHMEM_ADDRESS as an address hint, and pass xpti_shmem to the child process so that it can try to map at that same address. And if the child fails to do so, fallback to the non-shared memory behavior.

@@ +80,5 @@
> +    gXPTIShmemHeaders = (xptiShmemHeaders *)xpti_shmem;
> +
> +    if (parent) {
> +        // Create a new read-only fd for sharing with children, but
> +        // remove it from filesystem to prevent it from being abused.

Note that removing it from filesystem does *not* prevent abuse. Although, it's still worth removing it.
Attachment #680018 - Flags: review?(mh+mozilla) → review-
(In reply to Nicholas Nethercote [:njn] from comment #18)
> > #2: Tweak the XPT in-memory format to not have a ton of relocations (meaning
> > not use char* everywhere) so that more memory can be shared between
> > processes.
> 
> I barely understand this so forgive me if I'm talking nonsense... but all
> the xpti data (including strings) ends up in a single arena, and this
> arena's chunks are now 16 KiB, which is a multiple of 4 KiB.  So sharing
> doesn't seem like it would be that hard?

The problem is that the data contains pointers, so the addresses would need to be the same. If the data didn't contain pointers, it could be shared at any location in memory.
Thanks for the explanation, Mike.  So the char* fields are a problem, but there are also lots of other pointer fields for variable-sized arrays of things.  You could concoct some encoding that does everything inline without pointers but then lookups would be much harder.  Hmm.

</me-thinking-out-loud>
With some C++ magic, we could actually convert pointers to offsets. That would have a perf impact, though, but for the code currently using the pointers, it would be transparent.
I'm happy to delegate the review to glandium. However, I think that the patches in bug 685241 would probably be the better solution here, and has patches which I think just need to be updated.
Flags: needinfo?(benjamin)
> However, I think that the patches in bug 685241 would probably be the better solution 
> here, and has patches which I think just need to be updated.

AIUI khuey gave up on those patches in favor of this approach.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #24)
> I'm happy to delegate the review to glandium. However, I think that the
> patches in bug 685241 would probably be the better solution here, and has
> patches which I think just need to be updated.

bug 685241 doesn't solve anything for memory use, except if using prelinking at the same time.
or the trick mentioned in comment 23
Why doesn't bug 685241 solve memory use? Since the data is in libxul, doesn't it automatically share the codepage across B2G processes?
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #28)
> Why doesn't bug 685241 solve memory use? Since the data is in libxul,
> doesn't it automatically share the codepage across B2G processes?

B2G processed are fork+exec()ed, which means each process loads libxul and does its own relocations on it. Even if libxul is loaded at the same address by pure chance, relocations (from the use of pointers in the different data structures) will still induce non-shared pages. Prelinking, which makes the linker skip relocations altogether (or at least should), allows to actually share memory.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #28)
> Why doesn't bug 685241 solve memory use? Since the data is in libxul,
> doesn't it automatically share the codepage across B2G processes?

Not until we decrease the relocation count.
(In reply to Thinker Li [:sinker] from comment #8)
> (In reply to Chris Jones [:cjones] [:warhammer] from comment #2)
> > As things are currently implemented, we couldn't get a zygote process
> > through even early stages of XPCOM init while remaining forkable, so that's
> > not going to happen for a while (ever?).
> 
> We can share inner representation by putting it on a shared memory with a
> backing file.  By mapping the backing file at exactly the same location, we
> can share every bit among processes.

That problem has nothing to do with sharing memory.  We spawn threads during early XPCOM initialization and it would require quite a lot of (very risky) work to change that.
Attachment #680018 - Attachment is obsolete: true
Attachment #680018 - Flags: review?(khuey)
Attachment #686432 - Flags: review?(khuey)
Whiteboard: [slim:1MB] → [slim:1MB][MemShrink]
Important content issue, to make more memory available to apps.
Whiteboard: [slim:1MB][MemShrink] → [slim:1MB][MemShrink][tech-p2]
Whiteboard: [slim:1MB][MemShrink][tech-p2] → [slim:1MB][MemShrink:P2][tech-p2]
Has Nuwa rendered this bug moot?
Yeah, I think so.  Do you disagree Thinker?
Flags: needinfo?(tlee)
Agree.
Flags: needinfo?(tlee)
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
See Also: → 1438688
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: