Closed Bug 1325054 Opened 7 years ago Closed 7 years ago

Browsing Facebook causes hangs

Categories

(Core :: Networking: HTTP, defect, P2)

x86_64
Windows 7
defect

Tracking

()

RESOLVED FIXED
mozilla55
Performance Impact high
Tracking Status
firefox53 --- wontfix
firefox55 --- fixed

People

(Reporter: guijoselito, Assigned: hchang)

References

(Blocks 1 open bug)

Details

(Keywords: hang)

Attachments

(1 file)

After some time browsing around, loading some page or loading more content on Facebook makes Nightly hang for a couple of seconds.
After a hang I always went on about:telemetry and looked the "Browser Hangs" section.
Everytime it's exactly the same stack (comparing various hangs on different builds):

NtWaitForSingleObject (in wntdll.pdb)
WaitForSingleObjectExImplementation (in wkernel32.pdb)
WaitForSingleObject (in wkernel32.pdb)
_PR_MD_WAIT_CV (in nss3.pdb)
_PR_WaitCondVar (in nss3.pdb)
PR_WaitCondVar (in nss3.pdb)
mozilla::CondVar::Wait(unsigned int) (in xul.pdb)
mozilla::SyncRunnable::DispatchToThread(nsIEventTarget *,bool) (in xul.pdb)
mozilla::SyncRunnable::DispatchToThread(nsIEventTarget *,nsIRunnable *,bool) (in xul.pdb)
UrlClassifierDBServiceWorkerProxy::DoLocalLookup(nsACString_internal const &,nsACString_internal const &,nsTArray<mozilla::safebrowsing::LookupResult> *) (in xul.pdb)
nsUrlClassifierDBService::ClassifyLocalWithTables(nsIURI *,nsACString_internal const &,nsTArray<nsCString> &) (in xul.pdb)
mozilla::net::nsHttpChannel::BeginConnect() (in xul.pdb)
mozilla::net::nsHttpChannel::OnProxyAvailable(nsICancelable *,nsIChannel *,nsIProxyInfo *,nsresult) (in xul.pdb)
mozilla::net::nsAsyncResolveRequest::DoCallback() (in xul.pdb)
mozilla::net::nsAsyncResolveRequest::OnQueryComplete(nsresult,nsCString const &,nsCString const &) (in xul.pdb)
mozilla::net::ExecuteCallback::Run() (in xul.pdb)
nsThread::ProcessNextEvent(bool,bool *) (in xul.pdb)
NS_ProcessNextEvent(nsIThread *,bool) (in xul.pdb)
mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) (in xul.pdb)
MessageLoop::RunHandler() (in xul.pdb)
MessageLoop::Run() (in xul.pdb)
nsBaseAppShell::Run() (in xul.pdb)
nsAppShell::Run() (in xul.pdb)
nsAppStartup::Run() (in xul.pdb)
XREMain::XRE_mainRun() (in xul.pdb)
XREMain::XRE_main(int,char * * const,mozilla::XREAppData const &) (in xul.pdb)
XRE_main (in xul.pdb)
do_main (in firefox.pdb)
wmain (in firefox.pdb)
__scrt_common_main_seh (in firefox.pdb)
BaseThreadInitThunk (in wkernel32.pdb)
__RtlUserThreadStart (in wntdll.pdb)
_RtlUserThreadStart (in wntdll.pdb)
Guilherme, do you know how to use the profiler (http://new.cleopatra.io/)? A profile from there around the time you see these hangs would help (note that it'll contain URLs but not cookies).
Flags: needinfo?(guijoselito)
I got this one https://perfht.ml/2lNPSdK but it wasn't as the usual hangs. I loaded Facebook and Nightly completely hanged until it was given me the option to stop the page making Nightly slow. Hope it indicates something doable.
I'm still trying to get another of those hangs. Nightly seems to be working better these last couple of days.
Flags: needinfo?(guijoselito)
Ehsan, any thoughts on the profile from comment 2?

Thanks, Guilherme!
Flags: needinfo?(ehsan)
I don't understand this profile at all.  Large parts of the second content process' stack are missing.  Markus, can you please take a look?  Thanks!
Flags: needinfo?(ehsan) → needinfo?(mstange)
Yes, this is a really strange profile. The content process does have a few samples at the end, but I'd expect a lot more than those. Guilherme, can you try increasing the buffer size in the profiler settings (in the panel)?

In any case, the profiler is really bad at profiling content process hangs at the moment. When it grabs the profile, this "grab" message patiently waits in the content process main thread event queue until it's processed, so by the time you end up actually collecting the content process profile, the part of the hang that you were interested in might have long left the profile buffer. We have bug 1330185 about improving this.
Flags: needinfo?(mstange)
Finally Nightly hanged for a few seconds while loading some pictures and the list of friends online, I guess - https://perfht.ml/2kSs33H
I have an Average Joe notebook and I run the World Community Grid on 1 core, so my system is kind of slow. The stack in about:telemetry is exactly the same from comment 0.
This profile confirms that the stack in comment 0 is actually the correct one. Good!

So the main thread is dispatching a DoLocalLookupRunnable to the nsUrlClassifierDBService::BackgroundThread() and blocks until that runnable has run. So either the DoLocalLookupRunnable execution itself takes a lot of time on that thread, or the thread is busy with other stuff and takes a while until it gets around to processing the new DoLocalLookupRunnable.

Once bug 1340327 is fixed, can you try to get a profile that includes that thread? In order to do that, you need to change the value of the "Threads" textbox in the profiler panel settings to "GeckoMain,Compositor,URL Classifier". (This won't work in current Nightlies, due to bug 1340327. So you either need to use a Nightly from about two weeks ago, or we can just wait for that bug's fix to make it into a Nightly in a few days time).
It looks like the issue which has been fixed in Bug 1315097.
(In reply to Markus Stange [:mstange] (away until Feb 22) from comment #7)
> This profile confirms that the stack in comment 0 is actually the correct
> one. Good!
> 
> So the main thread is dispatching a DoLocalLookupRunnable to the
> nsUrlClassifierDBService::BackgroundThread() and blocks until that runnable
> has run. So either the DoLocalLookupRunnable execution itself takes a lot of
> time on that thread, or the thread is busy with other stuff and takes a
> while until it gets around to processing the new DoLocalLookupRunnable.
> 
> Once bug 1340327 is fixed, can you try to get a profile that includes that
> thread? In order to do that, you need to change the value of the "Threads"
> textbox in the profiler panel settings to "GeckoMain,Compositor,URL
> Classifier". (This won't work in current Nightlies, due to bug 1340327. So
> you either need to use a Nightly from about two weeks ago, or we can just
> wait for that bug's fix to make it into a Nightly in a few days time).

https://perfht.ml/2miKkrd
I think this is a dupe of bug 1334616.  I didn't realize this earlier since I think I didn't read comment 0 closely and just looked at the profile and got confused.  It's great that you can reproduce this, Henry may have questions for you.  :-)
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → DUPLICATE
Actually I just realized that bug 1334616 is different.  In that bug, the call into ClassifyLocalWithTables() comes from IsTrackerWhitelisted(), but here the call comes from nsHttpChannel::BeginConnect(), which is a totally different path.

Guilherme, do you have tracking protection enabled globally or are you using Facebook in a private browsing window with TP enabled?
Flags: needinfo?(guijoselito)
Henry, any chance you can look at this one too?
Status: RESOLVED → REOPENED
Flags: needinfo?(hchang)
Resolution: DUPLICATE → ---
Blocks: QF-Websites
Tracking protection enabled globally with the basic protection list.
Flags: needinfo?(guijoselito)
I would guess it's the same cause (ClassifyLocalWithTables) in another callsite [1].

Actually there are 4 callsites of ClassifyLocalWithTables for now

1) dom/base/nsDocument.cpp =>  nsDocument::PrincipalFlashClassification
2) dom/ipc/ContentParent.cpp => mozilla::dom::ContentParent::RecvClassifyLocal
3) netwerk/base/nsChannelClassifier.cpp => mozilla::net::nsChannelClassifier::IsTrackerWhitelisted
4) netwerk/protocol/http/nsHttpChannel.cpp => mozilla::net::nsHttpChannel::BeginConnect

(1) not sure if it's on main thread.
(2) is for e10s (not sure if it's being used)
(3) should be fixed by Bug 1341506. 
(4) this bug

I can check them individually but start from (4) first.

[1] http://searchfox.org/mozilla-central/rev/b1044cf7c2000c3e75e8181e893236a940c8b6d2/netwerk/protocol/http/nsHttpChannel.cpp#6115
Flags: needinfo?(hchang)
(In reply to Henry Chang [:henry][:hchang] from comment #14)
> I would guess it's the same cause (ClassifyLocalWithTables) in another
> callsite [1].
> 
> Actually there are 4 callsites of ClassifyLocalWithTables for now
> 
> 1) dom/base/nsDocument.cpp =>  nsDocument::PrincipalFlashClassification
> 2) dom/ipc/ContentParent.cpp =>
> mozilla::dom::ContentParent::RecvClassifyLocal
> 3) netwerk/base/nsChannelClassifier.cpp =>
> mozilla::net::nsChannelClassifier::IsTrackerWhitelisted
> 4) netwerk/protocol/http/nsHttpChannel.cpp =>
> mozilla::net::nsHttpChannel::BeginConnect
> 
> (1) not sure if it's on main thread.

It's on the content process's main thread which does a sync IPC to the parent process's main thread.  Both bad.

> (2) is for e10s (not sure if it's being used)

That's the parent side of (1).

> (3) should be fixed by Bug 1341506. 
> (4) this bug

I think (4) is definitely more important to investigate, since it will affect all TP usages.
(In reply to François Marier [:francois] from comment #16)
> https://bugzilla.mozilla.org/show_bug.cgi?id=1100024#c2 is related to (4).

Yeah.  I suspect we should have a way to suspend the channel to prevent it from creating sockets and whatnot momentarily when we get to BeginConnect().  I don't know the right Necko primitives for that.  If that's not currently supported, adding support for it sounds like a better solution to me than a blocking wait.
Blocks: UIJank
Jason - what are your thoughts in relation to comment 17?
Flags: needinfo?(jduell.mcbugs)
Some updates and summary:

Background: 

The root cause of this bug is the long blocking sync call 
"nsIURLClassifier.classifyLocalWithTables" on main thread.

Issues with the same root cause:

There are 3 callsites [1] of nsIURLClassifier.classifyLocalWithTables() in gecko.
(There used to be 4 but I have fixed one of them in Bug 1334616.) 
We should ideally change to use the async version [2] on all callsites.
Bug 1342333 is tracking this.

Solution in other aspect:

In the mean time, I am also working on Bug 1338017 and many dependent bugs
to minimize the blocking time. However, since minimizing the blocking time
is required to rewrite the url-classifier backend DB update threading, it's 
too risky to land before the coming branch day. (I've actually got all the
patches.)

Conclusion:

I haven't jumped into this bug so it's less likely I can turn the sync call
into async before the branch day. However, once Bug 1338017 is resolved, the
blocking should be significantly reduced. That being said, this bug should
still be outstanding for the "sync to async" work.

[1]

dom/base/nsDocument.cpp
13090	rv = uriClassifier->ClassifyLocalWithTables(classificationURI, // found in nsDocument::PrincipalFlashClassification
dom/ipc/ContentParent.cpp
5137	*aRv = uriClassifier->ClassifyLocalWithTables(uri, aTables, *aResults); // found in mozilla::dom::ContentParent::RecvClassifyLocal
netwerk/protocol/http/nsHttpChannel.cpp
5973	rv = classifier->ClassifyLocalWithTables(uri, tables, results); // found in mozilla::net::nsHttpChannel::BeginConnect

[2] http://searchfox.org/mozilla-central/rev/e844f7b79d6c14f45478dc9ea1d7f7f82f94fba6/netwerk/base/nsIURIClassifier.idl#88
Blocks: 1342333
Assignee: nobody → hchang
If the only concerns in https://bugzilla.mozilla.org/show_bug.cgi?id=1100024#c2
are the "DNS prefetch" and "Speculative connection", my patch should fix Bug 1100024
without using a synchronous call.

Let's start from the "original" BeginConnect(). What it did is

1) Sanity check and get/create mConnectionInfo for this channel.

2) Synchronously initialze mLocalBlockList

3) Do "DNS prefetch" if mLocalBlockList is false.

4) a) if mLocalBlockList is true:
        - (i)  nsChannelClassifier::Start() ==> the channel is suspended.
        - (ii) BeginConnectWithResult()     ==> Returns immediately since the channel has been suspended.
                                                Connect() will not be called.
   b) Otherwise
        - (i)  BeginConnectWithResult()     ==> Connect() and SpeculativeConnect() will be called.
        - (ii) nsChannelClassifier::Start() ==> Channel will be suspended but it only stops OnStart/OnData/OnStop.


So, my solution is to rewrtie BeginConnect() with BeginConnectWouldSucceed() and BeginConnectActual():

nsresult BeginConnect() 
{
  nsresult rv = BeginConnectWouldSucceed(); // Will only do (1)
  if (NS_FAILED(rv)) {
    return rv;
  }
  
  InitLocalBlockList( (aLocalBlockList) => {
    // Inside the callback.
    mLocalBlockList = aLocalBlockList; // Do (2)
    BeginConnectActual(); // Will do (3) and (4)
  });
  
  return NS_OK;
}

- BeginConnect(): remains its semantics except for (3.a below)

- BeginConnectWouldSucceed(): never makes connection but 
  might change the internal state. (probably needs a better name.)

- BeginConnectActual(): might make connection.

Following is the list for checking if BeginConnect() actually remains the semantics:
(Only 3.b is different from before.)

1) If any error occurs in (1), BeginConnect() will **synchronously** return error.

2) If no LOAD_CLASSIFY_URI, BeginConnect() will **synchronously** return 
   what ContinueBeginConnectWithResult() returns.

3) If LOAD_CLASSIFY_URI
   a) If the URI is a tracker, BeginConnectWithResult() will *asynchronously* be called
   b) Otherwise, BeginConnectWithResult() will *asynchronously* be called 
      (the original behavior is *synchronously be called and returns.*)

I cannot say if (3.b) matters or not. For example, if a non-tracker channel
would fail on calling Connect(), BeginConnect() will now return NS_OK the propagate
to asyncOpen (it used to return error.) The channel will still be aborted and
report OnStopRequest() after BeginConnectWithResult() asynchronously failed
so it shouldn't be considered incorrect behavior. (but I would not be surprised
if any caller of AsyncOpen() doesn't handle its return value very well.)
Attachment #8843890 - Flags: review?(mcmanus)
Hi Patrick,

Since you are the reviewer of Bug 1100024 so I assume you are familiar enough
and interested in reviewing this "follow-up" bug. In Bug 1100024, we introduced
a sync call to prevent "early" connection to the tracker URL. In this bug,
I identified all possible "early" connections (DNS prefetch and Speculative connection) 
and refactored BeginConnect() a little bit to defer DNS prefetch and Speculative connection
until we **asynchronously** know if it's a tracker URL.

More detailed explanation is in comment 21. Could you help review or forward to the
better reviewer you think of?

Thanks :)
Attachment #8843890 - Flags: review?(mcmanus) → review?(dd.mozilla)
Attachment #8843890 - Flags: review?(mcmanus) → review?(dd.mozilla)
Priority: -- → P2
Whiteboard: [qf:p1]
Comment on attachment 8843890 [details]
Bug 1325054 - Defer any possible connection establishment in BeginConnect until knowing if it's a tracker.

https://reviewboard.mozilla.org/r/117512/#review121030

Sorry for a delay. See my comments

::: netwerk/protocol/http/nsHttpChannel.cpp:5799
(Diff revision 3)
>  }
>  
> +namespace {
> +
> +class InitLocalBlockListXpcCallback final : public nsIURIClassifierCallback {
> +public:

You do not need make a new class just let nsHttpChannel implement nsIURIClassifierCallback.

::: netwerk/protocol/http/nsHttpChannel.cpp:5834
(Diff revision 3)
> +} // end of unnamed namespace/
> +
> +void
> +nsHttpChannel::InitLocalBlockList(const InitLocalBlockListCallback& aCallback)
> +{
> +    if (!(mLoadFlags & LOAD_CLASSIFY_URI)) {

Move this int BeginConnect. And make:
if (!(mLoadFlags & LOAD_CLASSIFY_URI)) {
  return BeginConnectionActual();
} else {
  InitLocalBlockList()
}
and add comment explaining each of these 2.

Probably you could return bool from InitLocalBlockLIst - succeeded or not and call BeginConnectionActual

::: netwerk/protocol/http/nsHttpChannel.cpp:5892
(Diff revision 3)
>  // functions that called BeginConnect if needed. Only AsyncOpen and
>  // OnProxyAvailable ever call BeginConnect.
>  nsresult
>  nsHttpChannel::BeginConnect()
>  {
> +    nsresult rv = BeginConnectWouldSucceed();

you can just remove BeginConnectWouldSucceed and move its content to BeginConnect. We have so many somethingsomethingBeginConnectionsomethingsomething :) (I am to blame for some of them)
Attachment #8843890 - Flags: review?(dd.mozilla)
Flags: needinfo?(jduell.mcbugs)
Comment on attachment 8843890 [details]
Bug 1325054 - Defer any possible connection establishment in BeginConnect until knowing if it's a tracker.

https://reviewboard.mozilla.org/r/117512/#review121030

Hi Dragana, 

Thanks for your review. I'd like to know your idea for my arguments (regarding the callback) in the comment. Feel free to drop the review if you don't buy it and I will revise it. 

Thanks!

> You do not need make a new class just let nsHttpChannel implement nsIURIClassifierCallback.

I intentionally created a new class for this specific callback because of

1) nsHttpChannel has inherited a tons of interface and I don't want to make it even more polymorphic for this local usage.
2) Splitting functions and "implicitly" associating them together is burdensome to read. For example, I would avoid the following approach:

```
nsresult 
nsHttpChannel BeginConnect() 
{
  // Do something
  InitLocalBlockList();
}

nsresult 
nsHttpChannel::OnClassifyComplete()
{
  BeginConnectActual();
}

```
When we read InitLocalBlockList(), it's quite a burden to keep in mind that "InitLocalBlockList will lead to OnClassifyComplete()" and follow OnClassifyComplete() to know what will actually be done. Instead, if we:

```
nsresult 
nsHttpChannel BeginConnect() 
{
  // Do something
  InitLocalBlockList([]() -> void {
    BeginConnectActual();  
  });
}

```
When we read these two lines, we immediately have an idea that InitLocalBlockList() will do first then BeginConnectActual() will be done in a callback fashion.


(Actually we can let nsHttpChannel implement nsIURIClassifierCallback and still go for this paradigm but it requires an extra local variable in nsHttpChannel to save the callback)

I am not insisting my approach but just explaining my rationale behind. If you think the former approach is still better I am also glad to do it :

> Move this int BeginConnect. And make:
> if (!(mLoadFlags & LOAD_CLASSIFY_URI)) {
>   return BeginConnectionActual();
> } else {
>   InitLocalBlockList()
> }
> and add comment explaining each of these 2.
> 
> Probably you could return bool from InitLocalBlockLIst - succeeded or not and call BeginConnectionActual

What I will do is:

Let InitLocalBlockList() return bool to indicate if mLocalBlockList will not be initialized until callback (i.e. whether we expect callback). If callback is NOT expected, we do BeginConnectActual() immediately and synchronously without handling the failure. Otherwise, we call BeginConnectActual in the callback and handle the failure on our own. (since BeginConnect() will return NS_OK first in this case.)

> you can just remove BeginConnectWouldSucceed and move its content to BeginConnect. We have so many somethingsomethingBeginConnectionsomethingsomething :) (I am to blame for some of them)

Couldn't agree more :) I will just move back BeginConnectWouldSucceed back to BeginConnect().
Attachment #8843890 - Flags: review?(mcmanus) → review?(dd.mozilla)
Comment on attachment 8843890 [details]
Bug 1325054 - Defer any possible connection establishment in BeginConnect until knowing if it's a tracker.

https://reviewboard.mozilla.org/r/117512/#review121030

> I intentionally created a new class for this specific callback because of
> 
> 1) nsHttpChannel has inherited a tons of interface and I don't want to make it even more polymorphic for this local usage.
> 2) Splitting functions and "implicitly" associating them together is burdensome to read. For example, I would avoid the following approach:
> 
> ```
> nsresult 
> nsHttpChannel BeginConnect() 
> {
>   // Do something
>   InitLocalBlockList();
> }
> 
> nsresult 
> nsHttpChannel::OnClassifyComplete()
> {
>   BeginConnectActual();
> }
> 
> ```
> When we read InitLocalBlockList(), it's quite a burden to keep in mind that "InitLocalBlockList will lead to OnClassifyComplete()" and follow OnClassifyComplete() to know what will actually be done. Instead, if we:
> 
> ```
> nsresult 
> nsHttpChannel BeginConnect() 
> {
>   // Do something
>   InitLocalBlockList([]() -> void {
>     BeginConnectActual();  
>   });
> }
> 
> ```
> When we read these two lines, we immediately have an idea that InitLocalBlockList() will do first then BeginConnectActual() will be done in a callback fashion.
> 
> 
> (Actually we can let nsHttpChannel implement nsIURIClassifierCallback and still go for this paradigm but it requires an extra local variable in nsHttpChannel to save the callback)
> 
> I am not insisting my approach but just explaining my rationale behind. If you think the former approach is still better I am also glad to do it :

I am not worried about adding one more interface.
ok, we can do it your way; this is a side code path. The runnable will hold reference to the nsHttpChannel so we are safe on that side.
Comment on attachment 8843890 [details]
Bug 1325054 - Defer any possible connection establishment in BeginConnect until knowing if it's a tracker.

https://reviewboard.mozilla.org/r/117512/#review121626

r=me with added the check whether the channel has been canceled. it is important.

::: netwerk/protocol/http/nsHttpChannel.h:294
(Diff revision 4)
>  
>  private:
>      typedef nsresult (nsHttpChannel::*nsContinueRedirectionFunc)(nsresult result);
>  
>      bool     RequestIsConditional();
> +    // Connection will only be established in this function.

Please extend this commment saying in which case this will be postponed, i.g. if we are waiting for initialization of the local black list.

Also add that dns look up is postponed as well.

::: netwerk/protocol/http/nsHttpChannel.cpp:6083
(Diff revision 4)
>  
>      if (!(mLoadFlags & LOAD_CLASSIFY_URI)) {
>          return ContinueBeginConnectWithResult();
>      }
>  
> +    RefPtr<nsHttpChannel> self = this;

Please add comment explaining what are you doing here. Thanks!

::: netwerk/protocol/http/nsHttpChannel.cpp:6106
(Diff revision 4)
> +    return NS_OK;
> +}
> +
> +nsresult
> +nsHttpChannel::BeginConnectActual()
> +{

please check if the channel has been canceled.
Attachment #8843890 - Flags: review?(dd.mozilla) → review+
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 97cef9da75bf -d 95c1e899eab6: rebasing 381891:97cef9da75bf "Bug 1325054 - Defer any possible connection establishment in BeginConnect until knowing if it's a tracker. r=dragana" (tip)
merging netwerk/protocol/http/nsHttpChannel.cpp
merging netwerk/protocol/http/nsHttpChannel.h
warning: conflicts while merging netwerk/protocol/http/nsHttpChannel.cpp! (edit, then use 'hg resolve --mark')
warning: conflicts while merging netwerk/protocol/http/nsHttpChannel.h! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
Pushed by hchang@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/af4ad0346213
Defer any possible connection establishment in BeginConnect until knowing if it's a tracker. r=dragana
sorry had to back this out for mochitest test failures in test_classify_ping.html in Android like
https://treeherder.mozilla.org/logviewer.html#?job_id=84198632&repo=autoland&lineNumber=2079

https://hg.mozilla.org/integration/autoland/rev/567080bf24b8
Flags: needinfo?(hchang)
Depends on: 1348626
Flags: needinfo?(hchang)
Pushed by hchang@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8ada08c1c886
Defer any possible connection establishment in BeginConnect until knowing if it's a tracker. r=dragana
https://hg.mozilla.org/mozilla-central/rev/8ada08c1c886
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Component: General → Networking: HTTP
Keywords: hang
Performance Impact: --- → P1
Whiteboard: [qf:p1]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: