Closed Bug 139802 Opened 22 years ago Closed 22 years ago

cert auth takes minutes

Categories

(Core :: DOM: UI Events & Focus Handling, defect, P1)

PowerPC
macOS
defect

Tracking

()

VERIFIED FIXED
mozilla1.1beta

People

(Reporter: jgmyers, Assigned: sfraser_bugs)

References

()

Details

(Keywords: regression, topembed+, Whiteboard: [adt2 RTM] [ETA 07/11])

Attachments

(3 files, 2 obsolete files)

Since about 0.9.9, certificate authentication is unusably slow on Mac OS X.

I timed connecting to https://aka.mcom.com and it takes 2 minutes 6 seconds
between the certificate expiration dialog and the cert picker dialog.

SSL/IMAP to tintin.mcom.com takes 2 minutes 49 seconds to bring up the cert
picker dialog.  During this time the throbber is moving but the progress meter
is frozen.  Browser windows do not accept new http content, so the netwerk
thread is probably blocked.

Bringing up the account manager and clicking on the "Select" button for S/MIME
certs takes 11 seconds before a cert picker dialog box picks up.  During this
time Mac OS X puts up the colored spinning disk cursor.
How large is your db?
cc relyea
cc nelsonb
cc javi
Priority: -- → P1
Target Milestone: --- → 2.3
Certificates7 is 338KB and Key Database3 is 16KB.

I exported my cert, moved the databases out of the way, and reimported my cert.
 That changed the times to 30 seconds for IMAP/SSL, 30 sec for HTTPS, and 3 sec
for the S/MIME cert picker.  That is still too slow.
I'm seeing this as well, just using the OS X nightly builds from
ftp.mozilla.org.  nominating for mozilla1.0 and nsbeta1.
Keywords: mozilla1.0, nsbeta1
Going to http://pki.mcom.com/tests.html and clicking the "Client Auth" link
hangs my build indefinitely (spinning beach ball).  Upping severity to blocker
(this is indeed blocking us verifying that there are no regressions from
updating Mozilla to use NSS_TIP_20020426.
Severity: normal → blocker
Brendan, do you think this should block RC2?
I still get the same performance problems on the 2002060103 trunk build.
Confirming on 6/3 Mac OSX trunk. OSX ssl performance has really slowed. You can 
also try this in-house test page. It used to take less than 2 seconds, now it is 
33 seconds.
Version: unspecified → 2.3
John,  In your original description you mentioned the elapsed time 
"between the certificate expiration dialog and the cert picker dialog."

Q: What certificate expiration dialog?  Is the AKA cert expired? 
(It's not when I visit it.)
At the time I reported this bug, AKA's certificate had expired.

*** This bug has been marked as a duplicate of 115368 ***
Status: NEW → RESOLVED
Closed: 22 years ago
Resolution: --- → DUPLICATE
I suggest this bug be reopened.  
This bug reports a problem that is specific to SSL/TLS client authentication.

I would ask the submittor, does making a signed S/MIME email message also 
take a comparable amount of time to the SSL/TLS handshake with client auth?

If so, then I would first suspect that the problem is that the MPI bignum 
library (used to perform the modular exponentiations for the RSA private key 
operation, common to S/MIME signatures and SSL client authentication) has 
been compiled with the wrong -D defines for the MAC's CPU, and hence has 
horribly suboptimal performance.  

IINM, the project file(s) used to build NSS on the MAC as part of the 
Mozilla build are not supplied by the NSS team, but rather are supplied 
by the Mozilla team.  I'd suggest that someone look into them to see if 
there were any changes to the project files or to the system include files
(e.g. <ctypes.h> <limits.h> <Types.h> and others) at about the time when
the performance degradation was first observed.

I'm happy to help some MAC developer investigate this problem.
Reopening
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Blocks: 115368
Another thought about this.  Mozilla builds NSS from the NSS_CLIENT_TAG,
right?  That tag has been moved sometime since the first of the year.
Can we determine if the performance degradation correlates to a movement
of the tag?  
Reassigning, per ssaux.
Assignee: ssaux → javi
Status: REOPENED → NEW
Please note that comment #2 suggests a performance problem around dbm.

I will do some more measurements tonight.
338 KB is peanuts.  
My cert7.db is 5304 KB, and client auth takes me a fraction of a second
on my Win2K box.
OTOH, If the elapsed time is before the cert picker, then it probably isn't
the private key op after all. :-(
This bug does not appear in 0.9.9 but does appear in 1.0RC1
*** Bug 115368 has been marked as a duplicate of this bug. ***
Per John's comment #18, I generated the differences between
the NSS and DBM in mozilla 0.9.9 and 1.0 RC1.  Note that there
are no changes to DBM and not that many changes to NSS.

The NSS in mozilla 0.9.9 is some beta release of NSS 3.4.
The NSS in mozilla 1.0 RC1 is very close to NSS 3.4 RTM plus
some bug fixes from NSS 3.4.1.
Did NSPR change between those releases?
In a Mozilla 1.0 build, I tried replacing each of NSPR20.shlb, NSS3.shlb,
NSSckbi.shlb, NSSStdLib.shlb, SSL3.shlb, Softoken3.shlb, and PIPPKI.shlb with
the version in 0.9.9.  Each library was done one at a time, with the original
version restored before trying the next library.  None of these significantly
reduced the timing on the test case.  I was not able to test replacing
SMIME3.shlb, PIPBOOT.shlb, or PIPNSS.shlb, as doing so caused Mozilla to not
function properly.
Were there any significant changes in necko during this time period? A hit in
SSL performance has been caused by necko changes in the past.
junruh: does the problem for this test case appear in the same builds as for bug
147979?

The slow client auth problem appears with the 4/29 branch and the 4/19 trunk, 
before general SSL download slowness appears. I'll test older builds to see when 
this slow client auth on Mac problem started happening.
The slowness appears sometime between 3/11 and 4/17, when there was a change 
from PSM 2.2 to PSM 2.3.
This build does not have a client auth problem
ftp://sweetlou/products/client/seamonkey/windows/32bit/x86/M099/
this build does
ftp://sweetlou/products/client/seamonkey/windows/32bit/x86/M100rc1/
regression
Keywords: regression
Javi,
can you take a look at what changes came up during that time frame.
marking adt2 rtm.
Keywords: nsbeta1nsbeta1+
Whiteboard: [adt2 RTM]
what are the chances we'll have a patch ready to be landed before the weeks out?
Whiteboard: [adt2 RTM] → [adt2 RTM] [ETA Needed]
topembed+ per EDT.
Keywords: topembedtopembed+
I did some logging in my debug build and found the following:

I ran SSL client auth twice, once with cert selection set to manual and another
set to select automatically.

With Select Automatically the time between calling connect on the SSL socket and
the final read was 1 minute 5 seconds.  We spent 36 seconds in the call to
CERT_FindCertsByUsage.  

With manual cert select the time between calling connect on the SSL socket and
the final read was 1 minute 12 seconds.  10 seconds were spent in asking the
user to select a cert (this includes bringing up the dialog and getting the
response).  In this case we spent 35 seconds in CERT_FindCertsByUsage.

At first glance it appears that CERT_FindCertsByUsage is causing the problems
and needs to be investigated to figure out why traversing the database looking
for user certs is taking so long on the Mac.
Upon closer examination, the time to find the private key associated with the
cert that's been chosen to present to the server also takes a long time.  46
seconds in the log I've attached below which is a select automatically log.
*** Bug 151400 has been marked as a duplicate of this bug. ***
Any progress on this?  Under Mac OS X, Mozilla is the only browser I have found
that support client cert authentication, so this is particularly important to
people who can only run OS X.
Does this performance issue also happen on Mac OS 9? What about Chimera (a MachO
buld that uses the Unix NSPR)? Is it worse on dual CPU Mac OS X machines?
Could someone who has access to a mac and knows how the profiling tools work run
the profiler on the mac and post the output.

I have spent the last several days making improvements to the
CERT_GetUserCertByUsage() (and underlying code), but most of the changes would
only affect large databases. I could dump all those patched into this bug and
hope that they have some positive effect, but the risk of some regression after
picking up such a large patch would be pretty high. If I knew exactly what
elements of the mac are really slow I could be more judicious in the patches I
propose.

bob
Chimera doesn't seem to have a fully implemented PSM so I can't check whether
the full client auth takes as long, but when it throws an error it happens in
fairly short order.
Perhaps the problem here is caused by an optimization that we did to reduce CPU
usage. Are any PLEvents flying around when doing cert auth? If not, I think
we're cranking down CPU usage (by calling WaitNextEvent with a big sleep time)
to the point where we're throttling NSPR threads. This would be bad.

The reason I think this is that Mozilla is not using lots of CPU during client
auth, which it would if the delay were compute-bound.
http://www.hilander.com/certauthsampler.txt

This is output from Apple's Sampler tool that comes with the dev kit.  I started
sampling every 20ms when I started the SSL session, and stopped once the page
successfully loaded.  The entire load took 45 seconds.  If I should be getting
profiling data differently let me know.
This profile shows that almost all the time is spent in WaitNextEvent or
EventAvail, which supports my theory that we're spending too much time yielding
to  other apps, and not enough time servicing NSPR threads. Someone should try
hacking nsMacMessagePump::BrowserIsBusy() to always return true, and see how
that affects cert auth performance.
I think that bug 105445 may contain the changes that caused this. This was
checked in on March 27.
OK. Need some way to deal with this without erasing the gain of 105445 for
embedding apps.
Backing out 105445 caused a noticeable improvement in performance.  I also see a
reduction in the amount of time required to traverse the database for user
certs.  Previous runs showed ~35 seconds to traverse my cert database, with the
patch backed out I now see a time of 8 seconds to traverse my cert db.

So it appears that the patch is contributing to the slow down.
I think "contributing" might be a little light on the language ... "solely
responsible for" might be more apt.

In any case, looks like we've found the culprit ... now we just need to strike a
compromise.
Even 8 seconds seems like an awfully long time. :-(
OK. I may have to make an IsBrowserBusy() routine available in the embedding API
in order to get back to where we were. That would require modification of the
embedding app's event pump which is what bug 105445 was meant to avoid.

Unless - is there something like a null PLEvent that this code could fire? How
is this code yielding to other apps while doing its work? 
8 seconds is indeed long, but it is a few orders of magnitude faster than the
current builds. We've at least found a major source of the slow-down, let's work
that first and then see where else improvements can be made. Maybe the
code-cleanup mentioned above will trim some more off that 8s.
Javi, can you show me where the affected code is? Another solution may be to add
calls to WakeUpProcess() to it. This would be better than reverting to an
IsBrowserBusy() in the message pump approach.
A fair amount of time is spent in
http://lxr.mozilla.org/seamonkey/ident?i=CERT_FindUserCertByUsage
but I don't see how you could expect NSS to sprinkle WakeUpProcess() throughout
its code.  The fundamental issue is that when one is using user-mode threads,
one cannot make long-blocking calls that the thread implementation doesn't know
about.  Otherwise, I/O-bound threads such as those calling NSS get starved
because the user-mode thread scheduler rarely gets a chance to schedule them.

The method I was testing was
http://lxr.mozilla.org/seamonkey/ident?i=nsNSS_SSLGetClientAuthData, but we're
only spending upwards of 45 seconds to 1 minute in this method in the worst
case.  The slow down is by a couple of minutes more than that.  So even if we
add calls to this method, I'm not sure we'll see a marked improvement.
Can somone summarize what NSPR threads are doing what work during the problem
period?
Our code gets called by necko.  We don't create any new threads or do any thread
management.  So I'm guessing whatever necko does in regards to threads is the
thread activity going on while this code is called.
Which necko protocols are active? Just file://, or also network stuff?
This makes me a little surprised, because I would have thought that necko would
be posting PLEvents. Maybe our Mac code isn't looking at all the event queues
that it needs to.
I would be more than happy to help, although I don't understand enough about the
Mozilla architecture to provide this info.  If somebody could give me a little
more info on what you want to see I would love to help.
This is happening during an SSL handshake.  A packet of data arrives from the
network and necko calls into NSS to handle the network activity.  NSS reads the
data then spends a fair amount of time processing the data.  Processing the data
involves running cryptographic algorithms, reading data from disk, and sometimes
throwing up dialog boxes.  During this processing, there is no network activity.
Solutions I can see are:

1) Add a Mac-only PR_WaitNextEvent() which permits NSPR to intercept blocking
WaitNextEvent calls and schedule other threads.

2) Add a call to NSPR which allows nsMacMessagePump::GetEvent() to know whether
or not there are other threads that want to run.
In response to comments 48 and 49,  and the idea of inserting new calls
into the relevant code...

The code in question is part of NSS.  NSS is not the exclusive property 
of Mozilla.  In fact, it is used in many other products, so inserting 
mozilla browser-specific calls into NSS is out of the question.

The code in question does not make any calls to yield the CPU because 
the functions in question are not expected to ever take more than a fraction
of a second.  That is why I commented that 8 seconds for that code seems
like an unusually long time.  It seems that the particular function for
searching for NSS certs has gotten an order of magnitude slower recently.
Rather than inserting some sort of CPU yielding function calls into it,
I think we should try to get that order of magnitude of performance back.
> 1) Add a Mac-only PR_WaitNextEvent() which permits NSPR to intercept blocking
> WaitNextEvent calls and schedule other threads.

It's not possible to schedule other threads while we're down in the
WaitNextEvent call. Mac OS 9 is a cooperatively scheduled OS, and NSPR threads
on OS 9 are themselves cooperatively scheduled. We can only be running NSPR
threads when our app has the CPU.
In reply to comment 59, the proposal is to have NSPR intercept the call,
scheduling the other threads around any native WaitNextEvent call.  Something like:

PR_WaitNextEvent(int timeout)
{
 
   WaitNextEvent(0);
   while (haven't got event && not timed out) {
        while (another thread wants CPU) {
              yield;
              WaitNextEvent(0);
        }
        WaitNextEvent(MIN(1 sec, timeout));
   }
}
        

    
Javi is not the right person to own this bug.
Can the solution described by John be coded and checked in on the trunk?

We need a mac expert to take ownership.
In looking for a solution to this problem, I appear to have blundered into the
main event.

I am not a Mac expert, but I can at least respond to Comment #36:

Access to cert-protected page from MacOS 9.1 (Mac Moz 1.1a), from MacOS Classic
under MacOS X (Mac Moz 1.1a), and from MacOS X (MacOS X Moz 1.1a).

OS 9.1 (Mac Moz1.1a)-time to password challenge: 1-2 sec; time to page: 1-2 sec
Classic(Mac Moz1.1a)-time to password challenge: 1-2 sec; time to page: 1-2 sec
OS X  (OS X Moz1.1a)-time to password challenge: 25-30 sec; time to page:>10min

For the OS X case, it seems that if I cancel the page access after providing the
password, and then access the protected page again (cert already successfully
validated), the page will appear in about 20-25 seconds.

I observed OS X Moz CPU usage via top: while waiting for the password challenge,
Moz CPU sat at about 10% of a reasonably idle system.  While waiting for page
(both after challenge and after page access restart) ~5%.

MacOS X version 10.1.5, single processor 400 MHz B&W Tower.  Accessing an
internal corporate page.
Attached patch First cut at a patch (obsolete) — Splinter Review
This patch solves the problem by adding an API on NSPR, Mac only, that returns
true if there are active threads in addition to the main thread. This is called
by code that decides how long to spend in WaitNextEvent.

I've tested cert auth wih this, and it solves the performance issues. The only
downside I've noticed is that the browser CPU usage bumps up every few seconds,
I think due to the memory flusher thread kicking in.

Conrad won't like this patch, because it won't work for embedding apps that
have  their own main event loop, and can't call an NSPR API.
To try to address Conrad's concerns, I tested an NSPR patch that calls
WakeUpProcess() whenever it schedules a thread that is not the main thread, or a
CPU idle thread; I thought this would have the same effect as the patch in this
bug, but it doesn't. It hardly affects the problem at all. The reason is that,
with NSPR threads being cooperatively scheduled, if your app isn't getting time
(because it's in WNE), then it isn't going to be switching between running
threads very often. WNE is really the driver here; it's in control.
Attached patch Patch version 2. (obsolete) — Splinter Review
This patch demonstrates a different approach, which is a little scarier, but
might be more to Conrad's liking. All the changes are in NSPR, in this case.

This patch puts a call to WakeUpProcess() in the Timer Task (which fires every
8ms whether or not the app has the CPU). WakeUpProcess() will be called when
we're ready for a thread switch. This patch is a little more heavyweight, since
it doesn't attempt to see what kind of threads are available to switch to (it
might just be the CPU idle thread).
could this also be the cause of annecdotal reports that IMAP header download is
much slower on macos than other platforms and 4.x? Isn't the IMAP stuff fairly
heavily threaded?
Simon, thanks for doing the 2 versions of the patch. I'll make an opt build
today and do the pageloader tests on it with both patches. Since you say that
the 1st patch, anyway, fixes the cert auth problem, what either patch does for
the general pageloader test (and CPU usage) needs to be weighed as well.

Even the 1st patch, which requires modification to the the app's event pump, is
a pretty small, clean modification so I'm not that opposed. Hopefully, we'll be
in a (Carbon Event) world soon where WNE isn't used and so calculation of sleep
time won't be needed.
I can verify that IMAP header downloads are relatively slow regardless of
enabling SSL for IMAP, on MacOS. Downloading 40 headers or so should complete in
a second or a fraction thereof, but typically takes 4-5 seconds. Especially when
communicating with a server that has practically no load.

Cheers.
Anecdotal? See bug 102797. These patches don't affect it, I think; there are
enough PLEvents flying around during header downloading.
bizarre. from the bug linked above, it seems that this (IMAP header downloads)
is a non-issue. yet I'm experiencing it ... I'll have to do some more formal
testing.

Cheers.
Moving this bug to the Browser product so that it shows up on radar.
Assignee: javi → sfraser
Component: Client Library → Event Handling
Product: PSM → Browser
Target Milestone: 2.3 → M1
Version: 2.3 → other
I did some page load tests on an optimized build with these patches. Results
(mean of 4 runs each):

Baseline          1200ms
Patch 1           1150ms
Patch 2           1070ms

so they actually help pageload time. The danger is that they'll cause mozilla to
suck more CPU. I'm still collecting data on that.
Status: NEW → ASSIGNED
Target Milestone: M1 → mozilla1.1beta
Comment on attachment 90147 [details] [diff] [review]
First cut at a patch

This patch should be avoided if possible because
it adds a Mac-only private function (PR_Mac_ThreadsBusy).
wtc: that header (pprthed.h) already contains 2 other mac-specific exported
functions (which are used for async DNS), so this isn't setting a precedent.
The first patch (attachment 90147 [details] [diff] [review]) doesn't seem to cause CPU usage to go up when
Mozilla is idle, nor when in MailNews, doing downloads etc. I like this patch
the best.
What's the CPU usage for patch #2? I still like it because of no change required
to the application.
Ooops, that was for patch 2. I didn't measure it for patch 1, because patch 2 is
better :)
Comment on attachment 90157 [details] [diff] [review]
Patch version 2.

gTimeManagerTaskDoesWUP should be declared static.  It
would be nice to have a comment explaining what this
variable means.
Attachment #90147 - Attachment is obsolete: true
Attachment #90157 - Attachment is obsolete: true
Comment on attachment 90693 [details] [diff] [review]
Patch v. 2 updated to address wtc's concerns

Simon,

It seems that the time manager task should call
the NSPR-defined function SignalIdleSemaphore
instead of WakeUpProcess, because NSPR's
_MD_PauseCPU calls WaitOnIdleSemaphore, which
doesn't call WaitNextEvent when we are running
on OS X.
Actually, the semaphore stuff is orthoganal to this particular issue. The
WaitNextEvent calls where we spend time are not the one in WaitOnIdleSemaphore()
(that never gets called on OS X), but rather in the application's main event loop.
Simon,

When the time manager task calls WakeUpProcess, is it possible
for the application to be in the NSPR function WaitOnIdleSemaphore?
Would that be a problem?
I think that could happen; we might be sitting in the semaphore (with the timer
task firing every 8ms), until a condition variable times out, at which point I
think we'd go into the '_PR_SET_RESCHED_FLAG' clause. But it should be harmless;
WakeUpProcess() is an interrupt-safe routine that just sets a flag in the OS
scheduler causing the app to get scheduled next time around.

<http://developer.apple.com/techpubs/mac/Processes/Processes-46.html>
Comment on attachment 90693 [details] [diff] [review]
Patch v. 2 updated to address wtc's concerns

r=wtc.	It would be nice if you could have a Mac developer review
this patch too.

Please check this in on the tip and NSPRPUB_PRE_4_2_CLIENT_BRANCH
of NSPR.
Attachment #90693 - Flags: review+
Comment on attachment 90693 [details] [diff] [review]
Patch v. 2 updated to address wtc's concerns

r=sdagley
Gotta love it when a patch fixes the bug, improves pageload perf and has no
additional idle overhead
Patch checked in to NSPRPUB_PRE_4_2_CLIENT_BRANCH and the NSPR tip. Keeping bug
open for possible 1.0 branch checkin.
Resolving as fixed, since the patch has been checked into the trunk. Added
adt1.0.1 for branch consideration.

Let's get this verified by john unruh tomorrow, so we can review it for
inclusion on the branch.
Blocks: 143047
Status: ASSIGNED → RESOLVED
Closed: 22 years ago22 years ago
Keywords: adt1.0.1
Resolution: --- → FIXED
Whiteboard: [adt2 RTM] [ETA Needed] → [adt2 RTM] [ETA 07/11]
Verified on 7/10 OSX and Mac 9.2.2 trunk. Client auth took around 3 seconds 
with automatic cert selection, and was very fast with manual cert selection. 
Also, the ssl speed test page took only 2 seconds - 
https://pki.mcom.com/color/color.html
Status: RESOLVED → VERIFIED
Is there anything else we should be testing besides this actual bug?
Yes. We need to check that the app doesn't use lots of CPU for no reason at any
time. So we need to run a basic test suite (browsing, mail, downloading etc)
with the CPU monitor open (or using top), keeping an eye on the CPU usage.
downloading info, when saving (using "save target link as") a 15mb file (the mac
8/9 mozilla 1.1a bits).

how    2002.07.09.13    2002.07.10.08
-------------------------------------
http       55 sec           15 sec
ftp        53 sec            8 sec


CPU usage (looking at the progress of the http download)
---------
2002.07.09.13: during the download, cpu peaked at around 25% then leveled off to
2.8% when finished.

2002.07.10.08: during the download, cpu peaked at around 58.2% then leveled off
to 1.4% when finished.
I checked POP3 and IMAP protocols with the build that exhibited the *old*
behavior (the 2002-07-09-13 trunk build running on OS X 10.1.5), as well as the
*current* trunk (2002-07-10-08).

Some basic observations using Process Viewer:

Old 9th build:
Downloading 1009 IMAP messages took 2 minute, 54 seconds and 57 milliseconds.

Current 10th build:
Downloading 1009 IMAP messages took 1 minute, 16 seconds and 85 milliseconds.

Granted this isn't an isolated mail test server, but these comparisons were done
literally within a few minutes of each other.

I didn't readily have a large pool of messages to test POP3's performance with,
but I was watching CPU usage in both cases, and found no alarming instances of
spiking.

CPU usage for most mail download operations hovered @ a 44-65% level, with a few
upward spikes, but nothing sustained.

At idle level, the app sits pretty quiet - LaunchCFM app reports anywhere from
0.3 % to somewhere near 4% (~3.7).

When running other applications concurrently with ours, we drop CPU usage and
yield to the other app I tested (as best as I can tell, since the Process ID Tab
of Process Viewer gives me a unique integer value, instead of the application's
name).  I quit IE, noticed the process ID of Mozilla, started downloading
headers, noticed CPU usage, launched IE (loading a page in IE yields 70% usage
or so), and noticed no spikes here in Mozilla, either.
in comment 91, i had been downloading from our local mozilla ftp and http
server, eg,
http://ftp.mozilla.org/pub/mozilla/releases/mozilla1.1a/mozilla-mac-11a-full.bin

simon suggested comparing these data with download from a more remote site,
where bandwidth would be more of an issue. so, i used an japanese moz mirror
site:
http://mirror.nucba.ac.jp/mirror/mozilla/mozilla/releases/mozilla1.0/mozilla-mac-10-full-instal.bin

i did three test runs for the japanese downloading for each build; here's the
summary:

* using 2002.07.09.13 trunk bits: CPU usage varied from a momentary peak of
16-45%, stabilizing to about 3-20% for most of the duration. download time
ranged from 9min 25sec to 10min 41sec.

* using 2002.07.10.08 trunk bits: CPU usage varied from a brief peak of 20-53%,
stabilizing to about 3-13% for the most of the duration. download time ranged
from as low as 5min 6sec to 10min 50sec.
Adding adt1.0.1+ on behalf of the adt pending sr=.  Please get drivers approval
and check into the Mozilla 1.0 branch asap.
Keywords: adt1.0.1adt1.0.1+
Comment on attachment 90693 [details] [diff] [review]
Patch v. 2 updated to address wtc's concerns

NSPR is exempt from Mozilla client's super-review requirement.
This patch has been reviewed by me (NSPR owner) and Steve
Dagley.
Attachment #90693 - Flags: superreview+
John can you verify that this is fixed on the trunk?
Sorry, I meant branch. I think it's fixed on the branch as well, but we should
also verify there.
Comment on attachment 90693 [details] [diff] [review]
Patch v. 2 updated to address wtc's concerns

This patch has not been checked in on the MOZILLA_1_0_BRANCH.
Comment on attachment 90693 [details] [diff] [review]
Patch v. 2 updated to address wtc's concerns

a=chofmann for 1.0.1. add  the fixed1.0.1 keyword after checking in on the
branch
Attachment #90693 - Flags: approval+
Checked in on the MOZILLA_1_0_BRANCH, added keyword. Thanks guys.
Verified on 7/16 Mac OSX branch.
Component: Event Handling → User events and focus handling
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: