Closed Bug 139802 Opened 23 years ago Closed 23 years ago

cert auth takes minutes

Categories

(Core :: DOM: UI Events & Focus Handling, defect, P1)

Product:
Core
Component:
DOM: UI Events & Focus Handling
Platform:
PowerPC
macOS
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla1.1beta

People

(Reporter: jgmyers, Assigned: sfraser_bugs)

References

(
URL
)

Details

(Keywords: regression, topembed+, Whiteboard: [adt2 RTM] [ETA 07/11])

Attachments

(3 files, 2 obsolete files)

Diffs between the NSS and DBM in mozilla 0.9.9 and 1.0 RC1
63.96 KB, patch
Details | Diff | Splinter Review
The log file I described.
1.75 KB, text/plain
Details
Patch v. 2 updated to address wtc's concerns
3.80 KB, patch
wtc
: review+
wtc
: superreview+
chofmann
: approval+
Details | Diff | Splinter Review
Since about 0.9.9, certificate authentication is unusably slow on Mac OS X. I timed connecting to https://aka.mcom.com and it takes 2 minutes 6 seconds between the certificate expiration dialog and the cert picker dialog. SSL/IMAP to tintin.mcom.com takes 2 minutes 49 seconds to bring up the cert picker dialog. During this time the throbber is moving but the progress meter is frozen. Browser windows do not accept new http content, so the netwerk thread is probably blocked. Bringing up the account manager and clicking on the "Select" button for S/MIME certs takes 11 seconds before a cert picker dialog box picks up. During this time Mac OS X puts up the colored spinning disk cursor.
How large is your db? cc relyea cc nelsonb cc javi
Priority: -- → P1
Target Milestone: --- → 2.3
Certificates7 is 338KB and Key Database3 is 16KB. I exported my cert, moved the databases out of the way, and reimported my cert. That changed the times to 30 seconds for IMAP/SSL, 30 sec for HTTPS, and 3 sec for the S/MIME cert picker. That is still too slow.
I'm seeing this as well, just using the OS X nightly builds from ftp.mozilla.org. nominating for mozilla1.0 and nsbeta1.
Keywords: mozilla1.0, nsbeta1
Going to http://pki.mcom.com/tests.html and clicking the "Client Auth" link hangs my build indefinitely (spinning beach ball). Upping severity to blocker (this is indeed blocking us verifying that there are no regressions from updating Mozilla to use NSS_TIP_20020426.
Severity: normal → blocker
Brendan, do you think this should block RC2?
I still get the same performance problems on the 2002060103 trunk build.
Confirming on 6/3 Mac OSX trunk. OSX ssl performance has really slowed. You can also try this in-house test page. It used to take less than 2 seconds, now it is 33 seconds.
URL: http://pki.mcom.com/tests.html
Version: unspecified → 2.3
John, In your original description you mentioned the elapsed time "between the certificate expiration dialog and the cert picker dialog." Q: What certificate expiration dialog? Is the AKA cert expired? (It's not when I visit it.)
At the time I reported this bug, AKA's certificate had expired.
*** This bug has been marked as a duplicate of 115368 ***
Status: NEW → RESOLVED
Closed: 23 years ago
Resolution: --- → DUPLICATE
I suggest this bug be reopened. This bug reports a problem that is specific to SSL/TLS client authentication. I would ask the submittor, does making a signed S/MIME email message also take a comparable amount of time to the SSL/TLS handshake with client auth? If so, then I would first suspect that the problem is that the MPI bignum library (used to perform the modular exponentiations for the RSA private key operation, common to S/MIME signatures and SSL client authentication) has been compiled with the wrong -D defines for the MAC's CPU, and hence has horribly suboptimal performance. IINM, the project file(s) used to build NSS on the MAC as part of the Mozilla build are not supplied by the NSS team, but rather are supplied by the Mozilla team. I'd suggest that someone look into them to see if there were any changes to the project files or to the system include files (e.g. <ctypes.h> <limits.h> <Types.h> and others) at about the time when the performance degradation was first observed. I'm happy to help some MAC developer investigate this problem.
Reopening
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Blocks: 115368
Another thought about this. Mozilla builds NSS from the NSS_CLIENT_TAG, right? That tag has been moved sometime since the first of the year. Can we determine if the performance degradation correlates to a movement of the tag?
Reassigning, per ssaux.
Assignee: ssaux → javi
Status: REOPENED → NEW
Please note that comment #2 suggests a performance problem around dbm. I will do some more measurements tonight.
338 KB is peanuts. My cert7.db is 5304 KB, and client auth takes me a fraction of a second on my Win2K box.
OTOH, If the elapsed time is before the cert picker, then it probably isn't the private key op after all. :-(
This bug does not appear in 0.9.9 but does appear in 1.0RC1
*** Bug 115368 has been marked as a duplicate of this bug. ***
Per John's comment #18, I generated the differences between the NSS and DBM in mozilla 0.9.9 and 1.0 RC1. Note that there are no changes to DBM and not that many changes to NSS. The NSS in mozilla 0.9.9 is some beta release of NSS 3.4. The NSS in mozilla 1.0 RC1 is very close to NSS 3.4 RTM plus some bug fixes from NSS 3.4.1.
Did NSPR change between those releases?
In a Mozilla 1.0 build, I tried replacing each of NSPR20.shlb, NSS3.shlb, NSSckbi.shlb, NSSStdLib.shlb, SSL3.shlb, Softoken3.shlb, and PIPPKI.shlb with the version in 0.9.9. Each library was done one at a time, with the original version restored before trying the next library. None of these significantly reduced the timing on the test case. I was not able to test replacing SMIME3.shlb, PIPBOOT.shlb, or PIPNSS.shlb, as doing so caused Mozilla to not function properly.
Were there any significant changes in necko during this time period? A hit in SSL performance has been caused by necko changes in the past.
junruh: does the problem for this test case appear in the same builds as for bug 147979?
The slow client auth problem appears with the 4/29 branch and the 4/19 trunk, before general SSL download slowness appears. I'll test older builds to see when this slow client auth on Mac problem started happening.
The slowness appears sometime between 3/11 and 4/17, when there was a change from PSM 2.2 to PSM 2.3. This build does not have a client auth problem ftp://sweetlou/products/client/seamonkey/windows/32bit/x86/M099/ this build does ftp://sweetlou/products/client/seamonkey/windows/32bit/x86/M100rc1/
regression
Keywords: regression
Javi, can you take a look at what changes came up during that time frame.
marking adt2 rtm.
Keywords: nsbeta1 → nsbeta1+
Whiteboard: [adt2 RTM]
what are the chances we'll have a patch ready to be landed before the weeks out?
Keywords: mozilla1.0 → mozilla1.0.1, topembed
Whiteboard: [adt2 RTM] → [adt2 RTM] [ETA Needed]
topembed+ per EDT.
Keywords: topembed → topembed+
I did some logging in my debug build and found the following: I ran SSL client auth twice, once with cert selection set to manual and another set to select automatically. With Select Automatically the time between calling connect on the SSL socket and the final read was 1 minute 5 seconds. We spent 36 seconds in the call to CERT_FindCertsByUsage. With manual cert select the time between calling connect on the SSL socket and the final read was 1 minute 12 seconds. 10 seconds were spent in asking the user to select a cert (this includes bringing up the dialog and getting the response). In this case we spent 35 seconds in CERT_FindCertsByUsage. At first glance it appears that CERT_FindCertsByUsage is causing the problems and needs to be investigated to figure out why traversing the database looking for user certs is taking so long on the Mac.
Upon closer examination, the time to find the private key associated with the cert that's been chosen to present to the server also takes a long time. 46 seconds in the log I've attached below which is a select automatically log.
*** Bug 151400 has been marked as a duplicate of this bug. ***
Any progress on this? Under Mac OS X, Mozilla is the only browser I have found that support client cert authentication, so this is particularly important to people who can only run OS X.
Does this performance issue also happen on Mac OS 9? What about Chimera (a MachO buld that uses the Unix NSPR)? Is it worse on dual CPU Mac OS X machines?
Could someone who has access to a mac and knows how the profiling tools work run the profiler on the mac and post the output. I have spent the last several days making improvements to the CERT_GetUserCertByUsage() (and underlying code), but most of the changes would only affect large databases. I could dump all those patched into this bug and hope that they have some positive effect, but the risk of some regression after picking up such a large patch would be pretty high. If I knew exactly what elements of the mac are really slow I could be more judicious in the patches I propose. bob
Chimera doesn't seem to have a fully implemented PSM so I can't check whether the full client auth takes as long, but when it throws an error it happens in fairly short order.
Perhaps the problem here is caused by an optimization that we did to reduce CPU usage. Are any PLEvents flying around when doing cert auth? If not, I think we're cranking down CPU usage (by calling WaitNextEvent with a big sleep time) to the point where we're throttling NSPR threads. This would be bad. The reason I think this is that Mozilla is not using lots of CPU during client auth, which it would if the delay were compute-bound.
http://www.hilander.com/certauthsampler.txt This is output from Apple's Sampler tool that comes with the dev kit. I started sampling every 20ms when I started the SSL session, and stopped once the page successfully loaded. The entire load took 45 seconds. If I should be getting profiling data differently let me know.
This profile shows that almost all the time is spent in WaitNextEvent or EventAvail, which supports my theory that we're spending too much time yielding to other apps, and not enough time servicing NSPR threads. Someone should try hacking nsMacMessagePump::BrowserIsBusy() to always return true, and see how that affects cert auth performance.
I think that bug 105445 may contain the changes that caused this. This was checked in on March 27.
OK. Need some way to deal with this without erasing the gain of 105445 for embedding apps.
Backing out 105445 caused a noticeable improvement in performance. I also see a reduction in the amount of time required to traverse the database for user certs. Previous runs showed ~35 seconds to traverse my cert database, with the patch backed out I now see a time of 8 seconds to traverse my cert db. So it appears that the patch is contributing to the slow down.
I think "contributing" might be a little light on the language ... "solely responsible for" might be more apt. In any case, looks like we've found the culprit ... now we just need to strike a compromise.
Even 8 seconds seems like an awfully long time. :-(
OK. I may have to make an IsBrowserBusy() routine available in the embedding API in order to get back to where we were. That would require modification of the embedding app's event pump which is what bug 105445 was meant to avoid. Unless - is there something like a null PLEvent that this code could fire? How is this code yielding to other apps while doing its work?
8 seconds is indeed long, but it is a few orders of magnitude faster than the current builds. We've at least found a major source of the slow-down, let's work that first and then see where else improvements can be made. Maybe the code-cleanup mentioned above will trim some more off that 8s.
Javi, can you show me where the affected code is? Another solution may be to add calls to WakeUpProcess() to it. This would be better than reverting to an IsBrowserBusy() in the message pump approach.
A fair amount of time is spent in http://lxr.mozilla.org/seamonkey/ident?i=CERT_FindUserCertByUsage but I don't see how you could expect NSS to sprinkle WakeUpProcess() throughout its code. The fundamental issue is that when one is using user-mode threads, one cannot make long-blocking calls that the thread implementation doesn't know about. Otherwise, I/O-bound threads such as those calling NSS get starved because the user-mode thread scheduler rarely gets a chance to schedule them.
The method I was testing was http://lxr.mozilla.org/seamonkey/ident?i=nsNSS_SSLGetClientAuthData, but we're only spending upwards of 45 seconds to 1 minute in this method in the worst case. The slow down is by a couple of minutes more than that. So even if we add calls to this method, I'm not sure we'll see a marked improvement.
Can somone summarize what NSPR threads are doing what work during the problem period?
Our code gets called by necko. We don't create any new threads or do any thread management. So I'm guessing whatever necko does in regards to threads is the thread activity going on while this code is called.
Which necko protocols are active? Just file://, or also network stuff? This makes me a little surprised, because I would have thought that necko would be posting PLEvents. Maybe our Mac code isn't looking at all the event queues that it needs to.
I would be more than happy to help, although I don't understand enough about the Mozilla architecture to provide this info. If somebody could give me a little more info on what you want to see I would love to help.
This is happening during an SSL handshake. A packet of data arrives from the network and necko calls into NSS to handle the network activity. NSS reads the data then spends a fair amount of time processing the data. Processing the data involves running cryptographic algorithms, reading data from disk, and sometimes throwing up dialog boxes. During this processing, there is no network activity.
Solutions I can see are: 1) Add a Mac-only PR_WaitNextEvent() which permits NSPR to intercept blocking WaitNextEvent calls and schedule other threads. 2) Add a call to NSPR which allows nsMacMessagePump::GetEvent() to know whether or not there are other threads that want to run.
In response to comments 48 and 49, and the idea of inserting new calls into the relevant code... The code in question is part of NSS. NSS is not the exclusive property of Mozilla. In fact, it is used in many other products, so inserting mozilla browser-specific calls into NSS is out of the question. The code in question does not make any calls to yield the CPU because the functions in question are not expected to ever take more than a fraction of a second. That is why I commented that 8 seconds for that code seems like an unusually long time. It seems that the particular function for searching for NSS certs has gotten an order of magnitude slower recently. Rather than inserting some sort of CPU yielding function calls into it, I think we should try to get that order of magnitude of performance back.
> 1) Add a Mac-only PR_WaitNextEvent() which permits NSPR to intercept blocking > WaitNextEvent calls and schedule other threads. It's not possible to schedule other threads while we're down in the WaitNextEvent call. Mac OS 9 is a cooperatively scheduled OS, and NSPR threads on OS 9 are themselves cooperatively scheduled. We can only be running NSPR threads when our app has the CPU.
In reply to comment 59, the proposal is to have NSPR intercept the call, scheduling the other threads around any native WaitNextEvent call. Something like: PR_WaitNextEvent(int timeout) { WaitNextEvent(0); while (haven't got event && not timed out) { while (another thread wants CPU) { yield; WaitNextEvent(0); } WaitNextEvent(MIN(1 sec, timeout)); } }
Javi is not the right person to own this bug. Can the solution described by John be coded and checked in on the trunk? We need a mac expert to take ownership.
In looking for a solution to this problem, I appear to have blundered into the main event. I am not a Mac expert, but I can at least respond to Comment #36: Access to cert-protected page from MacOS 9.1 (Mac Moz 1.1a), from MacOS Classic under MacOS X (Mac Moz 1.1a), and from MacOS X (MacOS X Moz 1.1a). OS 9.1 (Mac Moz1.1a)-time to password challenge: 1-2 sec; time to page: 1-2 sec Classic(Mac Moz1.1a)-time to password challenge: 1-2 sec; time to page: 1-2 sec OS X (OS X Moz1.1a)-time to password challenge: 25-30 sec; time to page:>10min For the OS X case, it seems that if I cancel the page access after providing the password, and then access the protected page again (cert already successfully validated), the page will appear in about 20-25 seconds. I observed OS X Moz CPU usage via top: while waiting for the password challenge, Moz CPU sat at about 10% of a reasonably idle system. While waiting for page (both after challenge and after page access restart) ~5%. MacOS X version 10.1.5, single processor 400 MHz B&W Tower. Accessing an internal corporate page.
Attached patch First cut at a patch (obsolete) — — Splinter Review
This patch solves the problem by adding an API on NSPR, Mac only, that returns true if there are active threads in addition to the main thread. This is called by code that decides how long to spend in WaitNextEvent. I've tested cert auth wih this, and it solves the performance issues. The only downside I've noticed is that the browser CPU usage bumps up every few seconds, I think due to the memory flusher thread kicking in. Conrad won't like this patch, because it won't work for embedding apps that have their own main event loop, and can't call an NSPR API.
To try to address Conrad's concerns, I tested an NSPR patch that calls WakeUpProcess() whenever it schedules a thread that is not the main thread, or a CPU idle thread; I thought this would have the same effect as the patch in this bug, but it doesn't. It hardly affects the problem at all. The reason is that, with NSPR threads being cooperatively scheduled, if your app isn't getting time (because it's in WNE), then it isn't going to be switching between running threads very often. WNE is really the driver here; it's in control.
Attached patch Patch version 2. (obsolete) — — Splinter Review
This patch demonstrates a different approach, which is a little scarier, but might be more to Conrad's liking. All the changes are in NSPR, in this case. This patch puts a call to WakeUpProcess() in the Timer Task (which fires every 8ms whether or not the app has the CPU). WakeUpProcess() will be called when we're ready for a thread switch. This patch is a little more heavyweight, since it doesn't attempt to see what kind of threads are available to switch to (it might just be the CPU idle thread).
could this also be the cause of annecdotal reports that IMAP header download is much slower on macos than other platforms and 4.x? Isn't the IMAP stuff fairly heavily threaded?
Simon, thanks for doing the 2 versions of the patch. I'll make an opt build today and do the pageloader tests on it with both patches. Since you say that the 1st patch, anyway, fixes the cert auth problem, what either patch does for the general pageloader test (and CPU usage) needs to be weighed as well. Even the 1st patch, which requires modification to the the app's event pump, is a pretty small, clean modification so I'm not that opposed. Hopefully, we'll be in a (Carbon Event) world soon where WNE isn't used and so calculation of sleep time won't be needed.
I can verify that IMAP header downloads are relatively slow regardless of enabling SSL for IMAP, on MacOS. Downloading 40 headers or so should complete in a second or a fraction thereof, but typically takes 4-5 seconds. Especially when communicating with a server that has practically no load. Cheers.
Anecdotal? See bug 102797. These patches don't affect it, I think; there are enough PLEvents flying around during header downloading.
bizarre. from the bug linked above, it seems that this (IMAP header downloads) is a non-issue. yet I'm experiencing it ... I'll have to do some more formal testing. Cheers.
Moving this bug to the Browser product so that it shows up on radar.
Assignee: javi → sfraser
Component: Client Library → Event Handling
Product: PSM → Browser
Target Milestone: 2.3 → M1
Version: 2.3 → other
I did some page load tests on an optimized build with these patches. Results (mean of 4 runs each): Baseline 1200ms Patch 1 1150ms Patch 2 1070ms so they actually help pageload time. The danger is that they'll cause mozilla to suck more CPU. I'm still collecting data on that.
Status: NEW → ASSIGNED
Target Milestone: M1 → mozilla1.1beta
Comment on attachment 90147 [details] [diff] [review] First cut at a patch This patch should be avoided if possible because it adds a Mac-only private function (PR_Mac_ThreadsBusy).
wtc: that header (pprthed.h) already contains 2 other mac-specific exported functions (which are used for async DNS), so this isn't setting a precedent.
The first patch (attachment 90147 [details] [diff] [review]) doesn't seem to cause CPU usage to go up when Mozilla is idle, nor when in MailNews, doing downloads etc. I like this patch the best.
What's the CPU usage for patch #2? I still like it because of no change required to the application.
Ooops, that was for patch 2. I didn't measure it for patch 1, because patch 2 is better :)
Comment on attachment 90157 [details] [diff] [review] Patch version 2. gTimeManagerTaskDoesWUP should be declared static. It would be nice to have a comment explaining what this variable means.
Attachment #90147 - Attachment is obsolete: true
Attachment #90157 - Attachment is obsolete: true
Comment on attachment 90693 [details] [diff] [review] Patch v. 2 updated to address wtc's concerns Simon, It seems that the time manager task should call the NSPR-defined function SignalIdleSemaphore instead of WakeUpProcess, because NSPR's _MD_PauseCPU calls WaitOnIdleSemaphore, which doesn't call WaitNextEvent when we are running on OS X.
Actually, the semaphore stuff is orthoganal to this particular issue. The WaitNextEvent calls where we spend time are not the one in WaitOnIdleSemaphore() (that never gets called on OS X), but rather in the application's main event loop.
Simon, When the time manager task calls WakeUpProcess, is it possible for the application to be in the NSPR function WaitOnIdleSemaphore? Would that be a problem?
I think that could happen; we might be sitting in the semaphore (with the timer task firing every 8ms), until a condition variable times out, at which point I think we'd go into the '_PR_SET_RESCHED_FLAG' clause. But it should be harmless; WakeUpProcess() is an interrupt-safe routine that just sets a flag in the OS scheduler causing the app to get scheduled next time around. <http://developer.apple.com/techpubs/mac/Processes/Processes-46.html>
Comment on attachment 90693 [details] [diff] [review] Patch v. 2 updated to address wtc's concerns r=wtc. It would be nice if you could have a Mac developer review this patch too. Please check this in on the tip and NSPRPUB_PRE_4_2_CLIENT_BRANCH of NSPR.
Attachment #90693 - Flags: review+
Comment on attachment 90693 [details] [diff] [review] Patch v. 2 updated to address wtc's concerns r=sdagley Gotta love it when a patch fixes the bug, improves pageload perf and has no additional idle overhead
Patch checked in to NSPRPUB_PRE_4_2_CLIENT_BRANCH and the NSPR tip. Keeping bug open for possible 1.0 branch checkin.
Resolving as fixed, since the patch has been checked into the trunk. Added adt1.0.1 for branch consideration. Let's get this verified by john unruh tomorrow, so we can review it for inclusion on the branch.
Blocks: 143047
Status: ASSIGNED → RESOLVED
Closed: 23 years ago → 23 years ago
Keywords: adt1.0.1
Resolution: --- → FIXED
Whiteboard: [adt2 RTM] [ETA Needed] → [adt2 RTM] [ETA 07/11]
Verified on 7/10 OSX and Mac 9.2.2 trunk. Client auth took around 3 seconds with automatic cert selection, and was very fast with manual cert selection. Also, the ssl speed test page took only 2 seconds - https://pki.mcom.com/color/color.html
Status: RESOLVED → VERIFIED
Is there anything else we should be testing besides this actual bug?
Yes. We need to check that the app doesn't use lots of CPU for no reason at any time. So we need to run a basic test suite (browsing, mail, downloading etc) with the CPU monitor open (or using top), keeping an eye on the CPU usage.
downloading info, when saving (using "save target link as") a 15mb file (the mac 8/9 mozilla 1.1a bits). how 2002.07.09.13 2002.07.10.08 ------------------------------------- http 55 sec 15 sec ftp 53 sec 8 sec CPU usage (looking at the progress of the http download) --------- 2002.07.09.13: during the download, cpu peaked at around 25% then leveled off to 2.8% when finished. 2002.07.10.08: during the download, cpu peaked at around 58.2% then leveled off to 1.4% when finished.
I checked POP3 and IMAP protocols with the build that exhibited the *old* behavior (the 2002-07-09-13 trunk build running on OS X 10.1.5), as well as the *current* trunk (2002-07-10-08). Some basic observations using Process Viewer: Old 9th build: Downloading 1009 IMAP messages took 2 minute, 54 seconds and 57 milliseconds. Current 10th build: Downloading 1009 IMAP messages took 1 minute, 16 seconds and 85 milliseconds. Granted this isn't an isolated mail test server, but these comparisons were done literally within a few minutes of each other. I didn't readily have a large pool of messages to test POP3's performance with, but I was watching CPU usage in both cases, and found no alarming instances of spiking. CPU usage for most mail download operations hovered @ a 44-65% level, with a few upward spikes, but nothing sustained. At idle level, the app sits pretty quiet - LaunchCFM app reports anywhere from 0.3 % to somewhere near 4% (~3.7). When running other applications concurrently with ours, we drop CPU usage and yield to the other app I tested (as best as I can tell, since the Process ID Tab of Process Viewer gives me a unique integer value, instead of the application's name). I quit IE, noticed the process ID of Mozilla, started downloading headers, noticed CPU usage, launched IE (loading a page in IE yields 70% usage or so), and noticed no spikes here in Mozilla, either.
in comment 91, i had been downloading from our local mozilla ftp and http server, eg, http://ftp.mozilla.org/pub/mozilla/releases/mozilla1.1a/mozilla-mac-11a-full.bin simon suggested comparing these data with download from a more remote site, where bandwidth would be more of an issue. so, i used an japanese moz mirror site: http://mirror.nucba.ac.jp/mirror/mozilla/mozilla/releases/mozilla1.0/mozilla-mac-10-full-instal.bin i did three test runs for the japanese downloading for each build; here's the summary: * using 2002.07.09.13 trunk bits: CPU usage varied from a momentary peak of 16-45%, stabilizing to about 3-20% for most of the duration. download time ranged from 9min 25sec to 10min 41sec. * using 2002.07.10.08 trunk bits: CPU usage varied from a brief peak of 20-53%, stabilizing to about 3-13% for the most of the duration. download time ranged from as low as 5min 6sec to 10min 50sec.
Adding adt1.0.1+ on behalf of the adt pending sr=. Please get drivers approval and check into the Mozilla 1.0 branch asap.
Keywords: adt1.0.1 → adt1.0.1+
Comment on attachment 90693 [details] [diff] [review] Patch v. 2 updated to address wtc's concerns NSPR is exempt from Mozilla client's super-review requirement. This patch has been reviewed by me (NSPR owner) and Steve Dagley.
Attachment #90693 - Flags: superreview+
John can you verify that this is fixed on the trunk?
Sorry, I meant branch. I think it's fixed on the branch as well, but we should also verify there.
Comment on attachment 90693 [details] [diff] [review] Patch v. 2 updated to address wtc's concerns This patch has not been checked in on the MOZILLA_1_0_BRANCH.
Comment on attachment 90693 [details] [diff] [review] Patch v. 2 updated to address wtc's concerns a=chofmann for 1.0.1. add the fixed1.0.1 keyword after checking in on the branch
Attachment #90693 - Flags: approval+
Keywords: mozilla1.0.1 → mozilla1.0.1+
Checked in on the MOZILLA_1_0_BRANCH, added keyword. Thanks guys.
Keywords: mozilla1.0.1+ → fixed1.0.1, mozilla1.0.1
Verified on 7/16 Mac OSX branch.
Keywords: fixed1.0.1 → verified1.0.1
Component: Event Handling → User events and focus handling
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: