Closed Bug 495318 Opened 15 years ago Closed 12 years ago

add support for LIST (SUBSCRIBED) part of IMAP LIST-EXTENDED command (RFC 5258)

Categories

(MailNews Core :: Networking: IMAP, defect)

defect
Not set
normal

Tracking

(thunderbird19 fixed)

RESOLVED FIXED
Thunderbird 19.0
Tracking Status
thunderbird19 --- fixed

People

(Reporter: Bienvenu, Assigned: dlech)

References

(Depends on 1 open bug, Blocks 3 open bugs, )

Details

Attachments

(2 files, 9 obsolete files)

Dovecot supports this (not sure who else does). It has the nice property that LIST-EXTENDED SUBSCRIBED gives you all the useful flags about a folder, like LIST does, but unlike LSUB. In particular, this will tell us things like /NoInferiors, which means we shouldn't even try to create sub-folders, which is important for Archive. I'm going to tentatively mark it blocking for that reason.
Flags: blocking-thunderbird3+
OS: Windows Vista → All
Hardware: x86 → All
I've fixed the Archive bug a different way, and the server it was happening on didn't support LIST-EXTENDED anyway, so this isn't a blocker. But it would still be very nice to have.
Flags: blocking-thunderbird3+ → wanted-thunderbird3+
Target Milestone: Thunderbird 3.0b4 → ---
This is not full LIST-EXTENDED support, just LIST (SUBSCRIBED) as mentioned in the first comment. And I'll come up with a test soon.
Attachment #664099 - Flags: review?(mozilla)
-      else if (token.Equals("EXTENDED-LIST", nsCaseInsensitiveCStringComparator()))
+      else if (token.Equals("LIST-EXTENDED", nsCaseInsensitiveCStringComparator()))

Outch! I've probably looked at this a dozen times and never noticed it.
Added some flags and flag parsing based on RFC document.
Attachment #664099 - Attachment is obsolete: true
Attachment #664099 - Flags: review?(mozilla)
Attachment #664130 - Flags: review?(mozilla)
Blocks: RFC6154
Compare bug 558659 comment 40, TODO Step 3
Comment on attachment 664130 [details] [diff] [review]
check for LIST-EXTENDED capibility when getting subscribed IMAP folders

 thx, David. I don't have a server to test this with, but it seems OK.
Attachment #664130 - Flags: review?(mozilla) → review+
Blocks: 317597
Assignee: mozilla → david
Attached patch related tests, etc. (obsolete) — Splinter Review
I have modified imapd.js to partially implement LIST-EXTENDED. I only implemented what was necessary to test this bug, but it should be very simple to complete the implementation if we need to.

In doing so, I also had to modify some of the GMail stuff. I have re-written the XLIST handler so that it works just like the official SPECIAL-USE imap extension. Again, implementing the rest of SPECIAL-USE should be rather trivial now.

I also threw in the CHILDREN imap extension for good measure (GMail uses it too).

I added a couple of tests in mailnews/base/test that test imapd.js to make sure that it spits out the proper responses when it is simulating LIST-EXTENDED and GMail servers.

Then finally, the test in mailnews/imap/test it the test to make sure the changes I made in actual mailnews code works properly.

I have run all tests in both mailnews/base/test and mailnews/imap/test to make sure I did not break anything. My GMail changes broke the couple of tests that atuljangra recently contributed, so I fixed them up.
Attachment #669415 - Flags: review?(mozilla)
Attached patch related tests, etc. v2 (obsolete) — Splinter Review
added an error check
Attachment #669415 - Attachment is obsolete: true
Attachment #669415 - Flags: review?(mozilla)
Attachment #669557 - Flags: review?(mozilla)
Blocks: 799821
No longer blocks: 317597
Blocks: 781443
Alias: RFC5258
Summary: add support for IMAP LIST-EXTENDED command → add support for IMAP LIST-EXTENDED command (RFC 5258)
Attachment #669557 - Flags: review?(irving)
Comment on attachment 669557 [details] [diff] [review]
related tests, etc. v2

Review of attachment 669557 [details] [diff] [review]:
-----------------------------------------------------------------

I feel a bit guilty about the nits because the code is great.

::: mailnews/base/test/unit/test_testsuite_fakeserver_imapd_gmail.js
@@ +1,5 @@
> +/* This Source Code Form is subject to the terms of the Mozilla Public
> + * License, v. 2.0. If a copy of the MPL was not distributed with this
> + * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
> +
> +// Test that imapd.js fakeserver correlecty emulates GMail server

correctly

@@ +31,5 @@
> +
> +// mbox mailboxes cannot contain both child mailboxes and messages, so this will
> +// be one test case.
> +function setupMailboxes()
> +{  

White space at end of line

@@ +43,5 @@
> +  gIMAPDaemon.createMailbox("[Gmail]/Trash", {specialUseFlag : "\\Trash"});
> +  gIMAPDaemon.createMailbox("test", {});
> +
> +  handler = gIMAPServer._readers[0]._handler;
> +  

White space

@@ +44,5 @@
> +  gIMAPDaemon.createMailbox("test", {});
> +
> +  handler = gIMAPServer._readers[0]._handler;
> +  
> +  // wait for imap pump to do it's thing or else we get memory leaks

its

Man, I'm picking all the nits today. Must be the rainy grey skies.

::: mailnews/base/test/unit/test_testsuite_fakeserver_imapd_list-extended.js
@@ +55,5 @@
> +
> +  // wait for imap pump to do it's thing or else we get memory leaks
> +  gIMAPInbox.updateFolderWithListener(null, asyncUrlListener);
> +  yield false;
> +}

Would it be worth the effort to extract the common setup and related code from all these tests? I'm not going to r- for it.

::: mailnews/imap/test/unit/test_gmailOfflineMsgStore.js
@@ +56,5 @@
> +  setup,
> +  updateFolder,
> +  selectInboxMsg,
> +  StreamMessageInbox,
> +  createAndUpdate,  

trailing whitespace

@@ +61,5 @@
> +  addFoo,
> +  updateFoo,
> +  selectFooMsg,
> +  StreamMessageFoo,
> +  crossStreaming,  

whitespace

@@ +75,5 @@
> +  setupIMAPPump("GMail");
> +
> +  gIMAPMailbox.specialUseFlag = "\\Inbox";
> +  gIMAPMailbox.subscribed = true;
> +  

whitespace

@@ +80,5 @@
> +  // need all mail folder to identify this as gmail server.
> +  gIMAPDaemon.createMailbox("[Gmail]", {flags : ["\\NoSelect"] });
> +  gIMAPDaemon.createMailbox("[Gmail]/All Mail", {subscribed : true,
> +                                                 specialUseFlag : "\\AllMail"});
> +  

whitespace

@@ +213,5 @@
> +
> +asyncUrlListener.callback = function(aUrl, aExitCode) {
> +  do_check_eq(aExitCode, 0);
> +};
> + 

whitespace

@@ +239,5 @@
>  
>      scriptStream.read(aCount);
>    }
>  };
> + 

whitespace

::: mailnews/imap/test/unit/test_listSubscribed.js
@@ +2,5 @@
> + * License, v. 2.0. If a copy of the MPL was not distributed with this
> + * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
> +
> +// Test that imapd.js fakeserver correlecty implements LIST-EXTENDED imap
> +// extension (RFC 5258 - http://tools.ietf.org/html/rfc5258)

Update the comment for this test

@@ +55,5 @@
> +  let rootFolder = gIMAPIncomingServer.rootFolder;
> +  let folder1 = rootFolder.getChildNamed("folder1");
> +  do_check_true(folder1.getFlag(nsMsgFolderFlags.ImapNoselect));
> +  do_check_false(folder1.getFlag(nsMsgFolderFlags.ImapNoinferiors));
> +  

white space

::: mailnews/test/fakeserver/imapd.js
@@ +613,5 @@
>  
> +// used by RFC 5258 and GMail (labels)
> +function parseMailboxList(aList) {
> +
> +  // strip enclosing parenthises

parentheses

@@ +1121,5 @@
> +      }
> +    }
> +    if (!this[listFunctionName])
> +      return 'BAD unknown LIST request options';
> +      

whitespace
Attachment #669557 - Flags: review?(irving) → review+
Attached patch related tests, etc. v3 (obsolete) — Splinter Review
Fixed nits. Thanks irving!

> Would it be worth the effort to extract the common setup and related code from
> all these tests? I'm not going to r- for it.
Yes, I think so. I plan on working on more related bugs, so I will take care of it on a later patch.
Attachment #669557 - Attachment is obsolete: true
Attachment #669557 - Flags: review?(mozilla)
Keywords: checkin-needed
What needs to be checked-in David ?
Can we hold off on checking this in for a bit? I may have just seen an issue with one of my IMAP accounts and I want to look into it.
Status: NEW → ASSIGNED
Keywords: checkin-needed
When I start up TB on a profile that connects to Mozilla's Zimbra server I get a popup warning box containing:

The current operation on 'Inbox' did not succeed. The mail server for account ireid@mozilla.com responded: LIST failed: wildcards not permitted in username.

The relevant lines from an imap:5 log appear to be:

2012-10-31 12:56:23.172058 UTC - 596119552[11febe550]: 22ad4000:mail.mozilla.com:A:SendData: 9 list (subscribed) "" "/home/*"
2012-10-31 12:56:23.312700 UTC - 596119552[11febe550]: ReadNextLine [stream=22886a58 nb=55 needmore=0]
2012-10-31 12:56:23.312710 UTC - 596119552[11febe550]: 22ad4000:mail.mozilla.com:A:CreateNewLineFromSocket: 9 NO LIST failed: wildcards not permitted in username
(Outch. We really need a testsuite that contacts real-life servers, on a separate test machine with Internet access.)
Attached patch fix v2 (obsolete) — Splinter Review
Interesting. Looks like I have resurrected Bug 417285. Since we are using LIST command from Lsub function instead of List function. I applied the same fix to the Lsub function. Do we want to log to Activity Manager like mentioned in Bug 417285 Comment 11?
Attachment #664130 - Attachment is obsolete: true
Attachment #677063 - Flags: review?(irving)
Logging to activity manager should be for things that users understand. I'm not sure most user know what LIST and LSUB are. I'd say no. We could log it in NSPR though.
(In reply to Ludovic Hirlimann [:Usul] from comment #18)
> Logging to activity manager should be for things that users understand. I'm
> not sure most user know what LIST and LSUB are. I'd say no. We could log it
> in NSPR though.

OK. Information is already in IMAP log for more ambitious users to find.
(In reply to David Lechner (:dlech) from comment #17)
> Created attachment 677063 [details] [diff] [review]
> fix v2

Looks like you attached a copy of the tests patch, rather than the fix
Attached patch fix v2 (obsolete) — Splinter Review
Oops. Here is the correct patch.
Attachment #677063 - Attachment is obsolete: true
Attachment #677063 - Flags: review?(irving)
Attachment #679912 - Flags: review?(irving)
Comment on attachment 679912 [details] [diff] [review]
fix v2

Review of attachment 679912 [details] [diff] [review]:
-----------------------------------------------------------------

r+ after the whitespace nits are fixed. The test patch looks good too, but needs to be updated because xpcshell.ini has conflicting changes. I'd like to land both at the same time.

::: mailnews/imap/src/nsImapServerResponseParser.cpp
@@ +848,5 @@
> +        // RFC 5258 \NonExistent implies \Noselect
> +        boxSpec->mBoxFlags |= kNoselect;
> +      }
> +      else if (!PL_strncasecmp(fNextToken, "\\Subscribed", 10))
> +        boxSpec->mBoxFlags |= kSubscribed;  

Trailing white space

@@ +850,5 @@
> +      }
> +      else if (!PL_strncasecmp(fNextToken, "\\Subscribed", 10))
> +        boxSpec->mBoxFlags |= kSubscribed;  
> +      else if (!PL_strncasecmp(fNextToken, "\\Remote", 6))
> +        boxSpec->mBoxFlags |= kRemote;  

Trailing white space

@@ +852,5 @@
> +        boxSpec->mBoxFlags |= kSubscribed;  
> +      else if (!PL_strncasecmp(fNextToken, "\\Remote", 6))
> +        boxSpec->mBoxFlags |= kRemote;  
> +      else if (!PL_strncasecmp(fNextToken, "\\HasChildren", 11))
> +        boxSpec->mBoxFlags |= kHasChildren;  

Trailing white space
Attachment #679912 - Flags: review?(irving) → review+
Attached patch related tests, etc. v4 (obsolete) — Splinter Review
Attachment #676879 - Attachment is obsolete: true
Attached patch fix v3Splinter Review
Attachment #679912 - Attachment is obsolete: true
Both patches need checked in. Tests v4 patch is r=bienvenu, Fix v3 patch is r=irving.
Keywords: checkin-needed
https://hg.mozilla.org/comm-central/rev/4991059f9bbb
https://hg.mozilla.org/comm-central/rev/f3c76cb43c77
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Flags: in-testsuite+
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 19.0
Unfortunately, I've just had to back out the tests, because there appears to be permanent orange on Windows Opt builds, and frequent orange on Linux opt builds:

https://hg.mozilla.org/comm-central/rev/dd55bf51eeae

For example:

https://tbpl.mozilla.org/php/getParsedLog.php?id=17064593&tree=Thunderbird-Trunk

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\mailnews\base\test\unit\test_testsuite_fakeserver_imapd_gmail.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>
*******************************************
Generator explosion!
Unhappiness at: c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_gmail.js:46
Because: TypeError: gIMAPServer._readers[0] is undefined
Stack:
  setupMailboxes@c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_gmail.js:46
  _async_driver@../../../resources/asyncTestUtils.js:156
  .run@c:\talos-slave\test\build\xpcshell\head.js:418
  
**** Async Generator Stack source functions:
  setupMailboxes
  _async_test_runner
*********
TypeError: gIMAPServer._readers[0] is undefined
-- Exception object --
+ fileName (string) 'c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_gmail.js'
+ lineNumber (number) 46
+ stack (string) 231 chars
+ columnNumber (number) 2
*
-- Stack Trace --
setupMailboxes@c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_gmail.js:46
_async_driver@../../../resources/asyncTestUtils.js:156
.run@c:\talos-slave\test\build\xpcshell\head.js:418

TEST-UNEXPECTED-FAIL | ../../../resources/asyncTestUtils.js | Generator explosion. ex: [error TypeError] async stack [undefined undefined] - See following stack:
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: do_throw :: line 451
JS frame :: ../../../resources/logHelper.js :: mark_failure :: line 620
JS frame :: ../../../resources/asyncTestUtils.js :: _async_driver :: line 178
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: .run :: line 418
native frame :: <unknown filename> :: <TOP_LEVEL> :: line 0


TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\mailnews\base\test\unit\test_testsuite_fakeserver_imapd_list-extended.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>
*******************************************
Generator explosion!
Unhappiness at: c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_list-extended.js:54
Because: TypeError: gIMAPServer._readers[0] is undefined
Stack:
  setupMailboxes@c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_list-extended.js:54
  _async_driver@../../../resources/asyncTestUtils.js:156
  .run@c:\talos-slave\test\build\xpcshell\head.js:418
  
**** Async Generator Stack source functions:
  setupMailboxes
  _async_test_runner
*********
TypeError: gIMAPServer._readers[0] is undefined
-- Exception object --
+ fileName (string) 'c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_list-extended.js'
+ lineNumber (number) 54
+ stack (string) 239 chars
+ columnNumber (number) 2
*
-- Stack Trace --
setupMailboxes@c:/talos-slave/test/build/xpcshell/tests/mailnews/base/test/unit/test_testsuite_fakeserver_imapd_list-extended.js:54
_async_driver@../../../resources/asyncTestUtils.js:156
.run@c:\talos-slave\test\build\xpcshell\head.js:418

TEST-UNEXPECTED-FAIL | ../../../resources/asyncTestUtils.js | Generator explosion. ex: [error TypeError] async stack [undefined undefined] - See following stack:
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: do_throw :: line 451
JS frame :: ../../../resources/logHelper.js :: mark_failure :: line 620
JS frame :: ../../../resources/asyncTestUtils.js :: _async_driver :: line 178
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: .run :: line 418
native frame :: <unknown filename> :: <TOP_LEVEL> :: line 0
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attached patch attempt to fix try server fails (obsolete) — Splinter Review
This will hopefully fix the fails on the try server. I left it as a separate patch for easy review. I will merge it into the other patch before checkin. I am also guessing that I should give this a go on the try server before requesting checkin again. Should I be asking for someone to do this for me or should I be requesting try server access so that I can do it myself (and if so, how do I go about doing this)?
Attachment #682333 - Flags: review?(irving)
You should be requesting try server access so you have them in the future.

It's documented at http://www.mozilla.org/hacking/committer/ - I don't know the level you'll need to access try servers.
David, you can get try server access easily. Read http://www.mozilla.org/hacking/committer/ for information.
Alternatively, I can do a try server run for you.
That would be L1 access, I guess. :)
(In reply to Atul Jangra [:atuljangra] from comment #30)

> Alternatively, I can do a try server run for you.

yes that would speed things up for this bug.
Cool so lets wait for irving's review, and then I'll do the try server run. :)
Atul, no need to wait for review. If you could run it now, then I can go to sleep and find out what happened in the morning :)

All 3 patches need to be pushed.
Pushed to try server: 
https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=960576197481

Hoping for good results :)
Results are out :-/
Build burning on Win64 opt and Linux Debug. And failure of two tests :(
https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=960576197481
(In reply to Atul Jangra [:atuljangra] from comment #36)
> Results are out :-/
> Build burning on Win64 opt and Linux Debug. And failure of two tests :(
> https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=960576197481

Win64 is a known build issue, Linux debug is some sort of infra issue (upload failed, I've alerted buildduty), both of these have been starred & rebuilds requested. I think both the unit test failures are known issues, I've starred one that I know is, and I've requested a rebuild of the second to make sure.
Thanks :-) I hope everything goes well before dlech wakes up :)
Try run for 960576197481 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=960576197481
Results (out of 22 total builds):
    success: 15
    warnings: 4
    failure: 3
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/atuljangra66@gmail.com-960576197481
(In reply to Atul Jangra [:atuljangra] from comment #40)
> P.S.: Did we just found a bug about Mozilla RelEng Bot?

Known issue with the bot, see bug 807778 (also linked to from https://wiki.mozilla.org/Thunderbird/Infrastructure/TryServer which is linked from the top of tbpl)
Atul, High five and thanks very much for the try server run!

Mark, Thanks for your input/assistance too!

If I am reading everything correctly, the Linux debug rebuild succeeded and the Win64 build can be ignored. So, I just need and r+ on the additional changes and then we can checkin again.
HI5 :-).
Ans yes, we are ready for a good checkin :)
For the record, the reason for the failures was that I was trying to cheat and grab an instance of an existing handler from the fakeserver imapd. I am guessing that in optimized builds that there was a timing issue and the handler had not been instantiated yet before I tried to reference it. 

To fix, I am now creating a new instance of a handler instead of trying to steal the existing one that may not exist yet. In order to do this, I had to move a few things from the .prototype section of the IMAP_RFC3501_handler to the constructor section in imapd.js. Object, etc. in .prototype are shared between all instances, so by having 2 handlers, extensions were getting mixed in twice and causing problems.
Attachment #682333 - Flags: review?(irving) → review+
Merged 2 patches. 

This is the only patch that needs to be checked in now. The other patch was not backed out. r=irving
Attachment #681556 - Attachment is obsolete: true
Attachment #682333 - Attachment is obsolete: true
Keywords: checkin-needed
Alias: RFC5258
Summary: add support for IMAP LIST-EXTENDED command (RFC 5258) → add support for LIST (SUBSCRIBED) part of IMAP LIST-EXTENDED command (RFC 5258)
Blocks: RFC5258
https://hg.mozilla.org/comm-central/rev/6996e05c74ee
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Unless I missed something , the milestone should be tb20
Target Milestone: Thunderbird 19.0 → Thunderbird 20.0
Comment on attachment 683304 [details] [diff] [review]
related tests, etc. v5

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 683304
User impact if declined: 
Testing completed (on m-c, etc.): yes
Risk to taking this patch (and alternatives if risky): low
String or UUID changes made by this patch: none


As pointed out by Joe, this patch landed just after the latest aurora branch. However, the other patch for this bug was landed before the branch. So, that means there is a patch in aurora without any tests. So, it would be beneficial to have this patch merged into aurora as well.
Attachment #683304 - Flags: approval-mozilla-aurora?
Changing milestone back because the patch that actually fixes the bug was not backed out before the branch, only the tests.
Target Milestone: Thunderbird 20.0 → Thunderbird 19.0
Attachment #683304 - Flags: approval-mozilla-aurora? → approval-comm-aurora?
Attachment #683304 - Flags: approval-comm-aurora? → approval-comm-aurora+
FWIW, I've been unable to connect to mozilla.com's Zimbra server since this patch landed.  I can confirm that older builds prior to 20121115 still work with it.  I think this is a bug in Zimbra's list (subscribed) support, and I've filed a ticket with Zimbra to investigate, but I thought I'd give you guys a heads-up.  The IMAP log from Thunderbird pretty clearly shows Zimbra hanging up after the list (subscribed) command is issued, and the logs on Zimbra's end show a crash and stack trace at that point, so I'm pretty positive this is Zimbra's fault and not Thunderbird's.  I'll follow up with a new bug if they try to pin it on Thunderbird. :)
It's been acknowledged as a bug in Zimbra, fix targeted for Zimbra 8.0.2 (they're not planning on backporting it to Zimbra 7.x at this point).

http://bugzilla.zimbra.com/show_bug.cgi?id=78794

Might want to keep an eye on this, and if they really don't backport it then do an exception for Zimbra 7.x to not use this? (the versions are clearly identified in the ID response)
Depends on: 816028
(In reply to Dave Miller [:justdave] (justdave@bugzilla.org) from comment #52)
> Might want to keep an eye on this, and if they really don't backport it then
> do an exception for Zimbra 7.x to not use this? (the versions are clearly
> identified in the ID response)

This just happened to one of our users, and we happen to be running Zimbra 7.2.7, which is the latest and last release of the 7.x branch. This won't be fixed on Zimbra, could this exception be implemented ?
Otherwise, is there a workaround for this problem ? Either server-side or client-side, but upgrading Zimbra is currently not an option.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: