[email/IMAP] IMAP folder logic does not handle completely missing parent folders, breaks folder sync and thereby sync

RESOLVED WORKSFORME

Status

Firefox OS
Gaia::E-Mail
RESOLVED WORKSFORME
3 years ago
2 years ago

People

(Reporter: brainslug, Unassigned)

Tracking

unspecified

Firefox Tracking Flags

(tracking-b2g:+, b2g-v1.4 affected, b2g-v2.0 affected, b2g-v2.1 fixed, b2g-v2.2 unaffected, b2g-master unaffected)

Details

(Whiteboard: [priority])

(Reporter)

Description

3 years ago
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140429 Firefox/24.0 Iceweasel/24.5.0 (Nightly/Aurora)
Build ID: 20140429160536

Steps to reproduce:

Set up imap+smtp account on ZTE OpenC 

Software: FFOS_US_EBAY_OPENCV1.0.0B01
OS version: 1.3.0.0
Platform version: 28.0
Build Identifier: 2014032033956
Git commit info: 2014-03-21 07:16:57 0cddd1cc


Actual results:

IMAP client can connect to server (993) and synchronize all folders and email headers quickly. New messages are marked as new. So far so good.
But opening any new message takes up to 50min before the message is loaded! After the first message successfully opened, opening any other message works instantaneously. There is no wifi traffic recorded in the "usage" app during the initial wait period.


Expected results:

Email body should be displayed immediately.
(Reporter)

Comment 1

3 years ago
I'm not a developer or programmer, so please let me know if I'm missing any important information. Thanks.
I suspect what's happening here is that our IMAP state machine freaks out and breaks.  The reason the message shows up after 50 minutes is that eventually the connection times out (the spec says at least 30 minutes) and then we reconnect and things work.  It's possible the problem is during the header fetching, which would pretty much explain this.  The 50 minutes could also be because of other requests getting stuck in our queue that will break the connection again and so it's more a question of multiple timeouts happening.

If you could provide a logcat and let us know what domain/IMAP server you're trying to talk to, that would be very handy.  Please see https://wiki.mozilla.org/Gaia/Email/RequiredBugInfo for details on how to gather the logcat.


If the problem isn't entirely deterministic, the workaround would be for you to close the email app by long-pressing on the physical home button then swiping up on the email app to kill it.  (Or click the little X in the upper left.)  You should definitely kill the email app once before trying to gather a log/etc.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: Opening email body takes a very long time → [email/IMAP] IMAP connection apparently hangs for up to 50 minutes, starts working again afterwards. Probably connection loss related.
(Reporter)

Comment 3

3 years ago
Hi Andrew, thanks for the quick reply!

I followed the instructions on how to get the logcat and got some information off the device. I created a log, but it's too big to paste it here, so I'm just pasting what I *think* might be helpful, but can email/upload the bigger log if necessary:

I/Gecko   ( 9866): WERR: Explosion while processing data TypeError: box.attribs is undefined
I/Gecko   ( 9866): WERR: Stack: properties._determineFolderType@app://email.gaiamobile.org/js/ext/mailapi/composite/configurator.js:3630
I/Gecko   ( 9866): walkBoxes@app://email.gaiamobile.org/js/ext/mailapi/composite/configurator.js:3761
I/Gecko   ( 9866): walkBoxes@app://email.gaiamobile.org/js/ext/mailapi/composite/configurator.js:3786
I/Gecko   ( 9866): properties._syncFolderComputeDeltas@app://email.gaiamobile.org/js/ext/mailapi/composite/configurator.js:3789
I/Gecko   ( 9866): processResponse@app://email.gaiamobile.org/js/ext/mailapi/imap/probe.js:833
I/Gecko   ( 9866): processData@app://email.gaiamobile.org/js/ext/mailapi/imap/probe.js:1116
I/Gecko   ( 9866): ImapConnection.prototype.connect/<@app://email.gaiamobile.org/js/ext/mailapi/imap/probe.js:549
I/Gecko   ( 9866): EventEmitter.prototype.emit@app://email.gaiamobile.org/js/ext/mailapi/worker-bootstrap.js:12184
I/Gecko   ( 9866): NetSocket.prototype._ondata@app://email.gaiamobile.org/js/ext/mailapi/composite/configurator.js:4096
I/Gecko   ( 9866): NetSocket/routerInfo<@app://email.gaiamobile.org/js/ext/mailapi/composite/configurator.js:4018
I/Gecko   ( 9866): receiveInstanceMessage@app://email.gaiamobile.org/js/ext/mailapi/worker-bootstrap.js:1045
I/GeckoDump( 9866): ERR: onerror reporting: box.attribs is undefined @ app://email.gaiamobile.org/js/ext/mailapi/composite/configurator.js : 3630


The mail server I'm trying to connect to is the university's mail server at 128.104.255.23:993 

Thank you very much for your help - getting this to work would be really great!
(Reporter)

Comment 4

3 years ago
One additional note - I just upgraded my phone to build identifier 20140505080626, git commit info 2014-05-04 23:25:52
The problem still exists, but I thought I'd mention that the software version on my phone is now slightly different than described in my first post.

Thanks!
I think you excerpted the relevant bit of the log, thank you!

This may be a variant on bug 925480.  That bug involves gmail though and your server seems to be using Dovecot which makes this somewhat surprising.  If it would be possible for you to follow the steps described at https://bugzilla.mozilla.org/show_bug.cgi?id=925480#c13 to help identify what is problematic about your folder structure, that would be quite helpful.  I'm wondering if it's possible there's a namespace/public folder situation that is causing us trouble...
(Reporter)

Comment 6

3 years ago
Interesting... This is the output I'm getting:

LIST-STATUS] Logged in
a2 XLIST "" "*"
a2 BAD Error in IMAP command XLIST: Unknown command.

Thaks a bunch!
Whoops.  You want to do:
  a2 LIST "" "*"

XLIST is gmail specific; my brain glossed over that.
(Reporter)

Comment 8

3 years ago
Just emailed you the full directory output. 

The only thing that my non-expert eye could spot is one folder name that has a ':' in it:

* LIST (\HasNoChildren) "." "INBOX.Help.BC: Management"

Truncating issue comes to mind, but that's just spraying buck-shot...

Thanks for taking a look.
(Reporter)

Comment 9

3 years ago
renamed that "suspicious" folder, restarted the mail client, but unfortunately the same problem. Looks like that's not the case.
(Reporter)

Comment 10

3 years ago
Thanks to Andrew's detective work, this problem might actually be fixed. There were two "orphaned" IMAP folders lying around on the server. After deleting both folders I can open emails now w/o problems. Will do some more testing and report back or close the bug.

Andrew rules!!
We'll call the problem worked-around for now :)  But glad that the email app is now working for you!  From our debugging, we determined the following:

Wacky folders of the following forms existed:
- ".INBOX.Folder.Subfolder" where there should not have been a leading "." and thereby both the "INBOX" and "Folder" parts were nonexistent parents.
- "INBOX.Foo.Bar" where there was no "INBOX.Foo" already.

It appears these were due to procmail filters on the server with the first one being the result of a typo.  After removing that folder, it appeared again without the leading "." suggesting that underlying software is now performing slightly more normalization in the folder creation process.

I'm still very much hoping we can upgraded our IMAP lib in bug 885110 for the v2.0 time-frame to address this.  Bug 925480 should be addressed at the same time and test cases for these definitely created.  (I don't think duping is quite appropriate since the case-sensitivity/magic issue in bug 925480 is distinct.)
See Also: → bug 925480
Summary: [email/IMAP] IMAP connection apparently hangs for up to 50 minutes, starts working again afterwards. Probably connection loss related. → [email/IMAP] IMAP folder logic does not handle completely missing parent folders, breaks folder sync and thereby sync
Whiteboard: [priority]
Duplicate of this bug: 1018550
This will be addressed by the switch to browserbox in bug 885110.  We may need to file a new bug and upstream issue for the ".INBOX.Folder.Subfolder" case which seems particularly crazy.
Depends on: 885110
Duplicate of this bug: 1060956

Comment 15

3 years ago
I've been reading this issue with interest, as I have experienced a very similar problem on my Open C (I thought it appropriate to append this issue with more info, rather than create a new/duplicate issue).

The situation I had is where my IMAP folder list wouldn't fully sync, and message sending seemed to hang. (Admittedly, I wasn't quite patient enough to wait 50 mins...)

I found that only the first half of my IMAP folders would sync into the folder list of the Email app.  But following Andrew's steps (comment #6 / comment #7) for retrieving the folder list from the IMAP server was enlightening, as it revealed that the IMAP folder list is not served alphabetically.

The cause of my partial folder list was that I had a \Noselect folder (in Thunderbird, this is a grey italic folder - see: http://kb.mozillazine.org/Grey_italic_folders) with a normal child folder.  These folders, and anything following them in the retrieved folder list, would not sync in the email app, resulting in similar symptoms as reported above.
Yeah, this definitely is the right place to append!

(In reply to dowie from comment #15)
> The cause of my partial folder list was that I had a \Noselect folder (in
> Thunderbird, this is a grey italic folder - see:
> http://kb.mozillazine.org/Grey_italic_folders) with a normal child folder. 
> These folders, and anything following them in the retrieved folder list,
> would not sync in the email app, resulting in similar symptoms as reported
> above.

Hm, we should be okay with \Noselect.  GMail, for one, marks its [Gmail] folder \Noselect.

It'd be great if you could provide the following:
- What version of Firefox OS you reproduced this problem on?
- An excerpt of the \Noselect folder and anything that preceded it that looks like it's an ancestor of that folder?  Alternately, provide the whole list.  Definitely if you do the latter, and maybe if you do the former, please feel free to send it to asuth@mozilla.com rather than post it here.  Alternately, consistent search/replace to sanitize is probably also fine.

I have some improved unit test coverage on a branch to help make sure we've totally addressed this family of errors, and I'd like to fold in whatever your situation is too even if it's already fixed in more recent versions of FxOS.  Thanks!

Comment 17

3 years ago
(In reply to Andrew Sutherland [:asuth] from comment #16)

> Hm, we should be okay with \Noselect.  GMail, for one, marks its [Gmail]
> folder \Noselect.
> 
> It'd be great if you could provide the following:
> - What version of Firefox OS you reproduced this problem on?

Originally I spotted the problem in Firefox OS 1.3 (ZTE Open C, UK variant, firmware: FFOS_EU_EBAY_OPENCV1.0.0B06), but it seems present in 2.0 too (in the emulator).  Haven't gone beyond that.

> - An excerpt of the \Noselect folder and anything that preceded it that
> looks like it's an ancestor of that folder?  Alternately, provide the whole
> list.  Definitely if you do the latter, and maybe if you do the former,
> please feel free to send it to asuth@mozilla.com rather than post it here. 
> Alternately, consistent search/replace to sanitize is probably also fine.



The original account in question is my work one, but I have set up a test account for you so I will email details over shortly.

If you need IMAP server logs (we're using Dovecot version 2.1.7 / Debian 7) or any additional testing, just let me know.

> I have some improved unit test coverage on a branch to help make sure we've
> totally addressed this family of errors, and I'd like to fold in whatever
> your situation is too even if it's already fixed in more recent versions of
> FxOS.  Thanks!

Glad to help!
(In reply to dowie from comment #17)
> The original account in question is my work one, but I have set up a test
> account for you so I will email details over shortly.

Thanks very much!  Looking at the folder list you created in the mail you sent me, the problem is indeed that the server is reporting a child before reporting its parent.  Our fix went in for v2.1 and v2.2 (as a byproduct of moving to browserbox for low level IMAP support), so we unfortunately do expect v2.0 to still have this problem.

You mention subscriptions in your mail; through v2.0 we don't pay any attention to subscriptions.  Starting in v2.1 we do, but only in an additive way, I think.  (We need to do more with supporting subscriptions.)

In the email you mention that the folders causing this problem are "virtual" folders that can only store other folders.  Can you clarify if they are in fact Dovecot virtual folders (using the "virtual" plugin) versus something storage-limitation related?  (Specifically, I know dovecot's mbox storage means that a folder either has other sub-folders or it has messages, but not both.)

Comment 19

3 years ago
(In reply to Andrew Sutherland [:asuth] from comment #18)

> In the email you mention that the folders causing this problem are "virtual"
> folders that can only store other folders.  Can you clarify if they are in
> fact Dovecot virtual folders (using the "virtual" plugin) versus something
> storage-limitation related?  (Specifically, I know dovecot's mbox storage
> means that a folder either has other sub-folders or it has messages, but not
> both.)

My mistake - I shouldn't have used this phrase.  The test account's Dovecot server uses Maildir as the storage method and the subdirectories only appear to be created when mail is stored in them for the first time (they are no subdirectories on the file system relating to the test folders currently).

The Dovecot virtual plug-in is present on the system and I have created a virtual folder in that test account, but this does not appear in the list of folders served out using the above method (in comment #7).

Hope that helps?
[priority] --> tracking-b2g:+ conversion
tracking-b2g: --- → +
As indicated on bug 18, the fix happened with bug 885110, so I'm resolving this as WFM.

Particularly for those users with a ZTE Open C using v1.3, the best option for you is to upgrade to a more recent version of Firefox OS.  I believe ZTE Open C devices are upgradeable.  Please see http://paulrouget.com/e/openc/ and the wiki links at the bottom of the most for more.

When upgrades are not an option, the problem can usually be worked around by ensure that all folders on the server have a parent.  When the problem is the server reporting the folders out of order, other than upgrading the server, the best hope is to remove/move the folders so that timestamps/inodes might be updated and thereby change the order the server reports things.

(This is motivated primarily by wanting to use this bug as a dupe-target.  Discussion on comment 15 through comment 18 was motivated by ensuring that the reported bug was due to the same underlying problem that was fixed.  Which it was.  I do apologize for not responding to comment 19; I was on vacation and the comment fell off my triage horizon.  I would have preferred to indicate in a timely fashion that the bug was as expected and the best options are workarounds and upgrades.)
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-b2g-v1.4: --- → affected
status-b2g-v2.0: --- → affected
status-b2g-v2.1: --- → fixed
status-b2g-v2.2: --- → unaffected
status-b2g-master: --- → unaffected
Resolution: --- → WORKSFORME
Duplicate of this bug: 1125629
Duplicate of this bug: 873793
You need to log in before you can comment on or make changes to this bug.