877159 - (attach-paradigm-fail) [Meta] Tracker bug for attachment paradigm failures - "attach/embed immediate snapshot" VS. "attach/embed later when sending"

Status: NEW

(Thunderbird :: Message Compose Window, defect)

Description

[Thomas D. (:thomas8)]

+++ This bug was initially created as a clone of Bug #378046 +++

There's a major wrong design paradigm in TB about the way we attach/embed things (messages, files, images, web pages from urls, etc.):

Instead of attaching/embedding an *immediate snapshot copy* of such things, TB just keeps a *reference to the original location*, and tries to *attach them _later_* when saving/closing draft or sending.

Then it starts to scream and spin around when those externally referenced things have been *moved, renamed, deleted, physically disconnected, touched by compaction or otherwise become unavailable* in the meantime. Worse, there's also *privacy violations* when the contents of those things have changed and the new contents get silently attached (wrong version of files, even entirely unrelated messages after compacting).

Add to that inconsistent behaviour depending on *methods* of attaching (MAPI vs. non-MAPI cases), and the fact that closing and reopening a composition will also change the status of your attachment without notice (e.g. from "attach from original file later when sending" to "snapshot copy somewhere down the road between attaching and sending"), and realize that the current UX is very unpredictable and confusing.

We have a lot of bugs exposing this general design problem, and folks like WADA, rsx11m and me keep coming to the same conclusion all over again: It's a general design problem which can only be addressed by taking an *immediate snapshot* of all things as soon as they get embedded or attached (at least as a *default behaviour*). Btw "immediate snapshot" is the default behaviour for all webmailers (when you upload your files), and the same behaviour we already have for Bigfiles and cases of attaching via Mapi (e.g. Explorer > file context > Send to > Email recipient).

Notwithstanding, there can be legitimate RFE's to establish a *new clear-cut feature* (which is currently only half-working and unpredictable) of "reference and *attach later when sending*" (especially helpful for attached files, probably not for most other things).

This meta bug is for collecting related bugs, with an intention of exposing the full extent of this design problem in a central place, so that it can be fully understood and addressed.

Comment 1

[rsx11m]

Thomas, thanks for creating this important tracker. There are probably more bugs around, but many of them may be in the Editor component and probably shouldn't be added here.

Also, you've made the bugs /depend/ on this one, where they'd usually /block/ the meta bug so that everybody is informed when an issue is resolved. The only bug from your current list this one should block is bug 579473 (attachUXtracker) as the higher-level meta bug.

Comment 2

[Thomas D. (:thomas8)]

(In reply to rsx11m from comment #1)
> Thomas, thanks for creating this important tracker. There are probably more
> bugs around, but many of them may be in the Editor component and probably
> shouldn't be added here.

I'll give that a second thought, but I think we should add Editor bugs IF they are about attachment paradigm as described and TB is affected (as in: "pasted image can't be sent because editor did not get an immediate snapshot of image data"). Otherwise of course, the focus is on TB and we don't want this to become a cesspool of editor's technical problems.

> Also, you've made the bugs /depend/ on this one, where they'd usually
> /block/ the meta bug so that everybody is informed when an issue is
> resolved. The only bug from your current list this one should block is bug
> 579473 (attachUXtracker) as the higher-level meta bug.

Sure, thanks! I realized that in the very second I pushed the button to create this bug, but it was already too late, sorry for bugspam.

Comment 3

[Thomas D. (:thomas8)]

Bug 532395 Sending reply to (or forward inline of) existing HTML message with embed images fails with never-ending error message: "attaching..." (this bug is kept for other cases than bug 453196, bugs pointed in bug 453196, and bug 817245)

Note Bug 532395, which currently acts as a landing pool for incoming bugs about replied-to/forwarded messages with inline images failing with "error attaching":

- 10 duplicates at the time of writing this
- 64 votes
- 37 reports on getsatisfaction

Most if not all of these probably fail because of wrong attachment paradigm (as described in this meta bug): instead of embedding the actual image as an *immediate snapshot*, TB just places an external reference to the image part in the originally replied-to/forwarded msg, e.g.

<img src="mailbox:.../Inbox%3Ennn?part=1.2&filename=EmbedImage.jpg">

...which then fails for saving as draft/sending when that referenced msg (having the image) for whatever reason gets moved, touched by compact, touched by auto-save into draft with new id, deleted etc, after which the external reference to such image breaks in composition.

Comment 4

[rsx11m]

Bug 557708 - Drag and drop of images from Firefox to Thunderbird as attachment results in BMP conversion, fails when sending - consists of two parts: One is unnecessary recoding from the original image format into BMP (Core problem), the other though related to this meta (i.e., the BMP is gone already at the time Thunderbird wants to access it when sending the message or saving it as draft).

Comment 5

[Jeff Barry]

Pointing to and using a piece of data which can go away is a major programming error with all sorts of possible outcomes.  Thomas D. is absolutely correct in marking this as a critical error.  This sort of bug can cause anything from confusion and lost messages (my case, bug ID 1185195) to privacy violations to addressing errors with resultant crashes.  When you are pointing at uncontrolled data you have NO IDEA what is going to happen.

Comment 7

[Thomas D. (:thomas8)]

rkent, fyi. Pls ping back when you've seen this.

(In reply to Jeff Barry from comment #5)
> Pointing to and using a piece of data which can go away is a major
> programming error with all sorts of possible outcomes.  Thomas D. is
> absolutely correct in marking this as a critical error.  This sort of bug
> can cause anything from confusion and lost messages (my case, bug ID
> 1185195) to privacy violations to addressing errors with resultant crashes. 
> When you are pointing at uncontrolled data you have NO IDEA what is going to
> happen.

Comment 8

[Thomas D. (:thomas8)]

Anecdotal user testimony:

(In reply to Jeff Barry from Bug 1185195 Comment 4)
> OK, my Bug 1185195 here is a symptom of another bug [Bug 453196].  It's clear from bug 532395,
> comment 114 by Thomas D. on 2014-04-16 10:01:57 PDT
> 
>     As hinted in Whiteboard field of this bug, this bug is to collect
>     bugs which report the same symptom ("Attaching..." endless spin),
>     but whose causes have not been verified yet. On the other hand, we
>     have (at least) two verified causes for this bug:
> 
>     Bug 453196 Embedded image URL points to another msg and that other
>     msg is deleted before saving/sending (so picture can't be found any
>     more)
> 
> I did exactly that.  I did a reply to the original eMail and then I moved
> the original and subsequently did a compaction, so the original image was
> gone from where it had been but the reply copy was still pointing at where
> the original image had been.  This also perfectly explains why there have
> been comments from people saying the problem is solved when they deleted the
> image from the reply and then pasted it back in: that changes the pointer to
> something which exists.
> 
> Sorry to be so blunt about it but this is an elementary programming bug. 
> You can only point at things which you can control.  This has been pointed
> out by another bug which, as far as I am concerned is a critical bug. 
> Comment 114 then goes on to say:
> 
>     I'm not even sure to what extent this has been acknowledged as a
>     general design problem by developers; but I've done my best to
>     present the problem and evidence from related bugs to support that
>     analysis, and other well-known bug triagers have come to the same
>     conclusion: In a nutshell, all things which are seen in your
>     composition should be copied immediately so that they are in the
>     exclusive control range of your composition, which would solve most
>     if not all of these bugs.
> 
> Exactly.  The real (and critical) bug in this case is:
> 
>     Bug 877159 - (attach-paradigm-fail) [Meta] Tracker bug for
>     attachment paradigm failures - "attach/embed immediate snapshot"
>     VS. "attach/embed later when sending"
> 
> which developers NEED TO FIX.
> 
> I think I have properly marked this bug as RESOLVED because it is yet
> another instance of a separately reported critical bug, 877159, which has
> yet to be addressed.
> 
> My thanks to Thomas D. for all his work on pointing out the seriousness of
> this problem.
> 
> Thank you,
> Jeff Barry

Comment 9

[Thomas D. (:thomas8)]

rkent, fyi: Bug 532395 (symptom cesspool for design problem described here in Bug 877159) currently has 13 duplicates and 102 votes.

Bug 532395 Comment 114 describes the status quo as of 2014-04.
We've improved over that by fixing Bug 854798; however I recall WADA saying somewhere that even that corner is not fully fixed yet.

needinfo:rkent for read/awareness receipt on this bug's comment 7 to comment 9.

Comment 10

[Kent James (:rkent)]

I think the NI is asking, do I read these bugs, and am I aware of these issues?

Yes, I read these bug reports, and am aware of these issues. That does not mean that I am prepared to move this to the top of my TODO list though. The critical issue though IMHO is not this bug, but the lack of developers to fix these bugs. I am currently focused more on this meta problem.

Comment 11

[Thomas D. (:thomas8)]

Sometimes we even fix this things :)
Bug 1315480, Bug 1317049

In fact, significant headway has been made in some key areas affected by this design problem.
But it's not the end of the line yet.