Open Bug 119964 (yEnc) Opened 23 years ago Updated 2 years ago

Support yEnc encoding (including multipart)

Categories

(MailNews Core :: MIME, enhancement)

Product:
MailNews Core
Component:
MIME
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: John.planb, Unassigned)

References

(Blocks 2 open bugs,
URL
)

Details

Attachments

(6 files, 1 obsolete file)

Article that used yEnc encoding
48.05 KB, text/plain
Details
A decoded version of the yenc-example
46.23 KB, image/jpeg
Details
yEncoded article posted to newsgroup
48.05 KB, message/rfc822
Details
jfd's patch, plus some minor nits
24.07 KB, patch
KaiE
: review-
Details | Diff | Splinter Review
patch for part 1 of multipart yenc
1.74 KB, patch
Details | Diff | Splinter Review
yet another robustness patch
3.12 KB, patch
Bienvenu
: review+
mscott
: superreview+
Details | Diff | Splinter Review 
There's a new encoding format (similar in design to UU but without it's 
30% overhead, and with built in error checkign) being batted around and 
adopted by a few newsreaders.  It's an 8 bit encoding (minus a few 
escaped characters; CR, LF, Nul), and is thus allows faster 
up/downloading and is less of a hit on the servers (allowing them to carry 
more articles).
Severity: normal → enhancement
Has word of this reached USEFOR yet?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Can somebody attach a sample message using this encoding. Also, would be nice to
have the corresponding RFC. Thanks
Status: NEW → ASSIGNED
Target Milestone: --- → Future

    
    

    
  


  

    
    

    
  
It was brought to USEFOR's attention, and the consensus there was that 
encoding methods weren't to be described by USEFOR.

There is no RFC, just the website and the programs using it (although it 
was discussed in news.software.nntp). 

    
    

    
  


  
I don't think the previous attempted worked (I tried to use plain-text and it
looks like the text was "normalized").

    
    

    
  
Doesn't look like the latest attempt to upload the attachment worked either 
(or at least I can't see how to download it without it having been changed).

Thoth and Xnews can post yEncoded binaries, so if anyone wants test 
articles posted to a specific group just ask (or has some idea as to how I 
could upload a non-image binary file that doesn't get trashed by bugzilla) 
just email me.

    
    

    
  
*** Bug 128837 has been marked as a duplicate of this bug. ***

    
    

    
  
Please support this: some porn is in yEnc form. :-)

    
    

    
  
yEnc multipart postings should also be supported
http://bugzilla.mozilla.org/show_bug.cgi?id=60981

    
    

    
  
http://groups.google.com/groups?q=yEnc&hl=en For some juicy discussion on yEnc

    
    

    
  
I suggest this be WONTFIXED. We shouldn't be encouraging people who devise crap
protocols like this without bothering to correct important deficiencies in them,
or to take the 6 months or so to work with the IETF and make them actually
interoperable.  (Yes, I'm biased.  OTOH, the author of the document referenced
in Comment #12 certainly knows more about handling binaries on Usenet than any
of us, and probably more than the author of yEnc :-)  Bottom line: encouraging
the fast-and-sloppy approach to protocols sticks in my craw.)

    
    

    
  
If you aren't planning on implementing this, throw it over to me. yEnc looks
like a pretty simple standard. As far as how to implement encoding schemes in
MailNews, I can probably figure it out. Any info would be helpful.

    
    

    
  
Chris: yEnc is still on sub version 2. I imagine it can be improved in future
versions. I'm glad someone had the guts to do this, although I haven't liked not
being able to see binaries anymore on Outlook Express. If mozilla supports this
and also bug 60981, I imagine we can pull the rug out from under Outlook Express
if they are slow in implementing it.



    
    

    
  
/. article on yEnc. http://slashdot.org/article.pl?sid=02/03/23/2154235

Jean, if you don't have time to implement this - I'll take it and either do it
myself or find someone to do it.

    
    

    
  
Remember this URL? http://www.exit109.com/~jeremy/news/yenc.html

"If you agree with me, what can you do to help? If you are the author of a
Usenet newsreader, you probably have to implement yEnc at least for decoding,
but you can leave out posting support to try to prevent this from spreading any
more. At the very least, if a MIME yEnc specification arrives but bypasses the
standards process, please, don't implement it. If you are a user, you can refuse
to post in yEnc."

I believe that he has a good idea here, as the proper place for the new 8-bit
encoding is MIME. I propose we only provide reading capabilities.

    
    

    
  
I may have underestimated the demand for the need to encode yEnc. It seems that
a lot of people are looking for this feature. Upon reflection, it appears to me
that not including it might be a bad idea. We can always rip it out once MIME is
made 8-bit.

    
    

    
  
Well, if you're interested in usage -- the server I use has various virtual 
groups, based upon article content, the group with formfeeds has gone 
from less than a thousand articles a day to more than 500 thousand 
articles a day.

So, I'd say that currently the posting volume is about 500,000 articles a 
day in yEnc format, and increasing as new clients are able to handle it.

    
    

    
  
Here is a quote from Juergen Helbring, creator of yEnc in reply to my email. 
(He gave me permission):

The eMail I've received right before yours was from a Netscape user
(Mac) who wanted to post and receive yEncoded posts. I told him to
send a feature request to Netscape. More and more people will follow.

On the "classical" music group there is meanwhile 100% of the binaries
in yEncoded format (They were the first who started yEnc'ing in
January 2002 - 12 weeks ago).

For Windows there are enough tools available to post binaries in yEnc.
Other platforms are not so well supported. Implementing also a
yEncoder might help people there.


If you need a _good_ reason why to implement also "encoding" - and
sending:

There is no sign that any other binary format than yEnc will be
created in the next few years (and I believe even in the next decade).

Those who are complaining most have not a single hour of time to do
constructive work on their own proposals. 
Even my biggets hope (Russ Albery - who said me he would write the doc
for "Content-Transfer-Encoding: yEnc" finally gave up and said that he
does not have the time do it - and he said that he does not have the
knowledge to do it).

Decide yourself what you are doing:

If you implement also sending (encoding) then the job is quickly - and
easily done. You'll have peace for the next few years.

If you dont do it now, then you will do it later.

All the "formal work" related to a change of the software will be
necessary again - and I am sure that also for you the "overhead" of
creating a new version is far more work than the 2-5 hours it will
take to expand the encoding/posting functions by yEnc.


If there would have been a _single_ step by the anti-yEnc'ies - a
single announcement, a single idea when anything else would be done,
then I'd say: Forget about yEnc. (I would have deleted yEnc instantly
and used something better). 
But I see no chance for alternatives from their side.


So it is all up to you.


If you want a "tip" for the implementation:
I would recommend to implement the de/encoder in a way that it is a
module which could be also used if yEnc comes as a CTE: x-yEnc
It might contain =ybegin, =ypart, =yend lines.


btw.: I did spent some hours on the "RFC" process - and I am now
pretty sure that yEnc will never become an RFC. The actual version is
not sophisticated enough - and the "next version" must be far better.
So it would take years.


Good luck !
-- 
Juergen


    
    

    
  
Sorry, that's Helbing.

    
    

    
  
As for usage, over the last few months I've noticed a large increase in the
usage of Yenc.

Some people love it and some people hate it. It seems most of the people who
hate it can't view it with their software.

So adding support for Yenc into Mozilla would be a useful feature for all who
uses newsgroups.

    
    

    
  
People are using this a lot lately. Supporting it will make a lot of 
people happy - its as easy as that. Supporting it doesnt mean that you 
*have* to use it. Those whining about what a crap protocol this is should
first come up with a better solution *and* get it accepted at least as
widely as yEnc.
And ... dont forget why VHS won over Video2000 :)

    
    

    
  
If somebody have some spare time to implement a yEnc decoder in Mime, he/she is
welcometo do it... and I would be happy to review it.
Keywords: helpwanted

    
    

    
  
I would like to suggest moving this to a mozdev project.
Most news reading folks don't hunt binaries, and those that do probably want a 
way to combine multiple posts like yEnc supports.
Implementing the decoder is pretty easy, I have rudimentary code for a while.
Finding the right user interface is the tricky part, and I doubt that we can create
a UI that is so hidden that it doesn't distract/confuse non-binary users and at
the same time exposes the functionality that binary-hunters want.
Having a optional package like a sidebar or something sounds like the way to go
for me. And people able to get porn from news servers find mozdev, too.

(Btw, I implemented my code as protocol handler, so that works only for single
part (if at all, it's been some time since I looked at it))

    
    

    
  
I'm adding the project to Mozdev.org

    
    

    
  
As a huge newsgroup fan, I can give some insight into what's going on in the 
big yEnc/Anti-yEnc battle.  In nearly all large binary groups (those with 
movies, games, cd images, etc.), yEnc has pretty much 100% saturation, but in 
picture groups, UUencoding and base64 are still King b/c there is little 
incentive to switch.  Apparently, the difference between 5 hours and 3.5 hours 
is much more noticable than 5 seconds and 3.5 seconds ;-)  I would very much 
like Mozilla to support yEnc, as it is 30% or more faster for posts & RFC or 
not, it's being widely used.  Thanks!

    
    

    
  
Jonas, so what? That yEnc is widely used for some groups is a fact. Nobody
has come up yet with something that has only a fraction of the acceptance.
If Mozilla supports the format, as ugly and bad it might be, it will be 
a reason for many people to use it in the current situation, if not it will
be a reason to drop it. 

I would propose to restrict this article to issues concerning the fix to this
bug. Everybody is free not to use the feature once it is implemented.

    
    

    
  
You people are behaving as idiotic as some of the Usenet users, that's really a
shame. You say yEnc is bad, but every argument you use against it is also an
argument against UUEncode. Do you support UUEncode? Yes. Can users post with
UUEncode? Yes. So this is a completely moronic hypocrisy! If you were really
concerned about that, you should have never implemented UUEncode in the first
place. yEnc is pretty much like UUEncode, it just encodes data differently,
that's the only difference. Whether yEnc will ever become MIME or not, who
cares? Right now it's not MIME, it is used in some binary NGs *EXCLUSIVELY* and
people are angry because it is not there in Mozilla. I permantnly rant about how
retarded Micro$oft is, just to see you are as retarted as them.

All the Usenet clients are supporting it, the only ones that are not is Outhouse
Excess and the Mozilla/Netscape client. That throws a very poor light on a
client that is otherwise known to always support the latest standards (like the
latest HTML/XML/CSS, picture formats, etc.) and it's a good reason to not use
Mozilla as news client in the first place.

    
    

    
  
http://yenc.mozdev.org/

I have been on vacation and haven't gotten around to creating the content such 
as goals and plan yet. As soon as I do that, I will announce the project, and 
we can get going on this. I need to talk to a few people on how best to 
implement this first, though.

    
    

    
  
What's the latest on the development of yEnc compatability in Mozilla?  yEnc 
has become the standard for posting binary files on usenet, save a few picture 
groups where they are slow to switch due to the number of Outlook Express, AOL, 
and WEBTV users that like to surf for pics.  (in other words, dial-up users 
largely that don't see much improvement with yEnc b/c they d/l small files one 
at a time anyway & they use **** software).  

    
    

    
  
*** Bug 175428 has been marked as a duplicate of this bug. ***

    
    

    
  
#30: that the majority of Usenet readers have implemented something does NOT
make it a standard. You shouldn't just do something because it can be done (or
has been done already). It exactly that way of thinking that Mozilla is trying
to combat: if you do something, do it right, if there is a standard you can use,
use it, if not, make one.

    
    

    
  
re comment #34 : Sorry, but this is just rubbish - a newsreader is a program that
should enable people to read news. If mozilla wants to provide functionality for
this it cannot ignore something that is extremely widely used. I doubt that 
Mozilla is designed as a tool to force people to "do it the right way" whatever
that should be. Dont forget that even Mozilla programmers might have varying views
of what "the right way" is, and it might become even more varied when we look
at the view of potential users. But this is a rather academical discussion,
because forcing "the right way" or "some way whatsoever" on people can only
be done by a product that is very widely used - and therefore definitely
not by mozilla. So please dont patronize people and their way how to do things. 
You are free not to use yenc as long as you wish even if it were implemented in 
mozilla. This is tool, not a bible.

    
    

    
  
leaving this up for the trolls.
I read this bug as a "fine as a mozdev project", and there it is.
If you want it, go there, contribute code.

    
    

    
  
mozdev has an yenc component. 
Also, there are newsgroups that try to combat yenc encoding.


    
    

    
  
I wouldn't say there are groups where people are "combatting" it so much 
as there are groups where OE is the primary newsreader and they don't 
like it because OE doesn't support it.  Of course there are some objecting 
because they are using Netscape and it doesn't support it either, but 
Netscape users seem (to me) more willing to either install the proxy 
server or use a different program.

    
    

    
  
Mind you, the author of the first link you posted
(http://www.exit109.com/~jeremy/news/yenc.html) says he has moved on from when
he wrote the article, and just accepts that yEnc is the predominant scheme for
encoding binaries on Usenet.

I'd say that we should support yEnc now, since it's the current (defacto)
standard, and that if and when someone writes something better than can be used
in more than just Mozilla, we can support that as well. I don't see how
supporting every encoding standard that comes along can be construed as a bad
idea: it lets the users make the choice as to what they want to use.

    
    

    
  
Well, one could argue that for Mozilla we might only want to use "real"
standards, standards that have passed through and approved by a standards body.
Look where the first years of HTML's unlimited growth has led us: the leading
browser in the world still doesn't supoort all the standards and no browser
seems to render exacty the same.
I don't say that this is the same ball park and supporting yEnc will inevitably
lead to problems but I do think that saying "it's a defacto standard" or
"everybody does it" is not reason enough to go and add it to the core.
But one thing I do know "just supporting every 'standard' that comes along" will
get us into trouble (and we would probably have to rename our Mozilla monster to
Hydra).

But.... realisticly speaking I do think we can hardly get around implementing
yEnc in some way or the other.

    
    

    
  
Okay, I agree with your comments on the general disarray of standards right now,
I was thinking more in terms of just simple encoding options (for instance, like
how file compression tools will support arj, rar, zip, sit, etc), we should try
to implement any and all encoding/decoding schemes that won't 'cause problems
for other parts of the browser. Sure yEnc isn't the best thought out scheme, but
I haven't seen any arguements saying that including support is going to break
other things in the browser.

    
  
Alias: yEnc

    
    

    
  
If someone wants to take http://yenc.mozdev.org/ just post a preliminary patch 
in this bug before I start working on it (which won't be for a long while) and 
its yours.

    
    

    
  
It's been a year since this bug was posted, so why isn't Mozilla yEnc compatable
yet?  Most news clients switched to yEnc within a few months of its appearance,
so I assumed that this would be implimented rather quickly.   Nearly all files I
download on Usenet use yEnc.   I've been suggesting Xnews (free download from
http://xnews.newsguy.com/ ) as an option for those who don't care to pay for the
agent version which supports yEnc and don't want to use yProxy for Outlook
Express... from what I have seen, most people are using Xnews now in the groups
I visit.   I don't see the purpose of having a newsreader bundled with mozilla
if it's not capable of downloading binaries.   Sooner or later, yEnc will likely
be the only posting format on Usenet.   The only place I see MIME or UUE is in
picture groups these days, and it's vanishing there quickly.

    
    

    
  
Wayne: Many news readers also have combine and decode abilities, yet Mozmail
doesn't. Multipart news messages have been around a lot longer and aren't
controversial. Therefore, there is no developer interest at the moment among the
main developers to implement this. If you want this done, find someone who
doesn't have many patches to write to make a preliminary patch and attach it
here and I will help them get set up using the http://yenc.mozdev.org/ Mozdev
project for a Mozilla extension.

Axel: Since the code for encoding is in c++, is there any way to make a c++ xpi
plugin?

    
    

    
  
So is anyone actually finally working on yenc support?

    
    

    
  
Not that I know of. If anyone wants to work on it, give me a buzz and I'll help
you get set up using: http://yenc.mozdev.org/

    
    

    
  
I have a basic implemetation of a yenc decoded in Mozilla. It doesn't support
multipart. It does not check the crc or the size of the data as this is a real
time stream decoder which does not bufferrize the data. The time we detect
something was wrong, the decoded data has been already pushed to the image
library and displayed to the user.

I'll cleanup my patch and post it soon...

    
    

    
  
Jean: You the man!

Do you want the Yenc mozdev project space I'm squatting on, or is your intent to
give what you've done so far so someone else can complete it?

        Attachment #64985 -
        Attachment mime type: application/octet-stream → text/plain

        Attachment #64985 -
        Attachment mime type: text/plain → message/rfc822

    
    

    
  

      
      
      
      

        Attached patch
          
          
        Proposed fix, v1 (obsolete)
        — 
          — Splinter Review
      

    


  
Keywords: helpwanted
Whiteboard: have fix

        Attachment #115174 -
        Flags: superreview?(sspitzer)

        Attachment #115174 -
        Flags: review?(cavin)

    
    

    
  
So, does this mean the next version of Mozilla will be able to display yEnc files?
The darn things seem to be everywhere..

        Attachment #115174 -
        Flags: review?(cavin) → review?(ssu)

    
    

    
  
would be nice if it make it... Reassign to ssu
Assignee: ducarroz → ssu
Status: ASSIGNED → NEW

    
    

    
  
(disclaimer)I'm not a reviewer(/disclaimer) I looked over the patch and it looks
clean to me.  Personally, I like how it makes the MimeDecoder more generic and
removes the assumption that UUDecoding is the only form of embeded text encoding.

Nice work Jean-Francois!

Any ideas on how to increase the visiblity for this?
It is a problem because it is not assigned and has no target milestone?  1.4b
seems like a reasonable milestone now that we have a patch....

I along with many others have been waiting patiently (a month and a half) for
this patch to be commited (or even formally reviewed).  I'm sure many people
would like to give it a spin.  What do we need to do to get this on the radar
for a r/sr?

Thanks again to Jean-Francois.

    
    

    
  
1)

+        if (out[-1] == nsCRT::CR || out[-1] == nsCRT::LF)

out is a char, and nsCRT::CR / ::LF are PRUnichar 

2a)

+		  if ((endOfLine - line) >= 13 && !nsCRT::strncmp (line, "=ybegin line=", 13))

use strncmp() instead of nsCRT::strcmp()

2b)

instead of 13 and "=ybegin line=", let's see some #defines:

#define YENC_BEGIN_LINE "=ybegin line="
#define YENC_BEGIN_LINE_LEN 13

3)

+        c= (unsigned char)*src;
+        if (c == 0x0A || c == 0x0D || c == 0x00)
+          break;

why cast to unsigned char?  those are \r \n and null.

4)

+	  PR_ASSERT(!uty->open_subpart);

remember, PR_ASSERT() calls abort() in debug builds.  is that what you want?  I
recommend NS_ASSERTION()

5)

+  /* take off newline. */
+  if (name[strlen(name)-1] == nsCRT::LF) name[strlen(name)-1] = 0;
+  if (name[strlen(name)-1] == nsCRT::CR) name[strlen(name)-1] = 0;

why not this?

int lastCharPos = strlen(name)-1;

if (name[lastCharPos] == '\n')
  name[lastCharPos] = 0;
else if (name[lastCharPos] == '\r')
  name[lastCharPos] = 0;

also, do we have to worry about both CRLF together??
Assignee: ssu → sspitzer

    
    

    
  
sorry, I take it back, nsCRT::CR and LF are not PRUnichar.

from nsCRT.h

class NS_COM nsCRT {
public:
  enum {
    TAB='\t'  /* Horizontal Tab */,
    LF='\n'   /* Line Feed */,
    VTAB='\v' /* Vertical Tab */,
    FF='\f'   /* Form Feed */,
    CR='\r'   /* Carriage Return */
  };

I'll take the patch, and drive it in.

    
    

    
  
doh!

we need it like this:

  if (name[strlen(name)-1] == nsCRT::LF) name[strlen(name)-1] = 0;
  if (name[strlen(name)-1] == nsCRT::CR) name[strlen(name)-1] = 0;

that works for CR, LF and CRLF

notice that we'd string the LF first, then the CR.

it's important to have the strlens(), and the length of the string can change!


        Attachment #115174 -
        Flags: superreview?(sspitzer)

        Attachment #115174 -
        Flags: review?(ssu)

    
    

    
  
this works, but I get assertions.

###!!! ASSERTION: nothing to write!: 'dest >= line && dest < src', file c:/trees
/mozilla/mailnews/mime/src/mimeenc.cpp, line 734

NS_ASSERTION(dest >= line && dest < src, "nothing to write!");

when I assert, dest == src.

ducarroz, any ideas?

note, as jfd explained, this only works for messages that are complete (not 
parts).

here's what I used to test:  
news://news.mcom.com:119/kgsp8vcj8na2f0nsudr229h9ma7uapvh3q@4ax.com

after I reviewed, tested (and landed), I realized that kaie has the strongest 
mime fu since ducarroz left.

kaie, can you review?  if you find any issues, we'll fix them.
Target Milestone: Future → mozilla1.4beta

    
    

    
  
> since ducarroz left

I should say, since ducarroz no longer works on mozilla, full time.

but he still reviews and hacks!

    
    

    
  
1.)
===

+  /* take off newline. */
+  if (name[strlen(name)-1] == nsCRT::LF) name[strlen(name)-1] = 0;
+  if (name[strlen(name)-1] == nsCRT::CR) name[strlen(name)-1] = 0;
+
+  /* Now try and figure out a type.
+   */
+  if (opt && opt->file_type_fn)
+	type = opt->file_type_fn(name, opt->stream_closure);
+  else
+	type = 0;
+
+  if (name_ret)
+	*name_ret = name;
+  else
+	PR_FREEIF(name);
+
+  if (type_ret)
+	*type_ret = type;
+  else
+	PR_FREEIF(type);

What about this instead? Can save a few cycles.

  /* take off newline. */
  size_t name_len = strlen(name);
  if (name[name_len-1] == nsCRT::LF) name[--name_len] = 0;
  if (name[name_len-1] == nsCRT::CR) name[--name_len] = 0;

  /* Now try and figure out a type.
   */
  if (type_ret)
    if (opt && opt->file_type_fn)
	  *type_ret = opt->file_type_fn(name, opt->stream_closure);

  if (name_ret)
	*name_ret = name;
  else
	PR_FREEIF(name);


2.)
===
+  char *line_end = data->line_buffer + data->line_buffer_size - 1;

Please declare as "const char*".


3.)
===
Let's declare a const for the 1000 == BUFFER_SIZE

Is the 997 value derived from the 1000 ?
Please use the constant if it is.


4.)
===
+    {
+      char *out = line + strlen(line);

It took me a while to understand why you are doing this.
Please add a comment where you say, this is necessary, because the buffer
might contain partial data from a previous call to the function.


5.)
===
Since we are not implementing an official standard,
what about attaching the draft of what you are implementing to this bug,
and add a comment to the sourcecode where you refer to the bug number?
Are you implementing this: http://www.yenc.org/yenc-draft.1.3.txt ?


6.)
===
+            if (new_line_size > data->line_buffer_size && new_line_size <= 997)
/* don't let bad value hurt us! */
I think you want to protect yourself from allocating a large block of memory.
But what happens in the case where new_line_size is indeed > 997?
Won't your code continue to process the data and write to a buffer that is too
small and corrupt memory?
Should processing be stopped in that situation?


    
    

    
  
And a question out of curiosity: The drafts I found do not mention "x-yencode",
where did you find that?


    
  

        Attachment #120031 -
        Flags: review?(kaie) → review-

    
    

    
  
yencode is not yet a standard. Therefore I choose to use the prefix "x-".
Anyway, this is only used internaly by Mozilla, could have been
"x-no-more-yencode-complain"

    
    

    
  
I don't know about the assert! does it happens only once per image! maybe it's
at the end of the decoding and this is a valid case where we should not assert?

    
    

    
  


  
Many yenc attachments are a single part but have a multipart format.
This patch against CVS adds support for these attachments..
It also handles part 1 of a multipart post (but this is just a side
effect).
This is NOT a complete multipart handler,
just a quick patch intended for single part attachments
which include the 'part=1' tag.

    
    

    
  
So, did the patch make it into 1.4b?  I installed the beta, but I don't see it.

    
  
Blocks: majorbugs

    
    

    
  


  
This does pretty much the same as Mark's patch (which I saw after I started on
this).	It ignores the additional multipart headers & part=val pairs.  It also
will ignore any future (and therefore unsupported) yenc directives by ignoring
all lines that start with =y except =yend.

    
  
Blocks: 212453

    
    

    
  
*** Bug 212453 has been marked as a duplicate of this bug. ***

    
    

    
  
Regarding comment 67, either bug 212453 is not a duplicate
or the description of this bug needs to change, I think.
It seems that the encoding is already supported just not
multipart.

It seems we should mark this bug as fixed and reopen
(undup) bug 212453 OR change the description of this bug to
mention multipart messages OR I'm confused.


    
    

    
  
Am I incorrect in saying that none of the patches attached to this bug have been
reviewed/super-reviewed/checked in? Bug 212453 comment 2 seems to indicate
otherwise.

If it is the case that the patches are not checked in, it would be a mistake to
mark this bug fixed, as it isn't fixed :)

In any case, multipart messages are part of the yEnc spec - if you don't support
multipart, you don't support yEnc. Thusly, multipart decoding is implicit in the
title "support new USENET encoding -- yEnc'. Having one bug for one part of
yEnc, and another for another part seems a bit unnecessary.

    
    

    
  
What is the difference between multipart yEnc and any other multipart message?
We don't handle any multipart, isn't that correct? That is covered in bug 60981,
right?

    
    

    
  
Problem is not that Mozilla can not handle multipart yencoded messages, but
problem is that many and many of singlepart yencoded messages encoded in
miltipart format (see more in comment #64
http://bugzilla.mozilla.org/show_bug.cgi?id=119964#c64).

    
    

    
  
So basically to the user, a multipart yEnc message should appear as a single
message header? In that case we will only grab the first part and the rest will
be ignored without the user knowing?

This is different than a split file sent in multiple postings that the user
would see like 100 headers and have to select something akin to "Combine and
decode" like in OE or "Join" in Agent, right? What I mean is that though it
would be sent as a multipart, it should be reconstructed and transparent to the
user, right?

I'm all for fully implementing a "standard" instead of partial. What additional
steps will it take?

    
    

    
  
Brian, I think what Alexander was getting at in comment 71 was that with yEnc,
it's quite possible to create a multipart message with one part, delivered as
one message. Some clients create multiparts as a matter of course. This is quite
valid according to the spec (http://www.yenc.org/yEnc-draft-1.txt). So even if
mozilla can't join message parts together automatically, it should be able to
parse yEnc's multipart format.

Attachment 121855 [details] [diff] allows mozilla to deal with such messages.

A single part multipart message looks something like this:

=ybegin part=1 line=128 size=100000 name=mybinary.dat
=ypart begin=1 end=100000
.... data
=yend size=100000 part=1 pcrc32=abcdef12 crc32=abcdef12

    
    

    
  
Ok, I think I get. So how would that appear to the user? In this case, it will 
only appear as one header to the user but still have the multipart as we see 
it (which we ignore), but if its a true multipart message, it will appear as 
multiple headers, with the main one looking correct but the rest looking like 
garbage to the average user?

Two examples: 
1 multipart message sent as 1 message - this will appear correctly
1 multipart message sent as 5 messages - we can't deal with this and the user 
will get 5 headers. Let's say its a .mpeg file, then they will only see the 
first 10 K (assuming a message is 10K long in this example), and therefore, 
its an .mpeg, they will only have the first 10K of the movie.

Is this correct?

    
    

    
  
The situation with these bugs is rather untidy.

This bug was for implementing yEnc, which has been done, so I guess it should be
marked fixed.  Bug 60981 is about combining multipart messages.  Bug 212453 was
opened for multipart yEnc but then duped to this.

Will fixing bug 60981 make multipart yEnc work?  If so, then bug 212453 is a
dupe of that bug, not this one.  If not, then we should either morph this bug
into being about multipart yEnc, or reopen bug 212453.

    
    

    
  
Fixing bug 60981 will not make multipart yEnc work.

The terminology surrounding multipart and yEnc is a bit ambiguous - simple
protocols (UUE, for instance) produce multiparts by generating a single part
encoding, and then splitting it amongst several messages. More complex (and
robust) protocols, such as yEnc and MIME, produce a multi-part encoding, which
is then posted, (often with) one part per message. 

Bug 60981 is concerned with automatically concatenating messages together; With
simple protocols, this will produce a single-part message. With protocols that
understand multipart, it will produce a single stream containing a multipart
encoding of several parts. As it stands, mozilla cannot parse these for yEnc,
only multipart yEnc encodings with a single part. 

To implement yEnc, a client must be able to parse both the single-part and
multipart encodings, as it is not illegal (merely unusual) to post multipart
yEnc coding (even with multiple parts) in a single message.

Bug 212453 was opened for the multipart yEnc encoding, however yEnc is not a
layered standard like MIME, MNG or MP[1-3]. It is not a valid implementation of
the standard to implement only a subset (such as single-part decoding). 

So, in my opinion, bug 212453 is rightly marked as a duplicate. Bug 119964
should not be marked fixed until mozilla can parse multipart yEnc (and anything
else in the spec that doesn't work yet), contained within a single stream 
(message body). If mozilla can do this, then a fix for bug 60981 will
automatically work for yEnc as well.

    
    

    
  
ok, cool. So we're clear that this bug is for multipart as well.
No longer blocks: 212453
Summary: RFE: Support new USENET encoding -- yEnc → Support yEnc encoding (including multipart)
Whiteboard: have fix

    
    

    
  
Sounds like we should keep the existing code in the tree hush-hush then and
continue onto the multipart before people notice :-)

    
    

    
  
I'm curious... It seems that people sometimes post split files in yEnc (i.e.
yEnc of pre-split files). Those would fall under single part files that would
need to be joined using bug 60981 and would still work OK, right?

It seems there are 3 ways a very large file could be posted:
- Pre split files posted individually in multiple messages of any encoding - bug
60981
- Server split messages of one large posting - would be joined via bug 60981,
then handled via the patch in this bug
- One large file that either was reconstructed by the server, or never split by
any server - would be fully handled via the patch in this bug

Did I miss anything? Sorry about the questions... I knew the answers to these
last year, but since have forgotten them or gotten confused. I'm not asking you
via private mail since ths can help other people. Usenet is one hell of a hacked
"standard" if you ask me :-)


    
    

    
  
I hope by 'Server split' you're not implying that the NNTP server is cutting the
message up? When an NNTP server encounters a message that is too large, it
doesn't split it, it throws it in the bin. Streams are always split into
messages by the client before being passed to the server.

There are several ways that a large file can be posted (from my memory,
apologies for omissions and inaccuracies) :

--- decodable at the moment ---

- Encoded as a single chunk using dumb protocol (UUE, base64) , chunk posted in
one message body.

- Encoded as a single chunk using MIME, chunk posted in one message body.

- Encoded as a many chunks using MIME, chunks all posted in one message body.

- Encoded as a single chunk using single part yEnc, chunk posted in one message
body.

--- decodable if message bodies are joined together (bug 60981) ---

- Encoded as a single chunk using dumb protocol (UUE, base64) , chunk split into
regular sections, and posted as several message bodies.

- Encoded using MIME, split into sections, and posted as several message bodies.
(as mozilla includes a full MIME parser. I think.)

- Encoded as a single chunk using single part yEnc, chunk split into regular
sections, and posted as several message bodies. (No client on earth posts like
this, and would probably end up on the 'broken tools' list if it did.

--- decodable using attachment 121855 [details] [diff] [review] ---

- Encoded as a single chunk using multipart yEnc, chunk posted in one message
body. (Some broken tools post like this)

--- not decodable with anything attached to bugzilla ---

- Encoded as as more than one chunk using multipart yEnc, chunks posted in same
message body (nothing posts like this)

- Encoded as as more than one chunk using multipart yEnc, chunks posted in
separate message bodies (almost every tool posts large files like this)


As it stands, mozilla cannot parse multipart yEnc messages with more than one
part, even if every part is in the same message body. Joining the messages
together (bug 60981) is not enough for multipart yEnc support, as mozilla must
also parse the structure of the multipart yEnc stream and decode the file. In
excatly the same way, joining all the message bodies together is not sufficient
to decode multipart MIME; you must also understand the structure of the encoding
method. Mozilla's MIME parser can understand MIME in multipart mode; Mozilla's
yEnc parser does not yet understand yEnc in multipart mode.

BTW, the term is "de-facto standard" :)

    
    

    
  
There is one other that you forgot (I'm just providing this for anyone reading)
that should be covered by bug 60981 -- pre-split files that are split as follows:

foo.bar.1
foo.bar.2
foo.bar.3
...

and then posted and individually encoded. We can handle such cases by allowing
people to select each attachment (or perhaps contect-click one file and select
"join similiar"), and choose "join" which would provide a list of files,
pre-sorted by filename, and allow the user to move the ordering around. Then it
would individually encode each attachment and combine it. Result would be:

foo.bar

I've never seen this done by a client, but there are tools out there for posting
of binaries for people who don't want to sit there while its done that do this
kind of thing and its pretty common in some groups (each group has its own rules).

    
    

    
  
There have been no new comments in four, almost five, months, so I thought I'd
ask... where do we stand with yEnc so far?  I have no coding skills whatsoever
or I'd help out with this myself, as I know it is stopping at least one person
from switching to Mozilla Mail and/or Mozilla Thunderbird.

    
    

    
  
yenc support in Mozilla 1.6 is almost useless, as about 50% of yenc posts use
the part 1 of 1 multipart format ("=ybegin part=1") which Mozilla just displays
as text. The patch has been posted here, so why can't it be checked in to the
code so that it will be included in the next binary release?

    
    

    
  
patches can't be checked in until they've been successfully reviewed. As far as
I can see nobody has even requested a review on the newer patches yet. So what's
needed is for someone to check that the patches still work, request reviews and
then do any necessary follow-up work.

    
    

    
  
*** Bug 258828 has been marked as a duplicate of this bug. ***

    
    

    
  
OK I just tried attachment 121855 [details] [diff] [review] and it still works in Thunderbird, on single
posts encoded using multipart yEnc. This is a big step in the right direction
(although not supporting >1 part), and I think it should be applied.

Who should review this?


    
    

    
  
Comment on attachment 127193 [details] [diff] [review]
yet another robustness patch

Seth, who isn't reading bugmail.

        Attachment #127193 -
        Flags: review?(sspitzer)

    
    

    
  
Andy: Can  you try attachment 127193 [details] [diff] [review], also? It's newer.

    
    

    
  
attachment 127193 [details] [diff] [review] applies and works just like the other one.

    
    

    
  
Andy: Did you send an email to the newsgroups?

    
    

    
  
Will do

    
    

    
  
Comment on attachment 127193 [details] [diff] [review]
yet another robustness patch

passing the review buck to david b.

        Attachment #127193 -
        Flags: review?(sspitzer) → review?(bienvenu)
Product: MailNews → Core
Assignee: sspitzer → bienvenu

    
    

    
  
Comment on attachment 127193 [details] [diff] [review]
yet another robustness patch

+  if ((endofline - s) < 5 || strncmp(s, "line=", 5)) return PR_FALSE;


can you put the return on its own line? That's the style we use, mostly so you
can set breakpoints on the return. Other than that, it looks OK.

        Attachment #127193 -
        Flags: superreview?(mscott)

        Attachment #127193 -
        Flags: review?(bienvenu)

        Attachment #127193 -
        Flags: review+

    
    

    
  
Comment on attachment 127193 [details] [diff] [review]
yet another robustness patch

modulo david's comments

        Attachment #127193 -
        Flags: superreview?(mscott) → superreview+

    
    

    
  
*** Bug 276826 has been marked as a duplicate of this bug. ***

    
    

    
  
I think my original report that Thunderbird did not support yEnc decoding was in
error.  The problem really appears to be that Thunderbird does not identify
posts that are actually incomplete multipart posts.  I checked some posts that
failed to decode under Thunderbird with my main email client Forte Agent and
that identified them as incomplete posts (so you know you cant decode them) with
an "incomplete" icon.   When all parts are available on the server, Agent
changes the icon to a "complete" icon.  Posts with "complete" icons are
automatically decoded by agent when downloaded.   Thunderbird does not appear to
identify incomplete multipart posts on the server so downloading the post just
shows garbage.  Using Agent I identified a couple of complete yEnc encoded posts
and then downloaded them using Thunderbird.  They decoded OK.  So can
Thunderbird identify incomplete multi-part posts on the server so you know not
to download them yet?

    
    

    
  
*** Bug 280850 has been marked as a duplicate of this bug. ***

    
    

    
  
*** Bug 277641 has been marked as a duplicate of this bug. ***

    
  
No longer blocks: majorbugs

    
    

    
  
I have never seen a more moronic response since the beta-max/VHS wars. It doesn't matter that YENC is not a formal standard. When you have easily 90 percent of all binaries encoded in yenc in the news groups, it is completely idiotic to not support it in thunderbird. I use FireFox for my browser, but will not use thunderbird for this specific reason. To the "purists" developers at mozilla, please grow up. You are cutting off your nose to spite your face. Yenc is NOT going away in the near future and you are deliberately stunting the growth of the thunderbird product. The attitude that you can’t support every protocol that comes along is just silly. That is like saying that you won’t support personal computers 30 years ago because only mainframes were “real” computers.

    
    

    
  
William: Mostly everyone agrees it would be a very nice-to-have and that it's a de-facto standard. However, the latest patch is from 2003, without multipart support (as far as I know). If you are interested in getting this fixed, please provide a patch or find someone who has time to write one. Obviously, without a patch, it's not going to happen. The developers are all very busy, lots of times on security issues and things like that which are all higher priority than yEnc support. However, it would be a nice thing to have.

Please don't abdicate in bugs (a lot of people already mirror your views) but instead voice your concerns about these things on the Mozillazine forums.

    
    

    
  
Abdicate wasn't the word I was looking for. Please don't make aggressive statements like that, attacking developers you don't really know, and saying things that have already been said in other comments in the bug. 

    
    

    
  
My apologies, I was frustrated with 20 Yenc attachments in a row failing to decode. I originally looked in Mozillazine  and they had one message that said, in effect, nothing. I discovered this bug and vented. It won't happen again.

    
    

    
  
- Mozilla 1.4 beta is long gone => Resetting target milestone.
- David, are you still interested in working on this bug? If you do, please take it again (I suppose esther-at-formerly-netscape is not a valid QA anymore).
Assignee: bienvenu → nobody
QA Contact: esther → mime
Target Milestone: mozilla1.4beta → ---

    
    

    
  
heippa 
noniina@gmail.com  
taasen

This might not be legal but anyhow -: 
Btw part of the linux code was actually "copied" from ddj about 1988-1990 by ...

Some approach would be using part of winvn yenc source c/c++ source
Try ex google


    
  
Blocks: 339983
Product: Core → MailNews Core

    
    

    
  
This bug is 9 years old, it has not been fixed up to now and lets face it, it won't ever gets fixed because nobody wants to work on it or maybe the Thunderbird developers are happy with the fact that Thunderbird is an inferior news client. So why don't we just close this bug. Pretty much all people seriously using Usenet have most likely stopped using TB years ago (just like I did) because it's pointless. I suggest setting status to resolved => won't fix.

    
  
Severity: normal → S3

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: