Open Bug 119964 (yEnc) Opened 23 years ago Updated 2 years ago

Support yEnc encoding (including multipart)

Categories

(MailNews Core :: MIME, enhancement)

Product:
MailNews Core
Component:
MIME
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: John.planb, Unassigned)

References

(Blocks 2 open bugs,
URL
)

Details

Attachments

(6 files, 1 obsolete file)

Article that used yEnc encoding
48.05 KB, text/plain
Details
A decoded version of the yenc-example
46.23 KB, image/jpeg
Details
yEncoded article posted to newsgroup
48.05 KB, message/rfc822
Details
jfd's patch, plus some minor nits
24.07 KB, patch
KaiE
: review-
Details | Diff | Splinter Review
patch for part 1 of multipart yenc
1.74 KB, patch
Details | Diff | Splinter Review
yet another robustness patch
3.12 KB, patch
Bienvenu
: review+
mscott
: superreview+
Details | Diff | Splinter Review
There's a new encoding format (similar in design to UU but without it's 30% overhead, and with built in error checkign) being batted around and adopted by a few newsreaders. It's an 8 bit encoding (minus a few escaped characters; CR, LF, Nul), and is thus allows faster up/downloading and is less of a hit on the servers (allowing them to carry more articles).
Severity: normal → enhancement
Has word of this reached USEFOR yet?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Can somebody attach a sample message using this encoding. Also, would be nice to have the corresponding RFC. Thanks
Status: NEW → ASSIGNED
Target Milestone: --- → Future
It was brought to USEFOR's attention, and the consensus there was that encoding methods weren't to be described by USEFOR. There is no RFC, just the website and the programs using it (although it was discussed in news.software.nntp).
I don't think the previous attempted worked (I tried to use plain-text and it looks like the text was "normalized").
Doesn't look like the latest attempt to upload the attachment worked either (or at least I can't see how to download it without it having been changed). Thoth and Xnews can post yEncoded binaries, so if anyone wants test articles posted to a specific group just ask (or has some idea as to how I could upload a non-image binary file that doesn't get trashed by bugzilla) just email me.
*** Bug 128837 has been marked as a duplicate of this bug. ***
Please support this: some porn is in yEnc form. :-)
yEnc multipart postings should also be supported http://bugzilla.mozilla.org/show_bug.cgi?id=60981
http://groups.google.com/groups?q=yEnc&hl=en For some juicy discussion on yEnc
I suggest this be WONTFIXED. We shouldn't be encouraging people who devise crap protocols like this without bothering to correct important deficiencies in them, or to take the 6 months or so to work with the IETF and make them actually interoperable. (Yes, I'm biased. OTOH, the author of the document referenced in Comment #12 certainly knows more about handling binaries on Usenet than any of us, and probably more than the author of yEnc :-) Bottom line: encouraging the fast-and-sloppy approach to protocols sticks in my craw.)
If you aren't planning on implementing this, throw it over to me. yEnc looks like a pretty simple standard. As far as how to implement encoding schemes in MailNews, I can probably figure it out. Any info would be helpful.
Chris: yEnc is still on sub version 2. I imagine it can be improved in future versions. I'm glad someone had the guts to do this, although I haven't liked not being able to see binaries anymore on Outlook Express. If mozilla supports this and also bug 60981, I imagine we can pull the rug out from under Outlook Express if they are slow in implementing it.
/. article on yEnc. http://slashdot.org/article.pl?sid=02/03/23/2154235 Jean, if you don't have time to implement this - I'll take it and either do it myself or find someone to do it.
Remember this URL? http://www.exit109.com/~jeremy/news/yenc.html "If you agree with me, what can you do to help? If you are the author of a Usenet newsreader, you probably have to implement yEnc at least for decoding, but you can leave out posting support to try to prevent this from spreading any more. At the very least, if a MIME yEnc specification arrives but bypasses the standards process, please, don't implement it. If you are a user, you can refuse to post in yEnc." I believe that he has a good idea here, as the proper place for the new 8-bit encoding is MIME. I propose we only provide reading capabilities.
I may have underestimated the demand for the need to encode yEnc. It seems that a lot of people are looking for this feature. Upon reflection, it appears to me that not including it might be a bad idea. We can always rip it out once MIME is made 8-bit.
Well, if you're interested in usage -- the server I use has various virtual groups, based upon article content, the group with formfeeds has gone from less than a thousand articles a day to more than 500 thousand articles a day. So, I'd say that currently the posting volume is about 500,000 articles a day in yEnc format, and increasing as new clients are able to handle it.
Here is a quote from Juergen Helbring, creator of yEnc in reply to my email. (He gave me permission): The eMail I've received right before yours was from a Netscape user (Mac) who wanted to post and receive yEncoded posts. I told him to send a feature request to Netscape. More and more people will follow. On the "classical" music group there is meanwhile 100% of the binaries in yEncoded format (They were the first who started yEnc'ing in January 2002 - 12 weeks ago). For Windows there are enough tools available to post binaries in yEnc. Other platforms are not so well supported. Implementing also a yEncoder might help people there. If you need a _good_ reason why to implement also "encoding" - and sending: There is no sign that any other binary format than yEnc will be created in the next few years (and I believe even in the next decade). Those who are complaining most have not a single hour of time to do constructive work on their own proposals. Even my biggets hope (Russ Albery - who said me he would write the doc for "Content-Transfer-Encoding: yEnc" finally gave up and said that he does not have the time do it - and he said that he does not have the knowledge to do it). Decide yourself what you are doing: If you implement also sending (encoding) then the job is quickly - and easily done. You'll have peace for the next few years. If you dont do it now, then you will do it later. All the "formal work" related to a change of the software will be necessary again - and I am sure that also for you the "overhead" of creating a new version is far more work than the 2-5 hours it will take to expand the encoding/posting functions by yEnc. If there would have been a _single_ step by the anti-yEnc'ies - a single announcement, a single idea when anything else would be done, then I'd say: Forget about yEnc. (I would have deleted yEnc instantly and used something better). But I see no chance for alternatives from their side. So it is all up to you. If you want a "tip" for the implementation: I would recommend to implement the de/encoder in a way that it is a module which could be also used if yEnc comes as a CTE: x-yEnc It might contain =ybegin, =ypart, =yend lines. btw.: I did spent some hours on the "RFC" process - and I am now pretty sure that yEnc will never become an RFC. The actual version is not sophisticated enough - and the "next version" must be far better. So it would take years. Good luck ! -- Juergen
Sorry, that's Helbing.
As for usage, over the last few months I've noticed a large increase in the usage of Yenc. Some people love it and some people hate it. It seems most of the people who hate it can't view it with their software. So adding support for Yenc into Mozilla would be a useful feature for all who uses newsgroups.
People are using this a lot lately. Supporting it will make a lot of people happy - its as easy as that. Supporting it doesnt mean that you *have* to use it. Those whining about what a crap protocol this is should first come up with a better solution *and* get it accepted at least as widely as yEnc. And ... dont forget why VHS won over Video2000 :)
If somebody have some spare time to implement a yEnc decoder in Mime, he/she is welcometo do it... and I would be happy to review it.
Keywords: helpwanted
I would like to suggest moving this to a mozdev project. Most news reading folks don't hunt binaries, and those that do probably want a way to combine multiple posts like yEnc supports. Implementing the decoder is pretty easy, I have rudimentary code for a while. Finding the right user interface is the tricky part, and I doubt that we can create a UI that is so hidden that it doesn't distract/confuse non-binary users and at the same time exposes the functionality that binary-hunters want. Having a optional package like a sidebar or something sounds like the way to go for me. And people able to get porn from news servers find mozdev, too. (Btw, I implemented my code as protocol handler, so that works only for single part (if at all, it's been some time since I looked at it))
I'm adding the project to Mozdev.org
As a huge newsgroup fan, I can give some insight into what's going on in the big yEnc/Anti-yEnc battle. In nearly all large binary groups (those with movies, games, cd images, etc.), yEnc has pretty much 100% saturation, but in picture groups, UUencoding and base64 are still King b/c there is little incentive to switch. Apparently, the difference between 5 hours and 3.5 hours is much more noticable than 5 seconds and 3.5 seconds ;-) I would very much like Mozilla to support yEnc, as it is 30% or more faster for posts & RFC or not, it's being widely used. Thanks!
Jonas, so what? That yEnc is widely used for some groups is a fact. Nobody has come up yet with something that has only a fraction of the acceptance. If Mozilla supports the format, as ugly and bad it might be, it will be a reason for many people to use it in the current situation, if not it will be a reason to drop it. I would propose to restrict this article to issues concerning the fix to this bug. Everybody is free not to use the feature once it is implemented.
You people are behaving as idiotic as some of the Usenet users, that's really a shame. You say yEnc is bad, but every argument you use against it is also an argument against UUEncode. Do you support UUEncode? Yes. Can users post with UUEncode? Yes. So this is a completely moronic hypocrisy! If you were really concerned about that, you should have never implemented UUEncode in the first place. yEnc is pretty much like UUEncode, it just encodes data differently, that's the only difference. Whether yEnc will ever become MIME or not, who cares? Right now it's not MIME, it is used in some binary NGs *EXCLUSIVELY* and people are angry because it is not there in Mozilla. I permantnly rant about how retarded Micro$oft is, just to see you are as retarted as them. All the Usenet clients are supporting it, the only ones that are not is Outhouse Excess and the Mozilla/Netscape client. That throws a very poor light on a client that is otherwise known to always support the latest standards (like the latest HTML/XML/CSS, picture formats, etc.) and it's a good reason to not use Mozilla as news client in the first place.
http://yenc.mozdev.org/ I have been on vacation and haven't gotten around to creating the content such as goals and plan yet. As soon as I do that, I will announce the project, and we can get going on this. I need to talk to a few people on how best to implement this first, though.
What's the latest on the development of yEnc compatability in Mozilla? yEnc has become the standard for posting binary files on usenet, save a few picture groups where they are slow to switch due to the number of Outlook Express, AOL, and WEBTV users that like to surf for pics. (in other words, dial-up users largely that don't see much improvement with yEnc b/c they d/l small files one at a time anyway & they use **** software).
*** Bug 175428 has been marked as a duplicate of this bug. ***
#30: that the majority of Usenet readers have implemented something does NOT make it a standard. You shouldn't just do something because it can be done (or has been done already). It exactly that way of thinking that Mozilla is trying to combat: if you do something, do it right, if there is a standard you can use, use it, if not, make one.
re comment #34 : Sorry, but this is just rubbish - a newsreader is a program that should enable people to read news. If mozilla wants to provide functionality for this it cannot ignore something that is extremely widely used. I doubt that Mozilla is designed as a tool to force people to "do it the right way" whatever that should be. Dont forget that even Mozilla programmers might have varying views of what "the right way" is, and it might become even more varied when we look at the view of potential users. But this is a rather academical discussion, because forcing "the right way" or "some way whatsoever" on people can only be done by a product that is very widely used - and therefore definitely not by mozilla. So please dont patronize people and their way how to do things. You are free not to use yenc as long as you wish even if it were implemented in mozilla. This is tool, not a bible.
leaving this up for the trolls. I read this bug as a "fine as a mozdev project", and there it is. If you want it, go there, contribute code.
mozdev has an yenc component. Also, there are newsgroups that try to combat yenc encoding.
I wouldn't say there are groups where people are "combatting" it so much as there are groups where OE is the primary newsreader and they don't like it because OE doesn't support it. Of course there are some objecting because they are using Netscape and it doesn't support it either, but Netscape users seem (to me) more willing to either install the proxy server or use a different program.
Mind you, the author of the first link you posted (http://www.exit109.com/~jeremy/news/yenc.html) says he has moved on from when he wrote the article, and just accepts that yEnc is the predominant scheme for encoding binaries on Usenet. I'd say that we should support yEnc now, since it's the current (defacto) standard, and that if and when someone writes something better than can be used in more than just Mozilla, we can support that as well. I don't see how supporting every encoding standard that comes along can be construed as a bad idea: it lets the users make the choice as to what they want to use.
Well, one could argue that for Mozilla we might only want to use "real" standards, standards that have passed through and approved by a standards body. Look where the first years of HTML's unlimited growth has led us: the leading browser in the world still doesn't supoort all the standards and no browser seems to render exacty the same. I don't say that this is the same ball park and supporting yEnc will inevitably lead to problems but I do think that saying "it's a defacto standard" or "everybody does it" is not reason enough to go and add it to the core. But one thing I do know "just supporting every 'standard' that comes along" will get us into trouble (and we would probably have to rename our Mozilla monster to Hydra). But.... realisticly speaking I do think we can hardly get around implementing yEnc in some way or the other.
Okay, I agree with your comments on the general disarray of standards right now, I was thinking more in terms of just simple encoding options (for instance, like how file compression tools will support arj, rar, zip, sit, etc), we should try to implement any and all encoding/decoding schemes that won't 'cause problems for other parts of the browser. Sure yEnc isn't the best thought out scheme, but I haven't seen any arguements saying that including support is going to break other things in the browser.
Alias: yEnc
If someone wants to take http://yenc.mozdev.org/ just post a preliminary patch in this bug before I start working on it (which won't be for a long while) and its yours.
It's been a year since this bug was posted, so why isn't Mozilla yEnc compatable yet? Most news clients switched to yEnc within a few months of its appearance, so I assumed that this would be implimented rather quickly. Nearly all files I download on Usenet use yEnc. I've been suggesting Xnews (free download from http://xnews.newsguy.com/ ) as an option for those who don't care to pay for the agent version which supports yEnc and don't want to use yProxy for Outlook Express... from what I have seen, most people are using Xnews now in the groups I visit. I don't see the purpose of having a newsreader bundled with mozilla if it's not capable of downloading binaries. Sooner or later, yEnc will likely be the only posting format on Usenet. The only place I see MIME or UUE is in picture groups these days, and it's vanishing there quickly.
Wayne: Many news readers also have combine and decode abilities, yet Mozmail doesn't. Multipart news messages have been around a lot longer and aren't controversial. Therefore, there is no developer interest at the moment among the main developers to implement this. If you want this done, find someone who doesn't have many patches to write to make a preliminary patch and attach it here and I will help them get set up using the http://yenc.mozdev.org/ Mozdev project for a Mozilla extension. Axel: Since the code for encoding is in c++, is there any way to make a c++ xpi plugin?
So is anyone actually finally working on yenc support?
Not that I know of. If anyone wants to work on it, give me a buzz and I'll help you get set up using: http://yenc.mozdev.org/
I have a basic implemetation of a yenc decoded in Mozilla. It doesn't support multipart. It does not check the crc or the size of the data as this is a real time stream decoder which does not bufferrize the data. The time we detect something was wrong, the decoded data has been already pushed to the image library and displayed to the user. I'll cleanup my patch and post it soon...
Jean: You the man! Do you want the Yenc mozdev project space I'm squatting on, or is your intent to give what you've done so far so someone else can complete it?
Attachment #64985 - Attachment mime type: application/octet-stream → text/plain
Attachment #64985 - Attachment mime type: text/plain → message/rfc822
Attached patch Proposed fix, v1 (obsolete) — — Splinter Review
Keywords: helpwanted
Whiteboard: have fix
Attachment #115174 - Flags: superreview?(sspitzer)
Attachment #115174 - Flags: review?(cavin)
So, does this mean the next version of Mozilla will be able to display yEnc files? The darn things seem to be everywhere..
Attachment #115174 - Flags: review?(cavin) → review?(ssu)
would be nice if it make it... Reassign to ssu
Assignee: ducarroz → ssu
Status: ASSIGNED → NEW
(disclaimer)I'm not a reviewer(/disclaimer) I looked over the patch and it looks clean to me. Personally, I like how it makes the MimeDecoder more generic and removes the assumption that UUDecoding is the only form of embeded text encoding. Nice work Jean-Francois! Any ideas on how to increase the visiblity for this? It is a problem because it is not assigned and has no target milestone? 1.4b seems like a reasonable milestone now that we have a patch.... I along with many others have been waiting patiently (a month and a half) for this patch to be commited (or even formally reviewed). I'm sure many people would like to give it a spin. What do we need to do to get this on the radar for a r/sr? Thanks again to Jean-Francois.
1) + if (out[-1] == nsCRT::CR || out[-1] == nsCRT::LF) out is a char, and nsCRT::CR / ::LF are PRUnichar 2a) + if ((endOfLine - line) >= 13 && !nsCRT::strncmp (line, "=ybegin line=", 13)) use strncmp() instead of nsCRT::strcmp() 2b) instead of 13 and "=ybegin line=", let's see some #defines: #define YENC_BEGIN_LINE "=ybegin line=" #define YENC_BEGIN_LINE_LEN 13 3) + c= (unsigned char)*src; + if (c == 0x0A || c == 0x0D || c == 0x00) + break; why cast to unsigned char? those are \r \n and null. 4) + PR_ASSERT(!uty->open_subpart); remember, PR_ASSERT() calls abort() in debug builds. is that what you want? I recommend NS_ASSERTION() 5) + /* take off newline. */ + if (name[strlen(name)-1] == nsCRT::LF) name[strlen(name)-1] = 0; + if (name[strlen(name)-1] == nsCRT::CR) name[strlen(name)-1] = 0; why not this? int lastCharPos = strlen(name)-1; if (name[lastCharPos] == '\n') name[lastCharPos] = 0; else if (name[lastCharPos] == '\r') name[lastCharPos] = 0; also, do we have to worry about both CRLF together??
Assignee: ssu → sspitzer
sorry, I take it back, nsCRT::CR and LF are not PRUnichar. from nsCRT.h class NS_COM nsCRT { public: enum { TAB='\t' /* Horizontal Tab */, LF='\n' /* Line Feed */, VTAB='\v' /* Vertical Tab */, FF='\f' /* Form Feed */, CR='\r' /* Carriage Return */ }; I'll take the patch, and drive it in.
doh! we need it like this: if (name[strlen(name)-1] == nsCRT::LF) name[strlen(name)-1] = 0; if (name[strlen(name)-1] == nsCRT::CR) name[strlen(name)-1] = 0; that works for CR, LF and CRLF notice that we'd string the LF first, then the CR. it's important to have the strlens(), and the length of the string can change!
Attachment #115174 - Flags: superreview?(sspitzer)
Attachment #115174 - Flags: review?(ssu)
this works, but I get assertions. ###!!! ASSERTION: nothing to write!: 'dest >= line && dest < src', file c:/trees /mozilla/mailnews/mime/src/mimeenc.cpp, line 734 NS_ASSERTION(dest >= line && dest < src, "nothing to write!"); when I assert, dest == src. ducarroz, any ideas? note, as jfd explained, this only works for messages that are complete (not parts). here's what I used to test: news://news.mcom.com:119/kgsp8vcj8na2f0nsudr229h9ma7uapvh3q@4ax.com after I reviewed, tested (and landed), I realized that kaie has the strongest mime fu since ducarroz left. kaie, can you review? if you find any issues, we'll fix them.
Target Milestone: Future → mozilla1.4beta
> since ducarroz left I should say, since ducarroz no longer works on mozilla, full time. but he still reviews and hacks!
1.) === + /* take off newline. */ + if (name[strlen(name)-1] == nsCRT::LF) name[strlen(name)-1] = 0; + if (name[strlen(name)-1] == nsCRT::CR) name[strlen(name)-1] = 0; + + /* Now try and figure out a type. + */ + if (opt && opt->file_type_fn) + type = opt->file_type_fn(name, opt->stream_closure); + else + type = 0; + + if (name_ret) + *name_ret = name; + else + PR_FREEIF(name); + + if (type_ret) + *type_ret = type; + else + PR_FREEIF(type); What about this instead? Can save a few cycles. /* take off newline. */ size_t name_len = strlen(name); if (name[name_len-1] == nsCRT::LF) name[--name_len] = 0; if (name[name_len-1] == nsCRT::CR) name[--name_len] = 0; /* Now try and figure out a type. */ if (type_ret) if (opt && opt->file_type_fn) *type_ret = opt->file_type_fn(name, opt->stream_closure); if (name_ret) *name_ret = name; else PR_FREEIF(name); 2.) === + char *line_end = data->line_buffer + data->line_buffer_size - 1; Please declare as "const char*". 3.) === Let's declare a const for the 1000 == BUFFER_SIZE Is the 997 value derived from the 1000 ? Please use the constant if it is. 4.) === + { + char *out = line + strlen(line); It took me a while to understand why you are doing this. Please add a comment where you say, this is necessary, because the buffer might contain partial data from a previous call to the function. 5.) === Since we are not implementing an official standard, what about attaching the draft of what you are implementing to this bug, and add a comment to the sourcecode where you refer to the bug number? Are you implementing this: http://www.yenc.org/yenc-draft.1.3.txt ? 6.) === + if (new_line_size > data->line_buffer_size && new_line_size <= 997) /* don't let bad value hurt us! */ I think you want to protect yourself from allocating a large block of memory. But what happens in the case where new_line_size is indeed > 997? Won't your code continue to process the data and write to a buffer that is too small and corrupt memory? Should processing be stopped in that situation?
And a question out of curiosity: The drafts I found do not mention "x-yencode", where did you find that?
Attachment #120031 - Flags: review?(kaie) → review-
yencode is not yet a standard. Therefore I choose to use the prefix "x-". Anyway, this is only used internaly by Mozilla, could have been "x-no-more-yencode-complain"
I don't know about the assert! does it happens only once per image! maybe it's at the end of the decoding and this is a valid case where we should not assert?
Many yenc attachments are a single part but have a multipart format. This patch against CVS adds support for these attachments.. It also handles part 1 of a multipart post (but this is just a side effect). This is NOT a complete multipart handler, just a quick patch intended for single part attachments which include the 'part=1' tag.
So, did the patch make it into 1.4b? I installed the beta, but I don't see it.
Blocks: majorbugs
This does pretty much the same as Mark's patch (which I saw after I started on this). It ignores the additional multipart headers & part=val pairs. It also will ignore any future (and therefore unsupported) yenc directives by ignoring all lines that start with =y except =yend.
Blocks: 212453
*** Bug 212453 has been marked as a duplicate of this bug. ***
Regarding comment 67, either bug 212453 is not a duplicate or the description of this bug needs to change, I think. It seems that the encoding is already supported just not multipart. It seems we should mark this bug as fixed and reopen (undup) bug 212453 OR change the description of this bug to mention multipart messages OR I'm confused.
Am I incorrect in saying that none of the patches attached to this bug have been reviewed/super-reviewed/checked in? Bug 212453 comment 2 seems to indicate otherwise. If it is the case that the patches are not checked in, it would be a mistake to mark this bug fixed, as it isn't fixed :) In any case, multipart messages are part of the yEnc spec - if you don't support multipart, you don't support yEnc. Thusly, multipart decoding is implicit in the title "support new USENET encoding -- yEnc'. Having one bug for one part of yEnc, and another for another part seems a bit unnecessary.
What is the difference between multipart yEnc and any other multipart message? We don't handle any multipart, isn't that correct? That is covered in bug 60981, right?
Problem is not that Mozilla can not handle multipart yencoded messages, but problem is that many and many of singlepart yencoded messages encoded in miltipart format (see more in comment #64 http://bugzilla.mozilla.org/show_bug.cgi?id=119964#c64).
So basically to the user, a multipart yEnc message should appear as a single message header? In that case we will only grab the first part and the rest will be ignored without the user knowing? This is different than a split file sent in multiple postings that the user would see like 100 headers and have to select something akin to "Combine and decode" like in OE or "Join" in Agent, right? What I mean is that though it would be sent as a multipart, it should be reconstructed and transparent to the user, right? I'm all for fully implementing a "standard" instead of partial. What additional steps will it take?
Brian, I think what Alexander was getting at in comment 71 was that with yEnc, it's quite possible to create a multipart message with one part, delivered as one message. Some clients create multiparts as a matter of course. This is quite valid according to the spec (http://www.yenc.org/yEnc-draft-1.txt). So even if mozilla can't join message parts together automatically, it should be able to parse yEnc's multipart format. Attachment 121855 [details] [diff] allows mozilla to deal with such messages. A single part multipart message looks something like this: =ybegin part=1 line=128 size=100000 name=mybinary.dat =ypart begin=1 end=100000 .... data =yend size=100000 part=1 pcrc32=abcdef12 crc32=abcdef12
Ok, I think I get. So how would that appear to the user? In this case, it will only appear as one header to the user but still have the multipart as we see it (which we ignore), but if its a true multipart message, it will appear as multiple headers, with the main one looking correct but the rest looking like garbage to the average user? Two examples: 1 multipart message sent as 1 message - this will appear correctly 1 multipart message sent as 5 messages - we can't deal with this and the user will get 5 headers. Let's say its a .mpeg file, then they will only see the first 10 K (assuming a message is 10K long in this example), and therefore, its an .mpeg, they will only have the first 10K of the movie. Is this correct?
The situation with these bugs is rather untidy. This bug was for implementing yEnc, which has been done, so I guess it should be marked fixed. Bug 60981 is about combining multipart messages. Bug 212453 was opened for multipart yEnc but then duped to this. Will fixing bug 60981 make multipart yEnc work? If so, then bug 212453 is a dupe of that bug, not this one. If not, then we should either morph this bug into being about multipart yEnc, or reopen bug 212453.
Fixing bug 60981 will not make multipart yEnc work. The terminology surrounding multipart and yEnc is a bit ambiguous - simple protocols (UUE, for instance) produce multiparts by generating a single part encoding, and then splitting it amongst several messages. More complex (and robust) protocols, such as yEnc and MIME, produce a multi-part encoding, which is then posted, (often with) one part per message. Bug 60981 is concerned with automatically concatenating messages together; With simple protocols, this will produce a single-part message. With protocols that understand multipart, it will produce a single stream containing a multipart encoding of several parts. As it stands, mozilla cannot parse these for yEnc, only multipart yEnc encodings with a single part. To implement yEnc, a client must be able to parse both the single-part and multipart encodings, as it is not illegal (merely unusual) to post multipart yEnc coding (even with multiple parts) in a single message. Bug 212453 was opened for the multipart yEnc encoding, however yEnc is not a layered standard like MIME, MNG or MP[1-3]. It is not a valid implementation of the standard to implement only a subset (such as single-part decoding). So, in my opinion, bug 212453 is rightly marked as a duplicate. Bug 119964 should not be marked fixed until mozilla can parse multipart yEnc (and anything else in the spec that doesn't work yet), contained within a single stream (message body). If mozilla can do this, then a fix for bug 60981 will automatically work for yEnc as well.
ok, cool. So we're clear that this bug is for multipart as well.
No longer blocks: 212453
Summary: RFE: Support new USENET encoding -- yEnc → Support yEnc encoding (including multipart)
Whiteboard: have fix
Sounds like we should keep the existing code in the tree hush-hush then and continue onto the multipart before people notice :-)
I'm curious... It seems that people sometimes post split files in yEnc (i.e. yEnc of pre-split files). Those would fall under single part files that would need to be joined using bug 60981 and would still work OK, right? It seems there are 3 ways a very large file could be posted: - Pre split files posted individually in multiple messages of any encoding - bug 60981 - Server split messages of one large posting - would be joined via bug 60981, then handled via the patch in this bug - One large file that either was reconstructed by the server, or never split by any server - would be fully handled via the patch in this bug Did I miss anything? Sorry about the questions... I knew the answers to these last year, but since have forgotten them or gotten confused. I'm not asking you via private mail since ths can help other people. Usenet is one hell of a hacked "standard" if you ask me :-)
I hope by 'Server split' you're not implying that the NNTP server is cutting the message up? When an NNTP server encounters a message that is too large, it doesn't split it, it throws it in the bin. Streams are always split into messages by the client before being passed to the server. There are several ways that a large file can be posted (from my memory, apologies for omissions and inaccuracies) : --- decodable at the moment --- - Encoded as a single chunk using dumb protocol (UUE, base64) , chunk posted in one message body. - Encoded as a single chunk using MIME, chunk posted in one message body. - Encoded as a many chunks using MIME, chunks all posted in one message body. - Encoded as a single chunk using single part yEnc, chunk posted in one message body. --- decodable if message bodies are joined together (bug 60981) --- - Encoded as a single chunk using dumb protocol (UUE, base64) , chunk split into regular sections, and posted as several message bodies. - Encoded using MIME, split into sections, and posted as several message bodies. (as mozilla includes a full MIME parser. I think.) - Encoded as a single chunk using single part yEnc, chunk split into regular sections, and posted as several message bodies. (No client on earth posts like this, and would probably end up on the 'broken tools' list if it did. --- decodable using attachment 121855 [details] [diff] [review] --- - Encoded as a single chunk using multipart yEnc, chunk posted in one message body. (Some broken tools post like this) --- not decodable with anything attached to bugzilla --- - Encoded as as more than one chunk using multipart yEnc, chunks posted in same message body (nothing posts like this) - Encoded as as more than one chunk using multipart yEnc, chunks posted in separate message bodies (almost every tool posts large files like this) As it stands, mozilla cannot parse multipart yEnc messages with more than one part, even if every part is in the same message body. Joining the messages together (bug 60981) is not enough for multipart yEnc support, as mozilla must also parse the structure of the multipart yEnc stream and decode the file. In excatly the same way, joining all the message bodies together is not sufficient to decode multipart MIME; you must also understand the structure of the encoding method. Mozilla's MIME parser can understand MIME in multipart mode; Mozilla's yEnc parser does not yet understand yEnc in multipart mode. BTW, the term is "de-facto standard" :)
There is one other that you forgot (I'm just providing this for anyone reading) that should be covered by bug 60981 -- pre-split files that are split as follows: foo.bar.1 foo.bar.2 foo.bar.3 ... and then posted and individually encoded. We can handle such cases by allowing people to select each attachment (or perhaps contect-click one file and select "join similiar"), and choose "join" which would provide a list of files, pre-sorted by filename, and allow the user to move the ordering around. Then it would individually encode each attachment and combine it. Result would be: foo.bar I've never seen this done by a client, but there are tools out there for posting of binaries for people who don't want to sit there while its done that do this kind of thing and its pretty common in some groups (each group has its own rules).
There have been no new comments in four, almost five, months, so I thought I'd ask... where do we stand with yEnc so far? I have no coding skills whatsoever or I'd help out with this myself, as I know it is stopping at least one person from switching to Mozilla Mail and/or Mozilla Thunderbird.
yenc support in Mozilla 1.6 is almost useless, as about 50% of yenc posts use the part 1 of 1 multipart format ("=ybegin part=1") which Mozilla just displays as text. The patch has been posted here, so why can't it be checked in to the code so that it will be included in the next binary release?
patches can't be checked in until they've been successfully reviewed. As far as I can see nobody has even requested a review on the newer patches yet. So what's needed is for someone to check that the patches still work, request reviews and then do any necessary follow-up work.
*** Bug 258828 has been marked as a duplicate of this bug. ***
OK I just tried attachment 121855 [details] [diff] [review] and it still works in Thunderbird, on single posts encoded using multipart yEnc. This is a big step in the right direction (although not supporting >1 part), and I think it should be applied. Who should review this?
Comment on attachment 127193 [details] [diff] [review] yet another robustness patch Seth, who isn't reading bugmail.
Attachment #127193 - Flags: review?(sspitzer)
Andy: Can you try attachment 127193 [details] [diff] [review], also? It's newer.
attachment 127193 [details] [diff] [review] applies and works just like the other one.
Andy: Did you send an email to the newsgroups?
Will do
Comment on attachment 127193 [details] [diff] [review] yet another robustness patch passing the review buck to david b.
Attachment #127193 - Flags: review?(sspitzer) → review?(bienvenu)
Product: MailNews → Core
Assignee: sspitzer → bienvenu
Comment on attachment 127193 [details] [diff] [review] yet another robustness patch + if ((endofline - s) < 5 || strncmp(s, "line=", 5)) return PR_FALSE; can you put the return on its own line? That's the style we use, mostly so you can set breakpoints on the return. Other than that, it looks OK.
Attachment #127193 - Flags: superreview?(mscott)
Attachment #127193 - Flags: review?(bienvenu)
Attachment #127193 - Flags: review+
Comment on attachment 127193 [details] [diff] [review] yet another robustness patch modulo david's comments
Attachment #127193 - Flags: superreview?(mscott) → superreview+
*** Bug 276826 has been marked as a duplicate of this bug. ***
I think my original report that Thunderbird did not support yEnc decoding was in error. The problem really appears to be that Thunderbird does not identify posts that are actually incomplete multipart posts. I checked some posts that failed to decode under Thunderbird with my main email client Forte Agent and that identified them as incomplete posts (so you know you cant decode them) with an "incomplete" icon. When all parts are available on the server, Agent changes the icon to a "complete" icon. Posts with "complete" icons are automatically decoded by agent when downloaded. Thunderbird does not appear to identify incomplete multipart posts on the server so downloading the post just shows garbage. Using Agent I identified a couple of complete yEnc encoded posts and then downloaded them using Thunderbird. They decoded OK. So can Thunderbird identify incomplete multi-part posts on the server so you know not to download them yet?
*** Bug 280850 has been marked as a duplicate of this bug. ***
*** Bug 277641 has been marked as a duplicate of this bug. ***
No longer blocks: majorbugs
I have never seen a more moronic response since the beta-max/VHS wars. It doesn't matter that YENC is not a formal standard. When you have easily 90 percent of all binaries encoded in yenc in the news groups, it is completely idiotic to not support it in thunderbird. I use FireFox for my browser, but will not use thunderbird for this specific reason. To the "purists" developers at mozilla, please grow up. You are cutting off your nose to spite your face. Yenc is NOT going away in the near future and you are deliberately stunting the growth of the thunderbird product. The attitude that you can’t support every protocol that comes along is just silly. That is like saying that you won’t support personal computers 30 years ago because only mainframes were “real” computers.
William: Mostly everyone agrees it would be a very nice-to-have and that it's a de-facto standard. However, the latest patch is from 2003, without multipart support (as far as I know). If you are interested in getting this fixed, please provide a patch or find someone who has time to write one. Obviously, without a patch, it's not going to happen. The developers are all very busy, lots of times on security issues and things like that which are all higher priority than yEnc support. However, it would be a nice thing to have. Please don't abdicate in bugs (a lot of people already mirror your views) but instead voice your concerns about these things on the Mozillazine forums.
Abdicate wasn't the word I was looking for. Please don't make aggressive statements like that, attacking developers you don't really know, and saying things that have already been said in other comments in the bug.
My apologies, I was frustrated with 20 Yenc attachments in a row failing to decode. I originally looked in Mozillazine and they had one message that said, in effect, nothing. I discovered this bug and vented. It won't happen again.
- Mozilla 1.4 beta is long gone => Resetting target milestone. - David, are you still interested in working on this bug? If you do, please take it again (I suppose esther-at-formerly-netscape is not a valid QA anymore).
Assignee: bienvenu → nobody
QA Contact: esther → mime
Target Milestone: mozilla1.4beta → ---
heippa noniina@gmail.com taasen This might not be legal but anyhow -: Btw part of the linux code was actually "copied" from ddj about 1988-1990 by ... Some approach would be using part of winvn yenc source c/c++ source Try ex google
Blocks: 339983
Product: Core → MailNews Core
This bug is 9 years old, it has not been fixed up to now and lets face it, it won't ever gets fixed because nobody wants to work on it or maybe the Thunderbird developers are happy with the fact that Thunderbird is an inferior news client. So why don't we just close this bug. Pretty much all people seriously using Usenet have most likely stopped using TB years ago (just like I did) because it's pointless. I suggest setting status to resolved => won't fix.
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: