Closed Bug 465797 (mailfuzz) Opened 16 years ago Closed 7 years ago

mail fuzzer [meta]

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gkw, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: meta, sec-other, Whiteboard: [sg:nse meta])

Attachments

(4 files, 8 obsolete files)

crashlog of Terminal.app
97.32 KB, text/plain
Details
reduced testcase
6 bytes, text/plain
Details
timemailfuzz.py
128 bytes, text/plain
Details
mailfuzz 0.1.4
12.00 KB, text/plain
Details
Attached file proof-of-concept (obsolete) — 
I came up with this on a sleepness night. I'm not sure if this tastes good but a try nonetheless.

Attached is a python script that generates a random "From:" email address in the email header of a mozmill test based on some set criteria. It is just a proof of concept and not comprehensive in any way.

Running the following command will generate the random email address every time -- it's near the last 1/4 of stdout output.

python test.py

(possible output include apples@mozilla.cn, bears@moco.org and cars@momo.org)

This operates within a mozmill test that jminta wrote for Thunderbird. The idea is that the mozmill test will be changed to generate random "From:" email addresses instead of tests@mozillamessaging.com that he originally specified in https://bugzilla.mozilla.org/show_bug.cgi?id=464714#c15

The script can be expanded to vary other conditions.

The ultimate idea is that an eventual mail fuzzer generates a random email, and sent to and from the POP/IMAP and SMTP fakeservers with mozmill simulating user actions of opening and viewing the mail, on dedicated fuzz machines, saving off malicious emails that are generated that cause problems e.g. crashing the program. Examples may include an obscure combination of Unicode characters that may cause Thunderbird to crash. But this is still some ways off yet.
Attachment #349023 - Attachment mime type: application/octet-stream → text/plain
A couple quick comments:

(1) We should keep fuzzing out of the normal test scripts. Those scripts should be deterministic, to make tracking down and fixing their failures more straightforward. Also, the fuzzer may need to be kept private, to prevent others from identifying exploits. We'll still want the normal scripts to be public however, so tinderbox can run them.

(2) Since I think we do want this to be separate, much of the test you've included can be removed. (E.g. the folder-creation stuff)

(3) I completely agree that we should expand this beyond the AddMessage method, and beyond the from field, but this is a great start.
Attached file eml generator - proof of concept 2 (obsolete) — 
In this revision, the eml generator script generates .eml files that TB can open. Inside, unicode email generation is also demonstrated.

python mailfuzz.py

will generate some emails that vary according to some set criteria (not random yet, but they're randomly chosen). If the probability of unicode content is hit by chance, an arbitrary unicode char is printed.

I will attach some sample .eml files that were generated by the script. These .eml files I've tested to open correctly either via the menu point-and-click, or via the command-line. The unicode char also automatically displays correctly as I've set the charset as appropriate.

Note that some parts of the email header are still arbitrarily set.
Attachment #349023 - Attachment is obsolete: true
I would like some thoughts about the security implications of this email fuzzer and how this can potentially fit into automated Thunderbird testing in the future. (e.g. mozmill? fakeserver sending/receiving?)
Attached file sample generated unicode email (obsolete) — 
shows one unicode char when opened in TB.
Attached file sample normal encoding generated email (obsolete) — 
to open from command-line, do this:

/Applications/Shredder.app/Contents/MacOS/thunderbird-bin ~/Desktop/b.eml

also, email messages seem to follow RFC822, so the generated output should comply with it.
http://tools.ietf.org/html/rfc822
another thing to add, the user agent varies according to the last digit.

e.g. the unicode email has Gecko/20081113 while the normal one has Gecko/20081115.

So it is quite possible to fuzz and spoof user agents to simulate other email clients, though I think they would have much different stuff in their email headers.
Keywords: meta
Whiteboard: [sg:nse meta]
Attached file mailfuzz 0.1.2 (obsolete) — 
mailfuzz 0.1.2

This generates emails with (the same standard headers as above, not complete yet) variable numbers of variable Unicode characters in the content, limiting to UTF-8, with ASCII characters coming out more often.

(0.1.1 used somewhat more complex formulae to calculate the Unicode characters' numeric values, 0.1.2 simply uses randint(a,b). )

Additionally, I'm suspecting whether having calling random.foo multiple times will have a performance impact, and whether it will affect the randomness.
Attachment #349122 - Attachment is obsolete: true
Attachment #349124 - Attachment is obsolete: true
Attachment #349125 - Attachment is obsolete: true
getting jcranmer on this, it'll be interesting if such eml generators can be adapted to generate news messages as well.
Interestingly, when you keep running |python mailfuzz.py| multiple times, you can occasionally get Terminal.app to crash (in Leopard). A crash log is attached.

This may be due to Terminal.app not being able to handle certain combinations of Unicode characters in stdout that get generated by mailfuzz 0.1.2, I've gotten it to crash with crash stacks that have similar stacktrace patterns three times already.

I may be stumbling upon Apple bugs when writing this fuzzer, though I myself am not very sure whether it's Terminal.app at fault.
Attached file reduced testcase — 
Reduced testcase which crashes Leopard's Terminal.app, filed as Apple Bug ID# 6411258.
Very interesting idea, it seems it already caught a crashing problem (just not in Thunderbird).

For my 2¢, I think this idea could go farther, such as:
1. Using long message headers to try to find stack overflows.
2. Randomly violate RFC 2822 (and RFC 5322) in really obtuse ways to make sure we don't crash when receiving flagrant violations.
3. More comprehensive RFC 2822/5322 coverage.
4. Multipart MIME. Also incorrect multipart MIME.

For testing Usenet, just ensuring the presence of a Newsgroup header would be sufficient.

I'll toy around with this when I get some more time.
(In reply to comment #11)
> I'll toy around with this when I get some more time.

mailfuzz 0.1.2 is still very much a conceptual idea, I plan to rewrite the code comprehensively (except the Unicode generation bit which took quite a while to figure out -- Python 2.5 has up to Unicode 4.1 support only. I think it should be easy to adapt the generation function around, getting it right is the hard bit). It's very messy in its current state.

If Terminal.app keeps crashing due to it being unable to read certain combinations of Unicode chars and Apple doesn't fix them quickly enough, I may have to go over to gnome-terminal in Ubuntu, where such bugs would still exist but where hopefully they'll be fixed faster.

But then again, even in its infant stages it has already shown its potential by finding a bug, just not yet in Mozilla products. mailfuzz will probably be able to find crashes, overflows, hangs, stuff like that automatically. And it has the potential to be very dangerous if fallen into the wrong hands.

Here's what I envision for the future of mailfuzz, which mailfuzz 0.1.2 is moving towards, but is some ways off:
- Randomly generated email messages with random content, some adhering to standards, some simply junk, but they'll be sort of separated in the fuzzer so it will be easy to disable generation of either RFC-standard input or non-RFC-standard noise. Includes Multipart MIME and basically throw every possible combination and length of correct/incorrect input at Thunderbird.
- These email messages can be morphed to Newsgroup messages as jcranmer put it, with the addition of a Newsgroup header.
- I haven't tried it, but if it's possible to put in something Lithium recognizes in the fuzzed mails, the email testcases can even be reduced using Lithium.

For automation, (very long term plan)
- pre-generate emails to-be-fuzzed, then open each one individually using mozmill in Thunderbird and close them out. Assert if necessary, report a problem and copy the guilty email to another directory  for analysis if Thunderbird crashes.
- Write companion protocol fuzzers that deliver such pre-generated emails via POP, IMAP or News and receive via SMTP within a "security-fied" automation machine zone. They should deliver properly and opened properly by mozmill at the receiving end, or xpcshell tests could be used if they suffice. (I'm not exactly an expert in this area though)
- Adopt a concept of fuzz machine clusters, with multiple machines doing these actions for different versions of Thunderbird, e.g. stable versions of 2.0, 3.0, and the nightlies, similar to that of jsfunfuzz.

For now, mailfuzz has to be working properly first, it'll be good to know if the ideas above taste good. :)
Generally speaking, this sounds like a great direction.  As far as Mac OS terminals go, you might want to check out iTerm.
Attached file mailfuzz 0.1.3 (obsolete) — 
Still messy -- as I've merely added more functions that make use of the Unicode generation functions, this time to generate email addresses. Need to add more ICANN top level domains too. Tweaked a bit of the character generation formula too.

I haven't run it against Python 2.6 / 3.0, though I'm tempted to. The Unicode function will probably benefit from 3.0's move to UTF-8 instead of being explicitly defined. I'll check out 2to3 too, if that makes the code cleaner / easier on top of the 3.0 Unicode benefits, I'll shift to 3.0 now that the final version of 3.0 has been released. Right now the eml generator apparently still generates malformed UTF-8 characters.
Attachment #350920 - Attachment is obsolete: true
If you change this so it has to run on Python 3.0 then any developer / test machine that wants to try this out will also need to run 3.0 (e.g. Macs seem to be on 2.5 at the moment). Not a show stopper, but worth remembering.
(In reply to comment #15)
> If you change this so it has to run on Python 3.0 then any developer / test
> machine that wants to try this out will also need to run 3.0 (e.g. Macs seem to
> be on 2.5 at the moment). Not a show stopper, but worth remembering.

True -- I'll see if it's really worth it. Haven't really tried 2.6 or 3.0 yet to be honest. It's just from what I read on 3.0's improvements in Unicode. (and removing all that u'\uFOO' stuff which had previously made my life a little miserable) Life gets complicated once Unicode is considered, but that seems to be where bugs lie more often as compared to ASCII or ISO-8859-1. 3.0 should be able to be installed through MacPorts once it hits the official repository.

Another issue I've encountered is that stdout output in both Terminal.app and iTerm look different, though |tee| outputs into the same file for both to view. I'm unsure why this is the case, but my fonts in both programs are Courier and in UTF-8 encoding.

note to self: I need to separate generation of compliant stuff from noise. It's still horribly mixed together.

mailfuzz isn't really complete such that it can be comprehensively tried out, but do feel free to try it and give feedback while noting all its current shortcomings.
Attached file timemailfuzz.py — 
Put this in the same directory as mailfuzz.py and run |python timemailfuzz.py|. It will run mailfuzz 100 times and average the timings over 3 runs.

Each run takes about half a minute on my not-so-recent MBP in Leopard.

I'm guessing this may help us estimate a sane number of Unicode characters to generate per email in mailfuzz.
Attached file mailfuzz 0.1.4 — 
Changes in 0.1.4:

- Separate generation of noise from valid input in email addresses.
- Completed all valid input for TLDs and country codes.
- Finished several more functions that generate random / valid data.
- More cruft removal (not yet complete).
(still Python 2.5, so far I've not had the chance to try the later versions yet.)

It should be possible to use (something)_timed_run.py to automatically run this fuzzer with mozmill in Thunderbird after pre-generating the emails beforehand, and a mozmill script that automatically opens each generated email which then closes it. It should also be possible to run it in Lithium this way by adding some lines (DDBEGIN / DDEND ?) to each email crash testcase (if any is found).
Attachment #351505 - Attachment is obsolete: true
Attached file mailfuzz 0.1.4 for Python 3.0 (obsolete) — 
I tried moving the script to Python 3.0 and here is the py3k version of 0.1.4.

Key differences in py3k so far:
- print statements turned into print functions.
- Banished unichr(), uses chr instead.
- Banished need to import sys for Unicode in stdout, and as well as the UTF-8 line at the top.
- Banished u'\uFOO' anymore, only just the actual \uFOO.

In the few performance-related tests I've run in py3k (3.0rc1) vs py2k (2.5.2), py3k seems slightly slower.

Thus, mailfuzz doesn't seem to be needing Python 3.0 anytime soon. :)
Attached file py3ktimemailfuzz.py (obsolete) — 
Attaching but obsoleting the 3.0 versions for archival purposes.
Attachment #352275 - Attachment is obsolete: true
Attachment #352276 - Attachment is obsolete: true
Gary is this usable ?
(In reply to comment #9)
> I may be stumbling upon Apple bugs when writing this fuzzer, though I myself am
> not very sure whether it's Terminal.app at fault.

Yup, it's a bug in 10.5's Terminal.app, the reduced testcase no longer crashes in 10.6's Terminal.app. I guess Apple does fix user-reported issues. :)

(In reply to comment #21)
> Gary is this usable ?

Yes, this is, at least partially.

python mailfuzz.py

Running this command generates .eml files with random characters and partially random fields each time, and the idea is to open each file up in Thunderbird and doing stuff with it. It's just that no one has got any time to further work on it.
Just to add some notes, having thought of this recently:

1. In the wild, I found a UTF-8 byte order mark lying around the ostensibly-ASCII header fields. I also found one header horribly maimed (I don't have the exact header on me right now, but it would be like "NNTP-Posting-Host: <high-bit garbage>stuff\r\n".
<nntp://news.eternal-september.org/de.alt.folklore.computer/4011> or <news:4d3d395a$0$305$4c56b896@news-read1.lambdanet.net> will get you that malformed header.

2. Libmime looks at more than just the header data, so we should probably add in some more varied content-types. HTML (both well- and ill-formed, to test the HTML stripper), various multipart/s, perhaps malformed message/receipts, and handling issues with signed/encrypted might be good.

3. Encodings is another thing to watch out for: everything from malformed UTF-8, 8-bit, base64, quoted-printable, yEnc, and uuencode.

4. We probably also want fuzzing on the EAI specs; in particular, I am not going to trust that every message/global with 8-bit stuff in its headers is actually UTF-8.
Gary, is it ok for you when I create an additional PeachPit to fuzz the MIME library?
(In reply to Christoph Diehl [:cdiehl] from comment #24)
> Gary, is it ok for you when I create an additional PeachPit to fuzz the MIME
> library?

Yes!
Summary: mail fuzzer → mail fuzzer [meta]
No progress, opening up, won't have time to work on this anytime soon.
Group: core-security
does this bug cover the needs of bug 482879 comment 26 ?
Flags: needinfo?
(In reply to Wayne Mery (:wsmwk) from comment #27)
> does this bug cover the needs of bug 482879 comment 26 ?

Possibly, although code to tie this up with the test harness will still have to be written.
Flags: needinfo?
Might be easier with mimejs.
This found bugs (just not in Thunderbird), and for what it's worth, its development has ended, so resolving FIXED.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
See Also: → 1505277
See Also: → 1641581
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: