812077 - Move Peach to Python 3

Status: RESOLVED FIXED

(mozilla.org :: Security Assurance, task)

Description

[Yvan Boily [:ygjb][:yvan]]

Description: Replace current XML module with one that is better supported.
Mentor: cdiehl
Duration: 400 hours
Requirements: Python and Python 3 knowledge. XML knowledge.
Goals:
The goal of this mentorship is to help move the Peach fuzzing framework over to Python 3.  We need to replace the current 4Suite XML library with one that is supported by Python 3.  This could be the default Python libraries or something that is better.

Comment 1

[moijes]

I'm having a look at https://wiki.mozilla.org/Security/Fuzzing/Peach . Will let you know within an hour if I'll work on this.

Comment 2

[Christoph Diehl [:posidron]]

Hi moijes, we have added many customizations to Peach since r2775. Please note that the wiki page refers to an earlier SVN version of the original author. Feel free to join us on irc.mozilla.org #security if you have any questions.

Comment 3

[Christoph Diehl [:posidron]]

What you would need to understand is how Peach works internally, how the state model operates and how the parser parses the XML pit into a DOM in order to mutate the data model.

The necessary code is in the folder "Peach/Engine" and the important files are:

- engine.py
This is our entry point. Take a look at the class "Engine" there you will find a method called "Run".

- dom.py
Here you will find the supported XML elements which you can use in a pit file.

- parser.py
This file is responsible for parsing the XML pit. Take a look at the class "ParseTemplate" there you will find a method called "parse".

- incoming.py
This code will parse the data of e.g. a sample file into a data model based on the structure which you defined inside the pit file. That would be everything between <DataModel></DataModel> in a pit file.

state.py
This is the state engine which interprets the "StateModel" and "Action" elements of a pit file and is responsible for how the fuzzing workflow shall behave.

There are of course a couple of more files which are responsible for the logic of the fuzzer like Publishers and Agents but let's not focus on those for now. The important part is to understand the parsing logic and to implement an Python 3 alternative.

Comment 4

[Christoph Diehl [:posidron]]

Attachment: dom.py diagram

Comment 5

[Christoph Diehl [:posidron]]

Attachment: engine.py - diagram

Comment 6

[Christoph Diehl [:posidron]]

Attachment: parser.py diagram

Comment 7

[Christoph Diehl [:posidron]]

Attachment: incoming.py diagram

Comment 8

[Christoph Diehl [:posidron]]

Attachment: state.py diagram

Comment 9

[moijes]

Hi

Where can I get the source code for Peach?

Comment 10

[Christoph Diehl [:posidron]]

Hi moijes, 

I will send you a mail with a link to download it from my Dropbox account.

Please understand that we can provide you only with a stripped down version of our MozPeach version at the current time - meaning a single Pit for testing and a limited amount of Agents.


The first step is to run the install.py script which will install some dependencies.

$ ./install.py

In the next step you need to adjust Config/peach.darwin or Config/peach.linux2 to your preferences. That means the paths need to point to your Firefox builds, most importantly make sure that the variable "default-browser" can get resolved without errors.

After those steps you can run MozPeach like this:

$ ./peach.py -pit Pits/Files/GIF/gif.xml -run Browser


If you want to debug a pit, the following command comes in handy:

$ ./peach.py -1 -debug -pit Pits/Files/GIF/gif.xml | grep Rating | less -R 


If you run into any bugs or problems, feel free to contact me. I have already began to port some of the stuff to Python 3, mainly the little things. The big outstanding part is the XML library stuff.

Comment 11

[Christoph Diehl [:posidron]]

:moijes, are you still looking into this? It would be helpful if you could give me a status update. :-)

Comment 12

[MikeLing]

Hello,

I would like to fix this bug.How can I get start with it？

Comment 13

[Christoph Diehl [:posidron]]

Hi MikeLing, thank you for showing interest in helping us and sorry for getting back to you so late but we were/are quite busy.

It looks like that we have made some good progress in moving key parts of Peach to Python 3 compatible libraries (eg. we are now using the lxml library and got rid of 4SuiteXML). This is a huge step which was made possible by Jesse Schwartzentruber.

There is still a huge amount of testing in front of us and we currently need to reorganize ourselves to figure out who will work on which part.

I will post updates here to let you and others know which items need further work and where help is greatly appreciated.

Thanks.

Comment 14

[MikeLing]

(In reply to Christoph Diehl [:cdiehl] from comment #13)
> Hi MikeLing, thank you for showing interest in helping us and sorry for

You are welcome. Could you tell me where and how can I start with it? Thank you.

Comment 15

[Christoph Diehl [:posidron]]

As said, we need to re-organize ourselves. The last parts we need to look at are depending on the XML library.

cPeach.zip for example can be compiled with Python 3 but whether we can or still need to use cPeach in conjunction with lxml is currently unclear and needs to be seen. 

However, we are not yet in a position to offer the lxml patch to the public because of outstanding legal paperwork with our partner. Therefore I can not offer you to look into anything ... not at the moment.

If you would like to contribute to the project in the future we can provide you either with a source package or you can submit your code to a public version on  GitHub. 

I will write updates here so that you will get notified when we need additional help.

In any case you are welcome to join #security on irc.mozilla.org so that we can get to know each other a bit better.

Comment 16

[Christoph Diehl [:posidron]]

@blisman
Sorry that I missed you in IRC the other day.
We are basically done in porting Peach to Python 3, we also have re-licensed the project to MIT. However, there are still some outstanding str()/bytes() issues which need to get solved in the parsing code. If you are interested, let me know your GitHub handle and we can add you to the GitHub project.

Comment 17

[MikeLing]

@Christoph 
Hi~ I have no time to take part in contribute myself into it because I'm busy with my postgraduate exam. 
So,could you tell me what's your progress now and is there any help can I offer? Thanks a lot!
(anyway, is you view my Linkdin profile that day？ :)

Comment 18

[naru.aditya]

Hello!
I would like to pitch in and offer help in any way possible. If there is anything I could do, do let me know!

Comment 19

[divmermarlav]

Hi @cdiehl. This bug seems to be active since years ago. Do you still need help? I have good skills with python 2 and 3, but I am not sure where to begin, so Im looking for some advice. Thanks

Comment 20

[bhavishyagopesh]

Hello, Didn't got any reply at #security, thus asking it here...Is this still on(I'm presently working on GSoC ,with cpython, and have got resonable experience with python)...would like to contribute if possible.

Comment 21

[Al Billings [:abillings - ex-MoCo]]

This is done as far as I know. Ni? Christoph.

Comment 22

[Christoph Diehl [:posidron]]

@bhavishyagopesh
Sorry, I did not notice any discussion there. You can find me on IRC - just ping posidron.

I am at this time not sure it would be worth the effort to look further into Peach since we are switching more and more our fuzzing infrastructure to different kinds of fuzzers which have drastic speed advantages, i.e LibFuzzer.

The future of Peach tends to lead (at the moment) to way where we may use it for corpus generation only or for network related things which we can not or should not only get tested outside of the browser. This however is as well a decreasing factor nowadays.

Comment 23

[franck.farmer456]

Hello,
Is this bug still open or has it been fixed? Can I work on it for my first bug?

Comment 24

[Christoph Diehl [:posidron]]

I think we can mark this as fixed. Sorry but thanks for the request!