Closed Bug 1359127 Opened 7 years ago Closed 7 years ago

Support importing mitmproxy certs and set up proxy settings for Firefox

Categories

(Release Engineering :: Applications: MozharnessCore, enhancement)

enhancement
Not set
normal

Tracking

(firefox55 fixed)

RESOLVED FIXED
Tracking Status
firefox55 --- fixed

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

Attachments

(5 files, 1 obsolete file)

I will be trying this.

I'm looking at AutoConfig via JavaScript:
https://wiki.mozilla.org/CA:AddRootToFirefox#AutoConfig_via_JavaScript


----------------
There are many great requests for measuring performance.  We are looking at web-page-replay as our #1 goal with new modern websites.  This would involve preliminary network shaping as well as SSL.  From what we can tell, the only way to get SSL to work is to use a cert_override.txt file with Firefox:
https://developer.mozilla.org/en-US/docs/Cert_override.txt

This MDN page is very old, and there is a lot here that we could use help with in making this work:
1) understanding all the options of creating a file and using it in a profile programatically
2) how to determine on a site like facebook.com which entries we would need to put in this .txt file
3) are there other ways to ignore SSL certificates in the browser?
4) what dangers would we face by using this method?
5) what type of coverage would we be ignoring by using this method? (I assume any loss is better than raw http)
-----------------
You don't want an override - there are functional differences to the code in the presence of overridden exceptions. Instead you want to load the cert from your fake CA into the firefox trust store for the test . I'm sure david will have better references.

https://wiki.mozilla.org/CA:AddRootToFirefox

it looks like the fake CA root used by WPR for MITM is checked into github as wpr_cert.pem (I'm just guessing from a quick look)

Just so you know, I forsee a few other problems with the web page replay approach
* iirc it does not speak h2 and lots of the targetted sites speak h2 (including google and fb) with firefox.
* on replay it resolves everything to localhost, which has unrealistic implications if you are running h2 regarding connection management

big picture: The shaping approach is ok for a first cut - but its only going to be a rough guide to reality. It makes a huge assumption about the only bottleneck being the last mile and also assumes that all the origins involved share that bottleneck fairly (in real life they don't). Plus packet loss considerations. Like I said, I'm ok with the first cut but its worth documenting why we know it will be different than live data. The H2 thing though, that you really ought to sort out now. 2/3 of all our https traffic is over h2 and honestly the server implementations can differ pretty widely.. google, facebook, and nginx do a much better job that the node based servers and it would be a shame if we tuned for something other than what our users are running (~20% of all http transactions are likely with google)
Depends on: 1359131
Trying to record with --no-ssl or --should_generate_certs is not working on first try:
https://github.com/chromium/web-page-replay/issues/94

I have not yet tried adding the cert. I will try next.
After meeting with keeler he said that he would need to look a bit more into web-page-replay to make the right recommendation.

I will meanwhile keep poking at the code to understand the code related to certificate and even try to load the certificate at the root level of the repository.

NI to keeler; he hopes to look at this tomorrow.
Flags: needinfo?(dkeeler)
keeler: I found an open PR from snorp with some information on how to do this: https://github.com/chromium/web-page-replay/pull/53/files
keeler: is there any preference that would allow Firefox to simply ignore any certificate issues?
To use SSL you need pyOpenSSL.
The minimum required version is 0.13.0 (which does not build on MacOS X; more poking required).

Newer versions of pyOpenSSL do things right and throw an exception which web-page-replay does not handle at the moment
https://github.com/openssl/openssl/issues/710
The issue is not necessarily with the certificates (which I managed to import) but TLS cyphersuites:
https://github.com/chromium/web-page-replay/issues/94#issuecomment-297102821
I assume we can use a precompiled version of this for windows, if not, we should find a way to build our own wheel or whatever packaging is needed so we can get pyopenssl.  That is of course if this is what we end up needing to make this work.
As far as I can tell, Firefox would have no problem with https and web-page-replay if you import the wpr_cert.pem and trust it to issue web server certificates (either via the certificate manager or the JS API in comment 0) and use the --should_generate_certs option, but web-page-replay seems to be broken in a few ways.

First, running `sudo ./replay.py --record archive.wpr` and then `curl http://www.example.com` results in:

(INFO) 2017-04-25 12:45:29,481 platformsettings.get_original_primary_nameserver:288  Saved original primary DNS nameserver: 10.248.75.120
(INFO) 2017-04-25 12:45:29,484 platformsettings.set_temporary_primary_nameserver:295  Changed temporary primary nameserver to 127.0.0.1
(WARNING) 2017-04-25 12:45:29,484 dnsproxy.__init__:292  DNS server started on 127.0.0.1:53
(WARNING) 2017-04-25 12:45:29,485 httpproxy.__init__:354  HTTP server started on 127.0.0.1:80
(WARNING) 2017-04-25 12:45:29,485 httpproxy.__init__:354  HTTPS server started on 127.0.0.1:443
(DEBUG) 2017-04-25 12:45:32,306 dnsproxy.handle:213  dnsproxy: s.youtube.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:33,078 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:33,078 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:33,080 httpclient.__call__:326  RealHttpFetch: www.example.com /
(DEBUG) 2017-04-25 12:45:33,081 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:33,081 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:33,082 httpclient.__call__:326  RealHttpFetch: www.example.com /
(DEBUG) 2017-04-25 12:45:33,083 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:33,083 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:33,084 httpclient.__call__:326  RealHttpFetch: www.example.com /
...
<snip many many lines>
(ERROR) 2017-04-25 12:45:34,195 httpproxy.get_request:374  Number of active connections (500) surpasses the supported limit of 500.
(DEBUG) 2017-04-25 12:45:34,196 httpclient.__call__:326  RealHttpFetch: www.example.com /
(DEBUG) 2017-04-25 12:45:34,196 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(DEBUG) 2017-04-25 12:45:34,197 dnsproxy.handle:213  dnsproxy: www.example.com. -> 127.0.0.1 (replay web proxy)
(ERROR) 2017-04-25 12:45:34,197 httpproxy.get_request:374  Number of active connections (501) surpasses the supported limit of 500.
<etc.>

What I think is happening here is that web-page-replay is receiving the (http) request from curl, attempting to fetch the real http://example.com, but then its own proxy machinery is intercepting the request, and so it attempts to fetch the real http://example.com, and so on, until it runs out of connections.

Running replay.py with --no-dns_forwarding works fine with http (as long as you tell the client to proxy to localhost:80), but this doesn't work with https, because the https intercepting proxy replay.py sets up isn't configured to accept proxied connections (it assumes everything is a normal https request), and the http proxy isn't smart enough to handle connections that are actually destined for an https url.

Even if the above were fixed, if --should_generate_certs is passed, the https proxy for some reason attempts to fetch the real certificate of each requested site so it can generate a new certificate for that site that's signed by the wpr_cert, but this is pointless because we already know what host we're connecting to, so we can just make a certificate that's valid for that host. It's also worse than pointless because it attempts to connect to the "real" host, but unless --no-dns_forwarding is specified (which, again, doesn't work), replay.py will intercept the connection with its proxy, attempt to connect to the real site, intercept that connection, etc. (it also neglects to set the server name indication extension in its request, which can cause some servers to return the wrong certificate anyway).

Long story short, it seems either my setup is broken in some way (this is Fedora 25) or web-page-replay needs some work before it can actually handle https.
Flags: needinfo?(dkeeler)
I don't think WPR is designed to run the WPR server on the same host as you're running the test on.
Summary: Load fake CA into Firefox for web-page-replay work → Load fake CA into Firefox for mitmproxy work
This is based on knowledge from here:
https://developer.mozilla.org/en-US/Firefox/Enterprise_deployment

Place the autoconfig.js file in one of these directories:
* on Windows    defaults\pref
* on Mac        Firefox.app/Contents/Resources/defaults/pref
* on Linux      defaults/pref

Place the mozilla.cfg file besides the firefox executable (Create a .cfg file in the Firefox program directory)

The mozilla.cfg code is based on these two sources:
* https://wiki.mozilla.org/CA:AddRootToFirefox#AutoConfig_via_JavaScript
* https://dxr.mozilla.org/mozilla-central/source/security/manager/tools/genRootCAHashes.js#12-18

The hash comes from ~/.mitmproxy/mitmproxy-ca.pem

I need to write a Python script that would write both the mozilla.cfg and autoconfig.js files in the right paths with the right hash.
Summary: Load fake CA into Firefox for mitmproxy work → Load mitmproxy certs and set up proxy settings into Firefox
Thank you :armen! I did the above, but when I try to record using mitmdump, and browse to a URL Firefox still says the connection is not secure. Perhaps I did something wrong... do I need to update the hash or something?
Flags: needinfo?(armenzg)
This patch also adds the proxy settings.
I tried this on a clean profile and with a brand new replay archive.

--no-pop is necessary to allow loading the same content more than just once

> mitmdump --anticache -w ~/mitmrecording-20170428-facebook_ssl.mp
> mitmdump --replay-kill-extra --no-pop -S ~/mitmrecording-20170428-facebook_ssl.mp

I will focus now on a script to automate generating these two files.
Attachment #8862865 - Attachment is obsolete: true
Attachment #8862900 - Attachment mime type: text/x-csrc → text/plain
Flags: needinfo?(armenzg)
I suggested trying mitmproxy-ca-cert.p12 as per http://docs.mitmproxy.org/en/stable/certinstall.html#windows
rwood: Let me know if you have a suggestion on where to put this code.
Do you have any initial feedback for the script?

I will add support for multiple platforms and test it on my Windows machine.
I should have it ready by Monday morning.
Attachment #8863010 - Flags: feedback?(rwood)
Comment on attachment 8863010 [details]
[WIP] autoconfig_files.py - Prototype for MacOS X

This is awesome! I would suggest putting it here:

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing

Then I can integrate it with mozharness, so it will work with ./mach talos-test locally as well as in CI.

Looks great so far! Few comments...

- In the prefs for the proxy configurations, shouldn't the IP be 127.0.0.1 (instead of 0.0.0.0)?

- We will definitely need error handling, what if the path is wrong, or the cert doesn't exist there, or reading or writing fails, etc.

- Along those same lines, can we have mitmproxy.run return a pass/fail value? If this fails then mozharness will need to know not to continue

- We should probably have init or a method that will check to see if the certificate is already in Firefox - then no need to add it again if it is already there
Attachment #8863010 - Flags: feedback?(rwood) → feedback+
I choose 0.0.0.0 because when replaying that is the host IP shows:
> armenzg@armenzg-mbp ~$ mitmdump --replay-kill-extra --no-pop -S ~/mitmrecording-20170428-facebook_ssl.mp
> Proxy server listening at http://0.0.0.0:8080

I will keep an eye for your remaining concerns when writing the rest of the code.
Component: Talos → Mozharness
Product: Testing → Release Engineering
QA Contact: jlund
Summary: Load mitmproxy certs and set up proxy settings into Firefox → Support importing mitmproxy certs and set up proxy settings for Firefox
Version: Version 3 → unspecified
Attachment #8863846 - Flags: review?(rail)
Attachment #8863847 - Flags: review?(rail)
rail let me know if you prefer me to pass the review to someone else.
(In reply to Patrick McManus [:mcmanus] from comment #18)
> see also https://bugzilla.mozilla.org/show_bug.cgi?id=1361099

Do you want me to add this to the preferences in this bug?

// When non empty all non-localhost DNS queries (including IP addresses)
// resolve to this value. The value can be a name or an IP address.
// domains mapped to localhost with localDomains stay localhost.
pref("network.dns.forceResolve", "");
Flags: needinfo?(mcmanus)
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #22)
> (In reply to Patrick McManus [:mcmanus] from comment #18)
> > see also https://bugzilla.mozilla.org/show_bug.cgi?id=1361099
> 
> Do you want me to add this to the preferences in this bug?
> 
> // When non empty all non-localhost DNS queries (including IP addresses)
> // resolve to this value. The value can be a name or an IP address.
> // domains mapped to localhost with localDomains stay localhost.
> pref("network.dns.forceResolve", "");

I anticipate you will want to use that (new) pref for this bug.. but it hasn't quite landed yet - very soon (waiting on try).
Flags: needinfo?(mcmanus)
Attachment #8863846 - Flags: review?(rail) → review?(bugspam.Callek)
Attachment #8863847 - Flags: review?(rail) → review?(bugspam.Callek)
I haven't been touching mozharness for ages. Let's try Callek ;)
Attachment #8863847 - Flags: review?(bugspam.Callek) → review?(aki)
I'm not really familiar with this area of mozharness, or proxies in general, hot-potato passed to aki.
Attachment #8863846 - Flags: review?(bugspam.Callek) → review?(aki)
Comment on attachment 8863846 [details]
Bug 1359127 - Add Firefox autoconfig support

https://reviewboard.mozilla.org/r/135578/#review138966

r+, with or without a shift to using functions instead of a class.

::: testing/mozharness/mozharness/mozilla/firefox/autoconfig.py:71
(Diff revision 1)
> +        with open(path, 'w') as fd:
> +            fd.write(contents)
> +
> +    def run_and_exit(self):
> +        self.write_file(self.cfg_path, self.cfg_contents)
> +        self.write_file(self.autoconfig_path, self.autoconfig_contents)

I think this class works.
I also think this could be a function rather than a class.

Essentially, you're running __init__() to pass in some variables, and then run_and_exit() to write 2 files using those variables.  If there's a function that takes the same config that you pass into __init__() as args or kwargs, then it could write those two files, and we don't need a class.
Attachment #8863846 - Flags: review?(aki) → review+
Comment on attachment 8863847 [details]
Bug 1359127 - Initial Mozharness support for mitmproxy

https://reviewboard.mozilla.org/r/135580/#review138968

This could also be a function.
Attachment #8863847 - Flags: review?(aki) → review+
Attachment #8863010 - Attachment mime type: text/x-python-script → text/plain
I've refactored as functions, tested it and requested for autoland.

Thank you aki!
Pushed by armenzg@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f7d6828ad9d9
Add Firefox autoconfig support r=aki
https://hg.mozilla.org/integration/autoland/rev/b06d3343f43a
Initial Mozharness support for mitmproxy r=aki
Comment on attachment 8863847 [details]
Bug 1359127 - Initial Mozharness support for mitmproxy

https://reviewboard.mozilla.org/r/135580/#review139692

::: testing/mozharness/mozharness/mozilla/mitmproxy.py:18
(Diff revision 2)
> +var Ci = Components.interfaces;
> +var certdb = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
> +var certdb2 = certdb;
> +
> +try {
> +certdb2 = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB2);

Forgive the after-the-fact comment, but this isn't necessary as of bug 643041 (which landed in 33).
(In reply to David Keeler [:keeler] (use needinfo?) from comment #33)
> Comment on attachment 8863847 [details]
> Bug 1359127 - Initial Mozharness support for mitmproxy
> 
> https://reviewboard.mozilla.org/r/135580/#review139692
> 
> ::: testing/mozharness/mozharness/mozilla/mitmproxy.py:18
> (Diff revision 2)
> > +var Ci = Components.interfaces;
> > +var certdb = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
> > +var certdb2 = certdb;
> > +
> > +try {
> > +certdb2 = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB2);
> 
> Forgive the after-the-fact comment, but this isn't necessary as of bug
> 643041 (which landed in 33).

No worries. What would the right way of doing it be?
Anything referencing nsIX509CertDB2 doesn't need to be there - this should work:

var certdb = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
var cert = "%(cert)s";
certdb.addCertFromBase64(cert, "C,,");

(note also there are only 2 arguments to addCertFromBase64 as of bug 857627, which shipped in 53)
(also also you really only need to set the "websites" trust bit for the second argument)
See Also: → 1485082
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: