Closed Bug 1414776 Opened 7 years ago Closed 6 years ago

Migrate SSL related Firefox UI tests away from mozqa.com to badssl.com

Categories

(Testing :: Firefox UI Tests, enhancement, P2)

Product:
Testing
Component:
Firefox UI Tests
Version:
Version 3
Type:
enhancement

Tracking

(firefox-esr60 fixed, firefox60 wontfix, firefox61 wontfix, firefox62 fixed, firefox63 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla63
Tracking Flags:
Tracking Status
firefox-esr60 --- fixed
firefox60 --- wontfix
firefox61 --- wontfix
firefox62 --- fixed
firefox63 --- fixed

People

(Reporter: fox2mike, Assigned: whimboo)

References

Details

Attachments

(2 files, 2 obsolete files)

Bug 1414776 - [fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org.
59 bytes, text/x-review-board-request
ato
: review+
Details
Bug 1414776 - [fxui] Reset security.mixed_content.upgrade_display_content preference for mixed content test.
59 bytes, text/x-review-board-request
ato
: review+
Details 
Tracker bug to make sure by the end of Q1 2018, all of the following SSL tests for the Firefox UI have been moved away from the mozqa.com subdomain. 

David Keeler will be leading this effort. 

The list of tests are : 

https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests/functional/security 

The list of domains from the tests above are :

no-ssl.mozqa.com
ssl-dv.mozqa.com
ssl-ev.mozqa.com
ssl-expired.mozqa.com
ssl-ov.mozqa.com
ssl-unknownissuer.mozqa.com
tlsv1-0.mozqa.com


In addition, Webops has the following configured on our loadbalancers as well :

dv.ssl.mozqa.com.
ssl-md5.mozqa.com.
ssl-selfsigned.mozqa.com.
sslv2.mozqa.com.
sslv3.mozqa.com.
tlsv1-1.mozqa.com.
tlsv1-2.mozqa.com.
NI'ing :keeler for information and ack for Q1 2018 planning. Thanks!
Flags: needinfo?(dkeeler)
I assume that we could split this work up into different sub bugs, which could be done as mentored bugs if all the information is present for contributors. Before that I would like to hear which direction we would like to go for these tests.
Note that many of these tests already have browser-chrome analogues; the duplicates should just be deleted.
(In reply to Shyam Mani [:fox2mike] from comment #0)
> no-ssl.mozqa.com

Pretty easy - mochitests can already access some predefined http domains (e.g. http://example.com) - we just need to make sure this works in this test suite.

> ssl-dv.mozqa.com

Similarly, mochitests can already access some example https "hosts" - we should see if something from https://dxr.mozilla.org/mozilla-central/rev/2535bad09d720e71a982f3f70dd6925f66ab8ec7/build/pgo/server-locations.txt will work

> ssl-ev.mozqa.com

A little more difficult. There is a framework in our xpcshell tests to enable an EV root in debug-only builds. We would have to import it into the mochitest server or something.

> ssl-expired.mozqa.com

If the ssl-dv.mozqa.com test works, we can replace this with https://expired.example.com

> ssl-ov.mozqa.com

This is unnecessary and can be removed.

> ssl-unknownissuer.mozqa.com

I don't think there's a corresponding testcase in server-locations.txt, but it would be fairly easy to make one.

> tlsv1-0.mozqa.com

https://tls1.example.com

FWIW, some of this functionality is already covered in https://dxr.mozilla.org/mozilla-central/rev/2535bad09d720e71a982f3f70dd6925f66ab8ec7/browser/base/content/test/about/browser_aboutCertError.js
Flags: needinfo?(dkeeler)
Had a chat with April, she's ok with us switching tests to badssl.com

Henrik,

Can you please switch the ones we can to badssl.com and let us know in the bug and then I can disable those endpoints on the load balancers. 

Thanks!
Flags: needinfo?(hskupin)
If there are tests that you need that aren't yet in badssl.com, let me know and I can try to make that happen. Alternatively, feel free to open up an issue:

https://github.com/chromium/badssl.com/issues

:jeff usually lets me work on it without too much issue. Last quarter I spent a bunch of time working on client certificates tests for platform security anyways.  :)
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Flags: needinfo?(hskupin)
(In reply to David Keeler [:keeler] (use needinfo) from comment #4)
> > no-ssl.mozqa.com
> 
> Pretty easy - mochitests can already access some predefined http domains
> (e.g. http://example.com) - we just need to make sure this works in this
> test suite.
 
I will use http://http.badssl.com/

> > ssl-dv.mozqa.com
> > ssl-ev.mozqa.com

For both we would need an equivalent given that none exist yet on badssl.com. April, could you check if this is possible?

> > ssl-expired.mozqa.com
> 
> If the ssl-dv.mozqa.com test works, we can replace this with
> https://expired.example.com

I will use https://expired.badssl.com/

> > ssl-ov.mozqa.com
> 
> This is unnecessary and can be removed.

done

> > ssl-unknownissuer.mozqa.com
> 
> I don't think there's a corresponding testcase in server-locations.txt, but
> it would be fairly easy to make one.

I will use https://self-signed.badssl.com/ here.

> > tlsv1-0.mozqa.com
> 
> https://tls1.example.com

I will use https://tls-v1-0.badssl.com:1010/


Additionally we have https://mozqa.com/data/firefox/security/mixed_content_blocked/index.html which covers a couple of unsecure inclusions of elements. I somewhat miss parts of that on badssl.com. April, can you please also have a look here?
Flags: needinfo?(april)
(In reply to Henrik Skupin (:whimboo) from comment #7)
> (In reply to David Keeler [:keeler] (use needinfo) from comment #4)
> > > ssl-dv.mozqa.com
> > > ssl-ev.mozqa.com
> 
> For both we would need an equivalent given that none exist yet on
> badssl.com. April, could you check if this is possible?

For ssl-dv.mozqa.com, you should be able to use pretty much any badssl domain name that successfully connects. Maybe https://mozilla-modern.badssl.com/ would be appropriate?
I would probably recommend https://mozilla-intermediate.badssl.com/

It seems unlikely that we'll ever recommend EV certs at any configuration level, but if we did it would probably start at the Modern level.
Flags: needinfo?(april)
Oh, as for EV certs, we would have to:

a) change up the badssl code to generate EV certs (medium effort)
b) buy (or ask Comodo or someone if they would donate) a publicly trusted EV cert

I'll check with my Google cohort to see if he has any opposition to the latter. I don't know why he would, but funding might be awkward. I have confidence that we'll figure it out somehow.
So Lucas Garron (my cohort at Google) has transitioned support for BadSSL to a different engineer, since he's no longer working at Google. I just sent them an introductory email and so hopefully I can get them on this bug as well.
I am CC'ing Chris Thompson, the Google engineer who will be assisting me with badssl.com.  :)
Summary: Migrate SSL related Firefox UI tests away from mozqa.com → Migrate SSL related Firefox UI tests away from mozqa.com to badssl.com
Priority: -- → P2
April, do you have any news regarding the EV certificate? Everything else is done now.
Flags: needinfo?(april)
I do! I contacted my friends over at Comodo, and it sounds as if they would be happy to issue a gratis EV certificate. They can't, however, relax the EV certification requirements. As such, Chris Thompson has reached out to Google's certificate management and legal departments to help gather up the necessary paperwork.

It's a bit awkward, because BadSSL is an open-source Apache licensed project, hosted by Google and kept under the Chromium GitHub repo, but it's also been jointly run by Mozilla and Google.

I have confidence that all these things can be worked out (especially as Google owns the domain), but I can't give any strict ETAs because it's currently out of my hands.

If it's necessary to have a site to test against in the interim, I would recommend either mozilla.org (which has an EV cert until November of 2018) or BMO (which has an EV cert until November 2019).

I'm shooting for getting EV support in this quarter, but again there's not much that I can do at this point. My goal is to get EV certs working on badssl.test (aka the docker site the codebase generates) sometime in the next couple weeks.
Flags: needinfo?(april)
"ssl-ev.mozqa.com" is not difficult to continue operating from a technical standpoint, as compared to things like "ssl-md5" and "sslv3", since the industry will continue to support EV for some time to come. So it should continue working until it's not needed anymore, without being a burden to IT.
Oh, perfect then. That works too. I will still work on getting extended-validation.badssl.com working ASAP.  :)
(In reply to Richard Soderberg  [:atoll] [:�] from comment #17)
> industry will continue to support EV for some time to come. So it should
> continue working until it's not needed anymore, without being a burden to IT.

Well, you mean as long as the mozqa.com host has not been decommissioned. I would really like to make the change for the ev tests to switch to something else, and get it landed for Firefox 60 which will be the next ESR release. Then we can ride the trains and then on 2018-08-20 all branches will have the change, and the Firefox 52ESR branch is no longer supported. I think that should fit nicely in our time schedule, right?
There’s still no-ssl (bug 1336254), right?
badssl.com has http.badssl.com, which might cover that use case?
Just submitted a PR for the test version of the EV site:

https://github.com/chromium/badssl.com/pull/335
(In reply to April King [:April] from comment #21)
> badssl.com has http.badssl.com, which might cover that use case?

Yes, thank you Richard for reminding us on that case. It is already fixed in the update I uploaded 2 weeks ago.

(In reply to Henrik Skupin (:whimboo) from comment #7)
> Additionally we have
> https://mozqa.com/data/firefox/security/mixed_content_blocked/index.html
> which covers a couple of unsecure inclusions of elements. I somewhat miss
> parts of that on badssl.com. April, can you please also have a look here?

April, I think you missed this one.
Flags: needinfo?(april)
badssl.com does have some mixed content tests, but there are a *lot* of them. Most of Tanvi's tests (and others) were moved here:

https://mixed-content-tests-mozilla.org/

Nobody mentioned those particular tests, so I didn't really have any idea they needed a new home. If they're needed given what is on badssl.com and the above site, please let me know and I can try to move them onto mixed-content-tests-mozilla.org.
Flags: needinfo?(april)
Shyam, could you work with Henrik to respond to comment 24 and also file a MOC decom bug for the DNS+monitoring for the endpoints no longer in use? I saw active work today on ensuring that monitoring of e.g. ssl-md5 continues to work, but if we don't need some of them monitored anymore, let's let them know.
Flags: needinfo?(smani)
Current DXR list of mozqa domains in use, for those just seeing this bug today:

https://dxr.mozilla.org/mozilla-central/search?q=mozqa.com&redirect=false


### these are all https

mozqa.com

ssl-dv.mozqa.com
ssl-ev.mozqa.com
ssl-ov.mozqa.com

tlsv1-0.mozqa.com

ssl-expired.mozqa.com
ssl-unknownissuer.mozqa.com
ssl-selfsigned.mozqa.com


### this is not https

no-ssl.mozqa.com
(In reply to April King [:April] from comment #24)
> badssl.com does have some mixed content tests, but there are a *lot* of
> them. Most of Tanvi's tests (and others) were moved here:
> 
> https://mixed-content-tests-mozilla.org/

I see. I quickly checked and we miss the plugin and stylesheet cases as far as I see. Can you please check that and if necessary add new test files to that site? I'm happy to use that one if necessary.

Also how does it go along with the EV certificate?

(In reply to Richard Soderberg  [:atoll] [:�] from comment #25)
> Shyam, could you work with Henrik to respond to comment 24 and also file a

You can also set ni? to me. :) 

(In reply to Richard Soderberg  [:atoll] [:�] from comment #26)
> Current DXR list of mozqa domains in use, for those just seeing this bug
> today:
> 
> https://dxr.mozilla.org/mozilla-central/search?q=mozqa.com&redirect=false

Also note that I will push a combined patch, which is attached but not finalized yet.
Flags: needinfo?(april)
No decoms will be possible until the changes are uplifted into ESR (either 52 or 60), so nothing for Shyam to do here for now.

The mct site autodeploys from https://github.com/mozilla/mixed-content-tests.git (which accepts pull requests) in case that simplifies things for y'all.
Flags: needinfo?(smani)
:whimboo, when I setup mixed-content-tests, I copied everything that had existed already. I may still have the old codebase, but all I did was update a zillion links and move things into GitHub. Are you sure it was there before?

I do have a plugin mixed content test that I wrote:
https://mixed-object-loader.badsite.io/

Feel free to use and/or copy that. It has a flash object loaded over HTTPS that makes an HTTP request.
Flags: needinfo?(april) → needinfo?(hskupin)
(In reply to April King [:April] from comment #29)
> :whimboo, when I setup mixed-content-tests, I copied everything that had
> existed already. I may still have the old codebase, but all I did was update
> a zillion links and move things into GitHub. Are you sure it was there
> before?

If you refer to https://mozqa.com/data/firefox/security/mixed_content_blocked/index.html then yes this page is around forever.

Something I didn't asked yet is actually do we have some automated tests in tree already, which might make this test obsolete? If not https://mixed-content-tests-mozilla.org/ has a huge number of tests, and I don't think that our tests should cover all those cases.

> I do have a plugin mixed content test that I wrote:
> https://mixed-object-loader.badsite.io/
> 
> Feel free to use and/or copy that. It has a flash object loaded over HTTPS
> that makes an HTTP request.

No, we don't use Flash. We would simply need a basic object node, which has a HTTP resource set for the data attribute.
Flags: needinfo?(hskupin) → needinfo?(april)
I'm not entirely familiar with all of what Tanvi's tests do, but it seems that is available here:

https://mixed-content-tests-mozilla.org/tvyas/mixedboth.html
Flags: needinfo?(april)
(In reply to April King [:April] from comment #31)
> I'm not entirely familiar with all of what Tanvi's tests do, but it seems
> that is available here:
> 
> https://mixed-content-tests-mozilla.org/tvyas/mixedboth.html

No, that is using Flash. In our case we have just:

        var o = document.createElement("object");
        o.type = "text/html";
        o.data = baseURL + "/scripts/object.html";
        document.getElementById("result3").appendChild(o);

So again, do we have automated tests somewhere running regularly against Firefox builds which test those features? If yes, how do those tests report results, and where are those visible?
Flags: needinfo?(april)
I have no idea, I actually just run and work on the public sites that are used for manual testing. Someone from seceng would have to answer that question for you.

Sorry!
Flags: needinfo?(april)
I worked with Henrik for a few minutes this morning and submitted a pull request containing the new content:

https://github.com/mozilla/mixed-content-tests/pull/9
The mixed content HTML testcase is available now:
http://mixed-content-tests-mozilla.org/mozqa/mixed_content_blocked/

I'm going to update this patch during this week.
Flags: needinfo?(hskupin)
While working on an updated patch I noticed that bug 1435733 made some changes in how mixed content is handled since Firefox 60. The preference `security.mixed_content.upgrade_display_content` has been set but never gets removed. This causes following tests to accidentally fail when run on their own. I will fix that with an additional commit on this patch series.
Depends on: 1435733
Flags: needinfo?(hskupin)
(In reply to Henrik Skupin (:whimboo) from comment #7)
> > > ssl-unknownissuer.mozqa.com
> > 
> > I don't think there's a corresponding testcase in server-locations.txt, but
> > it would be fairly easy to make one.
> 
> I will use https://self-signed.badssl.com/ here.

This actually doesn't work because of: AssertionError: u'MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT' != 'SEC_ERROR_UNKNOWN_ISSUER'

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7792bfeedd5ad0799561fed6f7d1108dee257c13&selectedJob=176759692

April, I cannot find anything on badssl.com which corresponds to that. Do you have a suggestion?
Flags: needinfo?(april)
You're probably looking for this one:

https://untrusted-root.badssl.com/

Let me know if that's not what you're looking for.  :)

Thanks!
Flags: needinfo?(april)
Yes! That's it! I will update the patch and we should have a final one now, except for the EV tests, which we decided can be done later.
Attachment #8943594 - Flags: review?(mjzffr)
Attachment #8972871 - Flags: review?(mjzffr)
Attachment #8943594 - Flags: review?(mjzffr)
There is a high-frequent intermittent failure for the mixed script content test:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=03204a3568005a9dbe4e8bd39c7438ee373fe359&selectedJob=176787597

Not sure yet what's causing this. It might be something broken with the https://mixed-content-tests-mozilla.org/mozqa/mixed_content_blocked/ testcase.
"Timed out after 5.1 seconds with message: Insecure script from iFrame has been blocked"

Maybe we're just failing to parse the successful blocking of insecure content as a success?
When I open https://mixed-content-tests-mozilla.org/mozqa/mixed_content_blocked/ I can already see that the script 2 gets loaded. So it's clearly a fault in that page and that a failure slipped in.
Who can diagnose this fault and help you solve it? I'm not a web developer and this is way outside of Infosec's scope, so you'll need to tap resources not currently tracking this bug.
The problem actually lays here:
https://github.com/mozilla/mixed-content-tests/pull/9/files#diff-648ebb9199229c92597563fc86035dadR5

Richard, could you come up with a follow-up fix to move this line to use "http://" instead?
Flags: needinfo?(atoll)
Comment on attachment 8943594 [details]
Bug 1414776 - [fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org.

https://reviewboard.mozilla.org/r/213966/#review247374

::: testing/firefox-ui/tests/functional/security/test_mixed_script_content_blocking.py:15
(Diff revision 4)
>  class TestMixedScriptContentBlocking(PuppeteerMixin, MarionetteTestCase):
>  
>      def setUp(self):
>          super(TestMixedScriptContentBlocking, self).setUp()
>  
> -        self.url = 'https://mozqa.com/data/firefox/security/mixed_content_blocked/index.html'
> +        self.url = 'https://mixed-content-tests-mozilla.org/mozqa/mixed_content_blocked/'

Note to myself that we first have to get the failure in that remote page fixed before we can push this patch to autoland.
Attachment #8943594 - Flags: review?(mjzffr)
Comment on attachment 8943594 [details]
Bug 1414776 - [fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org.

https://reviewboard.mozilla.org/r/213966/#review247642
Attachment #8943594 - Flags: review?(mjzffr) → review+
Comment on attachment 8972871 [details]
Bug 1414776 - [fxui] Reset security.mixed_content.upgrade_display_content preference for mixed content test.

https://reviewboard.mozilla.org/r/241424/#review247644
Attachment #8972871 - Flags: review?(mjzffr) → review+
https://github.com/mozilla/mixed-content-tests/pull/10 will fix the http:// once it's merged.
Flags: needinfo?(atoll)
The PR on github got landed. I will trigger another try build and if everything passes I will go ahead and request the landing of the patch.
Lets file a new bug for the EV certs, so that we can update the tests once the domain is ready.
Summary: Migrate SSL related Firefox UI tests away from mozqa.com to badssl.com → Migrate SSL related Firefox UI tests away from mozqa.com to badssl.com [except EV certificate tests]
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/77045639ed6c
[fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org. r=maja_zf
https://hg.mozilla.org/integration/autoland/rev/ed3b0284c348
[fxui] Reset security.mixed_content.upgrade_display_content preference for mixed content test. r=maja_zf
Blocks: 1459716
No longer blocks: 1459716
https://hg.mozilla.org/mozilla-central/rev/77045639ed6c
https://hg.mozilla.org/mozilla-central/rev/ed3b0284c348
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox62: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
All tests look fine so please uplift the test-only patch to esr60. Thanks.
Whiteboard: [checkin-needed-esr60]
Lets actually wait. Looks like we have some new intermittent failures.
Whiteboard: [checkin-needed-esr60]
Depends on: 1459910
Does this need to go to Beta as well? If not, please update the 61 status to wontfix.
status-firefox60: fix-optional → wontfix
(In reply to Ryan VanderMeulen [:RyanVM] from comment #59)
> Does this need to go to Beta as well? If not, please update the 61 status to
> wontfix.

We cannot land this yet due to a regression on bug 1459910. I would even suggest that we backout this patch for now until we can guarantee stable network access to tls-v1-0.badssl.com. The failure rate is actually pretty high.

(In reply to Arthur Iakab [arthur_iakab] from comment #56)
> https://hg.mozilla.org/mozilla-central/rev/77045639ed6c
> https://hg.mozilla.org/mozilla-central/rev/ed3b0284c348

Sheriffs, please backout those two patches. Thanks!
status-firefox61: affected → fix-optional
Backout by ncsoregi@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/c35f6059fced
Backed out 2 changesets due to a regression on bug 1459910. a=backout
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: mozilla62 → ---
Depends on: 1312632
Henrik - I've had a network monitor on tls-v1-0.badssl.com:1010 for over a week without a single reported outage, and I've setup a debug build of Firefox and not been able to cause a TLS-related error with Firefox and tls-v1-0.badssl.com on my side.

The macOS builds are significantly different from the Linux/Windows ones, aren't they? Do they take place in a different location?

Would it be possible to get someone from the Firefox networking team to take a look at this? I've done nearly everything I can to replicate the error, but as near as I can tell it seems confined entirely to try.

Thanks!
Thanks April. Maybe this is related to the test and triggers a misbehavior in Firefox. I submitted a new try build for testing:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=56357770a78b8edc82a3d4309c06bc88cc101ef8

When I can still reproduce the problem I will try with a one-click loaner to get more details.
Flags: needinfo?(hskupin)
Thanks, :whimboo. Sorry it's so nondeterministic.

I've been monitoring the TLS 1.0 site every few minutes for two weeks now and still haven't triggered a single outage alert.
The try build showed one of those failures:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=56357770a78b8edc82a3d4309c06bc88cc101ef8&selectedJob=180314323

I will retrigger some more jobs to see how often the test fails. I still wonder why this is OS X only.

Also I was able to reproduce it once when I run the test locally on Friday. I will try again today with HTTP logs enabled. Maybe I'm lucky and can fetch some details.
The failure rate on Linux64 debug is higher than on MacOS and fails slightly differently:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=56357770a78b8edc82a3d4309c06bc88cc101ef8&selectedJob=180523503

> AssertionError: 'tls-v1-0' not found in u'Secure Connection Failed'

Maybe using a one-click loaner will make it easier for me to reproduce.
Henrik, have you had any luck? My site monitors still haven't caught any outages.

Also, I'm updating the topic, since we now have:

https://extended-validation.badssl.com/
Summary: Migrate SSL related Firefox UI tests away from mozqa.com to badssl.com [except EV certificate tests] → Migrate SSL related Firefox UI tests away from mozqa.com to badssl.com
Sure. I will push an updated patch which includes the EV changes. I will then run another try build. If the failure for the outdated TLS test is still present we may want to mark the test as skipped for now. The priority should be to get all tests not using mozqa.com anymore now.
Flags: needinfo?(hskupin)
Attachment #8943594 - Attachment is obsolete: true
Attachment #8972871 - Attachment is obsolete: true
Attachment #8994522 - Flags: review?(ato)
Attachment #8994523 - Flags: review?(ato)
Comment on attachment 8994522 [details]
Bug 1414776 - [fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org.

https://reviewboard.mozilla.org/r/259080/#review266108

::: testing/marionette/puppeteer/firefox/firefox_puppeteer/api/security.py
(Diff revision 1)
>  
>          :returns: Address details as dictionary
>          """
> -        regex = re.compile('.*?L=(?P<city>.+?),ST=(?P<state>.+?),C=(?P<country>.+?)'
> +        regex = re.compile('.*?L=(?P<city>.+?),ST=(?P<state>.+?),C=(?P<country>.+?),')
> -                           ',postalCode=(?P<postal_code>.+?),STREET=(?P<street>.+?)'
> -                           ',serial')

The removal of those lines are fine for now given that it was hard-coded for mozqa.com. Now we just hard-code it for badssl.com. A refactoring can be done once it would be needed for other domains.
Comment on attachment 8994522 [details]
Bug 1414776 - [fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org.

https://reviewboard.mozilla.org/r/259080/#review266296

This appears technically fine, with the caveat that I’m somewhat
unfamiliar with this code.

One thing to note is that I’m not super-enthused about the fact
that we have tests running in automation hitting external network
resources.  I suppose for TLS tests it’s hard to emulate certificate
negotiation without making network requests, but I assume this is
a consideration you already made when these tests were written in
the first place.
Attachment #8994522 - Flags: review?(ato) → review+
Comment on attachment 8994523 [details]
Bug 1414776 - [fxui] Reset security.mixed_content.upgrade_display_content preference for mixed content test.

https://reviewboard.mozilla.org/r/259082/#review266298
Attachment #8994523 - Flags: review?(ato) → review+
Comment on attachment 8994522 [details]
Bug 1414776 - [fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org.

https://reviewboard.mozilla.org/r/259080/#review266296

That's why those tests are reported as Tier-2 level on Treeherder for a couple of years. It is expected to have such tests for Firefox UI tests because no other harness can handle that.
Btw. on MacOS the try build no longer showed the failure for the TLS tests, so I just triggered the landing of this patch. We can reopen bug 1459910 once it happens again frequently.
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a74087e45e13
[fxui] Change testcases from mozqa.com to badssl.com and mixed-content-tests-mozilla.org. r=ato
https://hg.mozilla.org/integration/autoland/rev/07d8a92836d3
[fxui] Reset security.mixed_content.upgrade_display_content preference for mixed content test. r=ato
(In reply to Andreas Tolfsen 「:ato」 from comment #73)
> One thing to note is that I’m not super-enthused about the fact
> that we have tests running in automation hitting external network
> resources.  I suppose for TLS tests it’s hard to emulate certificate
> negotiation without making network requests, but I assume this is
> a consideration you already made when these tests were written in
> the first place.

When we were considering alternatives to MOZQA.COM last year, we weren't able to find any existing "test a local TLS server" functionality that we could use to meet the needs of testing Firefox against (for example) SSL 3.0, TLS 1.0, and various other odd certificate configurations. Or in other words, you're right: This remains sub-optimal.

Long-term, I think it would make sense for our build engineers to integrate the (open source) BadSSL site into our build process, and then make use of it locally so that these tests are stable and under our direct control. But that's far beyond the narrower scope of replacing MOZQA.COM, and cannot be completed by the relevant QA team unassisted.
It is designed to be very easy to spin up in a docker container, so it could very easily be a local test as long as you add the generated certificate to the certificate store.

I know Google uses BadSSL.com for a lot of automated tests as well, and I haven't heard of any issues on their end. I've been unable to replicate this error with any other tests, so I suspect the problem is related to our network or build process, and at least the network stuff could be fixed with a docker container build.
(In reply to Atoll R. Soderberg [:atoll, :rsoderberg] from comment
#78)

> (In reply to Andreas Tolfsen 「:ato」 from comment #73)  
> 
> > One thing to note is that I’m not super-enthused about the
> > fact that we have tests running in automation hitting external
> > network resources.  I suppose for TLS tests it’s hard to emulate
> > certificate negotiation without making network requests, but I
> > assume this is a consideration you already made when these tests
> > were written in the first place.
> 
> When we were considering alternatives to MOZQA.COM last year,
> we weren't able to find any existing "test a local TLS server"
> functionality that we could use to meet the needs of testing
> Firefox against (for example) SSL 3.0, TLS 1.0, and various
> other odd certificate configurations. Or in other words, you're
> right: This remains sub-optimal.
> 
> Long-term, I think it would make sense for our build engineers to
> integrate the (open source) BadSSL site into our build process,
> and then make use of it locally so that these tests are stable and
> under our direct control. But that's far beyond the narrower scope
> of replacing MOZQA.COM, and cannot be completed by the relevant QA
> team unassisted.

Let me just start by saying that your arguments are sound and that I
trust what you’re doing here.

You might be interested to know that there’s a W3C Community
Group (CG) meeting at TPAC in Lyon in October, which mission
is to enable HTTPS/TLS over local networks.  I don’t know much
about their progress or manifest, but you can have a look here:
https://www.w3.org/community/httpslocal/
https://hg.mozilla.org/mozilla-central/rev/a74087e45e13
https://hg.mozilla.org/mozilla-central/rev/07d8a92836d3
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
status-firefox63: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Depends on: 1382494
Depends on: 1372020
I have seen a couple of intermittent failures which are all marked as deps now. Some of those are known to happen infrequently, and bug 1459910 needs special eyes for the next days. If we don't see a major fallout, I will request an uplift in the next couple of days.
Depends on: 1478848
The number of test failures are low on each trunk branch, as such we should uplift this test-only patch to beta and esr60 now. Note that it blocks decommissioning a host in SCL3. Thanks.
Whiteboard: [checkin-needed-beta][checkin-needed-esr60]
Depends on: 1479228
Depends on: 1479227
Depends on: 1479229
Note that since the patch was uplifted to beta we haven't had any failures on that branch. Those intermittents only seem to occur on trunk and mainly on MacOS. I will still keep watching...
Depends on: 1479593
Henrik,

Can I turn off all the SSL related VIPs in SCL3 (just turn off, not remove) so we can confirm that all the tests we need have been migrated? 

Thanks!
Flags: needinfo?(hskupin)
Can we make sure the logs do not contain clear evidence of Release Engineering processes hitting them?
(In reply to Shyam Mani [:fox2mike] from comment #86)
> Can I turn off all the SSL related VIPs in SCL3 (just turn off, not remove)
> so we can confirm that all the tests we need have been migrated? 

No. This patch hasn't been uplifted to ESR60 yet.
Flags: needinfo?(hskupin)
Thanks for uplifting, Sebastian!

Now lets wait a couple more days, maybe a week. I would like to have a better idea of failures before I accept stopping the webserver on mozqa.com. Also note that we still need the TPS fix landed.
Depends on: 1480794
Blocks: 1465216
Depends on: 1491284
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: