The following test fails today during the 9.0b5 testrun:
Fail: could not find element ID getMeOutOfHereButton
Created attachment 579990 [details] [diff] [review]
Changed the webpage which we access and used the one stated in the litmus testcase https://litmus.mozilla.org/show_test.cgi?id=8581
Al, do we already have an invalid SSL cert on mozqa.com which matches the one from https://overstock.com (Error code: ssl_error_bad_cert_domain)?
(In reply to Remus Pop (:RemusPop) from comment #1)
> Created attachment 579990 [details] [diff] [review]
> patch v1
> Changed the webpage which we access and used the one stated in the litmus
> testcase https://litmus.mozilla.org/show_test.cgi?id=8581
Can't we do this with a local test page? Cancelling review until we decide that
Created attachment 580043 [details] [diff] [review]
skip test (all branches)
Skips the test until we figure out which webpage to use in the test.
Comment on attachment 580043 [details] [diff] [review]
skip test (all branches)
We can skip the test for now - please put it in your queue because this is still a P1 fix needed
Vlad, please always remember to r? patches to me when you r+ it, otherwise I might miss it for check-in.
Created attachment 580081 [details] [diff] [review]
disable test (all) [checked-in]
Your commit message was backwards, here is the patch I have checked in.
Comment on attachment 580081 [details] [diff] [review]
disable test (all) [checked-in]
Note in previous comment the landing on 1.9.2 as this is failing across ALL versions of Firefox.
(In reply to Henrik Skupin (:whimboo) from comment #2)
> Al, do we already have an invalid SSL cert on mozqa.com which matches the
> one from https://overstock.com (Error code: ssl_error_bad_cert_domain)?
I know that Al mentioned before that we had all the valid cert types installed. I'm not sure if any "invalid" certs were installed though.
We have an expired cert installed, as far as invalid types. I haven't installed any for another domain that doesn't match mozqa.com (well, the expired one is for another domain but we're hosting that domain now too).
Thanks Al. Do you think one of those certs will work for this test? If not, should I open a dependency bug to get said invalid cert?
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #12)
> Thanks Al. Do you think one of those certs will work for this test? If not,
> should I open a dependency bug to get said invalid cert?
The litmus test is just looking for an untrusted connection when it expects one. Overstock.com versus www.overstock.com.
If you use https://summitbook.mozilla.org, we'll get the same sort of warning, because the cert is expired. That site is hosted on mozqa.com so it would seem to fit the bill.
Remus, please refactor this test to use the URL given in comment 13.
(In reply to Al Billings [:abillings] from comment #13)
> If you use https://summitbook.mozilla.org, we'll get the same sort of
> warning, because the cert is expired. That site is hosted on mozqa.com so it
> would seem to fit the bill.
No, that's a different thing. Please check the detailed SSL warning. This is different for summitbook.
Well, the problem here is we have a bunch of legal certs for *.mozqa.com, a self-signed cert for the same, and the expired cert for summitbook.
We don't have a simple "something.mozqa.com" cert that is a mismatch for the domain you load.
I'm not a security expert and can't tell how different those test cases will be. So I would propose that Anthony(?) gets in contact with the security guys and find out if we really need all those different invalid SSL tests, or if most of those are already covered by automated tests.
For the time being we can use the proposed external URL.
(In reply to Henrik Skupin (:whimboo) from comment #17)
> I'm not a security expert and can't tell how different those test cases will
> be. So I would propose that Anthony(?) gets in contact with the security
> guys and find out if we really need all those different invalid SSL tests,
> or if most of those are already covered by automated tests.
I can certainly get in touch with some devs. Which "different invalid SSL tests" do you refer to? All the tests we have already automated in Mozmill? The ones in Litmus?
> For the time being we can use the proposed external URL.
Maybe I missed it, can you please specify the URL?
The url is in the litmus case in comment 1.
Just some clarifications here...
https://overstock.com: Untrusted Connection due to ssl_error_bad_cert_domain
https://summitbook.mozilla.org: Untrusted Connection due to sec_error_expired_certificate
The Litmus testcase is not specific about the reason "Untrusted Connection" appears, only that it appears and that the "Get Me Out Of Here" button works. If this were a FFT for the particular symptom of the failure I would argue in favour of comment 2; but we aren't.
I am personally okay if we use summitbook.mozilla.org for this Mozmill test. Though, I would give Henrik a chance for rebuttal before moving forward with this refactor.
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #20)
> https://overstock.com: Untrusted Connection due to ssl_error_bad_cert_domain
> https://summitbook.mozilla.org: Untrusted Connection due to
As you can see here both failures are different. So please get in contact with the security guys before making an assumption that we should not differentiate between both. This can but doesn't have to be wrong.
Personally I think having separate tests for any failure makes sense and we should not mix different SSL cert failures and websites.
While I agree Henrik (I will reach out to security team about your question), but I think I didn't make my last question clear.
The purpose of this Litmus / Mozmill test is to check that the Untrusted Connection error page appears and the Get Me Out Of Here button works. It is not a test for the specific error messages. We should have a separate Litmus / Mozmill FFT for those in my opinion.
Hopefully that clarifies the point I was trying to make.
It depends on how elements on this pages are getting created. There is still the risk that for a special SSL failure the button will not exist. It's a black box for us.
Okay, so I guess the first step here is to figure out from the security team the general workflow for generating these pages under various conditions.
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #24)
> Okay, so I guess the first step here is to figure out from the security team
> the general workflow for generating these pages under various conditions.
Anthony, can you take this so that we get some progress on this bug?
(In reply to Henrik Skupin (:whimboo) from comment #25)
> Anthony, can you take this so that we get some progress on this bug?
I've emailed Sid Stamm who should be able to shed some light on this for us. I'll report back when I hear something from him.
Is your test racing the page load? The button is in the XUL document, so it's not dynamically generated as far as I know.
I don't think it's racing the page load. Here is the test:
Is it possible that the URL load and TLS negotiation is succeeding sometimes (not showing the error) and failing (showing error) some other times? The live mozilla.org works fine for me without www, and I get no cert error. What happens if you change it to https://irs.gov or one of the sites in comment 20?
I guess it's possible. Considering we are talking about an automated test, I'm wondering what we can do to simulate the conditions required for these error pages to appear on our own mozqa.com server.
The original page has been changed. So we cannot make use of mozilla.org anytime longer. I have checked mozqa.com and as it looks like we have to get bug 666966 fully fixed first. Once that happened we could make use of it.
Anthony, or would that be the wrong cert for this test?
bug 666966 was fixed last August.
Remus, can you please update the page to use https://ssl-selfsigned.mozqa.com/ ? That's the one we own and which should not change. Thanks.
I think you meant https://summitbook.mozilla.org.
No, that was a failure. This test is about an untrusted connection and not an expired certificate. It got mixed-up.
Created attachment 631890 [details] [diff] [review]
patch v2 (all branches)
Modified the visited page to the one stated in comment 33
If we pass we can backport the patch. Remus, on which branches we have to land? Can you please mark those in the status flags list?
All branches are affected and work with this fix.
Pushed to all other branches:
Which Litmus tests have been updated?
The weren't disabled.