Closed Bug 1064387 Opened 10 years ago Closed 9 years ago

Update the TLS certificate for https://www.mozilla.org to not use sha-1

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: ehsan.akhgari, Assigned: gozer)

References

()

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1319] )

      No description provided.
Assignee: server-ops → server-ops-webops
Component: Server Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
QA Contact: shyam → nmaul
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1226]
Assignee: server-ops-webops → gozer
This is a vague reference to mozilla.org? We have a lot of ssl certificates that match mozilla.org, what are you specifically referring to ?
Flags: needinfo?(ehsan.akhgari)
(In reply to Philippe M. Chiasson (:gozer) from comment #1)
> This is a vague reference to mozilla.org? We have a lot of ssl certificates
> that match mozilla.org, what are you specifically referring to ?

Erm, <https://www.mozilla.org/>?  I haven't triaged all of our TLS certs obviously, but we probably should.
Flags: needinfo?(ehsan.akhgari)
(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #2)
> (In reply to Philippe M. Chiasson (:gozer) from comment #1)
> > This is a vague reference to mozilla.org? We have a lot of ssl certificates
> > that match mozilla.org, what are you specifically referring to ?
> 
> Erm, <https://www.mozilla.org/>?  I haven't triaged all of our TLS certs
> obviously, but we probably should.

Yes, most certainly, we should do a full audit of all of ours. Nicely, today, DigiCert released a ssl cert dashboard doing exactly that.
Summary: Update the TLS certificate for mozilla.org to not use sha-1 → Update the TLS certificate for https://www.mozilla.org to not use sha-1
Upgraded to SHA-256 without issues.

I'll be looking at the global SSL cert audit later.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
We need to revert this. See bug 1060508.

TL;DR: SHA-2 certs on www.mozilla.org cost us around 145,000 Firefox downloads per week.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Jake Maul [:jakem] from comment #5)
> We need to revert this. See bug 1060508.
> 
> TL;DR: SHA-2 certs on www.mozilla.org cost us around 145,000 Firefox
> downloads per week.

Yes, please don't change SSL certs on www.mozilla.org without checking with #www or #webprod as we killed 1 million downloads recently by switching to SHA-2. A lot of the world is still running old browsers and come to our website to get Firefox.
Based on the dates here: https://www.digicert.com/sha-2-ssl-certificates.htm

Let's re-issue the cert for www.mozilla.org as SHA-1, expiring 2015-12-31. That gives us the maximum amount of time to support old users without breaking modern Chrome or IE. At that time, we may just have to start trying to detect WinXP users (ideally just pre-SP3, but that seems like it would be hard to detect) and force them to a non-SSL connection. Sucks, but better than giving them an error and making it impossible for them to download Firefox. Firefox is a great fix for them because it ships with its own SSL stack and thus avoids the underlying OS limitation.

Unless we want to drop support for WinXP entirely (to the point where they can't even *download* Firefox), this seems like the most reasonable solution.


Side note: bug 942515 indicates that Firefox is taking roughly the same measure soon, for anything expiring after 2017-01-01.
(In reply to Jake Maul [:jakem] from comment #7)
> Based on the dates here: https://www.digicert.com/sha-2-ssl-certificates.htm
> 
> Let's re-issue the cert for www.mozilla.org as SHA-1, expiring 2015-12-31.
> That gives us the maximum amount of time to support old users without
> breaking modern Chrome or IE. At that time, we may just have to start trying
> to detect WinXP users (ideally just pre-SP3, but that seems like it would be
> hard to detect) and force them to a non-SSL connection. Sucks, but better
> than giving them an error and making it impossible for them to download
> Firefox. Firefox is a great fix for them because it ships with its own SSL
> stack and thus avoids the underlying OS limitation.
> 
> Unless we want to drop support for WinXP entirely (to the point where they
> can't even *download* Firefox), this seems like the most reasonable solution.
> 
> 
> Side note: bug 942515 indicates that Firefox is taking roughly the same
> measure soon, for anything expiring after 2017-01-01.

+1. Let's keep SHA-1 on www.mozilla.org until we find a better solution. Switching to SHA-2 will kill 5% of out downloads and that has a direct impact on ongoing Firefox usage unless we have a better solution to deal with legacy browsers. Let's start the discussion now in a separate bug on how to handle legacy browsers before Dec 2015.
Hey gozer. just want to double check SHA-1 cert is on www.mozilla.org for the reasons above. If not, we are hurting downloads on the order of 5% of all users. Every bit of percentages matter now.
Flags: needinfo?(gozer)
(In reply to Chris More [:cmore] from comment #9)
> Hey gozer. just want to double check SHA-1 cert is on www.mozilla.org for
> the reasons above. If not, we are hurting downloads on the order of 5% of
> all users. Every bit of percentages matter now.

Correct, SHA-1 cert is back on www.mozilla.org
Flags: needinfo?(gozer)
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1226]
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1324]
"At this time, a site could use two certificates: ECDSA+SHA256 for modern clients and RSA+SHA1 for older clients."
https://community.qualys.com/blogs/securitylabs/2014/09/09/sha1-deprecation-what-you-need-to-know
(In reply to sjw from comment #11)
> "At this time, a site could use two certificates: ECDSA+SHA256 for modern
> clients and RSA+SHA1 for older clients."
> https://community.qualys.com/blogs/securitylabs/2014/09/09/sha1-deprecation-
> what-you-need-to-know

Interesting!

jakem: Is the above possible? Can we run multiple certificates?
Flags: needinfo?(nmaul)
Apache supports it: 

> This directive can be used multiple times (referencing different filenames) to support
> multiple algorithms for server authentication - typically RSA, DSA, and ECC.

http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslcertificatefile


Based on https://news.ycombinator.com/item?id=8276770 , it sounds like CloudFlare are implementing that feature for nginx, but whether/when it is open sourced is another matter...
Are we running Apache 2.4? I know at least the www.mozilla.org web servers are still on Apache 2.2, but they're not the ones that terminate the TLS connection. The key here though is to avoid doing anything that would jeopardize downloads, even from very old clients.
Also, not doing things that would jeopardize search engine rank given that search is the primary way people get to our download page. Google has mentioned in the future that they may start penalizing SHA-1 websites.
(In reply to Tom Fitzhenry from comment #13)
> Apache supports it: 
> 
> > This directive can be used multiple times (referencing different filenames) to support
> > multiple algorithms for server authentication - typically RSA, DSA, and ECC.
> 
> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslcertificatefile
> 
> Based on https://news.ycombinator.com/item?id=8276770 , it sounds like
> CloudFlare are implementing that feature for nginx, but whether/when it is
> open sourced is another matter...

The problem, for the time being, is that we would require Zeus (our SSL termination layer) to
support that feature.
Oh yes, if Zeus can't handle multiple SSL certs, than I guess we are blocked on multiple certs. I can't find docs if it is possible, so I assume it isn't.

Crazy idea: create a clone of bedrock and make a new virtual server with a SHA-1 cert and then upgrade current bedrock to SHA-256 and then redirect/rewrite legacy visitors to legacy clone at a different IP address. It would seem that we need to also resolve this same issue on mozilla.org, download.mozilla.org, and our SSL CDNs. I have been out of netops/webops for a long time to know if this is even possible. Probably a crazy idea that isn't feasible.
(In reply to Chris More [:cmore] from comment #17)
> Oh yes, if Zeus can't handle multiple SSL certs, than I guess we are blocked
> on multiple certs. I can't find docs if it is possible, so I assume it isn't.
> 
> Crazy idea: create a clone of bedrock and make a new virtual server with a
> SHA-1 cert and then upgrade current bedrock to SHA-256 and then
> redirect/rewrite legacy visitors to legacy clone at a different IP address.
> It would seem that we need to also resolve this same issue on mozilla.org,
> download.mozilla.org, and our SSL CDNs. I have been out of netops/webops for
> a long time to know if this is even possible. Probably a crazy idea that
> isn't feasible.

Nope, a good idea, but not feasible, because of the way SSL works, the certificate
selection needs to be part of the handshake, way too soon to be able to do that kind of redirecting.
Please let's stick to the recommended settings from https://wiki.mozilla.org/Security/Server_Side_TLS#Backward_Compatible_Ciphersuite

OpSec is working on a global https strategy that includes sha1 deprecation, but it's a tricky business that needs research and testing. I propose that we do the research and update the guidelines first, and then only modify production sites.
(In reply to Philippe M. Chiasson (:gozer) from comment #18)
> Nope, a good idea, but not feasible, because of the way SSL works, the
> certificate
> selection needs to be part of the handshake, way too soon to be able to do
> that kind of redirecting.

We would have to make the download landing page HTTP rather than HTTPS, detect the OS version in JS (which can be done) and then redirect accordingly to the SHA-1 or SHA-256 system.

Gerv
IMO there really is no good solution here. My gut reaction is to simply run with the SHA-1 cert for a while... by my reading, given it's expiration date and the way browsers are handling it, there shouldn't be any warning presented. In 6 months we can evaluate if it's become any more feasible to switch to a SHA-2 cert. If not, we simply wait out the time and switch when we no longer have a choice... or perhaps by then some other (better) solution will present itself.


Zeus can run multiple certs, but the only way it can differentiate which one to provide to the client is based on the Host header the client uses (aka: TLS SNI). Since everyone is providing the same header ("www.mozilla.org") this isn't helpful unless we somehow segregate users such that older, SHA-1-only clients use a different site than SHA-2-capable clients.

*After* the handshake (using a SHA-1 cert), it might be possible to do all sorts of heuristics in code to decide what to do. We could then set up a separate alias for www.mozilla.org (something like "www2.mozilla.org") and redirect people to it. That domain could run a good SHA-2 cert. I'm not sure if this is worth the effort, since the main site that everyone knows and uses would still need to be on SHA-1. We also introduce a bunch of new variables... what if the redirect fails, what if we redirect the wrong users, does everything work on both sites, etc. Lots of extra test-cases to worry about.

Another option would be to abandon the idea of requiring SSL from the start. Force everyone to non-SSL at first, then make a proper choice and redirect appropriately. This also might mean not using HTTP-Strict-Transport-Security. This has other ramifications- the idea behind moving to SSL was to ensure the integrity of the whole product delivery pipeline. Requiring HTTP (not HTTPS) at some point undermines that objective by introducing a point in the chain where the user could be intercepted.
Flags: needinfo?(nmaul)
No solution seems elegant enough to be maintainable long term. For the time being, we need to stick with SHA-1 certificates everywhere backward compatibility is required. Some sites that use advanced browser features (webrtc, ...) don't need backward compatibility and can move to sha-256 certs. But for all the others, we stick to SHA-1 by default.

In the meantime, we will experiment with various certificates settings, and see if there is a way to provide a sha-256 cert without breaking very old clients. Windows XP / IE6 pre-sp3 is the most visible issue, but I'm guessing there are dozens of scripts relying on ssl libraries that don't support sha-256, and will break with the move. We've seen that issue with Java 6 and DH key size before. If we want to make progress, we will first need a reliable way to log handshake failures, and detect clients that break.

Work is ongoing between OpSec and FF platform security team. We'll figure something out.
Summary: Update the TLS certificate for https://www.mozilla.org to not use sha-1 → Do not update the TLS certificate for https://www.mozilla.org, it must use sha-1
The long-term fix if you want to avoid having some HTTP in the mix is to have the landing page served from a site which can detect clients by examining the advertised cipher suite list in the SSL handshake. It's possible to detect older browsers this way (or newer, if you do it that way round), and then provide a SHA-1 cert, whereas everyone else gets SHA-256.

Of course, coding and deploying that is not trivial, and it'll need some careful testing. But we aren't the only site who needs this capability, so there may be opportunities for collaboration.

If you need advice from the certificate policy team (who have been rather involved in this issue), you know where to find us :-)

Gerv
(In reply to Julien Vehent [:ulfr] from comment #22)
> No solution seems elegant enough to be maintainable long term. For the time
> being, we need to stick with SHA-1 certificates everywhere backward
> compatibility is required. Some sites that use advanced browser features
> (webrtc, ...) don't need backward compatibility and can move to sha-256
> certs. But for all the others, we stick to SHA-1 by default.

Also maybe worth noting that some sights have a target audience that is less likely to be affected. Addons for example is probably extremely heavily weighted to Firefox users, who aren't affected. AUS is Firefox-only. MDN caters to web developers, who seem likely to be more up-to-date and therefore less likely to be affected. Services aimed at Mozillians are probably fine also (TBPL, wikimo, etc).

It's the ones aimed at non-Mozillians and/or non-Firefox users that are the bigger concern. Off the top of my head that means product delivery for sure, and possibly blogmo (press/PR usage) and SUMO ("my Firefox is broken so I have to use IE/Chrome to get help").
FYI, the plan is to show user-visible warnings on SHA-1 sites starting January 2016. That will be almost 2 years after XP EOLed, and 6 years after SP3 was released.

https://blog.mozilla.org/security/2014/09/23/phasing-out-certificates-with-sha-1-based-signature-algorithms/
FYI, I am thinking of a proposal to kill XP SP2 support in 2015 and XP SP3 support in 2016. MS will release the next version of Visual Studio in 2015 which is unlikely to support XP/Server 2003, but Mozilla will not upgrade immediately. I suggest to time the end of XP SP3 support just after an ESR release to help Custom Support customers.
(In reply to Yuhong Bao from comment #26)
> FYI, I am thinking of a proposal to kill XP SP2 support in 2015 and XP SP3
> support in 2016. MS will release the next version of Visual Studio in 2015
> which is unlikely to support XP/Server 2003, but Mozilla will not upgrade
> immediately. I suggest to time the end of XP SP3 support just after an ESR
> release to help Custom Support customers.

There is no existing proposal to kill XP SP2 support in 2015, FWIW.  Please ignore this comment.
Mozilla IT operates under the assumption that mozilla.org must be accessible by the highest possible number of people. We will maintain backward compatibility until a global decision to deprecate old clients is made at the leadership level.

When that happens, the changes will be reflected in https://wiki.mozilla.org/Security/Server_Side_TLS#Recommended_configurations . Until then, mozilla.org will apply the old configuration parameters.
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → WONTFIX
The current summary and resolution are confusing. Please remove "Do not" from the summary or change the resolution to "FIXED".
Summary: Do not update the TLS certificate for https://www.mozilla.org, it must use sha-1 → Update the TLS certificate for https://www.mozilla.org to not use sha-1
I don't you use normal HTTP? Then the website will be avaliable for everyone!
I think we users don't need that much security just to download a aplication.
Or create a separate website in normal HTTP just do download firefox and nothing else.
Erratum

Why don't you use normal HTTP? And then the website will be avaliable for everyone!
I think we users don't need that much security just to download a aplication.
Or create a separate website in normal HTTP just do download firefox and nothing else.
(In reply to Miguel from comment #30)
> I don't you use normal HTTP? Then the website will be avaliable for everyone!
Please no. Bad https is still better than no https. An attacker could replace the Download with some own crap.
(In reply to Miguel from comment #31)
> Erratum
> 
> Why don't you use normal HTTP? And then the website will be avaliable for
> everyone!
> I think we users don't need that much security just to download a aplication.
> Or create a separate website in normal HTTP just do download firefox and
> nothing else.

We switched to HTTPs from HTTP because as there is a security risk with distributing a download over HTTP as an attacker could get in the middle and serve up an infected download.
 > We switched to HTTPs from HTTP because as there is a security risk with
> distributing a download over HTTP as an attacker could get in the middle and
> serve up an infected download.

Opera browser is downloaded all the time using HTTP (non-ssl), and they don't think there's a security problem.
I have an idea: a user could download the web installer using HTTP and then the installer would download the rest of the files using HTTPS and/or the installer could verify the checksums from a separate server using SSL
(In reply to Miguel from comment #34)
> Opera browser is downloaded all the time using HTTP (non-ssl), and they
> don't think there's a security problem.
> I have an idea: a user could download the web installer using HTTP and then
> the installer would download the rest of the files using HTTPS and/or the
> installer could verify the checksums from a separate server using SSL

It's still dangerous. Because a man-in-the-middle can replace the installer with his Trojan, so that at the time you run the downloaded installer, the damage is already done.
(In reply to Julien Vehent [:ulfr] {Unavailable until July 13th} from comment #22)
> Work is ongoing between OpSec and FF platform security team. We'll figure
> something out.

What has been figured out? As of last month, the current download.mozilla.org site gives an SSL icon warning (lock has a yellow triangle) in the latest stable Chrome (42), because you are using a SHA-1 cert that expires after Dec 31st 2015. So everyone who comes to download Firefox using Chrome (hopefully a large group!) sees a warning. If you wanted an impact-free transition, you've run out of time... Switching that cert for one which expires in 2015 will fix that for now, but we need a strategy on this fast.

Gerv
Status: RESOLVED → REOPENED
Flags: needinfo?(chrismore.bugzilla)
Resolution: WONTFIX → ---
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1324] → [kanban:https://webops.kanbanize.com/ctrl_board/2/1319]
Assignee: gozer → server-ops-webops
Assignee: server-ops-webops → gozer
(In reply to Gervase Markham [:gerv] from comment #36)
> (In reply to Julien Vehent [:ulfr] {Unavailable until July 13th} from
> comment #22)
> > Work is ongoing between OpSec and FF platform security team. We'll figure
> > something out.
> 
> What has been figured out? As of last month, the current
> download.mozilla.org site gives an SSL icon warning (lock has a yellow
> triangle) in the latest stable Chrome (42), because you are using a SHA-1
> cert that expires after Dec 31st 2015. So everyone who comes to download
> Firefox using Chrome (hopefully a large group!) sees a warning. If you
> wanted an impact-free transition, you've run out of time... Switching that
> cert for one which expires in 2015 will fix that for now, but we need a
> strategy on this fast.
> 
> Gerv

I guess this might be a moot point with the mozilla.org certificate expiring at the end of 2015, but do anyone know what is the marketshare of XP SP2 on Firefox right now?
Right now my thinking is to move mozilla.org to a SHA2 cert at the end of 2015 (affecting only new Firefox downloads), and end XP SP2 support completely with the move to VS2015 in 2016.
(In reply to Yuhong Bao from comment #37)
> I guess this might be a moot point with the mozilla.org certificate expiring
> at the end of 2015, but do anyone know what is the marketshare of XP SP2 on
> Firefox right now?

No; you would need to ask the Metrics team. I believe we were losing a lot of downloads last time we tried to switch.

(In reply to Yuhong Bao from comment #38)
> Right now my thinking is to move mozilla.org to a SHA2 cert at the end of
> 2015 (affecting only new Firefox downloads), and end XP SP2 support
> completely with the move to VS2015 in 2016.

I think you should make sure you have signoff on that plan from the Firefox product managers; as I said above, last time we tried this, it lost us more downloads than we were comfortable with. That's why I filed bugs months ago to give the ops team time to implement some sort of split certificate solution. However, we don't seem to have got anywhere yet...

Gerv
Hi all,

This bug really isn't a good place to be discussing our future product plans for Bedrock and SHA-1 / XPSP2 support. As comment 28 (correctly) states:

(comment #28)
> Mozilla IT operates under the assumption that mozilla.org must be accessible
> by the highest possible number of people. We will maintain backward
> compatibility until a global decision to deprecate old clients is made at
> the leadership level.

The following question needs to be made by the module owners for Bedrock, whoever that is:

Should Bedrock support only SHA-1, only SHA-2, or dual SHA-1/SHA-2 based on client TLS version?

And it needs to be answered for three timelines:

2015 EOY: Our current SHA-1 certificate expires
2016 EOY: Certificate expiration date cutoff for SHA-1 certificates
2017+: The only way to support SHA-1 will be to issue a self-signed SSL certificate.

I'm re-closing this bug as there's no further actionable work for Webops to do here. If you open another bug to track the module owner conversations, please See Also this bug so we can keep track of things.

If anyone is considering reopening this bug, please do so only with Bedrock owner approval and with a clear statement of actionable changes we are approved to take on their behalf.

(For the record, I wholeheartedly support answering this question, and we need an answer - and, more specifically, a deprecation plan - as soon as one is decided upon.)
Status: REOPENED → RESOLVED
Closed: 10 years ago9 years ago
Flags: needinfo?(chrismore.bugzilla)
Resolution: --- → FIXED
Richard, you forwarded me mail last week that talked about Edgecast switching to SHA-2 certs. I'm not sure if our other CDN providers have already or will be making the switch as well, but with Edgecast switching any XP SP2 users that end up at Edgecast when downloading through www.mozilla.org will end up with a failure AFAICT.

It seems like we're going to end up with no reason not to do this if our CDNs stop supporting SHA-1.
Not necessarily; we may shunt XP SP2 users off to a different download location which uses a SHA-1 cert.

Laura Thomson is pulling together a team to consider all the ramifications of this and get a decision from the leadership about whether it's worth investing resources in making this continue to work. Let's not take any decisions before that happens (other than to keep things working, which means changing the www.mozilla.org cert to one which expires in 2015 sometime, to avoid Chrome warnings).

Gerv
(In reply to Gervase Markham [:gerv] from comment #43)
> Not necessarily; we may shunt XP SP2 users off to a different download
> location which uses a SHA-1 cert.
> 
> Laura Thomson is pulling together a team to consider all the ramifications
> of this and get a decision from the leadership about whether it's worth
> investing resources in making this continue to work. Let's not take any
> decisions before that happens (other than to keep things working, which
> means changing the www.mozilla.org cert to one which expires in 2015
> sometime, to avoid Chrome warnings).
> 
> Gerv

Sounds good to me. I'm trying to get XP SP2 usage information to help make a decision about what to do with code signing. Seems like that (bug 1178797) might be helpful here too.
Absolutely. You should check in with Laura - it's all part of the same issue.

Gerv
Ben & Gerv: Richard [:atoll] and I are leading the work on the SHA-1/SHA-2 issue and the impact to www.mozilla.org. bug 1178797 contains very useful information. Moving forward, it would be great if you could CC us on other bugs related to the issue. Thanks!
This bug is marked as resolved. Is there a follow-up bug for the migration to SHA-2 or should this one be reopened?
It shouldn't have been marked fixed either way; seems like INCOMPLETE or WONTFIX, with a new bug for attempting again in the future makes most sense?
Correct, sorry.

We are retaining SHA1 at this time. We will eventually deploy a solution of some kind that dual-stacks SHA1/SHA2 based on client version. CloudOps is working actively on a solution here - reach out to Travis by email/irc for more information, if you need it.

But, today, we're WONTFIX'd (bug status repaired to reflect) and no further action is due here.
Resolution: FIXED → WONTFIX
I'd be OK with simply renewing the SHA1 cert for an extra year if needed.
We'll take that under advisement. Thank you!
But remember that existing Firefox users would not be affected, only new downloads using IE on XP SP2.
Incorrect. There are several combinations of platform/browser - not just MSIE6 on XPSP2 - that are at risk, and certain components of Firefox use SHA1 in various ways that cause problems beyond the initial download. We formed a working group that studied this problem during Q2 and Q3. The results are not currently public as they contain private data, but if we are able to publish them someday, I will link them here for your review.
Yea, I know that if anything in Firefox uses Windows SChannel for connecting to Mozilla servers, they would be affected too, but I thought they wasn't doing that.
I provided a solution on bug 901393 comment 18, you may have a look there.
Unfortunately, the solution in bug 901393 comment 18 does not meet our needs. We must negotiate some kind of valid SSL to these clients, even if it's bad SSL, and so we'll have to continue as planned (per comment 49) with implementing a dual SHA1/SHA2 solution.
Mozilla has implemented mechanisms that will allow users of old browsers, including those which do not support SHA-2, to download Firefox. Firefox supports SHA-2 on all platforms, including older ones. Users should always make sure to update to the latest version of Firefox for the most-recent security updates and features by going to: https://www.mozilla.org/firefox/
Resolution: WONTFIX → INVALID
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.