Closed Bug 700955 Opened 14 years ago Closed 13 years ago

machine with lots of RAM for research group

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gal, Assigned: dumitru)

Details

Attachments

(3 files, 2 obsolete files)

Cindy's SSH public key, laptop
611 bytes, text/plain
Details
suggested "/etc/krb5.conf" for this machine
698 bytes, text/plain
Details
multiple SSH public keys for "liblit" account
1.78 KB, text/plain
Details
We need a machine with as much RAM as we can reasonably afford, and a couple fast CPUs and a fast disk for a static analysis project. 192GB seems affordable from what I can tell. Can you spec out a machine for us? It can be located in a data center. We only need remote access (Linux).
Assignee: server-ops → cshields
When do you need this? What is it for? (please justify this purchase, because chances are the RAM is going to make it expensive enough that we will have to justify it) AMD or Intel? "couple fast CPUs" - CPUs are not getting much faster, their cores are getting more numerous. How many cores do you need? How much disk? Once we have these answers I'll ask Rich to spec something out.
> When do you need this? As soon as possible; we have a research project that is currently blocked on it. > What is it for? (please justify this purchase, because chances are the RAM > is going to make it expensive enough that we will have to justify it) This is for Research: we are currently doing static analysis projects on the entire Mozilla codebase. Down the road we will be doing more and more advanced algorithms projects, in particular in static analysis, that will need a lot of RAM. > AMD or Intel? I don't think this matters much to us. Andreas can correct me. > "couple fast CPUs" - CPUs are not getting much faster, their cores are > getting more numerous. How many cores do you need? I would say "as many as reasonable." We don't have a specific number we need, but we will also be doing parallel algorithms work, so having many cores would be helpful for Rust and Servo, as well as some static analysis projects. > How much disk? I'd say at least 1TB. Thanks, Dave
We only need this machine part-time, so we can time-share if IT sees other uses for this.
Rich, can you come up with an Intel DL for this? Size does not matter, we'll have space in phx1 in a few weeks.
Corey - we can do a DL360 with 2 x 6 core procs/192 GB RAM - what would you like to do for hard drives ? 4 x 500gb or 1tb sata ?
(In reply to Rich Pomper from comment #5) > Corey - we can do a DL360 with 2 x 6 core procs/192 GB RAM - what would you > like to do for hard drives ? 4 x 500gb or 1tb sata ? 4 x 500GB SAS please. We'd like to avoid SATA.. thanks!
What's the status of this order? Thanks, Dave
dmoore: new phx1 module for this?
cshields: we have plenty of 1Gbe server capacity in the module right now, and we can accelerate the 10Gbe install if it would block this.
(In reply to Derek Moore from comment #9) > cshields: we have plenty of 1Gbe server capacity in the module right now, > and we can accelerate the 10Gbe install if it would block this. This is 1Gbe. Will ship it there, thanks!
configured DL360 shipped to PHoenix via Fed Ex Tracking no.: 040985925417394. ETA is 1/30
(In reply to Rich Pomper from comment #11) > configured DL360 shipped to PHoenix via Fed Ex Tracking no.: > 040985925417394. ETA is 1/30 Thanks Rich, seems like this was delivered to phx. CC'ing phong who's heading to the DC tomorrow.
Phong, this will go in the 1gig rack in phx1 module 8. Should be the only rack with a powered up 1g switch.
Assignee: cshields → dgherman
What will run on that server? Will it be a webapp or a standalone box?
Needs a Linux distro, with SSH access for: - Andreas Gal (gal@mozilla.com) -- needs sudo - Dave Herman (dherman@mozilla.com) -- needs sudo - Cindy Rubio-Gonzalez (crubio@cs.wisc.edu) -- doesn't need sudo Cindy does not have an LDAP account. She's a researcher at the University of Wisconsin helping us do analysis work. Thanks, Dave
Lets add sudo for Michael Bebenita and drop me from the list.
One more external collaborator: Cindy's advisor, Ben Liblit (liblit@cs.wisc.edu) -- doesn't need sudo either. Dave
We believe the best method for granting access would be via OpenVPN to the data center and then ssh to the host. I'm not sure what the protocol is for getting Cindy LDAP credentials as that is the authentication method for OpenVPN, but I'm pretty sure there is precedent for it.
Ravi, per my email, can we restrict Cindy's access to this server?
Yes, but I'm trying to see if we can in a time that is compatible with when this bug needs to be resolved.
If OpenVPN isn't ready for per user IPTables policies, how about the Juniper VPN. We could create a profile for Cindy that gives her ssh to this box.
Is there a chance to get this bug closed tomorrow? That would be great. Thanks!
Joe, yes, I think that is the best possible action at this point. I'll work on it in the morning. Andreas, we'll try.
Whats the status here?
Sorry, I wasn't able to complete this before my flight. I'm in London now catching up on the travel gap and will complete this over the weekend if not this evening. My concern is method of communicating the temporary password to Ben and Cindy. An IM with OTR or GPG encrypted email are preferred.
One thing that is missing are the host details (IP). Even though the accounts are created on the VPN ACLs to that host need to be created. Once provided we can get those in place and verify the VPN.
(In reply to Ravi Pina [:ravi] from comment #25) > An IM with OTR or GPG encrypted email are preferred. I can't speak for Cindy, but GPG encrypted e-mail is fine with me. My public key is available from your friendly neighborhood public key server, or at <http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x34DEE95F2C720028>.
(In reply to Ravi Pina [:ravi] from comment #26) > One thing that is missing are the host details (IP). Even though the > accounts are created on the VPN ACLs to that host need to be created. Once > provided we can get those in place and verify the VPN. staticanalysis1.dmz.phx1.mozilla.com
Sent GPG encrypted message with VPN credentials and access. I don't see host credential information in this bug so I assume they were also sent out of band.
Would it be possible to get my initial password over the phone? Thanks!
Ping -- Cindy is still waiting for help with her password. Can someone please address this ASAP? Thank you, Dave
I'm still waiting too. I do have VPN credentials from Ravi (per comment #29). But I still have no information about my username and password for staticanalysis1.dmz.phx1.mozilla.com once the VPN is up.
I am currently in London (UTC), tomorrow (Mar 1) is a travel day, and then I'll be in Paris (UTC+1) Mar 1-Mar 5. Can you find a time where the green overlaps with your time zone[1] and mail me your phone number and/or skype id to ravi@mozilla.com? [1] http://www.timeanddate.com/worldclock/meetingtime.html?p1=137&p2=75&p3=250&p4=136&p5=195&p6=236&p7=22
(In reply to Ravi Pina [:ravi] from comment #33) > Can you find a time where the green overlaps with your time zone[1] Who is "you" here? Me? Cindy? Dave? I'm guessing just Cindy, as I already have VPN access and I think that host access is not Ravi's job. Ravi, please clarify if I am mistaken.
(In reply to Ravi Pina [:ravi] from comment #33) > Can you find a time where the green overlaps with your time zone[1] and mail > me your phone number and/or skype id to ravi@mozilla.com? > Hi Ravi. I sent you an e-mail with the requested information. Thanks!
Juniper's SSH java client does not support SSH key authentication and is not very convenient to work with for the user. We should use Juniper's network connect (which requires some setting up) that let the user connect to SSH, through the VPN, with their own SSH client (putty, openssh, etc) and using SSH key authentication. This also avoid breaking our current Puppet policy on "no SSH password auth".
I got the SA OS upgraded and verified that the Network Connect (NC) loads on my host. I have some ACLs to update for this config yet.
I updated the NC policies and opened the flow to staticanalysis1.dmz.phx1.mozilla.com. Ben and Cindy, can you attach your ssh public key to this bug so it can get pushed to your user accounts on the host. You will still access https://vpn.mozilla.com/ (Cindy, I'll explain this on our call), but then you'll run the Network Connect which is essentially a VPN client. From there you can use ssh on your local host vs. having to use the Java client. I'm still fiddling with DNS, but until that is working well the IP is 10.8.74.5.
I spoke with Cindy and gave her the VPN credentials and instructions for connecting. Once the public SSH keys are received they will be pushed to the host and the team can begin using it.
Component: Server Operations → Release Engineering: Machine Management
Attached file Ben's SSH public key (obsolete) —
Attached file Ben's SSH public key, laptop (obsolete) —
Drat, looks like that Network Connect tool requires root access for some first-time setup. Cindy and I do not have root access to our desktop machines. I can probably ask our system administrators to do whatever is needed here, but I'd rather not waste their time on this until we're sure that it is actually working and necessary. For now, I'm going to use my laptop to connect rather than my work desktop, since I *do* have root access on my laptop. I'm replacing the SSH public key I attached earlier with the one I use from my laptop. (I use distinct SSH public keys for home/work/laptop.) Is there really no way to cross the VPN barrier using more standard tools on my side, such as vpnc or openconnect?
Attachment #602364 - Attachment is obsolete: true
VPN fail. Utter VPN fail. Once the Network Connect VPN client is up and running, I cannot reach 10.8.74.5 (the IP address given in comment #38). I also cannot resolve *any* hostnames: DNS is completely broken. It seems my DNS server My laptop is currently sitting on my private home network. This private network uses IP addresses from 192.168.0.0/24, with my DSL modem using typical NAT to the outside world. Before I start up the Juniper Network Connect tool, my machine's routing table looks like this: Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 wlan0 192.168.0.0 0.0.0.0 255.255.255.0 U 2 0 0 wlan0 After starting up the Network Connect tool, my machine's routing table looks like this: Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.2.74.200 0.0.0.0 UG 1 0 0 tun0 0.0.0.0 192.168.0.1 0.0.0.0 UG 10 0 0 wlan0 63.245.209.252 192.168.0.1 255.255.255.255 UGH 1 0 0 wlan0 192.168.0.0 0.0.0.0 255.255.255.0 U 2 0 0 wlan0 192.168.0.1 0.0.0.0 255.255.255.255 UH 1 0 0 wlan0 So it seems we have added a second default route (genmask "0.0.0.0") going through the VPN tunnel, using gateway 10.2.74.200. (That gateway address is in fact the address that the VPN client reports as assigned to my machine itself.) I don't care for the fact that the VPN wants to take over *all* traffic like this. I would prefer it be more selective, only being used for traffic destined for VPN-protected Mozilla hosts. But being the default route for all traffic isn't necessarily fatal, just perhaps inadvisable. Also interesting is that there's a new routing entry given for one specific address, 63.245.209.252. This is apparently the address of vpn.mozilla.com. Not sure why it's here now. It's also been added to "/etc/hosts" by the VPN client. {shrug} I mentioned earlier that DNS is broken. The VPN client has changed "/etc/resolve.conf" to the following: search sjc1.mozilla.com mozilla.org nameserver 10.2.74.125 nameserver 10.2.74.127 So in addition to wanting to own all of my network traffic, the VPN also wants Mozilla to manage all of my DNS lookups. I'd prefer you didn't, but even more so I'd prefer that your name servers actually be reachable. They are not. All DNS lookups fail. Neither name server responds to ping, and mtr to either one makes one hop to 10.200.200.200 after which it dead-ends. So to sum up, activating the Network Connect VPN client (1) does not allow me to reach the desired machine, and (2) prevents me from reaching any other machine outside of my private home network. This is not going well at all.
This bug was filed 3 month ago, and our collaborators still have no access. Please make fixing this bug a top priority. This has really taken too long.
Assignee: dgherman → ravi
This was tested without issue by IP. The VPN is not configured for split tunneling, bug I have gone do gor this profile for testing.
(In reply to Ravi Pina [:ravi] from comment #44) > bug I have gone do gor this profile for testing. Pardon?
Ben, are you able to physically take possession of the machine for the duration of the project? If IT is unable to get remote access to the machine resolved by end of business today, we will ship the machine to you.
Should this bug be in a component different than Release Engineering? Thanks.
Assignee: ravi → server-ops
Component: Release Engineering: Machine Management → Server Operations
QA Contact: cshields → phong
Assignee: server-ops → ravi
(In reply to Andreas Gal :gal from comment #46) > Ben, are you able to physically take possession of the machine for the > duration of the project? I'd need to check that with our system administration group. I don't know how they feel about non-UW-property gear in our machine rooms, but I can certainly ask. What would help me is if someone can post here a good description of the machine from a system administrator's perspective: size in rack units (assuming it is rack-mounted), power requirements, etc. I want to be able to give our sysadmins a clear picture of what I'd be asking them to support so that they can tell me if they're able to take that on or not. We'll also need to figure out who's paying for what. For example, I use my own research grants to pay a fee to our sysadmins for every machine I have on our departmental network. So yes, this is possible, but I need to do some homework. First need: a good description of the machine itself suitable for sending to my sysadmins. Can someone please provide that? Thanks!
Ravi, can you please provide the description requested in comment #48 in parallel of working on a last ditch effort to finally get access working here. Ben, if our IT isn't able to figure out remote access and we have to physically ship the machine and have your staff administer it, we will pay for shipping both ways and cover the admin expenses. Lets prepare for this contingency while Mozilla IT gives this another final shot. I want to unblock your work asap, its really important to us.
I thought I'd try the VPN again from work, where I am no longer behind a residential NAT. Before activating Network Connect, my routing tables look like this: Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 128.105.32.1 0.0.0.0 UG 0 0 0 wlan0 128.105.32.0 0.0.0.0 255.255.252.0 U 2 0 0 wlan0 After Network Connect starts up, my routing tables have been changed to this: Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 128.105.32.1 0.0.0.0 UG 0 0 0 wlan0 10.2.0.0 10.2.74.201 255.255.0.0 UG 1 0 0 tun0 10.8.0.0 10.2.74.201 255.255.0.0 UG 1 0 0 tun0 63.245.209.252 128.105.32.1 255.255.255.255 UGH 1 0 0 wlan0 128.105.32.0 0.0.0.0 255.255.252.0 U 2 0 0 wlan0 128.105.32.1 0.0.0.0 255.255.255.255 UH 1 0 0 wlan0 I guess that looks like a reasonable split-tunneling configuration: anything in 10.2.x.x or 10.8.x.x goes to the VPN, and the rest goes through my regular network. OK, good. Unfortunately I am still unable to reach any hosts of interest within those 10.{2,8}.x.x networks. The static analysis machine at 10.8.74.5 is unreachable via ssh, ping, traceroute, or mtr. Same goes for the two DNS servers at 10.2.74.{125,127}. traceroute can't even get one hop in: "1 * * *". That's a change from my previous attempt, where packets made it one hop to 10.200.200.200 before dead-ending. Not sure if that's helpful. By the way, the Network Connect GUI offers "Logs" and "Diagnostics" tabs. The diagnostics don't *seem* to contain any information I haven't already provided, but I'm no VPN expert. And I haven't looked into the logs at all. Let me know if you want any information from here. I'm happy to provide it.
Please grant Ben mpt-VPN access and provide him directions for openvpn & ssh to mpt-vpn.mozilla.com. Bumping bug so it'll page oncall. Infrasec is okay with this. I'd love to get this working through the SA but that's likely longer term. If you don't mind, what OS is this? I've only ever seen routes "installed" but unreadable on windows when OpenVPN isn't run with admin privs.
Severity: normal → major
We need to open a bug to create an account for Ben. Please attach your key to that bug.
(In reply to matthew zeier [:mrz] from comment #51) > Please grant Ben mpt-VPN access and provide him directions for openvpn & ssh > to mpt-vpn.mozilla.com. Note: Cindy Rubio González will need all of the same. If anything, getting her access is an even higher priority than getting me connected. > If you don't mind, what OS is this? I've only ever seen routes "installed" > but unreadable on windows when OpenVPN isn't run with admin privs. Fedora 16.
Pushing back to oncall. Please create as a contractor account, mpt-vpn access. Ben's key is already attached.
Assignee: ravi → server-ops
Assignee: server-ops → dgherman
Ben, I got your SSH key, thank you. I'll create LDAP accounts for you, but I also need Cindy's SSH public key to add to the server. Thank you.
(In reply to Dumitru Gherman [:dumitru] from comment #55) > Ben, I got your SSH key, thank you. I'll create LDAP accounts for you, but I > also need Cindy's SSH public key to add to the server. > Thank you. Hi Dumitru. I will get that to you in a couple of hours (I don't have my laptop in the office right now). Thanks!
LDAP and Zimbra account has been created. Ssh key has been attached and access to mpt-vpn granted. Sent account information to liblit@acm.org.
Ben, please disregard the email Ann sent to you. I will follow up with the right information in a minute. Cindy, sounds good, thank you!
Status: NEW → ASSIGNED
OK, I've received Dumitru's credentials message, and successfully changed my LDAP password. Yay. What do I do next? I have a vague sense I should be doing something exciting with mpt-vpn.mozilla.com, but I have no idea what. Some instructions, please?
Instructions coming in a few, working now in creating local accounts on the box itself. Will send you everything in a couple of minutes, stand by :)
Ben, Here are the instructions. You need a VPN client installed on your machine (Viscosity is a good one, or OpenVPN). Logins to the VPN are your fully qualified username (your full email address) and the password I emailed to you. Mozilla uses OpenVPN, an open-source, full-featured SSL VPN which implements OSI layer 2 or 3 secure network extension using the industry standard SSL/TLS protocol. Setup involves the following steps: - download/install OpenVPN binaries The files I sent you via email are common to all OS versions and include the pre-shared keys and configuration files. These files live in different locations on each OS. In Windows, it's usually in C:\Program Files\OpenVPN\config. Once you unpack the zip file with the configuration files in the OpenVPN, start OpenVPN client and it will connect to mpt-vpn.mozilla.com. After the VPN is successfully connected, use a SSH client to log into the server. Ben, your username is 'liblit', and you need to use your SSH key you sent me to log in. If all those sound complicated or confusing, I can give you a call tomorrow to walk you through the entire process to make sure you can access the server. Let me know a convenient time (I am in PST) and we can chat on the phone. Thank you!
(In reply to Dumitru Gherman [:dumitru] from comment #61) > Here are the instructions. Success! Using Dumitru's instructions along with the files he e-mailed separately, I can bring up the VPN connection and then SSH into host staticanalysis1.dmz.phx1.mozilla.com. Hooray! Importing the OpenVPN configuration was quite easy once I looked in the right place. I use GNOME under Fedora 16, with network connections managed by NetworkManager. First I installed the "NetworkManager-openvpn" RPM so that NetworkManager would have the right plugin to manage OpenVPN. Then I ran "nm-connection-editor" to bring up the connection editor. In the VPN tab, the Import button let me create a new VPN configuration using settings from the ".ovpn" file sent by Dumitru. Done! > Ben, your username is 'liblit', and you need to use your SSH key you sent me > to log in. Successful, yay! Question: can I provide several *additional* SSH keys? I use distinct keys for distinct machines: laptop, work, home, etc. [My intent in doing this is to contain the damage should any of those keys be compromised.] I looked at ".ssh/authorized_keys" on staticanalysis1.dmz.phx1.mozilla.com, and comments there reveal that it is "autogenerated [...] by pupper. While it can still be managed manually, it is definitely not recommended." So should I go through some other procedure to add keys to that file? Thanks, Dumitru. I'm very glad we finally made it this far.
Now that I can reach staticanalysis1.dmz.phx1.mozilla.com, I can tell we're going to need to get some more things installed here: scons, subversion, ocaml, etc. How should Cindy and I communicate our sysadmin-level needs to someone who can act on them? (Presumably it would be inappropriate for Cindy and I to have root access to the machine ourselves.) The good news is that this staticanalysis1 machine is running RHEL 6.2: the same Linux distribution I use every day. I'm quite familiar with this environment and I know that all the additional tools we'll need are available as standard, pre-packaged RPMs. So we'll need someone with admin powers, but our requests should all be quite simple ones. Whom should we talk to for things like this?
(In reply to Ben Liblit from comment #64) > Now that I can reach staticanalysis1.dmz.phx1.mozilla.com, I can tell we're > going to need to get some more things installed here: scons, subversion, > ocaml, etc. How should Cindy and I communicate our sysadmin-level needs to > someone who can act on them? (Presumably it would be inappropriate for > Cindy and I to have root access to the machine ourselves.) ideally file a bug with the RPMs you need installed - for now, you can just list them here.
Status: ASSIGNED → NEW
Ben, I can add another public key for your account if you need to. Just attach it to this bug. Also, can you tell me the entire list of packages you need to be installed? I installed subversion and scons. For ocaml can you let me know what exact packages do you need? Like Matthew said, for any sysadmin requests you can file a bug (and you can CC me since I am familiar with the setup) and we will help you. Cindy, how are things looking? Were you able to configure your VPN client and SSH into the server?
> I can add another public key for your account if you need to. Just attach it > to this bug. OK, I need to collect these from a few different places. I'll attach the updated collection of public keys soon. (I'm also happy to open a new Bugzilla request for that if you are eager to finally see *this* bug closed.) > Also, can you tell me the entire list of packages you need to be installed? > I installed subversion and scons. For ocaml can you let me know what exact > packages do you need? We will also need the following RPMs: automake db4-devel gcc-c++ krb5-workstation ocaml texlive-latex. Those are the exact RPM names; please install those plus their dependencies and we should be set to build and run our analysis tools. (I built this list by starting with a fairly bare-bones CentOS 6.2 virtual machine, then adding missing packages until I could build our tools and run through their regression tests. So that should be a pretty good approximation of what we'll need on this RHEL 6.2 box.) > Like Matthew said, for any sysadmin requests you can file a bug (and you can > CC me since I am familiar with the setup) and we will help you. OK, sounds good. Thank you all for your help!
In comment #67, I asked that the "krb5-workstation" RPM be installed. I'd also like you to install the attached file as "/etc/krb5.conf", replacing the default Kerberos 5 configuration file. My reason for requesting this is that UW–Madison Computer Sciences uses Kerberos for authentication. We will be checking out our code using Subversion tunneled through SSH to our UW CS machines. If you use the suggested "/etc/krb5.conf", then we should be able to use our UW CS Kerberos credentials to allow tunneled Subversion actions from Mozilla's machine to our machines without repeatedly typing our UW CS passwords. That should be both more convenient and more secure: the fewer times any password is typed while using a remote machine, the better. Thank you!
(In reply to myself from comment #67) > I need to collect these from a few different places. I'll attach the > updated collection of public keys soon. Done. Please add all of the attached SSH public keys to the authorized_keys list for the "liblit" account. There are three keys here in all, corresponding to work, home, and laptop. Thanks!
Attachment #602370 - Attachment is obsolete: true
Ben, I added the two additional keys under your account. I have also installed the requested packages and updated the krb5.conf file. I am waiting for Cindy's reply to see if she has everything in place before closing out the bug.
Thanks Dumitru. Looks like I missed one package: make. Would you please install that RPM as well? Sorry for missing that earlier.
(In reply to Ben Liblit from comment #71) > Thanks Dumitru. Looks like I missed one package: make. Would you please > install that RPM as well? Sorry for missing that earlier. Done.
Almost there! When I ssh, I get the following: debug1: Next authentication method: publickey debug1: Offering DSA public key: /home/rubio/.ssh/id_dsa debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic However, at the end I get a "permission denied" error message. I just compared with Ben. He gets the exact same three lines, but a "key accepted by server" message. Any suggestions? Thanks!
(In reply to Cindy Rubio Gonzalez from comment #73) > Almost there! When I ssh, I get the following: > > debug1: Next authentication method: publickey > debug1: Offering DSA public key: /home/rubio/.ssh/id_dsa > debug1: Authentications that can continue: > publickey,gssapi-keyex,gssapi-with-mic > > However, at the end I get a "permission denied" error message. I just > compared with Ben. He gets the exact same three lines, but a "key accepted > by server" message. Any suggestions? Thanks! Cindy, When you are logging in, use username 'crubio' instead of 'rubio'.
Got it. I have access now. Thanks!
Great! I'll go ahead and close this bug now. For any other requests for this server, please open a new bug using this form: https://bugzilla.mozilla.org/form:itrequest Select "Any other issue" and you can CC me. Don't forget to mention the server name the request is for (staticanalysis1.dmz.phx1) or this bug number. Thanks!
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Per email this server is still in use, so we have extended the warranty for 1 yr, until 2/15/2016. We will need to figure out how to fulfill this need with a replacement by then (AWS, replacement server, etc).
Thanks, Corey!
Thank you!
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: