Closed Bug 634770 Opened 13 years ago Closed 13 years ago

Deploy staging server for new TCM

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: carljm, Assigned: fox2mike)

References

Details

Attachments

(2 files, 1 obsolete file)

The new in-development Test Case Management system for QA is in two parts: a Java-based platform built by uTest that provides an HTTP API, and a Django-based UI frontend that speaks to that API. Both will need staging deployments on Mozilla infrastructure.

The platform repo is at https://github.com/mozilla/tcmplatform. It requires MySQL and jBoss. README file in that repo summarizes how we've done development setup for it; hopefully that provides the information you need for deployment.

The UI is in the ui/ subdirectory of https://github.com/mozilla/tcm. The README at https://github.com/mozilla/tcm/blob/master/ui/README.rst should provide the info needed to get it running.

Currently the only infrastructure dependencies (besides the platform server) are memcached and an email server (currently only used for sending error emails), though Redis will likely be added as a dependency within the next release or two.

If it's useful, I have Chef cookbooks that I currently  use for deploying this project under Nginx and Gunicorn, and can provide them.
Is this externally facing?
(In reply to comment #1)
> Is this externally facing?

I'm guessing no (it's still a very early-stage preview release for QA testing at this point), but Aakash will need to confirm that.
We use puppet for deployment, but Chef is similar enough that it might be useful to help us get it going faster.
> Is this externally facing?

It's not for the for-seeable future.
Attached file sample chef cookbooks (obsolete) —
Attached my chef cookbooks for setting up a server to host this application (both the platform and the UI). I'm not a "real" sysadmin, so won't vouch for having done anything the right way here, but these do work, and hopefully can be helpful as an example.
We've now branched a 0.1.X branch for 0.1 release fixes, and are working on new features in master. This staging deployment should come from the 0.1.X branch at this point, and later from future release branches.

If it would be better to have a single branch that always represents the "state that's ready for staging deployment" and not have to switch the deployed branch at each release, I can easily create and maintain a "staging-deploy" branch.
Whiteboard: [after ffx4]
If this is QA, what all will it need to be able to access?  I'm missing a lot of details to determine what kind of infrastructure this needs.  Something like this would help out a lot:  https://bugzilla.mozilla.org/show_bug.cgi?id=642173#c6
(In reply to comment #7)
> If this is QA, what all will it need to be able to access?  I'm missing a lot
> of details to determine what kind of infrastructure this needs.  Something like
> this would help out a lot: 
> https://bugzilla.mozilla.org/show_bug.cgi?id=642173#c6

Hi Corey,

I'm denied access to the bug you linked, so I'm not sure what details you're looking for. Can you give an example of what kinds of things you're thinking of when you ask "what it will need to be able to access"? I think all of the infrastructure dependencies are documented above or in the README files of the two github repos.

Or maybe Aakash can help provide the necessary info.
(In reply to comment #8)

> I'm denied access to the bug you linked, so I'm not sure what details you're
> looking for. Can you give an example of what kinds of things you're thinking of
> when you ask "what it will need to be able to access"? I think all of the
> infrastructure dependencies are documented above or in the README files of the
> two github repos.
> 

hrmm..  I guess a good starter is to ask "What does this website/app do?"
(In reply to comment #9)
> hrmm..  I guess a good starter is to ask "What does this website/app do?"

It is the upcoming replacement for Litmus. It provides a web-based system for testing managers to create manual testcases for software and assign them to testers, and for testers to log in and run those test cases, reporting passed/failed results on each test case.

It's split into a Java backend which stores data in MySQL and provides an HTTP API, and a Django frontend which talks over HTTP to the Java backend, and also relies on memcached. The Django front-end is the only piece that needs to be exposed to clients (currently internal-only, as Aakash mentioned above). Besides those elements there is nothing else that it needs access to, or needs to talk to.
Assignee: server-ops → jdow
Assignee: jdow → cshields
Carl are you on IRC?  (can you ping me - cshields)  I want to gather some more requirements here..  Probably easiest to do in real time.  Thanks
What we can do here is setup the front end servers to be on generic with a couple of nodes on the back end running the java procs.  If we can get by with it I would like to try doing this on the seamicro nodes.

The generic cluster should be available in a week or two (it is currently undergoing a move to phx) but for the java side we may need a new puppet module to get them going.  I haven't looked at the chef files yet but they might be a good start.  I'll work with Jake on this.
Assignee: cshields → nmaul
Whiteboard: [after ffx4]
Assigning to Corey to pick out 1-2 Seamicro nodes as the Java backend servers for this.
Assignee: nmaul → cshields
Sorry, got these assigned via IRC:

tcm-app1.seamicro.phx1
tcm-app2.seamicro.phx1
Assignee: cshields → nmaul
Just a note that there have been some minor changes in project layout for the UI; README is up to date but the Chef tarball configs here are out of date. Also I think I've fixed some minor bugs in the Chef configs since I uploaded? In any case, if you want the updated Chef configs let me know and I'll tar them up again and attach.
Assignee: nmaul → jthomas
I am not able to access the github repo listed above. Are these repo's marked private?
Hi Jason,

Apologies for the inconvenience.  We just changed the names of the repos to reflect the official naming of the product.  The name changes are as follows:

tcm --> caseconductor-ui
tcmplatform --> caseconductor-platform

Jason, what is the current status of this bug?
I am beginning to setup the puppet configurations to support the application. I will keep you updated as I make progress.
Jason, this is great.  And great timing as we would like to start using this at some point fairly soon.  Please feel free to ping me on IRC, if you like.  The team hangs out in the channel #caseconductor, as well.
Whiteboard: allhands
Whiteboard: allhands → [allhands]
Component: Server Operations → Server Operations: Web Operations
QA Contact: mrz → cshields
Hi Corey,

Can I lend any assistance to getting this going?  We'd like to start using this in-house for dog-fooding very soon.  Urgency is beginning to grow for this.  Perhaps you and I could chat to coordinate things.

Thanks, Cam
Assignee: jthomas → shyam
Whiteboard: [allhands]
Carl/Cameron : Can we use tomcat instead of jboss? or is jboss needed as a hard requirement?
FWIW, since https://github.com/mozilla/caseconductor-ui/commit/0061186c542d3f0b2be3007369ff422421772b4a everything needed to run the CC platform is present in the caseconductor-ui repo (a prebuilt war file, the db scripts, etc), at the correct versions for that version of the UI. So checking out the caseconductor-platform repo and building the war file with Maven should no longer be necessary at all.
(In reply to Shyam Mani [:fox2mike] from comment #21)
> Carl/Cameron : Can we use tomcat instead of jboss? or is jboss needed as a
> hard requirement?

I've never tried to run it under Tomcat. The platform authors at uTest don't run or test it under Tomcat, as far as I am aware.

Cameron can correct me if this is wrong, but I think the only real requirement is that it run :-) If you're able to get it to run under Tomcat instead of jBoss, go for it.

If Tomcat is your preference and you're not able to get it running, you could open a bug on the TCM-Platform component for uTest to fix that.
Shyam:
Oops.  Shyam: I just added Vadim Kissen as a CC on this bug.  He wrote the platform, so he knows whether it will run with Tomcat or not.  Vadim, would you chime in on Shyam's question of Tomcat vs. JBoss?
Carl/Cameron :

I'm working on converting the chef stuff to puppet. 

All the github links are 404'ing for me. https://github.com/mozilla/tcmplatform for example or the Readme links in comment #0. Could be a transient issue though.
Also, can I have an answer on tomcat? We have a lot of in-house tomcat experience and none with jboss, so it's a bug plus if we can use tomcat and not jboss.
(In reply to Aakash Desai [:aakashd] from comment #4)
> > Is this externally facing?
> 
> It's not for the for-seeable future.

(In reply to Carl Meyer from comment #10)
> > hrmm..  I guess a good starter is to ask "What does this website/app do?"
> 
> It is the upcoming replacement for Litmus.

These two statements are in conflict.  Which one is correct?

Litmus is a public-facing application.  If this is replacing Litmus, this is going to be public, eventually.
(In reply to Shyam Mani [:fox2mike] from comment #26)
> I'm working on converting the chef stuff to puppet. 

My Chef stuff has changed quite significantly since the tarball I uploaded here. Fixed various bugs in it, and more importantly started using the platform war file which is now pre-built and bundled in the UI repo, rather than building a new war file with maven every time. I'll upload the latest here so you can work off that instead.
 
> All the github links are 404'ing for me.
> https://github.com/mozilla/tcmplatform for example or the Readme links in
> comment #0. Could be a transient issue though.

No, not a transient issue; the project now has a name (Case Conductor) and the repos were re-named a month or so ago to reflect that: github.com/mozilla/caseconductor-ui and github.com/mozilla/caseconductor-platform. This change is also reflected in my latest Chef tarball.
Attachment #512980 - Attachment is obsolete: true
(In reply to Dave Miller [:justdave] from comment #28)
> (In reply to Aakash Desai [:aakashd] from comment #4)
> > > Is this externally facing?
> > 
> > It's not for the for-seeable future.
> 
> (In reply to Carl Meyer from comment #10)
> > > hrmm..  I guess a good starter is to ask "What does this website/app do?"
> > 
> > It is the upcoming replacement for Litmus.
> 
> These two statements are in conflict.  Which one is correct?
> 
> Litmus is a public-facing application.  If this is replacing Litmus, this is
> going to be public, eventually.

Dave,
Apologies for the confusion.  Yes, this will be externally facing.  We expect to launch 1.0 of the product sometime before the end of the year.  Possibly November sometime.
Shyam: here's the reply from Vadim:

============
We used to run our uTest platform on Tomcat before switching to JBoss, so I
don't see any issues with that. If they have Tomcat in-house they can just
build the app and deploy it there. The only issues that can arise are
ClassLoader conflicts between internal Tomcat libraries and the ones
application uses, but I can't tell for sure whether it will be an issue or
not....
============

Do you know how to avoid the problems he's referring to?
Cameron/Carl :

I'd like to set this up by hand first then and get it working on stage. I will then write up puppet as needed and then we'll go to production with puppet. 

Are there setup steps somewhere? Thanks!
Shyam: in each repo, there is a README file that has all the setup steps you should need.  If they're not clear, please let me know and I'll see about getting them fixed.

And let me know if I can help at all.  We're eager to start using this.  :)
(In reply to Cameron Dawson [:camd] from comment #34)
> Shyam: in each repo, there is a README file that has all the setup steps you
> should need.  If they're not clear, please let me know and I'll see about
> getting them fixed.
> 
> And let me know if I can help at all.  We're eager to start using this.  :)

As of last week, you shouldn't need to use the caseconductor-platform repo or its README at all, unless you need to modify the platform and rebuild its war file for some reason. The caseconductor-ui repo now has everything you need in it, including a prebuilt platform war file at the correct platform version, the database scripts, the mysql connector jar, etc. So just use the caseconductor-ui repo and reference its top-level README and the README inside the platform/ subdirectory.
(In reply to Cameron Dawson [:camd] from comment #34)

> And let me know if I can help at all.  

I don't see a wsgi file, we usually run mod_wsgi with apache, so that needs to be fixed. I've filed a bug for access to the database servers, so I can set that up. You might want to look at schematic for managing those (vs a shell script).

Also, do we have to install external dependencies? I see you have a requirements/* and we usually don't do that, just make use of vendor libraries instead : http://playdoh.readthedocs.org/en/latest/packages.html so you can fork playdoh's vendor and add stuff, or create your own vendor that I could use.

Thoughts? I'll add more stuff as I run into it.
(In reply to Shyam Mani [:fox2mike] from comment #36)
> I don't see a wsgi file, we usually run mod_wsgi with apache, so that needs
> to be fixed. I've filed a bug for access to the database servers, so I can
> set that up. You might want to look at schematic for managing those (vs a
> shell script).

I'll add a wsgi.py file.
 
> Also, do we have to install external dependencies? I see you have a
> requirements/* and we usually don't do that, just make use of vendor
> libraries instead : http://playdoh.readthedocs.org/en/latest/packages.html
> so you can fork playdoh's vendor and add stuff, or create your own vendor
> that I could use.

I have all the sdist tarballs present in https://github.com/mozilla/caseconductor-ui-reqs, which is a git submodule in requirements/dist (similar to how the vendor lib in amo is a submodule). I much prefer bundling sdists over using a vendor dir, as its less hassle to maintain, and doesn't require as much sys.path munging or separate handling of platform-specific/compiled dependencies. With the bundled sdists you still get super-fast installation that doesn't touch the network (by using "pip install requirements/prod.txt --no-index --find-links file:///path/to/sdists/dir", which is what the included bin/install-reqs script does).

If that's not adequate, I can create a vendor lib. I'll keep the requirements file and bundled sdists, and write a script to automatically build/rebuild the vendor module from that.
Carl,

I'd still prefer vendor, if we're to deploy this across 10 machines, I'd be happier skipping this one command (even though it's local command). 

The reason for this is we have an admin box, which holds all this stuff, and then gets pushed out to a number of nodes, via git. Would be easier with a vendor to do that vs using that + a command to create stuff.

Does that make sense? Would be awesome if we could stick to that. You guys could still maintain dist, but generate a vendor and have it checked in, so I can with a few git commands have everything I need to run this?
Sure, I'll create a vendor lib. It's ok if it's a separate git repo that's pulled in as a submodule?

Also, I'm not sure how this works if we end up needing compiled dependencies (we'll probably need PIL sooner or later, for example, and possibly lxml). Do you just install those at the system level with the OS package manager?

I've created bug 693715 for this and bug 693607 for the wsgi file; I'll try to knock those both out today.
(In reply to Carl Meyer from comment #39)
> Sure, I'll create a vendor lib. It's ok if it's a separate git repo that's
> pulled in as a submodule?

Sure, no problems.

> Also, I'm not sure how this works if we end up needing compiled dependencies
> (we'll probably need PIL sooner or later, for example, and possibly lxml).
> Do you just install those at the system level with the OS package manager?

Yes, that's what we do.

Thanks, again! Do update the readme files (if there are any changes).
(In reply to Shyam Mani [:fox2mike] from comment #40)
> Thanks, again! Do update the readme files (if there are any changes).

Will do.
Depends on: 693715, 693607
A couple of things I noticed with respect to the DB :

1) Dependency on the database name, tcm. Needs to be removed. We should be able to configure a DB name in the settings file and have it use that. This is true of the .sql scripts too

2) No password on the DB. This isn't acceptable, even for staging, surely not for production. We need to be able to set a username and password (and the app needs to find that from the settings file)

Look at https://github.com/mozilla/affiliates/blob/master/settings/local.py-dist for an example settings file. This is pretty much standard across all projects.

Also, you guys should look at schematic. Makes an initial setup _one_ step (you run ./manage.py syncdb) reads the DB values from the file, looks up the .sql files and punts whatever it needs into the database.  

I'm not going to stress on schematic for now, it's a nice to have..but I can hack around it if it's not something you guys want. But the other 2 points need to be addressed before we can go ahead. 

Thanks!
Attached file tomcat startup errors —
The war file seems to have hardcoded log paths, we can either make them configurable, or if that's not possible, where should they be created, so it doesn't error out?
(In reply to Shyam Mani [:fox2mike] from comment #42)
> A couple of things I noticed with respect to the DB :
> 
> 1) Dependency on the database name, tcm. Needs to be removed. We should be
> able to configure a DB name in the settings file and have it use that. This
> is true of the .sql scripts too
> 
> 2) No password on the DB. This isn't acceptable, even for staging, surely
> not for production. We need to be able to set a username and password (and
> the app needs to find that from the settings file)

I fully agree with both of these points, but they're both things that would need to be addressed in the platform; the Django codebase never talks directly to the database, only to the platform. The database scripts and the database connection configuration are both platform issues that Vadim would need to address.

Cameron, can you file platform bugs on both these issues?
 
> Look at
> https://github.com/mozilla/affiliates/blob/master/settings/local.py-dist for
> an example settings file. This is pretty much standard across all projects.

I'm not sure which part you're pointing to; we do something pretty similar with a local-settings file. The database stuff is not relevant for us as our Django code never talks to a database.
 
> Also, you guys should look at schematic. Makes an initial setup _one_ step
> (you run ./manage.py syncdb) reads the DB values from the file, looks up the
> .sql files and punts whatever it needs into the database.  
> 
> I'm not going to stress on schematic for now, it's a nice to have..but I can
> hack around it if it's not something you guys want. But the other 2 points
> need to be addressed before we can go ahead. 

We'd certainly use a schema migration tool if we were talking directly to the database.
(In reply to Shyam Mani [:fox2mike] from comment #43)
> Created attachment 566491 [details]
> tomcat startup errors
> 
> The war file seems to have hardcoded log paths, we can either make them
> configurable, or if that's not possible, where should they be created, so it
> doesn't error out?

Cam, can you create a platform bug for this? I have no idea about this side of things.
Depends on: 694053
Depends on: 694057
(In reply to Shyam Mani [:fox2mike] from comment #42)
> 2) No password on the DB. This isn't acceptable, even for staging, surely
> not for production. We need to be able to set a username and password (and
> the app needs to find that from the settings file)

Actually, on another review of the platform setup README, the database username and password are already configurable (via editing utest-ds.xml).

I think the database name is also configurable there, but that doesn't solve the problem that the platform db scripts all hardcode "tcm".
(In reply to Carl Meyer from comment #46)
> Actually, on another review of the platform setup README, the database
> username and password are already configurable (via editing utest-ds.xml).

Oh, I didn't read that at all...since you said we wouldn't have to build it, just the war will do...but I'll take a look at it when I wake up.
 
> I think the database name is also configurable there, but that doesn't solve
> the problem that the platform db scripts all hardcode "tcm".

True.
I said "just use the caseconductor-ui repo and reference its top-level README and the README inside the platform/ subdirectory" - this is mentioned in the latter.
Hi Shyam,

The vendor lib is created (https://github.com/mozilla/caseconductor-vendor-lib) and available as a submodule of caseconductor-ui in the requirements/vendor directory. I also added a WSGI script file at deploy/vendor.wsgi which sets up sys.path appropriately to use the vendor lib. Added a paragraph to the README briefly describing both of those.

Sorry I didn't get to this as soon as I anticipated - let me know if you have trouble with any of it, or if there's anything else you need!

Also, I updated the platform to the latest version, in which Vadim removed the hardcoded "USE tcm" statements from the SQL scripts, so platform database name, user, and password should all be configurable now.
Just an update:
This morning I noticed that the platform update scripts still mentioned "tcm" in a couple places which was breaking my ability to use a db of another name.  I fixed those and checked them in.

I then ran all my platform tests and they passed with the new DB just fine.  So I think we're good to go on that front.
Updated the bundled version of the platform to latest, so it includes Cam's fixes.
Depends on: 694677
Short update, I'll look into this by Wednesday or later, while I was waiting for the fixes to come in, I started on another project and I expect to wrap that up by Wednesday at the most.
Bug 694677 has been completed [per the comments, I am unable to verify myself], so all blockers should be removed at this point.
What is the status of this bug? Please confirm that there are no blockers keeping it from being worked on. Thanks
(In reply to Rebecca Billings from comment #54)
> What is the status of this bug? Please confirm that there are no blockers
> keeping it from being worked on. Thanks

I'm the blocker. I'm still working on Jenkins and this is blocked until that is completed, which should be sometime this week.
Shyam,

just check in for new status on this issue.

Also: when this is deployed and running, what will be the procedure for updating the code?  Updates to the github repo will be periodic.  Will I be able to update this machine?  or will I need to enter an IT bug to do that?

Thanks.
-Cam
Cam,

We're still trying to get the app running on tomcat. After fixing the log location, it still doesn't run and doesn't throw up any errors...so a couple of us are looking into it. 

As for deployments, dev can be automatically pushed. stage and prod will require IT bugs.
Shyam, thanks for the update.  I don't know why it wouldn't work under Tomcat, but it may be that it ONLY works with JBoss.  This is how all the deployments I'm aware of are using it.  

Just a heads up, in case it gets too painful and frustrating, going with JBoss may be a simpler path.
(In reply to Cameron Dawson [:camd] from comment #58)
> Just a heads up, in case it gets too painful and frustrating, going with
> JBoss may be a simpler path.

Not necessarily the case for us..  Nobody on the team has dealt with jboss before, and we have no puppet framework built up for it.
And this is an issue. We can't seem to get this working on tomcat, so that route is a no go. 

Do you have a step by step to get it running with JBoss? That'll make it faster vs me having to spend time catching up/figuring out JBoss, especially since this is temporary.
(In reply to Shyam Mani [:fox2mike] from comment #60)

> Do you have a step by step to get it running with JBoss? That'll make it
> faster vs me having to spend time catching up/figuring out JBoss, especially
> since this is temporary.

Also, we need to pay for JBoss, which is not something I'm inclined to suggest..if this is temporary. We can get a 30 day trial version though.
Shyam: I'll get you step by step JBoss setup in an hour or so.  It's pretty simple, really.

JBoss 5.1 is the one we use for CC, and it's on sourceforge.  So I think it's free:
http://sourceforge.net/projects/jboss/files/JBoss/JBoss-5.1.0.GA/
Steps are:

1. Unzip the the zip/tar file.
2. add these env variables.  Update to your system's settings:
 -  export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
 -  export JBOSS_HOME=$HOME/CodeLibraries/jboss-5.1.0.GA
3. Follow the instructions in platform/README.rst to copy files to JBoss for the platform.
4. Run JBoss like this: $JBOSS_HOME/bin/run.sh

let me know if you have more questions.
(In reply to Cameron Dawson [:camd] from comment #63)
> 1. Unzip the the zip/tar file.
> 2. add these env variables.  Update to your system's settings:
>  -  export
> JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
>  -  export JBOSS_HOME=$HOME/CodeLibraries/jboss-5.1.0.GA
> 3. Follow the instructions in platform/README.rst to copy files to JBoss for
> the platform.
> 4. Run JBoss like this: $JBOSS_HOME/bin/run.sh

In my experience on a Linux system, step 2 is not needed, it's just unzip, copy files in per the README, and run it.
(In reply to Carl Meyer from comment #64)
> In my experience on a Linux system, step 2 is not needed, it's just unzip,
> copy files in per the README, and run it.

This seems to work fine.
Carl,

I think we're running into some path issues with the vendor library, since running anything with manage.py errors out with :

[root@node200 caseconductor-ui]# ./manage.py shell
Traceback (most recent call last):
  File "./manage.py", line 33, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Carl,

I'm fox2mike on irc.mozilla.org, poke me when you're online and we'll work on this.
Hi Shyam: I think the error you're hitting is related to the steps in the caseconductor-ui/README.rst.

Follow that readme, but if you already have, then activate your virtualenv, and re-run these commands:
    git submodule init; git submodule update
and
    bin/install-reqs
Cameron,

We don't run virtualenvs. You might ask around in #webdev for help for how to run apps without virtualenvs (which was the whole point of the vendor library) :)
Shyam: ahh right.  Sorry, I forgot about your usage of the Vendor library.  

Then it sounds like either something is missing form the vendor library, or your connection to it isn't working.  Not sure, I don't really know much about vendor libraries.  

I'm sure Carl can elucidate.  :)
HOLY KAU!

https://caseconductor.allizom.org/ :D

So I didn't see instructions on how to create an admin login, so I'm not sure it can yet help create users etc aka CC_ADMIN_USER and CC_ADMIN_PASS are empty in the settings...any suggestions?
Shyam: this is awesome!  Thanks so much!

So, if you just migrate the table from camd.mv.mozilla.com over to this machine, then I'll be an admin and we'll be good to go.
Carl, Shyam: can you configure this machine so that the default role is "Tester" rather than admin?
Cam: it appears to me that that's already how it is configured. Shyam and I both signed up, and we are both just Testers.
You're welcome!

Next steps :

1) We get this into a "final", production ready state.
2) Carl/Cam will file an infrasec bug for reviewing the app
3) Once that's done, you can file a production deploy bug 
4) IT will deploy the app in production

I'm going to leave this open till we have the DB dump from Cam imported into the allizom site.
(In reply to Cameron Dawson [:camd] from comment #73)
> Carl, Shyam: can you configure this machine so that the default role is
> "Tester" rather than admin?

That is the case.

CC_NEW_USER_ROLE_ID = 6
Just an update here, we have been having some issues importing the DB from cam's machine to the DB server in phx1 and I think cam is going to work with mpressman today on that.
This is all good, thanks a lot for your help jason and matt and camd/carl for their patience :)
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: