133935 - Mirror Bugzilla database for Read Only in a few places around the world for our own disaster management

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Brian 'netdragon' Bober]

There needs to be a second bugzilla database somewhere else in the country, as
having two databases in the same area would not help if there were some kind of
natural disaster. This second shadow database need not be backed up to as often
and can come from the 1st shadow, as to not bog down the system.

Comment 1

[Bradley Baetz (:bbaetz)]

Currently the shadow db has to be on the same machine. You really want to just
use replication for this, and not have bugzilla involved at all, since querying
off a db on the other side of the world wouldn't be very speedy.

-> m.o, I guess, although I wuold guess that AOL's backup procedures are fairly
involved.

Comment 2

[Brian 'netdragon' Bober]

I thought about this, and I think the best idea would be to maybe monthly or
weekly have the shadow database get encrypted 128 bit or more (along with the
code) with only dawn and a few others having the key (and maybe some netscape
heads, etc) and then send it out encrypted to a few 3rd party people. Therefore,
if the data was lost because of a natural disaster or something, it would be
recoverable with the keys. It would have to be enough of an encryption method
that people couldn't crack it even with multiple computing. If we ever got  to
the point an old encrypted database possibly still stored on someone's computer
could be cracked, then it would be necessary to improve the encryption and have
everyone change their passwords. If the right method is used, it might be 20
years till that is ncessary even with parallel computing methosds like seti@home.

Daily:

Backup to shadow database (as usual)

Weekly or monthly:

Shadow db gets encrypted with local key and sent out to 3rd party people willing
to give up some space and bandwidth to backup bugzilla. It wouldn't affect usage
of b.m.o because only the shadow db would be used.

Speed of sending wouldn't be an issue because it wouldn't be done that often.

When its on the person's computer, its encrypted so they don't have access.

The more people that agree to use their computers, the better (within limits).
If this is done, then both the code and bugzilla could be recovered in a
disaster (with some work to get it back to usability) from people around the
world who have it stored.

Comment 3

[Brian 'netdragon' Bober]

Obviously we are not affiliated with AOL anymore, and I believe it would be wise
to backup Bugzilla to first a separate computer on the same network each day,
and use that computer then to mirror the database for Read Only to a few places
in the world using encryption to send out the data. This could actually be a
first step towards a distributed bugzilla database where you have a main server
for writes, but various mirrors for queries.

This would also be a good idea since think of it this way: What if the server
were in the World Trade Center? We know that California gets earthquakes. If the
servers hosting the FTP (which could be mirrored were lost) we could easily
reconstruct the code because its on developers' systems. What happens, though,
if we lose Bugzilla? We'd lose track of almost every bug we are working on and
would probably have to do go through the code and rewrite it in order to find
all the problems again.

Do we really want to play the odds like this?

Comment 4

[Gervase Markham [:gerv]]

I'm sure the Mozilla Foundation has a sensible backup policy...

Gerv

Comment 5

[Dave Miller [:justdave]]

We will have off-site replication of the Bugzilla database once we get our new
infrastructure up.  We'll have cross-colo replication of all databases between
Meer and OSL once we're done.  In the meantime, we do have daily backups, which
are not stored off-site, but are on a different server.

Comment 6

[Dave Miller [:justdave]]

This is a mass-reassign of bugs that I'm not actively working on right at this moment to the default component owner, since we now have a larger IT staff than just me.  These bugs will be getting redistributed to other sysadmins as sysadmin time becomes available.

Comment 7

[Justin Fitzhugh]

Backups are running and are stored offsite.