There needs to be a second bugzilla database somewhere else in the country, as having two databases in the same area would not help if there were some kind of natural disaster. This second shadow database need not be backed up to as often and can come from the 1st shadow, as to not bog down the system.
Currently the shadow db has to be on the same machine. You really want to just use replication for this, and not have bugzilla involved at all, since querying off a db on the other side of the world wouldn't be very speedy. -> m.o, I guess, although I wuold guess that AOL's backup procedures are fairly involved.
I thought about this, and I think the best idea would be to maybe monthly or weekly have the shadow database get encrypted 128 bit or more (along with the code) with only dawn and a few others having the key (and maybe some netscape heads, etc) and then send it out encrypted to a few 3rd party people. Therefore, if the data was lost because of a natural disaster or something, it would be recoverable with the keys. It would have to be enough of an encryption method that people couldn't crack it even with multiple computing. If we ever got to the point an old encrypted database possibly still stored on someone's computer could be cracked, then it would be necessary to improve the encryption and have everyone change their passwords. If the right method is used, it might be 20 years till that is ncessary even with parallel computing methosds like seti@home. Daily: Backup to shadow database (as usual) Weekly or monthly: Shadow db gets encrypted with local key and sent out to 3rd party people willing to give up some space and bandwidth to backup bugzilla. It wouldn't affect usage of b.m.o because only the shadow db would be used. Speed of sending wouldn't be an issue because it wouldn't be done that often. When its on the person's computer, its encrypted so they don't have access. The more people that agree to use their computers, the better (within limits). If this is done, then both the code and bugzilla could be recovered in a disaster (with some work to get it back to usability) from people around the world who have it stored.
Obviously we are not affiliated with AOL anymore, and I believe it would be wise to backup Bugzilla to first a separate computer on the same network each day, and use that computer then to mirror the database for Read Only to a few places in the world using encryption to send out the data. This could actually be a first step towards a distributed bugzilla database where you have a main server for writes, but various mirrors for queries. This would also be a good idea since think of it this way: What if the server were in the World Trade Center? We know that California gets earthquakes. If the servers hosting the FTP (which could be mirrored were lost) we could easily reconstruct the code because its on developers' systems. What happens, though, if we lose Bugzilla? We'd lose track of almost every bug we are working on and would probably have to do go through the code and rewrite it in order to find all the problems again. Do we really want to play the odds like this?
I'm sure the Mozilla Foundation has a sensible backup policy... Gerv
We will have off-site replication of the Bugzilla database once we get our new infrastructure up. We'll have cross-colo replication of all databases between Meer and OSL once we're done. In the meantime, we do have daily backups, which are not stored off-site, but are on a different server.
This is a mass-reassign of bugs that I'm not actively working on right at this moment to the default component owner, since we now have a larger IT staff than just me. These bugs will be getting redistributed to other sysadmins as sysadmin time becomes available.
Backups are running and are stored offsite.