Closed Bug 508406 Opened 12 years ago Closed 11 years ago

Speed up backupsnip

Categories

(Release Engineering :: General, defect)

x86
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: catlee)

References

Details

Attachments

(3 files)

Right now we backup the whole (releases) snippet store each time we want to push out more release snippets, but it takes around 50 minutes to pull tens of thousands of small files in a deep dir structure from the NFS mount. We should investigate doing this smarter, eg making a daily full backup and teaching backupsnip to only backup the directories/snippets that are going to change. This should be muuuuuch master, and may allow us to enforce that backupsnip is run before every snippet push (eg 3.5.2 followed by 3.0.13 followed by 3.0.13 MU).
Component: Release Engineering → Release Engineering: Future
Assignee: nobody → nthomas
Whiteboard: Q4
In addition to the smart-backup strategy of only backing up things we're changing, we could also start excluding old snippets. For example, the most recent backup has 232968 directories and files, and 56% of those are for the discontinued versions - Firefox 1.5 & 2.0, plus Thunderbird 1.5 (most of that is from the Fx2 partner nightmare). So there's a better than 2x speed up just from making a one-off backup and then excluding those directories in backupsnip. Comments ?
(In reply to comment #1)
> In addition to the smart-backup strategy of only backing up things we're
> changing, we could also start excluding old snippets. For example, the most
> recent backup has 232968 directories and files, and 56% of those are for the
> discontinued versions - Firefox 1.5 & 2.0, plus Thunderbird 1.5 (most of that
> is from the Fx2 partner nightmare). So there's a better than 2x speed up just
> from making a one-off backup and then excluding those directories in
> backupsnip. Comments ?

r=bhearsum on excluding snippets for any unsupported products
The old snippets are still present, and are backed up at
  /opt/aus2/snippets/backup/20091129-FX15-FX20-TB15-FINAL-BACKUP.tar.bz2
but we won't include them in any future backupsnip's. I tried defining EXCLUDES="--exclude='Firefox/1.5* ..." but the shell was breaking the quoting so I gave it away. Also adds 'time' since we do that frequently. 

Checking in backupsnip;
/cvsroot/mozilla/tools/release/bin/backupsnip,v  <--  backupsnip
new revision: 1.3; previous revision: 1.2
done
aus2-staging and staging-stage updated.
Attachment #415087 - Flags: checked-in+
37 minutes for a backupsnip run today, although it does depend a bit on how busy the nfs share is.
Status: some speedup achieved. There are basically three big chunks of data there now - Fx3.0, Fx3.5, and Tb2.0. The approach of comment #0 would maybe get a factor of 3 improvement.
Assignee: nrthomas → nobody
Whiteboard: Q4
Assignee: nobody → catlee
Component: Release Engineering: Future → Release Engineering
This patch generates a list of all directories at depth 3 in the staging directory.  This corresponds to the '20100115-Firefox-3.6rc2/Firefox/3.6b1/WINNT_x86-msvc', etc. directories.

Directories that don't exist in the live snippets directory are filtered out of this list.

We then tar up all these directories from the live snippets directory.

Switched to using gz instead of bz2 since it's faster as well.

runtime is a few minutes for 3.6 snippets, and 8 minutes for 3.5 snippets using this method.  We'd have to generate daily backups, and document recovery procedure as well.
Attachment #422236 - Flags: review?(nrthomas)
Comment on attachment 422236 [details] [diff] [review]
only backup directories that have changed

Looks good. We'll not save modification timestamps on Firefox/ and Firefox/$version but that seems unlikely to bite us in a recovery situation, which I imagine will be disable all/some updates, restore nightly backup somewhere, apply any updates you want to keep, swap out live directory.
Attachment #422236 - Flags: review?(nrthomas) → review+
Attached file Nightly backup script
Attachment #422341 - Flags: review?(nrthomas)
Attachment #422341 - Attachment mime type: application/octet-stream → text/plain
Comment on attachment 422341 [details]
Nightly backup script

Did you decide to stick with bz2 here ?
Attachment #422341 - Flags: review?(nrthomas) → review+
(In reply to comment #9)
> (From update of attachment 422341 [details])
> Did you decide to stick with bz2 here ?

Yes, the speed here isn't as critical, and we get a small space savings.
Comment on attachment 422236 [details] [diff] [review]
only backup directories that have changed

Checking in backupsnip;
/cvsroot/mozilla/tools/release/bin/backupsnip,v  <--  backupsnip
new revision: 1.4; previous revision: 1.3
done
Attachment #422236 - Flags: checked-in+
Comment on attachment 422341 [details]
Nightly backup script

RCS file: /cvsroot/mozilla/tools/release/bin/backupsnip-nightly,v
done
Checking in backupsnip-nightly;
/cvsroot/mozilla/tools/release/bin/backupsnip-nightly,v  <--  backupsnip-nightly
initial revision: 1.1
done
Attachment #422341 - Flags: checked-in+
backupsnip for 3.5.8 took 17 minutes compared to 42 minutes for 3.5.7.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Depends on: 544420
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.