Closed Bug 807289 Opened 12 years ago Closed 10 years ago

Use hardlinks when pushing to mirrors to speed it up

Categories

(Release Engineering :: Release Automation: Other, defect, P3)

x86_64
Linux
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: rail)

Details

Attachments

(1 file, 3 obsolete files)

We talked about how to improve the push to mirrors push time in the post mortem yesterday. Two ideas we came up with:
1) Symlink all the files we care about from releases -> candidates in the "push to mirrors builder". Replace them with real copies of the files in "postrelease". If we do this, we need to take care to do it atomically.
2) Hardlink files from releases -> candidates. This is fast and saves disk space. There's some leg work if we do it though, because these directories aren't on the same filesystem at the moment.
Either of these options will require changes to the pushToMirrors method of the stage-tasks.py script: https://github.com/mozilla/build-tools/blob/master/scripts/release/stage-tasks.py#L101.

Basically, one will need to replace the "rsync" command with a "find" command that filters out what we don't want and does the linking. Something like:
find /pub/mozilla.org/firefox/candidates/17.0b1 -not -name blah -and -not -name ack [...] -exec ln -s {} /pub/mozilla.org/firefox/releases/17.0b1/{}
This would be a really nice to have, but not a priority.
Priority: -- → P3
Product: mozilla.org → Release Engineering
Bug 988288 has put firefox/candidates back on the same partition as firefox/releases, and mobile, seamonkey, thunderbird, and xulrunner are in that state too, so hardlinks can be done now. 

An example rsync command argument is
 rsync -av --link-dest=/path/to/candidates/29.0b5-candidates/build1/ \
   /path/to/candidates/29.0b5-candidates/build1/ /path/to/releases/29.0b5/
Morphing to chose hardlinks as the solution. We should add cleanup of candidates later, since hardlinks allow for accidental modification of releases/ by futzing around in candidates/ (unusual, but not unheard of). We have to stop using old candidates dirs when making new releases first though.
Summary: use links of some sort when pushing to mirrors to speed it up → Use hardlinks when pushing to mirrors to speed it up
With the ever-onward growth of releases bumping this vol up, this bug may need to be bumped in priority a touch.

And/or we might need a tangential plan for fragging off some of the older releases, sometime in the next year.  Not a crisis.
Assignee: nobody → rail
Attached patch hardlinks-tools.diff (obsolete) — Splinter Review
Still need to be tested in staging, but the overall idea worked running the commands from shell.

1) copy candidates/xxx/buildN to candidates/xxx/.buildN using "cp -al"
2) delete not needed files by "rsync ... --delete-excluded"
3) move the "hidden" directory to its final destination
Attachment #8484490 - Flags: review?(nthomas)
I ran the commands manually in staging and they took 2-3 seconds. \o/
Comment on attachment 8484490 [details] [diff] [review]
hardlinks-tools.diff

Another incoming
Attachment #8484490 - Flags: review?(nthomas)
Attached file 6-log-run_script-stdio.gz (obsolete) —
the log output
WCPGW?!!

sent 51883 bytes  received 4442 bytes  112650.00 bytes/sec
total size is 32794695723  speedup is 582240.49

real    0m0.100s
user    0m0.014s
sys     0m0.033s
Attachment #8484527 - Attachment is obsolete: true
Attachment #8484561 - Attachment is obsolete: true
Attachment #8484527 - Flags: review?(nthomas)
Attachment #8484578 - Flags: review?(nthomas)
Comment on attachment 8484578 [details] [diff] [review]
hardlinks2-tools.diff

Looks good to me. You could test it out on http://dev-stage01.build.mozilla.org/pub/mozilla.org/firefox/candidates/31.0.0esr-candidates/
Attachment #8484578 - Flags: review?(nthomas) → review+
Comment on attachment 8484578 [details] [diff] [review]
hardlinks2-tools.diff

https://hg.mozilla.org/build/tools/rev/e44e9e285864

worked as expected in staging.
Attachment #8484578 - Flags: checked-in+
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/33.0b2-candidates/build1/logs/release-mozilla-beta-push_to_mirrors-bm84-build1-build13.txt.gz

Elapsed	50 sec

Win! \o/
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: