Use hardlinks when pushing to mirrors to speed it up

RESOLVED FIXED

Status

Release Engineering
Release Automation
P3
normal
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: bhearsum, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 3 obsolete attachments)

(Reporter)

Description

5 years ago
We talked about how to improve the push to mirrors push time in the post mortem yesterday. Two ideas we came up with:
1) Symlink all the files we care about from releases -> candidates in the "push to mirrors builder". Replace them with real copies of the files in "postrelease". If we do this, we need to take care to do it atomically.
2) Hardlink files from releases -> candidates. This is fast and saves disk space. There's some leg work if we do it though, because these directories aren't on the same filesystem at the moment.
(Reporter)

Comment 1

5 years ago
Either of these options will require changes to the pushToMirrors method of the stage-tasks.py script: https://github.com/mozilla/build-tools/blob/master/scripts/release/stage-tasks.py#L101.

Basically, one will need to replace the "rsync" command with a "find" command that filters out what we don't want and does the linking. Something like:
find /pub/mozilla.org/firefox/candidates/17.0b1 -not -name blah -and -not -name ack [...] -exec ln -s {} /pub/mozilla.org/firefox/releases/17.0b1/{}
(Reporter)

Comment 2

5 years ago
This would be a really nice to have, but not a priority.
Priority: -- → P3
Product: mozilla.org → Release Engineering
Bug 988288 has put firefox/candidates back on the same partition as firefox/releases, and mobile, seamonkey, thunderbird, and xulrunner are in that state too, so hardlinks can be done now. 

An example rsync command argument is
 rsync -av --link-dest=/path/to/candidates/29.0b5-candidates/build1/ \
   /path/to/candidates/29.0b5-candidates/build1/ /path/to/releases/29.0b5/
Depends on: 988288
Morphing to chose hardlinks as the solution. We should add cleanup of candidates later, since hardlinks allow for accidental modification of releases/ by futzing around in candidates/ (unusual, but not unheard of). We have to stop using old candidates dirs when making new releases first though.
Summary: use links of some sort when pushing to mirrors to speed it up → Use hardlinks when pushing to mirrors to speed it up

Comment 5

3 years ago
With the ever-onward growth of releases bumping this vol up, this bug may need to be bumped in priority a touch.

And/or we might need a tangential plan for fragging off some of the older releases, sometime in the next year.  Not a crisis.
(Assignee)

Updated

3 years ago
Assignee: nobody → rail
(Assignee)

Comment 6

3 years ago
Created attachment 8484490 [details] [diff] [review]
hardlinks-tools.diff

Still need to be tested in staging, but the overall idea worked running the commands from shell.

1) copy candidates/xxx/buildN to candidates/xxx/.buildN using "cp -al"
2) delete not needed files by "rsync ... --delete-excluded"
3) move the "hidden" directory to its final destination
Attachment #8484490 - Flags: review?(nthomas)
(Assignee)

Comment 7

3 years ago
I ran the commands manually in staging and they took 2-3 seconds. \o/
(Assignee)

Comment 8

3 years ago
Comment on attachment 8484490 [details] [diff] [review]
hardlinks-tools.diff

Another incoming
Attachment #8484490 - Flags: review?(nthomas)
(Assignee)

Comment 9

3 years ago
Created attachment 8484527 [details] [diff] [review]
hardlinks-tools-4.diff

http://dev-master1.srv.releng.scl3.mozilla.com:8444/builders/release-mozilla-beta-push_to_mirrors/builds/6

Elapsed	38 secs
Attachment #8484490 - Attachment is obsolete: true
Attachment #8484527 - Flags: review?(nthomas)
(Assignee)

Comment 10

3 years ago
Created attachment 8484561 [details]
6-log-run_script-stdio.gz

the log output
(Assignee)

Comment 11

3 years ago
Created attachment 8484578 [details] [diff] [review]
hardlinks2-tools.diff

WCPGW?!!

sent 51883 bytes  received 4442 bytes  112650.00 bytes/sec
total size is 32794695723  speedup is 582240.49

real    0m0.100s
user    0m0.014s
sys     0m0.033s
Attachment #8484527 - Attachment is obsolete: true
Attachment #8484561 - Attachment is obsolete: true
Attachment #8484527 - Flags: review?(nthomas)
Attachment #8484578 - Flags: review?(nthomas)
Comment on attachment 8484578 [details] [diff] [review]
hardlinks2-tools.diff

Looks good to me. You could test it out on http://dev-stage01.build.mozilla.org/pub/mozilla.org/firefox/candidates/31.0.0esr-candidates/
Attachment #8484578 - Flags: review?(nthomas) → review+
(Assignee)

Comment 13

3 years ago
Comment on attachment 8484578 [details] [diff] [review]
hardlinks2-tools.diff

https://hg.mozilla.org/build/tools/rev/e44e9e285864

worked as expected in staging.
Attachment #8484578 - Flags: checked-in+
(Assignee)

Comment 14

3 years ago
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/33.0b2-candidates/build1/logs/release-mozilla-beta-push_to_mirrors-bm84-build1-build13.txt.gz

Elapsed	50 sec

Win! \o/
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.