Use hardlinks when pushing to mirrors to speed it up



7 years ago
5 years ago


(Reporter: bhearsum, Assigned: rail)


Firefox Tracking Flags

(Not tracked)



(1 attachment, 3 obsolete attachments)

We talked about how to improve the push to mirrors push time in the post mortem yesterday. Two ideas we came up with:
1) Symlink all the files we care about from releases -> candidates in the "push to mirrors builder". Replace them with real copies of the files in "postrelease". If we do this, we need to take care to do it atomically.
2) Hardlink files from releases -> candidates. This is fast and saves disk space. There's some leg work if we do it though, because these directories aren't on the same filesystem at the moment.

Comment 1

6 years ago
Either of these options will require changes to the pushToMirrors method of the script:

Basically, one will need to replace the "rsync" command with a "find" command that filters out what we don't want and does the linking. Something like:
find /pub/ -not -name blah -and -not -name ack [...] -exec ln -s {} /pub/{}

Comment 2

6 years ago
This would be a really nice to have, but not a priority.
Priority: -- → P3
Product: → Release Engineering
Bug 988288 has put firefox/candidates back on the same partition as firefox/releases, and mobile, seamonkey, thunderbird, and xulrunner are in that state too, so hardlinks can be done now. 

An example rsync command argument is
 rsync -av --link-dest=/path/to/candidates/29.0b5-candidates/build1/ \
   /path/to/candidates/29.0b5-candidates/build1/ /path/to/releases/29.0b5/
Depends on: 988288
Morphing to chose hardlinks as the solution. We should add cleanup of candidates later, since hardlinks allow for accidental modification of releases/ by futzing around in candidates/ (unusual, but not unheard of). We have to stop using old candidates dirs when making new releases first though.
Summary: use links of some sort when pushing to mirrors to speed it up → Use hardlinks when pushing to mirrors to speed it up

Comment 5

5 years ago
With the ever-onward growth of releases bumping this vol up, this bug may need to be bumped in priority a touch.

And/or we might need a tangential plan for fragging off some of the older releases, sometime in the next year.  Not a crisis.


5 years ago
Assignee: nobody → rail

Comment 6

5 years ago
Posted patch hardlinks-tools.diff (obsolete) — Splinter Review
Still need to be tested in staging, but the overall idea worked running the commands from shell.

1) copy candidates/xxx/buildN to candidates/xxx/.buildN using "cp -al"
2) delete not needed files by "rsync ... --delete-excluded"
3) move the "hidden" directory to its final destination
Attachment #8484490 - Flags: review?(nthomas)

Comment 7

5 years ago
I ran the commands manually in staging and they took 2-3 seconds. \o/

Comment 8

5 years ago
Comment on attachment 8484490 [details] [diff] [review]

Another incoming
Attachment #8484490 - Flags: review?(nthomas)

Comment 10

5 years ago
Posted file 6-log-run_script-stdio.gz (obsolete) —
the log output

Comment 11

5 years ago

sent 51883 bytes  received 4442 bytes  112650.00 bytes/sec
total size is 32794695723  speedup is 582240.49

real    0m0.100s
user    0m0.014s
sys     0m0.033s
Attachment #8484527 - Attachment is obsolete: true
Attachment #8484561 - Attachment is obsolete: true
Attachment #8484527 - Flags: review?(nthomas)
Attachment #8484578 - Flags: review?(nthomas)
Comment on attachment 8484578 [details] [diff] [review]

Looks good to me. You could test it out on
Attachment #8484578 - Flags: review?(nthomas) → review+

Comment 13

5 years ago
Comment on attachment 8484578 [details] [diff] [review]

worked as expected in staging.
Attachment #8484578 - Flags: checked-in+

Comment 14

5 years ago

Elapsed	50 sec

Win! \o/
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.