Use hardlinks when pushing to mirrors to speed it up



Release Engineering
Release Automation
5 years ago
3 years ago


(Reporter: bhearsum, Assigned: rail)


Firefox Tracking Flags

(Not tracked)



(1 attachment, 3 obsolete attachments)



5 years ago
We talked about how to improve the push to mirrors push time in the post mortem yesterday. Two ideas we came up with:
1) Symlink all the files we care about from releases -> candidates in the "push to mirrors builder". Replace them with real copies of the files in "postrelease". If we do this, we need to take care to do it atomically.
2) Hardlink files from releases -> candidates. This is fast and saves disk space. There's some leg work if we do it though, because these directories aren't on the same filesystem at the moment.

Comment 1

5 years ago
Either of these options will require changes to the pushToMirrors method of the script:

Basically, one will need to replace the "rsync" command with a "find" command that filters out what we don't want and does the linking. Something like:
find /pub/ -not -name blah -and -not -name ack [...] -exec ln -s {} /pub/{}

Comment 2

5 years ago
This would be a really nice to have, but not a priority.
Priority: -- → P3
Product: → Release Engineering
Bug 988288 has put firefox/candidates back on the same partition as firefox/releases, and mobile, seamonkey, thunderbird, and xulrunner are in that state too, so hardlinks can be done now. 

An example rsync command argument is
 rsync -av --link-dest=/path/to/candidates/29.0b5-candidates/build1/ \
   /path/to/candidates/29.0b5-candidates/build1/ /path/to/releases/29.0b5/
Depends on: 988288
Morphing to chose hardlinks as the solution. We should add cleanup of candidates later, since hardlinks allow for accidental modification of releases/ by futzing around in candidates/ (unusual, but not unheard of). We have to stop using old candidates dirs when making new releases first though.
Summary: use links of some sort when pushing to mirrors to speed it up → Use hardlinks when pushing to mirrors to speed it up

Comment 5

3 years ago
With the ever-onward growth of releases bumping this vol up, this bug may need to be bumped in priority a touch.

And/or we might need a tangential plan for fragging off some of the older releases, sometime in the next year.  Not a crisis.
Assignee: nobody → rail
Created attachment 8484490 [details] [diff] [review]

Still need to be tested in staging, but the overall idea worked running the commands from shell.

1) copy candidates/xxx/buildN to candidates/xxx/.buildN using "cp -al"
2) delete not needed files by "rsync ... --delete-excluded"
3) move the "hidden" directory to its final destination
Attachment #8484490 - Flags: review?(nthomas)
I ran the commands manually in staging and they took 2-3 seconds. \o/
Comment on attachment 8484490 [details] [diff] [review]

Another incoming
Attachment #8484490 - Flags: review?(nthomas)
Created attachment 8484527 [details] [diff] [review]

Elapsed	38 secs
Attachment #8484490 - Attachment is obsolete: true
Attachment #8484527 - Flags: review?(nthomas)
Created attachment 8484561 [details]

the log output
Created attachment 8484578 [details] [diff] [review]


sent 51883 bytes  received 4442 bytes  112650.00 bytes/sec
total size is 32794695723  speedup is 582240.49

real    0m0.100s
user    0m0.014s
sys     0m0.033s
Attachment #8484527 - Attachment is obsolete: true
Attachment #8484561 - Attachment is obsolete: true
Attachment #8484527 - Flags: review?(nthomas)
Attachment #8484578 - Flags: review?(nthomas)
Comment on attachment 8484578 [details] [diff] [review]

Looks good to me. You could test it out on
Attachment #8484578 - Flags: review?(nthomas) → review+
Comment on attachment 8484578 [details] [diff] [review]

worked as expected in staging.
Attachment #8484578 - Flags: checked-in+

Elapsed	50 sec

Win! \o/
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.