The default bug view has changed. See this FAQ.

Optimize snippets directories

RESOLVED WONTFIX

Status

Release Engineering
Release Automation
P3
normal
RESOLVED WONTFIX
5 years ago
4 years ago

People

(Reporter: rail, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [releases][automation][updates])

Attachments

(7 attachments, 8 obsolete attachments)

1.29 KB, text/plain
Details
1.38 KB, text/plain
Details
7.09 KB, text/plain
Details
40.80 KB, patch
rail
: review+
Details | Diff | Splinter Review
13.84 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
89.10 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
1.07 KB, patch
rail
: review+
Details | Diff | Splinter Review
(Assignee)

Description

5 years ago
It would be great to optimize snippets so that we have symlinks for duplicating directories instead of copies.

Nick suggested to post process aus2{,.test,.beta} directories being generated by patcher.
(Assignee)

Comment 1

5 years ago
Created attachment 587664 [details]
ddupes.py

Example usage:

# pass all build IDs
PYTHONPATH=tools/lib/python python ddupes.py aus2/*/*/*/*

# convert absolute symlinks to relative
symlinks -cr aus2
Attachment #587664 - Flags: feedback?(nrthomas)
(Assignee)

Comment 2

5 years ago
My only concern here is the fact that we have to use rsync --force in pushsnip to replace directories with symlinks (and remove old partials, yay!).
(Assignee)

Comment 3

5 years ago
Comment on attachment 587664 [details]
ddupes.py

Talked with catlee about this:

* we can't use the generated snippets because generated structure doesn't represents the directory layout on aus2-state. Pushing final snippets would destroy test snippets and vice versa.
* The scripts could be used to optimize live snippets:
 ** Define which releases should be optimized
 ** Remove those releases from the patcher config (save into another file for historical reasons) so patcher won't touch the directories in the future.
 ** The script should ignore partial.txt files for old releases.
Attachment #587664 - Flags: feedback?(nrthomas)
(Assignee)

Updated

5 years ago
Priority: P2 → P3
(Assignee)

Updated

5 years ago
Priority: P3 → P2
(Assignee)

Comment 4

5 years ago
I played with snippets and this is what I have now:

==================================================
aus=/dev/shm/aus2/Firefox
versions="4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 7.0 7.0.1"

echo -n > optimized_dirs.txt
for v in $versions; do
    for channel in $aus/$v/*/*/*/{beta,betatest,release,releasetest}; do
        if [ -f $channel/partial.txt -a -f $channel/complete.txt ]; then
            path=$(echo $channel | sed "s,$aus/,,")
            platform=$(echo $path | awk -F/ '{print $2}')
            buildID=$(echo $path | awk -F/ '{print $3}')
            echo "$aus/$v/$platform/$buildID" >> optimized_dirs.txt
            rm -vf $channel/partial.txt
        fi
    done
done

sort -u optimized_dirs.txt > optimized_dirs.txt.new
mv optimized_dirs.txt.new optimized_dirs.txt


# slightly modified version which read directories from a file
PYTHONPATH=tools/lib/python python \
    ddupes.py optimized_dirs.txt

for d in `cat optimized_dirs.txt`; do
    if [ ! -L $d ]; then
        # print not optimized dirs
        echo $d
    fi
done
==================================================

It:
1) deletes partial.txt from some old releases

files: 118962 vs 67272
du: 339M vs 183M

2) link duplicate dirs

files: 67272 vs 19170
du: 183M vs 75M

3) print directories which we are unable to symlink
63 dirs out of 170

There shouldn't be any problem with 1), it's straight forward. It could be also applied to other (3.6.x) directories. It should reduce backupsnip time.

I have some concerns about 2) because it can't optimize directories for some releases when we changed the list of locales and those builds can't be removed from the patcher config (otherwise they wouldn't get updates to the latest release). As possible workaround we can leave those builds in the patcher config.

Ideas, comments?
(Assignee)

Comment 5

5 years ago
Created attachment 592170 [details]
ddupes.py
Attachment #587664 - Attachment is obsolete: true
(Assignee)

Comment 6

5 years ago
Created attachment 592171 [details]
optimize 4.0b8 -> 9.0

This script optimizes 4.0b8 -> 9.0 snippets so we could remove them from the patcher configs.

Before: 472M, 120759 files
After: 8.0M, 22608 files
Attachment #592171 - Flags: feedback?(bhearsum)
(Assignee)

Updated

5 years ago
Attachment #592171 - Flags: feedback?(nrthomas)
Comment on attachment 592171 [details]
optimize 4.0b8 -> 9.0

Wow, there's a lot to digest in this.

>####################################################
># special case for 4.0.1, cck :(, point channels to 5.0
># do we need them?

Kev is the best person to ask.

># ppc is not supported for 4.0b8+

I have it as b7 onwards at https://wiki.mozilla.org/index.php?title=User:NThomas:Mac_BUILDTARGET, double check ?

>####################################################
># rm beta channels for 4.0rc1, 4.0.1, 5.0, 5.0.1, 6.0, 6.0.1, 6.0.2
># we generated beta snippets for these releases :(

So these are the releases we published on the beta channel ? Have we managed to move the beta users off already, so we don't need to provide a path forward ? I see that it mixes things up if we have to.

>function rm_partials() {
>for v in $versions 4.0.1; do
>    if [ ! -L $aus/$v/Darwin_x86_64-gcc3 ]; then
>        rm -rvf $aus/$v/Darwin_x86_64-gcc3
>        ln -svf Darwin_x86_64-gcc3-u-i386-x86_64 $aus/$v/Darwin_x86_64-gcc3
>    fi

Nit-ish, clearer if all the mac specific stuff is put together. This plus the 'Darwin platform name changes' block slightly earlier.

>    # 7.0 build2 started before updates are generated, no snippets available

If there are users from ftp downloads we can retrieve the buildID and create a path. Are the buildN's from buildIDs in the snippet dirs ? I may have an independent source to cross check against.

>function magic() {

Nit, helps readability if function order matches call order.

>PYTHONPATH=/home/rail/work/mozilla/build/tools/lib/python python \
>    ddupes.py optimized_dirs.txt

Does this dedupe.py differ a lot from attachment 592170 [details] ?

>    symlinks -crs $aus/$v

Woo, learned something!

>Before: 472M, 120759 files
>after: 8.0M, 22608 files

This is very nice! To get a rough speed up you should count files+dirs though.

>How to check:
>rsync --copy-links -r /dev/shm/aus2/Firefox/ ~/tmp/aus.new/Firefox/
>for v in 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 4.0; do diff -ur --exclude partial.* aus.old/Firefox/$v aus.new/Firefox/$v >>aus.diff; done

Haven't run this yet, but I'm planning to. It would be fantastic to see this for 3.6 too, if you have time.
Attachment #592171 - Flags: feedback?(nrthomas) → feedback+
(Assignee)

Comment 8

5 years ago
Created attachment 593250 [details]
optimize 4.0b8 -> 9.0

Updated version
* don't delete beta channels for release builds. symlink them to the latest beta
* address comments.
(Assignee)

Comment 9

5 years ago
Created attachment 593253 [details] [diff] [review]
patcher config patch

* would be good to keep removed versions in a separate file to easier find them.
(Assignee)

Comment 10

5 years ago
Created attachment 593304 [details]
optimize 4.0b8 -> 9.0

The previous one without unneeded exit
Attachment #592171 - Attachment is obsolete: true
Attachment #593250 - Attachment is obsolete: true
Attachment #592171 - Flags: feedback?(bhearsum)
Attachment #593304 - Flags: review?(nrthomas)
(Assignee)

Comment 11

5 years ago
Comment on attachment 593253 [details] [diff] [review]
patcher config patch

I tested every removed build ID and all of them are symlinks or locale/channel are symlinks (for 4.0.1 and some others, for example).
Attachment #593253 - Attachment description: possible patcher config patch → patcher config patch
Attachment #593253 - Flags: review?(nrthomas)
(Assignee)

Updated

5 years ago
Attachment #593253 - Flags: review?(bhearsum)
(Assignee)

Updated

5 years ago
Attachment #593304 - Flags: review?(bhearsum)
Attachment #593253 - Flags: review?(bhearsum) → review+
Comment on attachment 593304 [details]
optimize 4.0b8 -> 9.0

># last release is 9.0/*/20111216140209
>lrv=9.0
>lrid=20111216140209
># last beta 9.0/*/20111212185108
>lbv=9.0
>lbid=20111212185108

I wouldn't mind if lrv/vrid/lbv/lbid were more verbose - they're really not intuitive for me.


> # special case for 4.0.1, cck :(, point channels to 5.0
> # do we need them?

Actually, these yahoo builds only have partials blocked, which means they fall back to the real completes. Currently, they're pointing at 10.0, and I think they will after your changes too?

> for localedir in 4.0.1/*/*/*; do
>    for channel in betatest release releasetest; do
>        if [ ! -L  $localedir/$channel ]; then
>            rm -rvf $localedir/$channel
>            s=${localedir/4.0.1/${lrv}}
>            s=${s/20110413222027/${lrid}}
>            ln -sv ../../../../$s/$channel $localedir/$channel
>        fi
>    done
> done

Per IRC with Kev, we don't need this anymore:
14:14 < bhearsum> ok, so we definitely don't need those anymore, because there's no way for them to get 
                  partials anymore
14:14 < kev> updater has a bug where, if there is a distribution directory in the app dir, updates fail
14:14 < bhearsum> we can remove the blocks, and they'll continue to get complete updates to the latest firefox
14:14 < kev> so the snippets give them full updates
14:14 < bhearsum> kev: but only fails with partials?
14:14 < rail> ok, I see
14:15 < kev> the issue is that firefox in that release was unsigned
14:15 < kev> so the validation checks fail and it kicks off a full update
14:15 < bhearsum> ah, right
14:15 < kev> and then that breaks
14:15 < bhearsum> so if it's getting a full update in the first place, there's nothing to worry about
14:15 < rail> sounds like we're safe to remove partials then
14:15 < kev> yeah, it just has to be an explicit full
14:15 < bhearsum> yup
14:15 < bhearsum> great!
14:15 < rail> yay!
14:15 < kev> yeah. partials will never work


># ppc is not supported for 4.0b7+
>rm -rvf 4.0b8/Darwin*ppc*
>rm -rvf 4.0b9/Darwin*ppc*
>rm -rvf 4.0b8/Darwin_Universal-gcc3
>rm -rvf 4.0b9/Darwin_Universal-gcc3

Shouldn't you be removing Universal/ppc snippets for 4.0b7 too?



>function rm_partials() {
>for v in $versions 4.0.1; do
>    for channel in $aus/$v/*/*/*/{beta,betatest,release,releasetest}; do
>        if [ ! -L $channel ]; then
>            rm -vf $channel/partial.*
>        fi
>    done
>done
>

Shouldn't be needed anymore, either.


Other than the minor things above, looks good!
Attachment #593304 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 13

5 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #12)
> I wouldn't mind if lrv/vrid/lbv/lbid were more verbose - they're really not
> intuitive for me.

Just to reduce the width. :) I can rename those.
 
> Actually, these yahoo builds only have partials blocked, which means they
> fall back to the real completes. Currently, they're pointing at 10.0, and I
> think they will after your changes too?

They would be pointing to a directory which is in the patcher config, so it would be up to date.

I'm removing yahoo snippets and the special case.

> Shouldn't you be removing Universal/ppc snippets for 4.0b7 too?

I'm not touching 4.0b7 since it's not in the patcher config and not affected by backupsnip/pushsnip right now. In this bug I wanted to concentrate on supported builds.

> Other than the minor things above, looks good!

Thanks! I'll run a couple of test runs then attach the changed script.
(Assignee)

Comment 14

5 years ago
Created attachment 593552 [details]
optimize 4.0b8 -> 9.0

No mercy for 4.0.1 yahoo snippets!
Attachment #593304 - Attachment is obsolete: true
Attachment #593304 - Flags: review?(nrthomas)
Attachment #593552 - Flags: review?(nrthomas)
(Assignee)

Comment 15

5 years ago
Updated stats:

Before:

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type f |wc -l
152170

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type d |wc -l
115860

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type l |wc -l
0


After:

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type f |wc -l
2028

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type d |wc -l
3486

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type l |wc -l
2949
Created attachment 596654 [details]
optimize 5.0.1 -> 9.0.1

There are so many edge cases in the really old builds that I'm being super cautious about signing off of the whole thing. I think we could use most of the script and get a lot of the wins straight away. So I'd be happy to
* nuke the partials block for 4.0.1 yahoo
* remove the old partial.txt
* ddupe 8.0.1, 8.0, 7.0.1, 7.0, 6.0.2, 6.0.1, 6.0, 5.0.1
* use 9.0.1 instead of 9.0 as the last release, use 10.0b6 instead of 9.0b6 as last beta (since we're up to 10.0.1 and 11.0b2 now)

In the 4.0b8-10.0 dirs that takes us from 173872/131067/0 to 50834/41442/2859 for files/directories/symlinks.

Within 5.0 and 4.0* there is more to chase down but I can do that later. I haven't had a chance to get this staged yet.
Created attachment 596655 [details]
Notes on aus_*.diff
Comment on attachment 596654 [details]
optimize 5.0.1 -> 9.0.1

Must be doing something wrong because this leaves a bunch of dirs with files, which kinda defeats the point of deduping.
Attachment #596654 - Attachment is obsolete: true
Comment on attachment 593253 [details] [diff] [review]
patcher config patch

>Index: mozBeta-branch-patcher2.cfg
>-        past-update   9.0b5 9.0b6 betatest releasetest beta
>         past-update   9.0b6 10.0b1 betatest releasetest beta
>         past-update   10.0b1 10.0b2 betatest releasetest beta
>         past-update   10.0b2 10.0b3 betatest releasetest beta
>         past-update   10.0b3 10.0b4 betatest releasetest beta
>         past-update   10.0b4 10.0b5 betatest releasetest beta

Lets do this before 11.0b3 regardless of how we handle deduping/cleanup, because the backupsnip and pushsnip will be much quicker operating on fewer directories. We can also extend the removals to make
 past-update   10.0b6 11.0b1 betatest releasetest beta
the first line of older updates.

This will create a two-step update for people on older builds, ie old -> 10.0b6 -> latest, but that's OK until we symlink.

>-            <4.0b10>

I wouldn't mind if this stuff stayed in here for historical purposes. If you want to remove it, can we leave a comment like 
  # see rev X for details of older versions
Not sure if the patcher config bumper will preserve that or not.

>Index: mozRelease-branch-patcher2.cfg
>-        past-update   4.0rc1 4.0rc2 betatest releasetest release
>-        past-update   4.0rc2 4.0.1 betatest releasetest release
>-        past-update   4.0.1 5.0 betatest releasetest release
>-        past-update   5.0 5.0.1 betatest releasetest release
>-        past-update   5.0.1 6.0 betatest releasetest release
>-        past-update   6.0 6.0.1 betatest releasetest release
>-        past-update   6.0.1 6.0.2 betatest releasetest release
>-        past-update   6.0.2 7.0 betatest releasetest release
>-        past-update   7.0 7.0.1 betatest releasetest release
>-        past-update   7.0.1 8.0 betatest releasetest release
>-        past-update   8.0 8.0.1 betatest releasetest release
>-        past-update   8.0.1 9.0 betatest releasetest release
>         past-update   9.0 9.0.1 betatest releasetest release

We can remove 9.0 -> 9.0.1 too. Same comment about a comment for history. Lets leave this hunk off landing until we do the symlinking.
Attachment #593253 - Flags: review?(nrthomas) → review+
(Assignee)

Comment 20

5 years ago
Created attachment 596909 [details]
optimize 4.0b8 -> 9.0

Updated script which doesn't use ddupes.py.
Attachment #593552 - Attachment is obsolete: true
Attachment #593552 - Flags: review?(nrthomas)
Run in MPT & PHX:

cd /opt/aus2/incoming/3/Firefox

# cleanup yahoo partials
for d in 4.0.1/*/*/*/*yahoo*; do 
  rm -rvf $d
done
rm -rfv 4.0.1/Darwin_x86-gcc3-u-i386-x86_64/20110413222027/ja
rm -rfv 4.0.1/Darwin_x86_64-gcc3-u-i386-x86_64/20110413222027/ja
rm -rfv 4.0.1/Linux_x86-gcc3/20110413222027/ja-JP-mac
rm -rfv 4.0.1/Linux_x86_64-gcc3/20110413222027/ja-JP-mac
rm -rfv 4.0.1/WINNT_x86-msvc/20110413222027/ja-JP-mac

# cleanup (most of the) old partials
versions="4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0"
for v in $versions; do
    for channel in $v/*/*/*/{beta,betatest,release,releasetest}; do
        if [ ! -L $channel ]; then
            rm -vf $channel/partial.*
        fi
    done
done
------

Which is 20585 less directories, and 80637 less files to deal with.
Comment on attachment 593253 [details] [diff] [review]
patcher config patch

Rail, we talked about me writing a change to the patcher config based on an old -> 9.0b6 -> latest path, to speed up snippet operations until we dedupe. I think the mozBeta-branch-patcher2.cfg part of this is patch is the same as what I'd be doing so lets go with that. If you wanted a new patch as a way to avoid confusion from half-landing this one then feel free to carry the r+ to a new attachment.
(Assignee)

Comment 23

5 years ago
Created attachment 597008 [details] [diff] [review]
patcher config patch

Refreshed patch. Carrying on r+
Attachment #593253 - Attachment is obsolete: true
Attachment #597008 - Flags: review+
(Assignee)

Comment 24

5 years ago
Comment on attachment 597008 [details] [diff] [review]
patcher config patch

 cvs commit -m "Bug 717214 - Optimize snippets directories. p=rail,r=nthomas"
cvs commit: Examining .
Checking in mozBeta-branch-patcher2.cfg;
/cvsroot/mozilla/tools/patcher-configs/mozBeta-branch-patcher2.cfg,v  <--  mozBeta-branch-patcher2.cfg
new revision: 1.51; previous revision: 1.50
done
Attachment #597008 - Flags: checked-in+
(Assignee)

Comment 25

5 years ago
Created attachment 597128 [details] [diff] [review]
update verify configs
Attachment #597128 - Flags: review?(nrthomas)
(Assignee)

Comment 26

5 years ago
Created attachment 597148 [details] [diff] [review]
update verify configs

* old betas should get updates to 11.0b2
* use releases directory instead of candidates
Attachment #597128 - Attachment is obsolete: true
Attachment #597128 - Flags: review?(nrthomas)
Attachment #597148 - Flags: review?(nrthomas)
Comment on attachment 597148 [details] [diff] [review]
update verify configs

http://hg.mozilla.org/build/tools/rev/138047297b59
Attachment #597148 - Flags: review?(nrthomas)
Attachment #597148 - Flags: review+
Attachment #597148 - Flags: checked-in+
(Assignee)

Comment 28

5 years ago
(In reply to Nick Thomas [:nthomas] from comment #27)
> Comment on attachment 597148 [details] [diff] [review]
> update verify configs
> 
> http://hg.mozilla.org/build/tools/rev/138047297b59

So, this one didn't work, since we use chunked verify script and it just splits update verify files into smaller chunks.
(Assignee)

Comment 29

5 years ago
Created attachment 597965 [details] [diff] [review]
drop 4.0b8-9.0b5
Attachment #597965 - Flags: review?(bhearsum)
Attachment #597965 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 30

5 years ago
Comment on attachment 597965 [details] [diff] [review]
drop 4.0b8-9.0b5

http://hg.mozilla.org/build/tools/rev/8039a034c96a
Attachment #597965 - Flags: checked-in+
(Assignee)

Comment 31

5 years ago
Nick, what do you think about not touching betas for now and optimize the release snippets only? I can prep another script for this.
(Assignee)

Updated

5 years ago
Priority: P2 → P3
Created attachment 603071 [details] [diff] [review]
[cvs] backupsnip down to the buildID level, instead of buildTarget

Before:
./backupsnip Firefox-10.0.2esr-build1-test
Running /usr/bin/time -p -o /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.time /bin/tar cfvz /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.tar.gz Firefox/10.0.1/Darwin_x86-gcc3-u-i386-x86_64 Firefox/10.0.1/Darwin_x86_64-gcc3-u-i386-x86_64 ...

./backupsnip Firefox-10.0.2esr-build1-test
Running /usr/bin/time -p -o /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.time /bin/tar cfvz /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.tar.gz Firefox/10.0.1/Darwin_x86-gcc3-u-i386-x86_64/20120208062825 Firefox/10.0.1/Darwin_x86_64-gcc3-u-i386-x86_64/20120208062825 ...

45k vs 8.5k files+dirs to touch. This will help for esr and rapid releases, and marginal speedup for betas.
Attachment #603071 - Flags: review?(rail)
(Assignee)

Comment 33

5 years ago
Comment on attachment 603071 [details] [diff] [review]
[cvs] backupsnip down to the buildID level, instead of buildTarget

lgtm
Attachment #603071 - Flags: review?(rail) → review+
Comment on attachment 603071 [details] [diff] [review]
[cvs] backupsnip down to the buildID level, instead of buildTarget

Checking in backupsnip;
/cvsroot/mozilla/tools/release/bin/backupsnip,v  <--  backupsnip
new revision: 1.8; previous revision: 1.7

aus2-staging updated.
Attachment #603071 - Flags: checked-in+
Mass move of bugs to Release Automation component.
Component: Release Engineering → Release Engineering: Automation (Release Automation)
Flags: checked-in+
No longer blocks: 714371
We lost a bunch of checked-in+ flags when this bug moved, indicating a set up problem with the component.
(In reply to Nick Thomas [:nthomas] from comment #36)
> We lost a bunch of checked-in+ flags when this bug moved, indicating a set
> up problem with the component.

I re-opened bug 739370 to investigate this.
(Assignee)

Comment 38

5 years ago
Phoenix is fast enough and balrog is FTW!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.