717214 - Optimize snippets directories

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, defect, P3)

Description

[Rail Aliiev [:rail]]

It would be great to optimize snippets so that we have symlinks for duplicating directories instead of copies.

Nick suggested to post process aus2{,.test,.beta} directories being generated by patcher.

Comment 1

[Rail Aliiev [:rail]]

Attachment: ddupes.py

Example usage:

# pass all build IDs
PYTHONPATH=tools/lib/python python ddupes.py aus2/*/*/*/*

# convert absolute symlinks to relative
symlinks -cr aus2

Comment 2

[Rail Aliiev [:rail]]

My only concern here is the fact that we have to use rsync --force in pushsnip to replace directories with symlinks (and remove old partials, yay!).

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 587664 [details]
ddupes.py

Talked with catlee about this:

* we can't use the generated snippets because generated structure doesn't represents the directory layout on aus2-state. Pushing final snippets would destroy test snippets and vice versa.
* The scripts could be used to optimize live snippets:
 ** Define which releases should be optimized
 ** Remove those releases from the patcher config (save into another file for historical reasons) so patcher won't touch the directories in the future.
 ** The script should ignore partial.txt files for old releases.

Comment 4

[Rail Aliiev [:rail]]

I played with snippets and this is what I have now:

==================================================
aus=/dev/shm/aus2/Firefox
versions="4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 7.0 7.0.1"

echo -n > optimized_dirs.txt
for v in $versions; do
    for channel in $aus/$v/*/*/*/{beta,betatest,release,releasetest}; do
        if [ -f $channel/partial.txt -a -f $channel/complete.txt ]; then
            path=$(echo $channel | sed "s,$aus/,,")
            platform=$(echo $path | awk -F/ '{print $2}')
            buildID=$(echo $path | awk -F/ '{print $3}')
            echo "$aus/$v/$platform/$buildID" >> optimized_dirs.txt
            rm -vf $channel/partial.txt
        fi
    done
done

sort -u optimized_dirs.txt > optimized_dirs.txt.new
mv optimized_dirs.txt.new optimized_dirs.txt


# slightly modified version which read directories from a file
PYTHONPATH=tools/lib/python python \
    ddupes.py optimized_dirs.txt

for d in `cat optimized_dirs.txt`; do
    if [ ! -L $d ]; then
        # print not optimized dirs
        echo $d
    fi
done
==================================================

It:
1) deletes partial.txt from some old releases

files: 118962 vs 67272
du: 339M vs 183M

2) link duplicate dirs

files: 67272 vs 19170
du: 183M vs 75M

3) print directories which we are unable to symlink
63 dirs out of 170

There shouldn't be any problem with 1), it's straight forward. It could be also applied to other (3.6.x) directories. It should reduce backupsnip time.

I have some concerns about 2) because it can't optimize directories for some releases when we changed the list of locales and those builds can't be removed from the patcher config (otherwise they wouldn't get updates to the latest release). As possible workaround we can leave those builds in the patcher config.

Ideas, comments?

Comment 5

[Rail Aliiev [:rail]]

Attachment: ddupes.py

Comment 6

[Rail Aliiev [:rail]]

Attachment: optimize 4.0b8 -> 9.0

This script optimizes 4.0b8 -> 9.0 snippets so we could remove them from the patcher configs.

Before: 472M, 120759 files
After: 8.0M, 22608 files

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 592171 [details]
optimize 4.0b8 -> 9.0

Wow, there's a lot to digest in this.

>####################################################
># special case for 4.0.1, cck :(, point channels to 5.0
># do we need them?

Kev is the best person to ask.

># ppc is not supported for 4.0b8+

I have it as b7 onwards at https://wiki.mozilla.org/index.php?title=User:NThomas:Mac_BUILDTARGET, double check ?

>####################################################
># rm beta channels for 4.0rc1, 4.0.1, 5.0, 5.0.1, 6.0, 6.0.1, 6.0.2
># we generated beta snippets for these releases :(

So these are the releases we published on the beta channel ? Have we managed to move the beta users off already, so we don't need to provide a path forward ? I see that it mixes things up if we have to.

>function rm_partials() {
>for v in $versions 4.0.1; do
>    if [ ! -L $aus/$v/Darwin_x86_64-gcc3 ]; then
>        rm -rvf $aus/$v/Darwin_x86_64-gcc3
>        ln -svf Darwin_x86_64-gcc3-u-i386-x86_64 $aus/$v/Darwin_x86_64-gcc3
>    fi

Nit-ish, clearer if all the mac specific stuff is put together. This plus the 'Darwin platform name changes' block slightly earlier.

>    # 7.0 build2 started before updates are generated, no snippets available

If there are users from ftp downloads we can retrieve the buildID and create a path. Are the buildN's from buildIDs in the snippet dirs ? I may have an independent source to cross check against.

>function magic() {

Nit, helps readability if function order matches call order.

>PYTHONPATH=/home/rail/work/mozilla/build/tools/lib/python python \
>    ddupes.py optimized_dirs.txt

Does this dedupe.py differ a lot from attachment 592170 [details] ?

>    symlinks -crs $aus/$v

Woo, learned something!

>Before: 472M, 120759 files
>after: 8.0M, 22608 files

This is very nice! To get a rough speed up you should count files+dirs though.

>How to check:
>rsync --copy-links -r /dev/shm/aus2/Firefox/ ~/tmp/aus.new/Firefox/
>for v in 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 4.0; do diff -ur --exclude partial.* aus.old/Firefox/$v aus.new/Firefox/$v >>aus.diff; done

Haven't run this yet, but I'm planning to. It would be fantastic to see this for 3.6 too, if you have time.

Comment 8

[Rail Aliiev [:rail]]

Attachment: optimize 4.0b8 -> 9.0

Updated version
* don't delete beta channels for release builds. symlink them to the latest beta
* address comments.

Comment 9

[Rail Aliiev [:rail]]

Attachment: patcher config patch

* would be good to keep removed versions in a separate file to easier find them.

Comment 10

[Rail Aliiev [:rail]]

Attachment: optimize 4.0b8 -> 9.0

The previous one without unneeded exit

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 593253 [details] [diff] [review]
patcher config patch

I tested every removed build ID and all of them are symlinks or locale/channel are symlinks (for 4.0.1 and some others, for example).

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 593304 [details]
optimize 4.0b8 -> 9.0

># last release is 9.0/*/20111216140209
>lrv=9.0
>lrid=20111216140209
># last beta 9.0/*/20111212185108
>lbv=9.0
>lbid=20111212185108

I wouldn't mind if lrv/vrid/lbv/lbid were more verbose - they're really not intuitive for me.


> # special case for 4.0.1, cck :(, point channels to 5.0
> # do we need them?

Actually, these yahoo builds only have partials blocked, which means they fall back to the real completes. Currently, they're pointing at 10.0, and I think they will after your changes too?

> for localedir in 4.0.1/*/*/*; do
>    for channel in betatest release releasetest; do
>        if [ ! -L  $localedir/$channel ]; then
>            rm -rvf $localedir/$channel
>            s=${localedir/4.0.1/${lrv}}
>            s=${s/20110413222027/${lrid}}
>            ln -sv ../../../../$s/$channel $localedir/$channel
>        fi
>    done
> done

Per IRC with Kev, we don't need this anymore:
14:14 < bhearsum> ok, so we definitely don't need those anymore, because there's no way for them to get 
                  partials anymore
14:14 < kev> updater has a bug where, if there is a distribution directory in the app dir, updates fail
14:14 < bhearsum> we can remove the blocks, and they'll continue to get complete updates to the latest firefox
14:14 < kev> so the snippets give them full updates
14:14 < bhearsum> kev: but only fails with partials?
14:14 < rail> ok, I see
14:15 < kev> the issue is that firefox in that release was unsigned
14:15 < kev> so the validation checks fail and it kicks off a full update
14:15 < bhearsum> ah, right
14:15 < kev> and then that breaks
14:15 < bhearsum> so if it's getting a full update in the first place, there's nothing to worry about
14:15 < rail> sounds like we're safe to remove partials then
14:15 < kev> yeah, it just has to be an explicit full
14:15 < bhearsum> yup
14:15 < bhearsum> great!
14:15 < rail> yay!
14:15 < kev> yeah. partials will never work


># ppc is not supported for 4.0b7+
>rm -rvf 4.0b8/Darwin*ppc*
>rm -rvf 4.0b9/Darwin*ppc*
>rm -rvf 4.0b8/Darwin_Universal-gcc3
>rm -rvf 4.0b9/Darwin_Universal-gcc3

Shouldn't you be removing Universal/ppc snippets for 4.0b7 too?



>function rm_partials() {
>for v in $versions 4.0.1; do
>    for channel in $aus/$v/*/*/*/{beta,betatest,release,releasetest}; do
>        if [ ! -L $channel ]; then
>            rm -vf $channel/partial.*
>        fi
>    done
>done
>

Shouldn't be needed anymore, either.


Other than the minor things above, looks good!

Comment 13

[Rail Aliiev [:rail]]

(In reply to Ben Hearsum [:bhearsum] from comment #12)
> I wouldn't mind if lrv/vrid/lbv/lbid were more verbose - they're really not
> intuitive for me.

Just to reduce the width. :) I can rename those.
 
> Actually, these yahoo builds only have partials blocked, which means they
> fall back to the real completes. Currently, they're pointing at 10.0, and I
> think they will after your changes too?

They would be pointing to a directory which is in the patcher config, so it would be up to date.

I'm removing yahoo snippets and the special case.

> Shouldn't you be removing Universal/ppc snippets for 4.0b7 too?

I'm not touching 4.0b7 since it's not in the patcher config and not affected by backupsnip/pushsnip right now. In this bug I wanted to concentrate on supported builds.

> Other than the minor things above, looks good!

Thanks! I'll run a couple of test runs then attach the changed script.

Comment 14

[Rail Aliiev [:rail]]

Attachment: optimize 4.0b8 -> 9.0

No mercy for 4.0.1 yahoo snippets!

Comment 15

[Rail Aliiev [:rail]]

Updated stats:

Before:

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type f |wc -l
152170

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type d |wc -l
115860

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type l |wc -l
0


After:

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type f |wc -l
2028

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type d |wc -l
3486

find 4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0 -type l |wc -l
2949

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: optimize 5.0.1 -> 9.0.1

There are so many edge cases in the really old builds that I'm being super cautious about signing off of the whole thing. I think we could use most of the script and get a lot of the wins straight away. So I'd be happy to
* nuke the partials block for 4.0.1 yahoo
* remove the old partial.txt
* ddupe 8.0.1, 8.0, 7.0.1, 7.0, 6.0.2, 6.0.1, 6.0, 5.0.1
* use 9.0.1 instead of 9.0 as the last release, use 10.0b6 instead of 9.0b6 as last beta (since we're up to 10.0.1 and 11.0b2 now)

In the 4.0b8-10.0 dirs that takes us from 173872/131067/0 to 50834/41442/2859 for files/directories/symlinks.

Within 5.0 and 4.0* there is more to chase down but I can do that later. I haven't had a chance to get this staged yet.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Notes on aus_*.diff

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 596654 [details]
optimize 5.0.1 -> 9.0.1

Must be doing something wrong because this leaves a bunch of dirs with files, which kinda defeats the point of deduping.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 593253 [details] [diff] [review]
patcher config patch

>Index: mozBeta-branch-patcher2.cfg
>-        past-update   9.0b5 9.0b6 betatest releasetest beta
>         past-update   9.0b6 10.0b1 betatest releasetest beta
>         past-update   10.0b1 10.0b2 betatest releasetest beta
>         past-update   10.0b2 10.0b3 betatest releasetest beta
>         past-update   10.0b3 10.0b4 betatest releasetest beta
>         past-update   10.0b4 10.0b5 betatest releasetest beta

Lets do this before 11.0b3 regardless of how we handle deduping/cleanup, because the backupsnip and pushsnip will be much quicker operating on fewer directories. We can also extend the removals to make
 past-update   10.0b6 11.0b1 betatest releasetest beta
the first line of older updates.

This will create a two-step update for people on older builds, ie old -> 10.0b6 -> latest, but that's OK until we symlink.

>-            <4.0b10>

I wouldn't mind if this stuff stayed in here for historical purposes. If you want to remove it, can we leave a comment like 
  # see rev X for details of older versions
Not sure if the patcher config bumper will preserve that or not.

>Index: mozRelease-branch-patcher2.cfg
>-        past-update   4.0rc1 4.0rc2 betatest releasetest release
>-        past-update   4.0rc2 4.0.1 betatest releasetest release
>-        past-update   4.0.1 5.0 betatest releasetest release
>-        past-update   5.0 5.0.1 betatest releasetest release
>-        past-update   5.0.1 6.0 betatest releasetest release
>-        past-update   6.0 6.0.1 betatest releasetest release
>-        past-update   6.0.1 6.0.2 betatest releasetest release
>-        past-update   6.0.2 7.0 betatest releasetest release
>-        past-update   7.0 7.0.1 betatest releasetest release
>-        past-update   7.0.1 8.0 betatest releasetest release
>-        past-update   8.0 8.0.1 betatest releasetest release
>-        past-update   8.0.1 9.0 betatest releasetest release
>         past-update   9.0 9.0.1 betatest releasetest release

We can remove 9.0 -> 9.0.1 too. Same comment about a comment for history. Lets leave this hunk off landing until we do the symlinking.

Comment 20

[Rail Aliiev [:rail]]

Attachment: optimize 4.0b8 -> 9.0

Updated script which doesn't use ddupes.py.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Run in MPT & PHX:

cd /opt/aus2/incoming/3/Firefox

# cleanup yahoo partials
for d in 4.0.1/*/*/*/*yahoo*; do 
  rm -rvf $d
done
rm -rfv 4.0.1/Darwin_x86-gcc3-u-i386-x86_64/20110413222027/ja
rm -rfv 4.0.1/Darwin_x86_64-gcc3-u-i386-x86_64/20110413222027/ja
rm -rfv 4.0.1/Linux_x86-gcc3/20110413222027/ja-JP-mac
rm -rfv 4.0.1/Linux_x86_64-gcc3/20110413222027/ja-JP-mac
rm -rfv 4.0.1/WINNT_x86-msvc/20110413222027/ja-JP-mac

# cleanup (most of the) old partials
versions="4.0b8 4.0b9 4.0b10 4.0b11 4.0b12 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0 7.0.1 8.0 8.0.1 9.0"
for v in $versions; do
    for channel in $v/*/*/*/{beta,betatest,release,releasetest}; do
        if [ ! -L $channel ]; then
            rm -vf $channel/partial.*
        fi
    done
done
------

Which is 20585 less directories, and 80637 less files to deal with.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 593253 [details] [diff] [review]
patcher config patch

Rail, we talked about me writing a change to the patcher config based on an old -> 9.0b6 -> latest path, to speed up snippet operations until we dedupe. I think the mozBeta-branch-patcher2.cfg part of this is patch is the same as what I'd be doing so lets go with that. If you wanted a new patch as a way to avoid confusion from half-landing this one then feel free to carry the r+ to a new attachment.

Comment 23

[Rail Aliiev [:rail]]

Attachment: patcher config patch

Refreshed patch. Carrying on r+

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 597008 [details] [diff] [review]
patcher config patch

 cvs commit -m "Bug 717214 - Optimize snippets directories. p=rail,r=nthomas"
cvs commit: Examining .
Checking in mozBeta-branch-patcher2.cfg;
/cvsroot/mozilla/tools/patcher-configs/mozBeta-branch-patcher2.cfg,v  <--  mozBeta-branch-patcher2.cfg
new revision: 1.51; previous revision: 1.50
done

Comment 25

[Rail Aliiev [:rail]]

Attachment: update verify configs

Comment 26

[Rail Aliiev [:rail]]

Attachment: update verify configs

* old betas should get updates to 11.0b2
* use releases directory instead of candidates

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 597148 [details] [diff] [review]
update verify configs

http://hg.mozilla.org/build/tools/rev/138047297b59

Comment 28

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #27)
> Comment on attachment 597148 [details] [diff] [review]
> update verify configs
> 
> http://hg.mozilla.org/build/tools/rev/138047297b59

So, this one didn't work, since we use chunked verify script and it just splits update verify files into smaller chunks.

Comment 29

[Rail Aliiev [:rail]]

Attachment: drop 4.0b8-9.0b5

Comment 30

[Rail Aliiev [:rail]]

Comment on attachment 597965 [details] [diff] [review]
drop 4.0b8-9.0b5

http://hg.mozilla.org/build/tools/rev/8039a034c96a

Comment 31

[Rail Aliiev [:rail]]

Nick, what do you think about not touching betas for now and optimize the release snippets only? I can prep another script for this.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] backupsnip down to the buildID level, instead of buildTarget

Before:
./backupsnip Firefox-10.0.2esr-build1-test
Running /usr/bin/time -p -o /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.time /bin/tar cfvz /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.tar.gz Firefox/10.0.1/Darwin_x86-gcc3-u-i386-x86_64 Firefox/10.0.1/Darwin_x86_64-gcc3-u-i386-x86_64 ...

./backupsnip Firefox-10.0.2esr-build1-test
Running /usr/bin/time -p -o /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.time /bin/tar cfvz /opt/aus2/snippets/backup/20120305-3-pre-Firefox-10.0.2esr-build1-test.tar.gz Firefox/10.0.1/Darwin_x86-gcc3-u-i386-x86_64/20120208062825 Firefox/10.0.1/Darwin_x86_64-gcc3-u-i386-x86_64/20120208062825 ...

45k vs 8.5k files+dirs to touch. This will help for esr and rapid releases, and marginal speedup for betas.

Comment 33

[Rail Aliiev [:rail]]

Comment on attachment 603071 [details] [diff] [review]
[cvs] backupsnip down to the buildID level, instead of buildTarget

lgtm

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 603071 [details] [diff] [review]
[cvs] backupsnip down to the buildID level, instead of buildTarget

Checking in backupsnip;
/cvsroot/mozilla/tools/release/bin/backupsnip,v  <--  backupsnip
new revision: 1.8; previous revision: 1.7

aus2-staging updated.

Comment 35

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

We lost a bunch of checked-in+ flags when this bug moved, indicating a set up problem with the component.

Comment 37

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #36)
> We lost a bunch of checked-in+ flags when this bug moved, indicating a set
> up problem with the component.

I re-opened bug 739370 to investigate this.

Comment 38

[Rail Aliiev [:rail]]

Phoenix is fast enough and balrog is FTW!