1199197 - Consider using local mbsdiff cache for release l10n repacks

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

Attachment: cache_diffs-tools.diff

Using local cache for generating partial updates optimized funsize 40-50% (with chunks enabled). 

Even though we are looking forward for release promotion, where partials will be generated separately from builds/repacks, it'd be great to improve the current situation. The change is relatively small, but it'd be great to test it on Windows in staging. We have never used this on windows, I suspect that we may hit path issues (/ vs \\) and path length issues.


I just copied the script we use in funsize from https://github.com/mozilla/funsize/blob/master/docker/funsize-update-generator/scripts/mbsdiff_hook.sh and added 2 lines of code to the existing wrapper.

Comment 1

[Rail Aliiev [:rail]]

Nick, can I use your staging setup to test this? landing+tagging+retriggering a repack should do the trick

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Feel free to land your stuff in users/stage-ffxbld/ repos and rerun l10n jobs, but you'll need bug 1200094 fixed first (upload host is broken). I'm also planning to do a 41.0 build2 once all the changes are finalised across firefox/mobile/thunderbird.

Comment 3

[Rail Aliiev [:rail]]

I pushed to
remote:   https://hg.mozilla.org/users/stage-ffxbld/tools/rev/5d34736f4f61
remote:   https://hg.mozilla.org/users/stage-ffxbld/tools/rev/9adc6f6b41d2

Linux is still in progress, but looking promising.

Mac is failing to run `openssl sha512`, but the script doesn't handle that properly:

/builds/slave/rel-m-rel-m64_rpk_1-0000000000/fs-cache// -> /builds/slave/rel-m-rel-m64_rpk_1-0000000000/mozilla-release/obj-l10n/dist/current.work/Contents/MacOS/XUL.patch.bz2
/builds/slave/rel-m-rel-m64_rpk_1-0000000000/fs-cache///CodeResources.patch.bz2 -> /builds/slave/rel-m-rel-m64_rpk_1-0000000000/mozilla-release/obj-l10n/dist/current.work/Contents/MacOS/XUL.patch.bz2/CodeResources.patch.bz2
Successful retrieved /builds/slave/rel-m-rel-m64_rpk_1-0000000000/mozilla-release/obj-l10n/dist/current.work/Contents/MacOS/XUL.patch.bz2 from local cache!
file "Contents/MacOS/XUL" found in funsize, diffing skipped
      patch "Contents/MacOS/XUL.patch" "Contents/MacOS/XUL"
diffing "Contents/Info.plist"
openssl:Error: 'sha512' is an invalid command

Comment 4

[Rail Aliiev [:rail]]

Yup, linux looks fine:

diffing "libxul.so"
`/builds/slave/rel-m-rel-lx_rpk_1-00000000000/fs-cache/bceb366ef8a47d187403740fbeedf9b0cbd84a33e37af79b46f8fad99532c0b07a96afced9b1d2a367f1a35fc2a1e42e93b523e5674f0407ec04c6fe55e31c85/592f96dcc09e07d7b2965546c949eeb0f68ea0cd991ed3e91b77a92cff2bec0cf675cd99eccf907106873efb718536ab6dbed584b44ac74f852c78720a77a173' -> `/builds/slave/rel-m-rel-lx_rpk_1-00000000000/mozilla-release/obj-l10n/dist/current.work/libxul.so.patch.bz2'
Successful retrieved /builds/slave/rel-m-rel-lx_rpk_1-00000000000/mozilla-release/obj-l10n/dist/current.work/libxul.so.patch.bz2 from local cache!
file "libxul.so" found in funsize, diffing skipped
      patch "libxul.so.patch" "libxul.so"

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

'openssl dgst -sha512' is probably more portable.

Comment 6

[Rail Aliiev [:rail]]

Windows complained about `openssl sha512` as well. I reverted the patch:

remote:   https://hg.mozilla.org/users/stage-ffxbld/tools/rev/686f74e407fd
remote:   https://hg.mozilla.org/users/stage-ffxbld/tools/rev/607b51088652

Comment 7

[Rail Aliiev [:rail]]

Pushed again:

remote:   https://hg.mozilla.org/users/stage-ffxbld/tools/rev/ef819c0a3edb
remote:   https://hg.mozilla.org/users/stage-ffxbld/tools/rev/b30d30ac5cfd

Comment 8

[Rail Aliiev [:rail]]

Interesting:

$ grep -c "File is not in the locale cache" linux.log 
69
$ grep -c "found in funsize, diffing skipped" linux.log
108

$ grep -c "File is not in the locale cache" mac.log    
108
$ grep -c "found in funsize, diffing skipped" mac.log  
30

I wonder if this is related to Mac signing...

Windows is still trying to purge builds...

Comment 9

[Rail Aliiev [:rail]]

Sweet.

$ grep -c "File is not in the locale cache" win32.log 
88
$ grep -c "found in funsize, diffing skipped" win32.log 
128

Comment 10

[Rail Aliiev [:rail]]

Attachment: l10n_diff_cache.diff

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8656950 [details] [diff] [review]
l10n_diff_cache.diff

Review of attachment 8656950 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine to me. In this local-only implementation $LOCAL_CACHE_DIR must have a value for any caching to happen, so consider making that a mandatory argument. Did you do a comparison between the old method and this for the staging release ? IIRC the bucket is versioned, so it should be possible to grab them, and compare contents and 'mar -T' output.

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8656950 [details] [diff] [review]
l10n_diff_cache.diff

https://hg.mozilla.org/build/tools/rev/0c36841d6cd2

Comment 13

[Rail Aliiev [:rail]]

./mar -T firefox-40.0.2-41.0.partial.mar 
Signature block found with 1 signature
1 additional block found:
  - Product Information Block:
    - MAR channel name: firefox-mozilla-release
    - Product version: 41.0

SIZE    MODE    NAME
1476    0666    softokn3.dll.patch
176     0666    application.ini.patch
1782    0666    sandboxbroker.dll.patch
....