1232031 - Intermittent repack failures due to truncated mar files

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Typically shows up:

Traceback (most recent call last):
  File "/builds/slave/rel-m-esr38-m64_rpk_8-00000000/scripts/scripts/l10n/create-release-repacks.py", line 134, in createRepacks
    mozillaDir=mozillaDir, mozillaSrcDir=mozillaSrcDir)
  File "/builds/slave/rel-m-esr38-m64_rpk_8-00000000/scripts/lib/python/build/l10n.py", line 207, in repackLocale
    cwd=path.join(nativeDistDir, 'previous'), env=env)
  File "/builds/slave/rel-m-esr38-m64_rpk_8-00000000/scripts/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['perl', '../../../tools/update-packaging/unwrap_full_update.pl', '../../../..//firefox-38.3.0esr.ru.complete.mar']' returned non-zero exit 

The previous complete mar files are smaller than they should be.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Check size after d/l

Lets see if a retry solves the problem. It may be that Cloudfront is caching the wrong size and this won't help, but lets hope it does.

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8697672 [details] [diff] [review]
[tools] Check size after d/l

Review of attachment 8697672 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/release/download.py
@@ +77,5 @@
>      local_f.write(remote_f.read())
>      local_f.close()
> +
> +    expected_size = int(remote_f.info()['Content-Length'])
> +    actual_size =  os.stat(destFileName).st_size

I'd use os.path.getsize(), which uses os.stat() under the hood.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8697672 [details] [diff] [review]
[tools] Check size after d/l

Landed with getsize():
https://hg.mozilla.org/build/tools/rev/b8b1fb9b95eb

Now we wait to see if it helps, looking for 'File is truncated' messages and retries.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

We didn't hit any job failures with 38.0.5 build2 because the new code used retry 3 times, eg

Downloading http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/38.3.0esr-candidates/build2/update/linux-x86_64/lij/firefox-38.3.0esr.complete.mar to firefox-38.3.0esr.lij.complete.mar
retry: Calling <function downloadUpdateIgnore404 at 0x2a87d70> with args: ('archive.mozilla.org', 'firefox', '38.3.0esr', 2, 'linux64', 'lij'), kwargs: {}, attempt #2
Downloading http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/38.3.0esr-candidates/build2/update/linux-x86_64/lij/firefox-38.3.0esr.complete.mar to firefox-38.3.0esr.lij.complete.mar

Strange that the warning (File is truncated ....') didn't show up given the master code sets logging.INFO. And actually, we could have prevented this much earlier by not using .../mozilla.org/... in the path, and hence avoiding the proxy that serves those requests.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Tweaks

Use log.info when we detect truncated file, and probably avoid them altogether by going direct to S3 by dropping mozilla.org from the request path.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8699302 [details] [diff] [review]
Tweaks

https://hg.mozilla.org/build/tools/rev/80741989e34f

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Reopening to fix tests.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Remove /mozilla.org/ from makeReleasesDir(), fix tests

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8699617 [details] [diff] [review]
Remove /mozilla.org/ from makeReleasesDir(), fix tests

https://hg.mozilla.org/build/tools/rev/39da3b2f7be5

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

In the event of any bustage I suggest backing out 39da3b2f7be5 and 80741989e34f. That would get you to a working but not logging state.