957455 - Valgrind builds don't specify --mirror or --bundle (intermittent HTTP Error 413: Request Entity Too Large errors)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

A typical linux desktop build does:
 [blah blah] hgtool.py \
 --mirror http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound \
 --bundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg \
 http://hg.mozilla.org/integration/mozilla-inbound build

The mirror option points to a set of hg servers reserved for build machines, and bundle uses a snapshot to save on network traffic (the snapshot is updated weekly). If it needs to clone, hgtool will try the bundle first, then the mirror, before hitting hg.m.o.


The valgrind script has something much simpler that results in:
 hgtool.py --rev aac565918cf5bde25d0934bbb01bf043e01af31a \
 http://hg.mozilla.org/try src

This results in trying to clone the entire try repo, and sometimes we get errors:
command: START
command: hg clone -U http://hg.mozilla.org/try /builds/hg-shared/try
command: cwd: /builds/slave/try-l64-valgrind-0000000000000
command: output:
requesting all changes
abort: HTTP Error 413: Request Entity Too Large
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/try-l64-valgrind-0000000000000/scripts/buildfarm/utils/../../lib/python/util/commands.py", line 47, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['hg', 'clone', '-U', 'http://hg.mozilla.org/try', '/builds/hg-shared/try']' returned non-zero exit status 255
command: END (12.68s elapsed)

hg is requesting a very large number of heads via the HTTP header.

Comment 1

[Nicholas Nethercote [inactive]]

Thanks for catching this, nthomas.

I wonder if bug 957304 is related to this.

Comment 3

[Daniel Holbert [:dholbert]]

I just hit this (and filed a dupe bug) FWIW. My log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=32723408&tree=Try

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Sync hgtool.py call with linux desktop builds

Adds --mirror and --bundle, and wraps the whole lot in a retry.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Sync hgtool.py call with linux desktop builds

Whoops, the OS went all Maverick on me.

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8357507 [details] [diff] [review]
[tools] Sync hgtool.py call with linux desktop builds

Review of attachment 8357507 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/valgrind/valgrind.sh
@@ +16,5 @@
>  
>      BRANCHES_JSON=$SCRIPTS_DIR/buildfarm/maintenance/production-branches.json
>  
>      HG_REPO=$($JSONTOOL -k ${branch}.repo $BRANCHES_JSON)
> +    HG_MIRROR=${HG_REPO/hg.\mozilla\.org/hg-internal.dmz.scl3.mozilla.com}

A nit: the first backslash should escape ".", not "m".

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8357507 [details] [diff] [review]
[tools] Sync hgtool.py call with linux desktop builds

https://hg.mozilla.org/build/tools/rev/9c3c3cc72800

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

The landing included the nit, good catch.

I've seen jobs on several trees using the internal hg mirror, nothing using a bundle yet as we haven't caught a slave without a hg share already.

Comment 10

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=32879532&tree=Try seems to have tried to use a bundle, and failed, and then 413 failed the job.

Comment 11

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=32879212&tree=Try on the other hand, rather than silently failing to unbundle, just timed out unbundling.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

The first one is just weird - the unbundle succeeded AFAICT looking at the code. The second one is just taking too long to unbundle. I suspect download time on the bundle.

Comment 13

[Phil Ringnalda (:philor)]

Oy, there wasn't much running on other trees, so I didn't even notice that it was at the bug 957502 time of day; the push after me also timed out, but before and after the every evening bustage, things went fine.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

I'm open to making the timeout longer, if the network fixes we're working on don't get us out of the woods. We'd have to do that for the whole valgrind script rather than just the clone.