Make hgtool.py not complain about missing bundles

RESOLVED FIXED

Status

Release Engineering
General Automation
P4
normal
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: armenzg, Assigned: catlee)

Tracking

({sheriffing-P1})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
This is not a high priority IMHO.

https://tbpl.mozilla.org/php/getParsedLog.php?id=17678234&tree=Elm&full=1#error0

Trying to use bundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/elm.hg
command: START
command: hg unbundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/elm.hg
command: cwd: e:\builds\hg-shared\projects\elm
command: output:
abort: HTTP Error 404: Not Found
command: ERROR
Traceback (most recent call last):
  File "e:/builds/moz2_slave/elm-w32/tools/buildfarm/utils\../../lib/python\util\commands.py", line 40, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "C:\mozilla-build\python27\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['hg', 'unbundle', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/elm.hg']' returned non-zero exit status -1
command: END (0.11s elapsed)

Problem unbundling/pulling from http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/elm.hg
Traceback (most recent call last):
  File "e:/builds/moz2_slave/elm-w32/tools/buildfarm/utils\../../lib/python\util\hg.py", line 142, in clone
    unbundle(bundle, dest)
  File "e:/builds/moz2_slave/elm-w32/tools/buildfarm/utils\../../lib/python\util\hg.py", line 506, in unbundle
    run_cmd(['hg', 'unbundle', bundle], cwd=dest)
  File "e:/builds/moz2_slave/elm-w32/tools/buildfarm/utils\../../lib/python\util\commands.py", line 40, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "C:\mozilla-build\python27\lib\subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['hg', 'unbundle', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/elm.hg']' returned non-zero exit status -1
Using bundles failed; falling back to clone
Attempting to clone from mirrors
I like having this detailed output from hgtool. Consumers should be smart enough to figure out what happened through log output + return code.
(Assignee)

Comment 2

5 years ago
classic case of wanting more informational logs to assist in debugging vs. having concise logs.

in any case, I think hgtool is being pretty spammy here, the traceback is output twice.

maybe hgtool can handle the output, and turn an "abort: HTTP Error 404: Not Found" into an informational message like "Got 404 when fetching bundle; falling back to clone"

Comment 3

5 years ago
"abort: " lines are picked up by TBPL's parser, so this has the potential to cause confusion.

We cannot remove the regex for these lines from TBPL, since in other places, they are the real error. Please can we do similar to what mozharness has done and suppress things like this (or else replace with warning style strings not "abort:")?
Whiteboard: [sheriff-want]
(Assignee)

Comment 4

5 years ago
(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #3)
> "abort: " lines are picked up by TBPL's parser, so this has the potential to
> cause confusion.
> 
> We cannot remove the regex for these lines from TBPL, since in other places,
> they are the real error. Please can we do similar to what mozharness has
> done and suppress things like this (or else replace with warning style
> strings not "abort:")?

The complication here is that these messages are coming directly from hg, hgtool is merely passing them through.
(Assignee)

Comment 5

5 years ago
Maybe hgtool could do a HEAD request for the bundle url before trying to use it?

Comment 6

5 years ago
(In reply to Chris AtLee [:catlee] from comment #4)
> The complication here is that these messages are coming directly from hg,
> hgtool is merely passing them through.

Yeah I was meaning we could just suppress the stdout/stderr

Comment 7

5 years ago
(This caused me to miss the real cause of bug 822651)

Updated

5 years ago
Keywords: sheriffing-P1
Whiteboard: [sheriff-want]
(Assignee)

Updated

5 years ago
Assignee: nobody → catlee
(Assignee)

Comment 8

5 years ago
Created attachment 735759 [details] [diff] [review]
make unbundling quieter
Attachment #735759 - Flags: review?(jhopkins)
Comment on attachment 735759 [details] [diff] [review]
make unbundling quieter

If you want to avoid the 'abort' output being passed from hg (and confusing tbpl), you may want to change include_stderr to False here:

+        get_output(['hg', 'unbundle', bundle], cwd=dest, include_stderr=True)
Attachment #735759 - Flags: review?(jhopkins) → review+
(Assignee)

Comment 10

5 years ago
Actually, I think that include_stderr means that get_output captures stderr as well as stdout, and so prevents the "abort" error from appearing in the log. That was my intent at least!
(Assignee)

Updated

5 years ago
Attachment #735759 - Flags: checked-in+
(Assignee)

Comment 11

5 years ago
I think this is fixed?
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Priority: -- → P4
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.