743304 - After SSH failure, Android XUL mozilla-central nightly builder spams a bunch of "SyntaxError: invalid syntax" when running retry.py / balrog-client.py

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Daniel Holbert [:dholbert]]

Android XUL mozilla-central nightly on 2012-04-06 03:12:22 PDT for push da0d07b5ca1e
https://tbpl.mozilla.org/php/getParsedLog.php?id=10692174&tree=Firefox

That^ nightly build seems to have hit an SSH issue ("ssh: connect to host aus2-staging.mozilla.org port 22: Connection refused"), but it had much spammier / more cryptic issues afterwards, which I suspect might be a bug in our scripts that is only exposed on exceptions like this.

The chunk after that SSH failure has 5 repeated failures like this:
{{{
retry: Calling <function run_with_timeout at 0xb7cf510c> with args: (['python', '/builds/slave/m-cen-andrd-xul-ntly/tools/scripts/updates/balrog-client.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['python', '/builds/slave/m-cen-andrd-xul-ntly/tools/scripts/updates/balrog-client.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py']
Traceback (most recent call last):
  File "/builds/slave/m-cen-andrd-xul-ntly/tools/scripts/updates/balrog-client.py", line 11, in <module>
    from balrog.client.cli import NightlyRunner
  File "/builds/slave/m-cen-andrd-xul-ntly/tools/lib/python/balrog/client/cli.py", line 7, in <module>
    from balrog.client.api import API
  File "/builds/slave/m-cen-andrd-xul-ntly/tools/lib/python/balrog/client/api.py", line 4, in <module>
    import requests
  File "/builds/slave/m-cen-andrd-xul-ntly/tools/lib/python/vendor/requests-0.10.8/requests/__init__.py", line 25, in <module>
    from . import utils
  File "/builds/slave/m-cen-andrd-xul-ntly/tools/lib/python/vendor/requests-0.10.8/requests/utils.py", line 232
    return b"".join(L)
             ^
SyntaxError: invalid syntax
retry: Failed, sleeping 1 seconds before retrying
}}}

...followed by...
{{{
retry: Giving up on <function run_with_timeout at 0xb7cf510c>
Unable to successfully run ['python', '/builds/slave/m-cen-andrd-xul-ntly/tools/scripts/updates/balrog-client.py', '--build-properties', 'buildprops_balrog.json', '--api-root', 'https://aus4-admin-dev.allizom.org', '--verbose', '--credentials-file', 'BuildSlaves.py'] after 5 attempts
program finished with exit code 1
}}}


The SSH failure looks intermittent/connectivity-related (and is probably the proximate cause of bustage), but this syntax error looks like a real bug that we should fix.

Comment 1

[Daniel Holbert [:dholbert]]

Attachment: relevant part of log

Here's the relevant chunk of this log, from the failed SSH command up to the balrog-client.py syntax error issues (at the bottom).

Comment 2

[[e-mail] (:bhearsum)]

Rail, can you take a look? It sounds like we might be using the wrong Python here? Or we have a buggy python-requests library?

Comment 3

[Chris AtLee [:catlee]]

looks like it's being run with python2.5 which doesn't support b"".

Comment 4

[Rail Aliiev [:rail]]

Attachment: set PYTHON26 env for android{,-xul}

This should fix the problem

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 615728 [details] [diff] [review]
set PYTHON26 env for android{,-xul}

Seeing similar errors on Windows too.

Comment 6

[Rail Aliiev [:rail]]

Attachment: set PYTHON26 env for android{,-xul}, win32

Waiting for 2.7 on win32...

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 616584 [details] [diff] [review]
set PYTHON26 env for android{,-xul}, win32

Once I'm done with bug 616584, I can deploy this.

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 616584 [details] [diff] [review]
set PYTHON26 env for android{,-xul}, win32

http://hg.mozilla.org/build/buildbot-configs/rev/5e7816c35792

Comment 9

[Rail Aliiev [:rail]]

Worked fine for win32 nightlies. Android balrog submissions use proper python version and don't generate the reported error. However, it still can't submit updates. The corresponding issue is tracked in bug 746937,

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 616584 [details] [diff] [review]
set PYTHON26 env for android{,-xul}, win32

http://hg.mozilla.org/build/buildbot-configs/rev/b89597d25dfd

It causes win32 signing falures:

'bash' '-c' 'd:/mozilla-build/python27/python.exe e:/builds/moz2_slave/rel-m-beta-w32-bld/tools/release/signing/signtool.py --cachedir /e/builds/moz2_slave/rel-m-beta-w32-bld/signing_cache -t /e/builds/moz2_slave/rel-m-beta-w32-bld/token -n /e/builds/moz2_slave/rel-m-beta-w32-bld/nonce -c e:/builds/moz2_slave/rel-m-beta-w32-bld/tools/release/signing/host.cert -H signing1.build.scl1.mozilla.com:9120 -H signing2.build.scl1.mozilla.com:9120 -f gpg -f mar ""build/obj-firefox/dist/update/win32/en-US/firefox-13.0b2-13.0b3.partial.mar""'

....

IOError: [Errno 2] No such file or directory: '\\build/obj-firefox/dist/update/win32/en-US/firefox-13.0b2-13.0b3.partial.mar\\'

Comment 11

[Rail Aliiev [:rail]]

Attachment: Remove extra quotes

The patch removes extra quotes. Passed staging.

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 624606 [details] [diff] [review]
Remove extra quotes

http://hg.mozilla.org/build/buildbotcustom/rev/b5ca2e87801b

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 616584 [details] [diff] [review]
set PYTHON26 env for android{,-xul}, win32

http://hg.mozilla.org/build/buildbot-configs/rev/d518706b4646

Comment 14

[Mike Taylor [:bear]]

in production 2012-05-17 0750 PDT

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 616584 [details] [diff] [review]
set PYTHON26 env for android{,-xul}, win32

The win32 portion of this patch was backed out in bug 758064 due to win32 release l10n repack bustage.

Comment 16

[Chris AtLee [:catlee]]

maybe msys is doing it? http://www.mingw.org/wiki/Posix_path_conversion

Comment 17

[Rail Aliiev [:rail]]

Closing this one and reopening bug 758064, which has more details related to the win32 part of the bug.