1315253 - Autoland stuck due to exception thrown during error handling

Status: RESOLVED FIXED

(Conduit Graveyard :: Transplant, defect)

Description

[Carsten Book [:Tomcat]]

We had some long tree closures because of different reasons and now that the trees are reopen, it seems autoland is stuck. 

I tried to land https://reviewboard.mozilla.org/r/90676/diff/1#index_header but so far it didn't happen

Comment 1

[Carsten Book [:Tomcat]]

gps can you take a look at this thx !

Comment 2

[Gregory Szorc [:gps]]

I'm looking into it. The daemon seemed to be stuck in an infinite loop. I stopped the daemon, cleared out the working directory, and started the daemon. Let's see what happens...

Comment 3

[Gregory Szorc [:gps]]

Autoland is running again.

Locally adding some extra logging, I found the following:

    15:51:55,136 autoland INFO base revision: 6ca50bdebc9e
    15:52:02,106 autoland INFO rebase output: rebasing 349669:6ca50bdebc9e "Bug 1310019 - web extension API : chrome.tabs.query doesn’t get tabs title at first call, r=kmag" (tip)
    merging browser/components/extensions/ext-browserAction.js
    merging browser/components/extensions/ext-utils.js
    merging browser/components/extensions/test/browser/browser_ext_browserAction_popup.js
    warning: conflicts while merging browser/components/extensions/ext-browserAction.js! (edit, then use 'hg resolve --mark')
    warning: conflicts while merging browser/components/extensions/test/browser/browser_ext_browserAction_popup.js! (edit, then use 'hg resolve --mark')
    unresolved conflicts (see hg resolve, then hg rebase --continue)
     
    15:52:02,203 autoland ERROR Traceback (most recent call last):
      File "autoland.py", line 294, in main
        handle_pending_transplants(logger, dbconn)
      File "autoland.py", line 120, in handle_pending_transplants
        commit_descriptions)
      File "/home/ubuntu/version-control-tools/autoland/autoland/transplant.py", line 34, in transplant
        commit_descriptions=commit_descriptions)
      File "/home/ubuntu/version-control-tools/autoland/autoland/transplant.py", line 134, in _transplant
        return False, formulate_hg_error(['hg'] + cmd, output)
      File "/home/ubuntu/version-control-tools/autoland/autoland/transplant.py", line 19, in formulate_hg_error
        return 'hg error in cmd: ' + ' '.join(cmd) + ': ' + output
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 87: ordinal not in range(128) 

I'll submit patches shortly.

Comment 4

[Gregory Szorc [:gps]]

Attachment: autoland: properly log errored command (bug 1315253);

Previously, we'd log the last assignment of "cmd" which was from the
"rewritecommitdescriptions" invocation a few lines above.

Review commit: https://reviewboard.mozilla.org/r/90726/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90726/

Comment 5

[Gregory Szorc [:gps]]

Attachment: autoland: don't pass unicode values into transplant (bug 1315253);

Production autoland got into an infinite loop because
formulate_hg_error() in transplant.py was raising a UnicodeDecodeError
trying to coerce str and unicode types because one of the command
arguments was unicode. The unicode instance appeared to originate with
the database.

This commit coerces some values from the database back to str to avoid
the type matching. For good measure, we also add some asserts to
transplant() to guard against unwanted unicode types.

Review commit: https://reviewboard.mozilla.org/r/90728/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90728/

Comment 6

[Steven MacLeod [:smacleod]]

Comment on attachment 8807611 [details]
autoland: properly log errored command (bug 1315253);

https://reviewboard.mozilla.org/r/90726/#review90440

Comment 7

[Steven MacLeod [:smacleod]]

Comment on attachment 8807612 [details]
autoland: don't pass unicode values into transplant (bug 1315253);

https://reviewboard.mozilla.org/r/90728/#review90442

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f463cf3738b6
autoland: properly log errored command ; r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/374b33eb3a71
autoland: don't pass unicode values into transplant ; r=smacleod