1195463 - Windows sendchange failure: exceptions.KeyError: partialMarUrl/packageUrl

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[aleth [:aleth]]

Busts comm-central Linux and Windows builds, e.g.
https://treeherder.mozilla.org/logviewer.html#?job_id=22748&repo=comm-central
https://treeherder.mozilla.org/logviewer.html#?job_id=22751&repo=comm-central

 ========= Started sendchange (results: 2, elapsed: 0 secs) (at 2015-08-15 15:48:31.016943) =========
[Failure instance: Traceback: <type 'exceptions.KeyError'>: 'packageUrl'
/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py:260:addCallback
/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py:249:addCallbacks
/builds/buildbot/build1/lib/python2.7/site-packages/twisted/internet/defer.py:441:_runCallbacks
/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/buildstep.py:769:_startStep_2
--- <exception caught here> ---
/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/steps/misc.py:109:start
/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/properties.py:108:render
/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/properties.py:106:render
/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/properties.py:194:render
/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/properties.py:169:__getitem__
/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/properties.py:50:__getitem__
]
========= Finished sendchange (results: 2, elapsed: 0 secs) (at 2015-08-15 15:48:31.017954) =========

Comment 1

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22750
repository: comm-central
start_time: 2015-08-15T14:51:49
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-383
buildname: TB Linux comm-central build
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'

Comment 2

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22757
repository: comm-central
start_time: 2015-08-16T03:17:46
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-309
buildname: TB Linux comm-central nightly
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
exceptions.KeyError: 'partialMarUrl'

Comment 3

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22766
repository: comm-central
start_time: 2015-08-17T03:03:05
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-1052
buildname: TB Linux comm-central nightly
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
exceptions.KeyError: 'partialMarUrl'

Comment 4

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22748
repository: comm-central
start_time: 2015-08-15T14:51:49
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-1061
buildname: TB Linux x86-64 comm-central build
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'

Comment 5

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22759
repository: comm-central
start_time: 2015-08-16T03:17:47
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-398
buildname: TB Linux x86-64 comm-central nightly
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
exceptions.KeyError: 'partialMarUrl'

Comment 6

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22756
repository: comm-central
start_time: 2015-08-15T14:51:49
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-382
buildname: TB Linux comm-central leak test build
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
PROCESS-CRASH | test_assert.js | application crashed [unknown top frame]

Comment 7

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22764
repository: comm-central
start_time: 2015-08-17T03:03:07
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-269
buildname: TB Linux x86-64 comm-central nightly
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
exceptions.KeyError: 'partialMarUrl'

Comment 8

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22749
repository: comm-central
start_time: 2015-08-15T14:51:49
who: aleth[at]instantbird[dot]org
machine: bld-linux64-spot-534
buildname: TB Linux x86-64 comm-central leak test build
revision: ab6c50b76ed3

IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/bundleclone.py'
PROCESS-CRASH | test_assert.js | application crashed [unknown top frame]

Comment 9

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22761
repository: comm-central
start_time: 2015-08-16T03:02:42
who: aleth[at]instantbird[dot]org
machine: b-2008-ix-0010
buildname: TB WINNT 5.2 comm-central nightly
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255
exceptions.KeyError: 'partialMarUrl'

Comment 10

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22762
repository: comm-central
start_time: 2015-08-17T03:03:46
who: aleth[at]instantbird[dot]org
machine: b-2008-ix-0091
buildname: TB WINNT 5.2 comm-central nightly
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255
exceptions.KeyError: 'partialMarUrl'

Comment 11

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22752
repository: comm-central
start_time: 2015-08-15T14:51:51
who: aleth[at]instantbird[dot]org
machine: b-2008-ix-0153
buildname: TB WINNT 5.2 comm-central build
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255

Comment 12

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22751
repository: comm-central
start_time: 2015-08-15T14:51:51
who: aleth[at]instantbird[dot]org
machine: b-2008-ix-0127
buildname: TB WINNT 5.2 comm-central leak test build
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255

Comment 13

[aleth [:aleth]]

Appears to have resolved itself on Linux for some reason.

Windows still fails with exceptions.KeyError: 'partialMarUrl'

Comment 14

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22767
repository: comm-central
start_time: 2015-08-18T03:02:09
who: aleth[at]instantbird[dot]org
machine: b-2008-ix-0118
buildname: TB WINNT 5.2 comm-central nightly
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255
exceptions.KeyError: 'partialMarUrl'

Comment 15

[aleth [:aleth]]

Windows now fails with [Failure instance: Traceback: <type 'exceptions.KeyError'>: 'packageUrl']

No obvious reason for the change.

Comment 16

[Richard Marti (:Paenglab)]

Nick, please can you say what could be wrong here? The mar files and also the binaries are on the ftp server.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

mshal, you might know more here. It appears that packageUrl is not being parsed out of make upload output, eg at http://buildbot-master85.bb.releng.scl3.mozilla.com:8001/builders/TB%20WINNT%205.2%20comm-central%20build/builds/32. This is using the old-school method, no mach automation or TC here.

aleth, Paenglab - it would be super helpful to know when this first started.

Comment 18

[aleth [:aleth]]

(In reply to Nick Thomas [:nthomas] from comment #17)
> aleth, Paenglab - it would be super helpful to know when this first started.

August 15 - which means the earliest possible time for when it actually broke would be sometime August 14.

Comment 19

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22893
repository: comm-central
start_time: 2015-08-26T03:02:19
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0113
buildname: TB WINNT 5.2 comm-central nightly
revision: e6c5bce2b8fb

ValueError: I/O operation on closed file
exceptions.KeyError: 'partialMarUrl'

Comment 20

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22779
repository: comm-central
start_time: 2015-08-19T03:02:15
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0143
buildname: TB WINNT 5.2 comm-central nightly
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255
exceptions.KeyError: 'partialMarUrl'

Comment 21

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22876
repository: comm-central
start_time: 2015-08-25T03:03:47
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0129
buildname: TB WINNT 5.2 comm-central nightly
revision: e6c5bce2b8fb

CalledProcessError: Command '['hg', 'log', '-r', u'e6c5bce2b8fb6859970e1197ca0027a251693dc1', '--template', '{node}']' returned non-zero exit status 255
exceptions.KeyError: 'partialMarUrl'

Comment 22

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22915
repository: comm-central
start_time: 2015-08-27T03:02:08
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0075
buildname: TB WINNT 5.2 comm-central nightly
revision: cc0929afb85b

exceptions.KeyError: 'partialMarUrl'

Comment 23

[Michael Shal [:mshal]]

I think the problem is that previously, the output from post_upload.py passed straight through to the terminal, whereas now it is captured using check_output(). Somewhere along the way, the line endings are becoming '\r\n' instead of '\n', which the regex doesn't recognize. Eg: stdout in parse_make_upload() looks like:

'mozmake.exe[1]: Entering directory \'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb/mail/installer\'\r\nmkdir -p `dirname \'../../dist//thunderbird-43.0a1.en-US.win32.checksums\'`\r\n...

Just replacing \r\n with \n makes the regex work again.

Comment 24

[Michael Shal [:mshal]]

Attachment: make-upload-newlines

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8653633 [details] [diff] [review]
make-upload-newlines

Could also use
  "\n".join([stdout, stderr]).strip()
by the looks, but I know you checked the windows case already and swapping would require more verification.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

I take it back about strip(), that's not gonna work. Full steam ahead with the patch you have.

Comment 27

[Michael Shal [:mshal]]

Comment on attachment 8653633 [details] [diff] [review]
make-upload-newlines

https://hg.mozilla.org/build/buildbotcustom/rev/f4df0521313a

Comment 28

[Richard Marti (:Paenglab)]

Hmm. now we get: exceptions.KeyError: 'partialMarUrl'. See: http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2015/08/2015-08-30-03-04-40-comm-central/comm-central-win32-nightly-bm86-build1-build15.txt.gz

Comment 29

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22983
repository: comm-central
start_time: 2015-08-30T03:05:52
who: mozilla[at]kewis[dot]ch
machine: b-2008-ix-0154
buildname: TB WINNT 5.2 comm-central nightly
revision: 8fe4ce806f4b

exceptions.KeyError: 'partialMarUrl'

Comment 30

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22990
repository: comm-central
start_time: 2015-08-30T04:03:18
who: mozilla[at]kewis[dot]ch
machine: b-2008-ix-0004
buildname: TB WINNT 5.2 comm-central leak test build
revision: 8fe4ce806f4b

CalledProcessError: Command '['hg', 'log', '-r', u'8fe4ce806f4b680f846ff16417c0c6d6942cba39', '--template', '{node}']' returned non-zero exit status 255

Comment 31

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22992
repository: comm-central
start_time: 2015-08-30T04:03:11
who: mozilla[at]kewis[dot]ch
machine: b-2008-ix-0169
buildname: TB WINNT 5.2 comm-central build
revision: 8fe4ce806f4b

Comment 32

[Philipp Kewisch [:Fallen][📱📆]]

Michael, any ideas why this may still be an issue?

Comment 33

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23008
repository: comm-central
start_time: 2015-08-31T03:03:12
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0158
buildname: TB WINNT 5.2 comm-central nightly
revision: 8fe4ce806f4b

exceptions.KeyError: 'partialMarUrl'

Comment 34

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23020
repository: comm-central
start_time: 2015-09-01T03:03:34
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0125
buildname: TB WINNT 5.2 comm-central nightly
revision: 94a77adb9a03

exceptions.KeyError: 'partialMarUrl'

Comment 35

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23066
repository: comm-central
start_time: 2015-09-03T03:02:21
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0170
buildname: TB WINNT 5.2 comm-central nightly
revision: 6bcf01ef7591

exceptions.KeyError: 'partialMarUrl'

Comment 36

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23028
repository: comm-central
start_time: 2015-09-02T03:02:26
who: richard[dot]marti[at]gmail[dot]com
machine: b-2008-ix-0112
buildname: TB WINNT 5.2 comm-central nightly
revision: 94a77adb9a03

CalledProcessError: Command '['hg', 'log', '-r', u'94a77adb9a033affe78d7818b86d49cf776e8d85', '--template', '{node}']' returned non-zero exit status 255
exceptions.KeyError: 'partialMarUrl'

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Looks like the fix wasn't merged to production, fixed with https://hg.mozilla.org/build/buildbotcustom/rev/d4e96f6e1349, it'll deploy in 40 minutes. Rerunning the windows builds on c-c 35d8f81ac152 to test.

Comment 38

[Nick Thomas [:nthomas] (UTC+12)]

Oops, should have waited for the actual deployment before rebuilding. There third pair of windows builds on 35d8f81ac152 will have it.

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

And they were green.