940990 - final verification times out without any output

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

Wed Nov 20 06:04:32 PST 2013:  Command called: 'final-verification.sh' 'mozBeta-firefox-linux.cfg' 'mozBeta-firefox-linux64.cfg' 'mozBeta-firefox-mac64.cfg' 'mozBeta-firefox-win32.cfg'
Wed Nov 20 06:04:32 PST 2013:  From directory: '/builds/slave/rel-m-beta-fnl_verf-0000000000/tools/release'
Wed Nov 20 06:04:32 PST 2013:  
Wed Nov 20 06:04:32 PST 2013:  Parsing arguments...
Wed Nov 20 06:04:32 PST 2013:  Checking one or more config files have been specified...
Wed Nov 20 06:04:32 PST 2013:  Checking whether MAX_PROCS is a number...
Wed Nov 20 06:04:32 PST 2013:  Checking specified config files all exist relative to directory '/builds/slave/rel-m-beta-fnl_verf-0000000000/tools/release/updates':
Wed Nov 20 06:04:32 PST 2013:  
Wed Nov 20 06:04:32 PST 2013:    * 'mozBeta-firefox-linux.cfg' ok
Wed Nov 20 06:04:32 PST 2013:    * 'mozBeta-firefox-linux64.cfg' ok
Wed Nov 20 06:04:32 PST 2013:    * 'mozBeta-firefox-mac64.cfg' ok
Wed Nov 20 06:04:32 PST 2013:    * 'mozBeta-firefox-win32.cfg' ok
Wed Nov 20 06:04:32 PST 2013:  
Wed Nov 20 06:04:32 PST 2013:  All checks completed successfully.
Wed Nov 20 06:04:32 PST 2013:  
Wed Nov 20 06:04:32 PST 2013:  Starting stopwatch...
Wed Nov 20 06:04:32 PST 2013:  

command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1200.077583

Comment 1

[[e-mail] (:bhearsum)]

I assume this means it's taking a long time to build its report. I wonder if there's some way we can stream some of the report...any idea Pete?

It would be helpful if Buildbot would send SIGTERM first too, that way we could print what we have before dying =\.

Comment 2

[Pete Moore [:pmoore][:pete]]

This is likely due to https://bugzilla.mozilla.org/show_bug.cgi?id=863602#c5

Maybe time for a python rewrite :)

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Pete Moore [:pete][:pmoore] from comment #2)
> This is likely due to https://bugzilla.mozilla.org/show_bug.cgi?id=863602#c5
> 
> Maybe time for a python rewrite :)

Would be nice :). In the meantime we need to do something here. The simplest thing is probably just to bump up the timeout for the step.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: double the timeout

Comment 5

[[e-mail] (:bhearsum)]

In production.

Comment 6

[Justin Wood (:Callek)]

something[s] here made it to production