683417 - retry.py didn't actually kill process tree for a timed-out pushsnip

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Process list

In bug 683412 we timed out a pushsnip, but on inspection of the processes list on aus2-staging there was sync to Phoenix still running. This was actually advantageous, but I bet it wasn't expected.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Hit this again today. Based on http://superuser.com/questions/20679/why-does-my-remote-process-still-run-after-killing-an-ssh-session, we should give the -t argument to ssh.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] use -t with ssh, bump timeout for pushsnip

For 10.0b6 the pushsnip of the test snippets failed on all five of retry attempts, often while pushing to PHX. This should help by not leaving running processes slowing down NFS (ever moar!), and by setting the pushsnip timeout to the same 2 hours that backupsnip gets.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 591341 [details] [diff] [review]
[buildbotcustom] use -t with ssh, bump timeout for pushsnip

http://hg.mozilla.org/build/buildbotcustom/rev/83f17929c032

Comment 4

[[e-mail] (:bhearsum)]

This landed in production today.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

May not be working, from the 3.6.26 build2 log:
  Pseudo-terminal will not be allocated because stdin is not a terminal.
which could be fallout from having 
  using PTY: False
when making the call from buildbot.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

It does look like the processes are being cleaned up properly though. Needs more investigation.

Comment 7

[Justin Wood (:Callek)]

(In reply to Nick Thomas [:nthomas] from comment #5)
> May not be working, from the 3.6.26 build2 log:
>   Pseudo-terminal will not be allocated because stdin is not a terminal.
> which could be fallout from having 
>   using PTY: False
> when making the call from buildbot.

Likely we need (or want) to add |usePTY=True| to the RetryingShellCommand step specifically.

The default of the steps is of course to use whatever the slave is configured as (and we currently configure it to "do not use a PTY") each ShellCommand-based step can be setup to override this behavior with that arg.

Comment 8

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Probably needs to be ssh -tt based on the man page:

-t      Force pseudo-tty allocation.  This can be used to execute arbitrary screen-based programs on a remote machine, which can be very useful, e.g. when implementing menu services.  Multiple -t options force tty allocation, even if ssh has no local tty.

Comment 10

[[e-mail] (:bhearsum)]

pushsnip is going away in the forseeable future, probably not going to fix this