852242 - Stop re-trying imaging pandas if the image is bad

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

It seems that SUT is not coming up and we're depleting the pool of pandas.

2013-03-18T11:33:22 syslog Imaging complete. Rebooting
2013-03-18T11:33:22 statemachine entering state b2g_rebooting
2013-03-18T11:33:22 syslog Submitting lifeguard event at http://10.12.128.33/api/device/panda-0100/event/b2g_rebooting/
2013-03-18T11:35:25 statemachine entering state b2g_pinging
2013-03-18T11:35:26 sut connecting to SUT agent
2013-03-18T11:35:26 statemachine entering state sut_verifying
2013-03-18T11:36:16 sut connecting to SUT agent
2013-03-18T11:37:06 sut connecting to SUT agent
2013-03-18T11:37:56 sut connecting to SUT agent
2013-03-18T11:38:46 sut connecting to SUT agent
2013-03-18T11:39:36 sut connecting to SUT agent
2013-03-18T11:40:26 sut connecting to SUT agent
2013-03-18T11:41:16 sut connecting to SUT agent
2013-03-18T11:42:06 sut connecting to SUT agent
2013-03-18T11:42:56 sut connecting to SUT agent
2013-03-18T11:43:46 sut connecting to SUT agent
2013-03-18T11:43:53 statemachine ignored event free in state sut_verifying
2013-03-18T11:44:36 sut connecting to SUT agent
2013-03-18T11:45:26 sut connecting to SUT agent
2013-03-18T11:46:16 sut connecting to SUT agent
2013-03-18T11:47:06 statemachine device has failed: failed_sut_verifying
2013-03-18T11:47:06 statemachine entering state failed_sut_verifying

Comment 1

[Armen [:armenzg]]

On the buildbot side:

https://tbpl.mozilla.org/php/getParsedLog.php?id=20786403&tree=Mozilla-Inbound&full=1#error0

11:23:39     INFO - #####
11:23:39     INFO - ##### Running request-device step.
11:23:39     INFO - #####
11:23:39     INFO - Getting output from command: ['/builds/panda-0100/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())']
11:23:39     INFO - Copy/paste: /builds/panda-0100/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
11:23:39     INFO - Reading from file tmpfile_stdout
11:23:39     INFO - Output received:
11:23:39     INFO -  /builds/panda-0100/test/build/venv/lib/python2.7/site-packages
11:23:39     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
11:23:39     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
11:23:40     INFO - Got request, url=http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/51225/
11:23:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:24:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:25:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:26:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:27:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:28:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:29:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:30:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:31:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:32:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:33:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:34:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:35:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:36:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:37:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:38:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:39:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:40:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:41:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:42:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
11:43:41    ERROR - INFRA-ERROR: Request did not become ready in time
11:43:41  WARNING - # TBPL RETRY #
11:43:41    FATAL - Retries limit exceeded
11:43:41    FATAL - Exiting -1

Comment 2

[Armen [:armenzg]]

Attachment: stop re-trying

Comment 3

[Armen [:armenzg]]

Comment on attachment 726275 [details] [diff] [review]
stop re-trying

checked-in and deployed.

Comment 4

[Armen [:armenzg]]

The jobs are turning orange instead of red or purple. I will look into it.

Comment 5

[Armen [:armenzg]]

Attachment: wip - tbpl_failure when it is a known failure

I have to test this first.

Comment 6

[Armen [:armenzg]]

Attachment: stop the re-trying

This will at least stop the retrying and slow down the bleeding.

Comment 7

[Armen [:armenzg]]

Comment on attachment 726679 [details] [diff] [review]
stop the re-trying

checked-in and merged to production.