Failures in some Amazon EC2 B2G builds ("Connection to the other side was lost in a non-clean fashion.")



7 years ago
10 months ago


(Reporter: emorley, Assigned: catlee)


Firefox Tracking Flags

(Not tracked)


(Whiteboard: [briarpatch][ec2])



7 years ago
Might just be a transient network issue, but figured since the EC2 builds are still pretty new, wouldn't hurt to file in case we can improve logging output or reliability somehow.

B2G gb_armv7a_gecko-debug mozilla-inbound build on 2012-07-17 10:04:38 PDT for push b2aeb8be3ded
slave: bld-linux64-ec2-005

B2G gb_armv7a_gecko mozilla-inbound build on 2012-07-17 10:04:38 PDT for push b2aeb8be3ded
slave: bld-linux64-ec2-004

B2G gb_armv7a_gecko mozilla-inbound build on 2012-07-17 10:21:38 PDT for push 2b31d5caf239
slave: bld-linux64-ec2-002

Not sure who has been dealing with the AWS work, other than what I read in , so don't really know who to CC.
ed: thanks for this. Yes, all very new, so yes, we're very interested in shaking out any surprises.

catlee: any ideas?
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee

Comment 2

7 years ago
those look like "regular" connection timeouts. the builds were running for a very long time before finally losing their connections.
this is from the monitoring code - it makes an entry that the slave was idle and then a bit later goes to do a shutdown.

twice now the slave has started a job in the minutes between the marking of idle and the shutdown request.

I have disabled this code for the short term until I can test and fix it.


7 years ago
Assignee: nobody → bear
Whiteboard: [briarpatch][ec2]


7 years ago
Assignee: bear → catlee

Comment 4

7 years ago
I can't promise this will never happen again, but the immediate cause of this particular case is fixed. We now verify that buildbot has shut down before turning off the VM.
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: → Release Engineering
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.