Closed Bug 817024 Opened 13 years ago Closed 12 years ago

Tegras/Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>..."

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [purple][android_tier_1])

+++ This bug was initially created as a clone of Bug #711725 +++ eg: https://tbpl.mozilla.org/php/getParsedLog.php?id=17492506&tree=Mozilla-Inbound remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly. remoteFailed: [Failure instance: Traceback: <class 'twisted.spread.pb.DeadReferenceError'>: Calling Stale Broker
https://tbpl.mozilla.org/php/getParsedLog.php?id=17493320&tree=Mozilla-Inbound (first starting seeing these maybe 2-3 days ago)
coop, could you find an owner for this please.
Flags: needinfo?(coop)
Information from etherpad regarding Android releng crashes * ~250 failures in the last 2 months * 80% these fail on non test/install/connection steps * more heavily found on tegras * perhaps cp is killing the buildslave instance? ** Same as above buildbot-master side issue, likely.
(In reply to Kim Moir [:kmoir] from comment #386) > * perhaps cp is killing the buildslave instance? > ** Same as above buildbot-master side issue, likely. Do we need better logging here?
Flags: needinfo?(coop)
Summary: Tegras disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>..." → Tegras/Pandas disconnect with "remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>..."
This bug is showing up on the top failures and is 100% unrelated to test jobs or installing the product. I looked at the last 10 logs pasted in here and this is all in a step related to buildbot specifics. Can we get traction on this? It seems silly to have so much noisy in the results and bug tracking when this appears to not be an issue with the product, tests, or core tools.
Flags: needinfo?(bugspam.Callek)
(In reply to Joel Maher (:jmaher) from comment #704) > This bug is showing up on the top failures and is 100% unrelated to test > jobs or installing the product. I looked at the last 10 logs pasted in here > and this is all in a step related to buildbot specifics. > > Can we get traction on this? It seems silly to have so much noisy in the > results and bug tracking when this appears to not be an issue with the > product, tests, or core tools. Yea, you'll be happy to hear I have ideas on improving this. I already did the largest part of it just finished up this week, which is allowing the devices to always exit buildbot when they are done a job. The buildbot master side was deployed by catlee over the last weeks, the buildbot.tac had to be manually deployed to every device directory by hand, one at a time and is now done. And buildbot is now already marking the job to take advantage of that potential. Next step is a coord with sheriffs about removing the "hard kill buildbot when we see error.flg" code, and watching for problems around that, and/or a decrease in instances of this (and similar) bugs.
Flags: needinfo?(bugspam.Callek)
Product: mozilla.org → Release Engineering
Found in triage. A bunch of work landed over the last few weeks, improving things. The rate of failures for this problem (and for bug#711725) seem to have decreased significantly. It used to happen multiple-times-per-day almost every day; now, the last report here is from a month ago. (Hopefully, I did not just jinx it!). Lets leave open for a few more weeks to see if it is really FIXED_WITH_WOODEN_STAKE_THROUGH_HEART.
Component: Other → General Automation
Last two starrings were mis-stars, happy to call this fixed for now (we'd probably be better off with a new bug if it comes back anyway).
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.