509604 - better error reporting for builders using graph server post

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[alice nodelman [:alice] [:anode]]

Working with catlee's proposed solution from bug 476208:
diff --git a/steps/test.py b/steps/test.py
--- a/steps/test.py
+++ b/steps/test.py
@@ -402,22 +402,24 @@
               found = True
           if not found:
               self.stdio.addStderr("results not added, response: \n" +
contents)
               raise Exception("graph server did not add results successfully")
         except Exception, e:
           self.stdio.addStderr(str(e))
           raise

-    def postFailed(self, testlongname):
+    def postFailed(self, testlongname, res=None):
         # This function is called when getPage() fails and simply sets
         # self.error = True so postFinished knows that something failed.
         self.error = True
         self.stdio.addStderr('\nEncountered error when trying to post %s\n' %
\
           testlongname)
+        if res:
+            self.stdio.addStderr(str(res))

     def constructString(self, machine, testname, branch, sourcestamp, buildid,
date, val):
         info_format = "%s,%s,%s,%s,%s,%s\n"
         str = ""
         str += "START\n"
         str += "AVERAGE\n"
         str += info_format % (machine, testname, branch, sourcestamp, buildid,
date)
         str += "%.2f\n" % (float(val))
@@ -439,17 +441,17 @@
             testname, testlongname, testval, prettyval = res
             testval = str(testval).strip(string.letters)
             data = self.constructString(self.resultsname, testlongname,
self.branch, self.sourcestamp, self.buildid, self.timestamp, testval)
             content_type, body = post_file.encode_multipart_formdata([("key",
"value")], [("filename", "data", data)])
             headers = {'Content-Type': content_type, 'Content-Length' :
str(len(data))}
             d = getPage(self.graphurl, timeout=self.timeout, method='POST',
headers=headers, postdata=body)
             d.addCallback(self.doTinderboxPrint, testlongname, testname,
                           prettyval)
-            d.addErrback(lambda x: self.postFailed(testlongname))
+            d.addErrback(lambda res: self.postFailed(testlongname, res))
             deferreds.append(d)

         # Now, once *everything* has finished we need to tell Buildbot
         # that this step is complete.
         dl = DeferredList(deferreds)
         dl.addCallback(self.postFinished)

     def postFinished(self, results):

Comment 1

[alice nodelman [:alice] [:anode]]

Currently testing on staging build environment.

Comment 2

[alice nodelman [:alice] [:anode]]

Attachment: collect errors upon graph server post failure

Tested on stage - looks fine there.

If you have a chance, maybe this can get rolled into tomorrow's downtime?

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 393682 [details] [diff] [review]
collect errors upon graph server post failure

changeset:   381:9a96ef6c9065

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Landed in downtime this morning, lets wait for an error to see if it is handled any better.

Comment 5

[[e-mail] (:bhearsum)]

Much better error messages now:
(view as text)

results not added, response: 
(2003, "Can't connect to MySQL server on 'tm-b01-master01' (111)")
graph server did not add results successfully
Encountered error when trying to post refcnt_leaks
[Failure instance: Traceback: <type 'exceptions.Exception'>: graph server did not add results successfully
/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/web/client.py:132:handleResponse
/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/web/client.py:269:page
/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py:229:callback
/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py:294:_startRunCallbacks
--- <exception caught here> ---
/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/defer.py:307:_runCallbacks
/tools/buildbotcustom/buildbotcustom/steps/test.py:405:doTinderboxPrint
]



(This happened because of the IT downtime last night)

Comment 6

[alice nodelman [:alice] [:anode]]

Success!