hgpoller fails when adding commits with long comments

RESOLVED FIXED

Status

Release Engineering
General Automation
--
blocker
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: catlee, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Assignee)

Description

a year ago
Possibly related to the recent TCW.

We're seeing problems where no new jobs are being scheduled on mozilla-inbound. The logs on buildbot-master81 indicate that this is because the 'comments' data is being truncated on insertion.
(Assignee)

Updated

a year ago
Severity: normal → blocker
Log snippet from bm81:

2016-10-10 07:28:15-0700 [HTTPPageGetter,client] adding change, who r@hev.cc, 165 files, rev=25bc09ae4be007dfe33f133c6a4d7ff1d0e51a42, branch=integration/mozilla-inbound, repository=, comments Bug 1303690 - Baldr: MIPS: Take alignment hints into account when compiling load/store. r=bbouvier

        ---
         .../jit/mips-shared/CodeGenerator-mips-shared.cpp  |  72 +++++++++++++-
         js/src/jit/mips-shared/CodeGenerator-mips-shared.h |   7 ++
         js/src/jit/mips-shared/LIR-mips-shared.h           | 104 +++++++++++++++++++++
         js/src/jit/mips-shared/Lowering-mips-shared.cpp    |  46 ++++++++-
         .../jit/mips-shared/MacroAssembler-mips-shared.cpp |  89 ++++++++++++++++++
         .../jit/mips-shared/MacroAssembler-mips-shared.h   |   6 ++
         js/src/jit/mips32/CodeGenerator-mips32.cpp         |  68 +++++++++++++-
         js/src/jit/mips32/CodeGenerator-mips32.h           |   7 ++
         js/src/jit/mips32/LOpcodes-mips32.h                |   4 +
         js/src/jit/mips32/MacroAssembler-mips32.cpp        |  87 +++++++++++++++++
         js/src/jit/mips32/MacroAssembler-mips32.h          |   6 ++
         js/src/jit/mips64/CodeGenerator-mips64.cpp         |  47 +++++++++-
         js/src/jit/mips64/CodeGenerator-mips64.h           |   7 ++
         js/src/jit/mips64/LOpcodes-mips64.h                |   4 +
         js/src/jit/mips64/MacroAssembler-mips64.cpp        |  74 +++++++++++++++
         js/src/jit/mips64/MacroAssembler-mips64.h          |   6 ++
         16 files changed, 625 insertions(+), 9 deletions(-), category None, project
2016-10-10 07:28:15-0700 [HTTPPageGetter,client] <HgPoller for https://hg.mozilla.org/integration/mozilla-inbound>: polling failed, result
2016-10-10 07:28:15-0700 [HTTPPageGetter,client] Traceback (most recent call last):
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.result = callback(self.result, *args, **kw)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbotcustom/changes/hgpoller.py", line 154, in succeeded
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.d.callback(result)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/twisted/internet/defer.py", line 318, in callback
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self._startRunCallbacks(result)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self._runCallbacks()
2016-10-10 07:28:15-0700 [HTTPPageGetter,client] --- <exception caught here> ---
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.result = callback(self.result, *args, **kw)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbotcustom/changes/hgpoller.py", line 438, in processData
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.parent.addChange(c)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbot-0.8.2_hg_8b87b4974e3c_production_0.8-py2.7.egg/buildbot/changes/manager.py", line 114, in addChange
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.parent.addChange(change)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbot-0.8.2_hg_8b87b4974e3c_production_0.8-py2.7.egg/buildbot/master.py", line 1178, in addChange
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.db.addChangeToDatabase(change)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbot-0.8.2_hg_8b87b4974e3c_production_0.8-py2.7.egg/buildbot/db/connector.py", line 308, in addChangeToDatabase
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.runInteractionNow(self._txn_addChangeToDatabase, change)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbot-0.8.2_hg_8b87b4974e3c_production_0.8-py2.7.egg/buildbot/db/connector.py", line 212, in runInteractionNow
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     return self._runInteractionNow(interaction, *args, **kwargs)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbot-0.8.2_hg_8b87b4974e3c_production_0.8-py2.7.egg/buildbot/db/connector.py", line 237, in _runInteractionNow
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     result = interaction(c, *args, **kwargs)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/buildbot-0.8.2_hg_8b87b4974e3c_production_0.8-py2.7.egg/buildbot/db/connector.py", line 326, in _txn_addChangeToDatabase
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     t.execute(q, values)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     self.errorhandler(self, exc, value)
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]   File "/builds/buildbot/build_scheduler/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2016-10-10 07:28:15-0700 [HTTPPageGetter,client]     raise errorclass, errorvalue
2016-10-10 07:28:15-0700 [HTTPPageGetter,client] _mysql_exceptions.DataError: (1406, "Data too long for column 'comments' at row 1")
Comment hidden (mozreview-request)
(Assignee)

Updated

a year ago
Assignee: nobody → catlee
maybe at some time a hg hook or so would be nice to prevent such problems and so tree closures

Comment 4

a year ago
mozreview-review
Comment on attachment 8799417 [details]
Bug 1308932 - Truncate commit message comments before adding to DB

https://reviewboard.mozilla.org/r/84606/#review83148
Attachment #8799417 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Comment 5

a year ago
https://hg.mozilla.org/build/buildbotcustom/rev/e80e5cbe992faea36d3f16f4fca8bab2a3ab2066
Bug 1308932 - Truncate commit message comments before adding to DB r=Callek
builds are now pending, reopen trees
For future reference, 6 hours' worth of missing builds/tests should probably lead to us holding off on reopening until results come in, rather than reopening at the first sign of pending builds.
Flags: needinfo?(cbook)
(Assignee)

Comment 8

a year ago
We deployed this patch, and then used the manhole to reset the hgpoller's idea of lastPushID to the last push we did builds on.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Flags: needinfo?(cbook)
You need to log in before you can comment on or make changes to this bug.