Closed Bug 1077137 Opened 10 years ago Closed 10 years ago

twistd logs not being sent because "Message size exceeds fixed limit"

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: nthomas)

Details

Traceback (most recent call last):
  File "/builds/buildbot/build1/tools/buildfarm/maintenance/watch_twistd_log.py", line 247, in <module>
    hostname, exceptions, options.name)
  File "/builds/buildbot/build1/tools/buildfarm/maintenance/watch_twistd_log.py", line 117, in send_msg
    s.sendmail(fromaddr, [addr], m.as_string())
  File "/tools/python27/lib/python2.7/smtplib.py", line 722, in sendmail
    raise SMTPSenderRefused(code, resp, from_addr)
smtplib.SMTPSenderRefused: (552, '5.3.4 Message size exceeds fixed limit', 'cltbld@buildbot-master84.srv.releng.scl3.mozilla.com')


There's probably two issues to deal with here:
1) Why are the messages so big?
2) Split the mail if the messages are too big.
The db reboot yesterday gave us a metric crapton of tracebacks in the logs, bug 1076623 to handle that a little more gracefully. To fix this up we need to go around and touch the marker file, which tells the script which timestamp to start at.
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
Actually, it looks like the files (eg bm11:~cltbld/.bm111-tests1-windows-last-time.txt) have an epoch timestamp instead.
Assignee: nobody → nthomas
Priority: -- → P2
Shoved 1412298000 (Thu Oct 2 18:00:00 2014 Pacific) into the stale files just prior to the 7pm cron job.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
And again today for bm81, 82, 84, 85, 86, 105. A gap as big as 7am to 4pm Pacific.
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.