twistd logs not being sent because "Message size exceeds fixed limit"

RESOLVED FIXED

Status

Release Engineering
Buildduty
P2
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: bhearsum, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
Traceback (most recent call last):
  File "/builds/buildbot/build1/tools/buildfarm/maintenance/watch_twistd_log.py", line 247, in <module>
    hostname, exceptions, options.name)
  File "/builds/buildbot/build1/tools/buildfarm/maintenance/watch_twistd_log.py", line 117, in send_msg
    s.sendmail(fromaddr, [addr], m.as_string())
  File "/tools/python27/lib/python2.7/smtplib.py", line 722, in sendmail
    raise SMTPSenderRefused(code, resp, from_addr)
smtplib.SMTPSenderRefused: (552, '5.3.4 Message size exceeds fixed limit', 'cltbld@buildbot-master84.srv.releng.scl3.mozilla.com')


There's probably two issues to deal with here:
1) Why are the messages so big?
2) Split the mail if the messages are too big.
(Assignee)

Comment 1

3 years ago
The db reboot yesterday gave us a metric crapton of tracebacks in the logs, bug 1076623 to handle that a little more gracefully. To fix this up we need to go around and touch the marker file, which tells the script which timestamp to start at.
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
(Assignee)

Comment 2

3 years ago
Actually, it looks like the files (eg bm11:~cltbld/.bm111-tests1-windows-last-time.txt) have an epoch timestamp instead.
(Assignee)

Updated

3 years ago
Assignee: nobody → nthomas
Priority: -- → P2
(Assignee)

Comment 3

3 years ago
Shoved 1412298000 (Thu Oct 2 18:00:00 2014 Pacific) into the stale files just prior to the 7pm cron job.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Comment 4

3 years ago
And again today for bm81, 82, 84, 85, 86, 105. A gap as big as 7am to 4pm Pacific.
You need to log in before you can comment on or make changes to this bug.