Traceback (most recent call last): File "/builds/buildbot/build1/tools/buildfarm/maintenance/watch_twistd_log.py", line 247, in <module> hostname, exceptions, options.name) File "/builds/buildbot/build1/tools/buildfarm/maintenance/watch_twistd_log.py", line 117, in send_msg s.sendmail(fromaddr, [addr], m.as_string()) File "/tools/python27/lib/python2.7/smtplib.py", line 722, in sendmail raise SMTPSenderRefused(code, resp, from_addr) smtplib.SMTPSenderRefused: (552, '5.3.4 Message size exceeds fixed limit', 'firstname.lastname@example.org') There's probably two issues to deal with here: 1) Why are the messages so big? 2) Split the mail if the messages are too big.
The db reboot yesterday gave us a metric crapton of tracebacks in the logs, bug 1076623 to handle that a little more gracefully. To fix this up we need to go around and touch the marker file, which tells the script which timestamp to start at.
Actually, it looks like the files (eg bm11:~cltbld/.bm111-tests1-windows-last-time.txt) have an epoch timestamp instead.
Shoved 1412298000 (Thu Oct 2 18:00:00 2014 Pacific) into the stale files just prior to the 7pm cron job.
And again today for bm81, 82, 84, 85, 86, 105. A gap as big as 7am to 4pm Pacific.