security bug reports need to be sent manually since 2019-12-22 went missing
Categories
(bugzilla.mozilla.org :: Administration, defect, P1)
Tracking
()
People
(Reporter: aryx, Assigned: dkl)
References
Details
Every Sunday at 12:00(pm) UTC, Bugzilla sends an email with the subject 'Security Bugs Report for YYYY-MM-DD' to the security group. Starting last week, there is also a 'Security Bugs Report (moderate & low) for YYYY-MM-DD'.
Yesterday, none of these emails arrived here.
Related to the work in bug 1597800?
|
||
Updated•6 years ago
|
|
|
||
Comment 1•6 years ago
|
||
The work in bug 1597800 is stable and I just tested it by sending the email to myself for the same time frame.
I manually triggered it for 2019-12-22 12PM UTC. Aryx, can you confirm that it was received now?
I will follow up with ops when they get back from break to see why the cron failed.
|
Reporter | |
Comment 2•6 years ago
|
||
Yes, thank you. I received them on Monday. Will comment here if the next cron run fails again on Sunday.
|
|
Reporter | |
Comment 3•6 years ago
|
||
Israel, can you kick off both reports once more, please?
|
|
||
Comment 4•6 years ago
|
||
Manually triggered those reports for 12/29, should be in everyone's inbox now. Our ops guys are off till next Monday and I will coordinate with them to see what we need to do. Keeping the needinfo on myself as a reminder and also I'll probably have to kick off next Sunday's report manually it looks like.
|
|
||
Comment 5•6 years ago
•
|
||
Report for this week should be in everyone's inbox. Talking with ops to see what happened with the crons.
|
|
||
Comment 6•6 years ago
|
||
Ops says they've fixed the problem and it should run fine next week. Please re-open this bug or file a new one if the report isn't in your inbox again.
|
|
Reporter | |
Comment 7•6 years ago
|
||
No reports received today, please investigate.
|
|
||
Comment 8•6 years ago
|
||
Report has been manually sent out and I'm chatting with ops again to see if we figure out why the cron fails. Sorry!
|
|
||
Comment 9•6 years ago
|
||
ni ckolos since this will probably come up again this week and I wont be able to assist anymore. ckolos mentioned it was a quoting issue in the wrapper script that ops has deployed with puppet.
If it fails, aryx you can ni dkl who can run perl scripts/secbugsreport.pl 2020 01 19 12 00 00 +0000 on the admin node this coming sunday.
|
|
Reporter | |
Comment 10•6 years ago
|
||
David, can you run the script mentioned by Israel when you start to work on Tuesday morning? Thank you in advance.
(In reply to Israel Madueme [:imadueme] from comment #9)
If it fails, aryx you can ni dkl who can run
perl scripts/secbugsreport.pl 2020 01 19 12 00 00 +0000on the admin node this coming sunday.
@Israel: Good luck for the future.
|
Assignee | |
Comment 11•6 years ago
|
||
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #10)
David, can you run the script mentioned by Israel when you start to work on Tuesday morning? Thank you in advance.
(In reply to Israel Madueme [:imadueme] from comment #9)
If it fails, aryx you can ni dkl who can run
perl scripts/secbugsreport.pl 2020 01 19 12 00 00 +0000on the admin node this coming sunday.
I have run the script just now without error. Please let me know if you received it correctly and if I can help with anything else.
|
|
Reporter | |
Comment 12•6 years ago
|
||
Thanks, the reports have arrived. Has the quoting issue mentioned in comment 9 be fixed (answer is not obvious to me from comment 11)?
|
||
Comment 13•6 years ago
|
||
Thank you, that sends the report for this week (I got them, thanks!). It doesn't fix the underlying problem of the automation being broken which I think is what this bug is actually about. reopening
|
|
||
Updated•6 years ago
|
|
|
Assignee | |
Comment 14•6 years ago
|
||
(In reply to Daniel Veditz [:dveditz] from comment #13)
Thank you, that sends the report for this week (I got them, thanks!). It doesn't fix the underlying problem of the automation being broken which I think is what this bug is actually about. reopening
Ah Ok. So you did not receive the automated one but you received the one I did manually. Will follow up with cloudops and see what is up.
|
|
Assignee | |
Comment 15•6 years ago
|
||
(In reply to David Lawrence [:dkl] from comment #14)
(In reply to Daniel Veditz [:dveditz] from comment #13)
Thank you, that sends the report for this week (I got them, thanks!). It doesn't fix the underlying problem of the automation being broken which I think is what this bug is actually about. reopening
Ah Ok. So you did not receive the automated one but you received the one I did manually. Will follow up with cloudops and see what is up.
Talk with ckolos and we are going to deploy another fix tomorrow which should help with the next run. Sorry for the delay.
|
|
Reporter | |
Comment 16•6 years ago
|
||
No reports were received yesterday. Did the deploy get postponed?
It didn't get post-poned, but the most recent deploy seems to have changed something in the container that has created a new, fun, and exciting error when running the script. I will catch up with :dkl and get this sorted.
:dkl has been advised and knows the fix. We should have a rollout in the next day.
correction, this is a problem in the container execution for the cron task. Ops issue. Mea culpa.
|
|
Reporter | |
Comment 20•6 years ago
|
||
Please run the reports for 2020-02-02 and the 'moderate & low' also for 2020-01-26. The high and critical one got sent twice (one on Monday, one on Wednesday - testing?).
I've manually run both reports. My apologies for these continued hassles with this process. I promise that it's not something I'm ignoring.
|
|
Reporter | |
Comment 22•6 years ago
|
||
Thanks. Can you check that the script finished successfully? No new mails have arrived here.
They didn't emit any error messages, but I've re-run them for good measure
[root@ip-172-31-23-91 scripts]# docker exec -it push /bin/bash
app@ip-172-31-23-91:/app$ ./scripts/secbugsreport.pl "2020 01 26 12 00 00 +0000"
app@ip-172-31-23-91:/app$ ./scripts/secbugsreport.pl "2020 02 02 12 00 00 +0000"
app@ip-172-31-23-91:/app$
:dkl - is is possible that there some environmental settings that are being sourced that would prevent the script from running properly when invoked via a docker exec ?
|
|
||
Comment 25•6 years ago
|
||
Still no mail. I don't know if it's a clue, but when dkl ran it manually in Berlin (twice) only the critical+high report was sent (see comment 20).
|
|
Reporter | |
Comment 26•6 years ago
|
||
There was a report for the week ending 2020-01-26 today and one for 2020-02-20 which should have been for 2020-02-02. Can you generate it for the later, please?
done. Please check and NI me
|
|
Reporter | |
Comment 28•6 years ago
|
||
Thanks, the high+critical report has arrived and looks like expected.
Woo! Thank you :aryx!
|
|
||
Comment 30•6 years ago
|
||
Still only the critical+high report being sent. The moderate+low report has not been sent for any of the dates mentioned in comment 26
I am running the report exactly as specified in comment 9 with no errors in the output. I don't know why only the critical+high report would be sent and the moderate+low would be skipped.
|
|
Reporter | |
Comment 32•6 years ago
|
||
The "Security Bugs Report (moderate & low) for 2020-02-05" sent yesterday shows up is an empty email with a "noname" html attachment that can't be displayed.
|
|
Assignee | |
Updated•6 years ago
|
|
|
||
Comment 33•6 years ago
|
||
I received a 2020-02-09 report for crit+high bugs, and given the time it was sent (4:00am Sunday, exactly) I'm guessing the automation must be working again. Still no "moderate & low" report though.
|
|
Assignee | |
Comment 34•6 years ago
|
||
All of these issues should be resolved as part of todays production deployment. Please open new bug if there are any issues going forward.
Description
•