1146037 - release-drivers list is missing some emails from at least 2015-03-20

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Mail, task)

Description

[Robert Kaiser]

Yesterday, some of my emails that I sent to release-drivers@m.o did not appear on the list.

First, I could work around that by slightly changing the message and re-sending, but later, when I signed off the chemspill, I could not get any variant of the email to the list. Same with the sign-off today.

I do not get any reply that something was blocked, and there is nothing in the moderation queue according to Lawrence, so my guess is that the messages get stuck in postini or so.

We had the same issue with at least one message from Florin yesterday as well.

Comment 1

[Robert Kaiser]

Setting this to major for now, it may even be critical as this list is the backbone of our communication for releases.

Comment 2

[Lawrence Mandel [:lmandel] (use needinfo)]

I tried to sign in to postini but don't seem to have the right access for the release-drivers list.

Comment 3

[Ryan Watson [:w0ts0n]]

It's a weekend and people aren't online today, going to move this down to normal. Will move it back up on Monday and find someone to look at it.

Comment 4

[Ed Lim [:limed]]

Will need to have a message ID for the missing mail for us to trace the mail

Comment 5

[Ed Lim [:limed]]

(In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #2)
> I tried to sign in to postini but don't seem to have the right access for
> the release-drivers list.

We don't use postini anymore for mozilla.org its google message security. CC'ing jen for more info on this

Comment 6

[Lawrence Mandel [:lmandel] (use needinfo)]

I didn't know we'd made a change for mailman lists. In case it's helpful, here's the list's page:
https://mail.mozilla.org/listinfo/release-drivers

Comment 7

[Jennifer Hayashi [:jen]]

Hi Lawrence -

We moved off Postini to Google for junk mail filtering in late January.   The Google Message Security (junk mail filtering) is just a mailbox now - but with SSO we've had some problems accessing it without an account.  Let me see if I can delegate that mailbox to your account.

Give it about 30 minutes then hit refresh twice in your google webmail interface.  If you go to your name in the top right hand corner you should see release-drivers as a delegated mailbox that you can switch to.

Comment 8

[Robert Kaiser]

(In reply to Ed Lim [:limed] from comment #4)
> Will need to have a message ID for the missing mail for us to trace the mail

I can give you the original IDs of a few messages that the list received them with.


Florin's message that didn't make it was (according to the copy I received directly):

From: "Florin Mezei" <[e-mail]>
To: "'Robert Kaiser'" <[e-mail]>
Cc: <[e-mail]>,
	<[e-mail]>,
	<[e-mail]>
Subject: [desktop] Firefox 37.0 Beta 7 (build1) Testing
Date: Fri, 20 Mar 2015 18:47:40 +0200
Message-ID: <029a01d0632d$94d50150$be7f03f0$@softvisioninc.eu>


Here are some of mine that were swallowed (according to my Sent folder):

From: Robert Kaiser <[e-mail]>
Subject: [Desktop] QE signing off on 36.0.3 release
To: release-drivers <[e-mail]>
message-id: <[e-mail]>
Date: Sat, 21 Mar 2015 00:03:26 +0200

From: Robert Kaiser <[e-mail]>
Subject: [Desktop] 36.0.3 on release channel signed off by QE
To: Mozilla release drivers <[e-mail]>
message-id: <[e-mail]>
Date: Sat, 21 Mar 2015 00:31:49 +0200

From: Robert Kaiser <[e-mail]>
Subject: [Desktop] 36.0.4 on release channel signed off by QE
To: Mozilla release drivers <[e-mail]>
message-id: <[e-mail]>
Date: Sat, 21 Mar 2015 21:19:23 +0100

Comment 9

[Jennifer Hayashi [:jen]]

Thanks for the message ids.  That helps.  It looks like these messages were marked as spam that's why they weren't delivered.  But there's not enough information to understand why it was marked as spam.  I've opened a ticket with Google to get more information and see if we can adjust or tweak the behavior.

Comment 10

[Jennifer Hayashi [:jen]]

Robert - can I get the full headers of your messages?

Comment 11

[Robert Kaiser]

Attachment: KaiRo message "Firefox 36.0.3 signed off on the release-localtest channel by QE"

I'm attaching the full messages with headers for those 3 samples (note I had more than those that were swallowed as spam) I mentioned in comment #8, this is the first one.

Comment 12

[Robert Kaiser]

Attachment: KaiRo message "36.0.3 on release channel signed off by QE"

Comment 13

[Robert Kaiser]

Attachment: KaiRo message "[Desktop] QE signing off on 36.0.3 release"

Comment 14

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Jennifer Hayashi [:nej] from comment #7)
> Give it about 30 minutes then hit refresh twice in your google webmail
> interface.  If you go to your name in the top right hand corner you should
> see release-drivers as a delegated mailbox that you can switch to.

I can see the delegated mailbox but get the following message when I try to access it:

"Login to Gmail has been disabled because your messages are being redirected to a non-Gmail mailbox."

Comment 15

[Jennifer Hayashi [:jen]]

Yah I didn't realize that.  working with support to see if we can redirect mozilla.org mail to quarantine you can get to and edit settings.

Comment 16

[Lawrence Mandel [:lmandel] (use needinfo)]

Any update on this? While I haven't received any further reports of messages being dropped, it is disconcerting not to know if we are missing release related messages from this list.

Comment 17

[Jennifer Hayashi [:jen]]

sorry about that.  I think we finally have a fix.    Any mail quarantined for mozilla.org recipients will now go to an admin quarantine so we can release it.  The old setup was going to individual quarantine's that no one could access.

Comment 18

[Lawrence Mandel [:lmandel] (use needinfo)]

Is the admin quarantine something that that list admins should have access to or is that something that someone else will help with? In the former case, how do we access it? In the latter case, who do we speak with when we have a problem?

Comment 19

[Jennifer Hayashi [:jen]]

The admin quarantine is something only the google admins would have access to.  If you were missing an email you would submit a servicenow request and we could investigate.  Really - all gmail questions should be tracked there since End-User services is the primary contact and you'll get faster support.

I realize that can be a pain, so I think what we need to do is have the list owners start filtering emails on the mailing lists so I can turn off the "Aggressive Spam Filtering" setting on mozilla.org.     We've been trying to balance people who complain they get too much junk mail - to others like you who get legitimate email trapped.

If we turn off the aggressive spam filtering, Google should still do an initial check before passing it on. Then at the mailing list level - the owners would add a setting like this:
https://mana.mozilla.org/wiki/display/SD/Adding+spam+filtering+to+a+mailing+list+on+LISTS.mozilla.org
which should at least minimize what junk mail makes it through the Google filters.

I'm still trying to see what other options we have to tweak.   I'll send a message to all mailman list owners before we make any changes I suggested above.

Comment 20

[Lawrence Mandel [:lmandel] (use needinfo)]

We've hit this issue again with mail from Ioana Chiorian. The e-mail that was blocked in this case was QE sign-off for the mobile 37.0.2 release. (Kind of important.)

Have you determined why some legitimate e-mails are being flagged as spam?

Is there any way that I can get access to the list of e-mails flagged as spam so that I can approve the ones that need to go through (and check for any additional e-mails that may have been caught)?

Comment 21

[Liz Henry (:lizzard)]

Looks like we are hitting this issue again today for signoff for the desktop release from KaiRo.

Comment 22

[Liz Henry (:lizzard)]

I filed a ServiceNow IT request for this and referenced this bug. The ServiceNow issue is INC0015635.

Comment 23

[Ed Lim [:limed]]

I released what I think are legitimate emails to [e-mail]. At this time we don't have a real answer on how we can get administrators of the list to have access to the quarantine, as mentioned in comment 19 we're still trying to tweak some stuff to make it work for everyone

Comment 24

[Robert Kaiser]

(In reply to Liz Henry (:lizzard) from comment #21)
> Looks like we are hitting this issue again today for signoff for the desktop
> release from KaiRo.

Not in that specific case, I did forget to send it when I could but did so an hour ago and it got to the list.

Comment 25

[Jennifer Hayashi [:jen]]

I changed the mozilla.org to stop agressively filtering spam.  Let me know if that helps.   If we start getting a bunch of complaints of junk mail coming in I may need to revert back this change.  But I did send out an email to all mozilla.org list owners to add spam filters on the mailing lists.   So hopefully we can strike a balance where your legit email isn't blocked and the list owners don't get tons of junk mail.  
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=40055836