Closed
Bug 756259
Opened 14 years ago
Closed 14 years ago
mx1.mail.corp.phx1.m.c is modifying message subjects based on spam scores
Categories
(Infrastructure & Operations :: Infrastructure: Other, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: Atoll, Assigned: limed)
Details
Attachments
(2 files)
Please consider adjusting the internal-only email server to not mangle messages it thinks are spammy - if there's any reason for it to be running spam checks at all, given that Postini handles that for external mail anyways.
Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO
zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by
zmmbox2.mail.corp.phx1.mozilla.com with LMTP; Thu, 17 May 2012 11:21:01
-0700 (PDT)
Received: from mx1.mail.corp.phx1.mozilla.com (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id C28A5106604
for <rsoderberg@mozilla.com>; Thu, 17 May 2012 11:21:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
X-Spam-Flag: YES
X-Spam-Score: 7.446
X-Spam-Level: *******
X-Spam-Status: Yes, score=7.446 required=6.31 tests=[HTML_MESSAGE=0.001,
HTML_TAG_BALANCE_BODY=1.247, MIME_HTML_ONLY=2.199,
TO_NO_BRKTS_DIRECT=3.999] autolearn=no
Received: from confluence1.webapp.phx1.mozilla.com (unknown [63.245.216.223])
by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 3BD10F2704
for <rsoderberg@mozilla.com>; Thu, 17 May 2012 11:21:01 -0700 (PDT)
Date: Thu, 17 May 2012 11:21:01 -0700 (PDT)
From: "Aaron Hill (Confluence)" <mana@mozilla.org>
To: rsoderberg@mozilla.com
Message-ID: <888859992.1278.1337278861237.JavaMail.confluence@confluence1.webapp.phx1.mozilla.com>
subject: ***SPAM*** [mana] Netops > VLAN assignments
Comment 1•14 years ago
|
||
It's submitted mail on smtp.mozilla.org that's getting scanned (precisely because it didn't come in via postini).
Correct fix here is to whitelist mana as a sender. We copied over the amavis config files from the old mail servers, it's possible we never added phx's IP range?
Assignee: server-ops → justdave
Component: Server Operations → Server Operations: Infrastructure
QA Contact: phong → jdow
Comment 2•14 years ago
|
||
Just to confirm, the IP range whitelists in amavis on the relays were WAY out of date. The ones in postfix did get updated.
We do *not* have these files in puppet yet (they will be soon). I need to split for a few hours to take care of some kid stuff. If someone else wants to grab this before I get back, the affected files are /etc/amavisd.conf on mx1 and mx2.mail.corp.phx1. It goes in @mynetworks on line 45. The correct IP whitelist is available by running "postconf mynetworks" on either server.
Comment 3•14 years ago
|
||
"service amavisd restart" after editing.
Updated•14 years ago
|
Group: infra
Comment 4•14 years ago
|
||
OK, this should be set now
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
I've seen several emails like this since you closed the bug. Interestingly, none before that. (We get a lot of these update emails from this host while doing a release.)
Not all messages from that host are so marked - I'll attach another one from the same time that did not get marked.
See attachments for spam/not spam on nearly identical emails from an internal server
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Nearly identical email within a few minutes of the other that was (correctly) not marked as spam
Comment 8•14 years ago
|
||
The one that got marked as spam was processed by mx2, the one that wasn't was processed by mx1. I *thought* I restarted amavis after copying the config over, but maybe I didn't. I'll go kick it again.
Comment 9•14 years ago
|
||
I've just got this in the buildbot logs on dev-master01:
Failure: twisted.mail.smtp.SMTPDeliveryError: 451 4.6.0 Content scanner malfunction
>>> DATA
<<< 354 End data with <CR><LF>.<CR><LF>
>>> .
<<< 451 4.6.0 Content scanner malfunction
$ grep ^DS /etc/mail/sendmail.cf
DSmail.build.mozilla.org
Comment 10•14 years ago
|
||
You hit it in the 20 seconds between when we pushed a broken config change and it recovering with the fixed config. 4xx errors are non-fatal, just means "try again".
Your best guard against that is to set your local sendmail or postfix to use smtp.mozilla.org for a relayhost and submit to 127.0.0.1 from your scripts, then it'll queue and try again if it gets errors without breaking your script.
Comment 11•14 years ago
|
||
This hit us again yesterday, with emails from buildbot about yesterday's FF13.0b6 release being flagged as spam.
Comment 12•14 years ago
|
||
I sent one example to cshields earlier today. Here's another example which hit our TB13.0b4 release in progress tonight.
-------- Original Message --------
Subject: Fwd: ***SPAM*** [release] Thunderbird 13.0b4 build1: ready for releasetest testing
Date: Wed, 30 May 2012 00:43:23 -0400
From: Rail Aliiev <rail@mozilla.com>
To: thunderbird-drivers@mozilla.org
I'm not sure if this passed to the mailing list.
This means that QA can skip testing of updates on the betatest channel
and use the releasetest channel instead.
-------- Original Message --------
Subject: ***SPAM*** [release] Thunderbird 13.0b4 build1: ready for
releasetest testing
Date: Tue, 29 May 2012 21:32:32 -0700
From: release@mozilla.com
To: Callek@..., akeybl@..., mbanner@...,
release@mozilla.com
Mirrors are absorbed enough for releasetest testing of Thunderbird
13.0b4 build1.
-buildbot
_______________________________________________
Thunderbird-drivers mailing list
Thunderbird-drivers@mozilla.org
https://mail.mozilla.org/listinfo/thunderbird-drivers
Comment 13•14 years ago
|
||
I need the full headers for it to be useful to me at all. (in particular all the X-Spam and Received stuff)
Comment 14•14 years ago
|
||
Return-Path: release@mozilla.com
Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO
zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by
zmmbox3.mail.corp.phx1.mozilla.com with LMTP; Tue, 29 May 2012 12:13:49
-0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 8918D1068F4;
Tue, 29 May 2012 12:13:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
X-Spam-Flag: YES
X-Spam-Score: 7.198
X-Spam-Level: *******
X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2,
MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no
Received: from buildbot-master13.build.scl1.mozilla.com (unknown [63.245.222.66])
by mx2.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id 4AF42F22ED;
Tue, 29 May 2012 12:13:49 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf8"
Date: Tue, 29 May 2012 12:13:45 -0700
subject: ***SPAM*** [release] Firefox 13.0b6 build1: ready for release
From: release@mozilla.com
To: Callek@gmail.com, akeybl@mozilla.com, release@mozilla.com
Message-Id: <20120529191349.8918D1068F4@zmmta1.mail.corp.phx1.mozilla.com>
TWlycm9ycyBhcmUgYWJzb3JiZWQgZW5vdWdoIGZvciByZWxlYXNlIG9mIEZpcmVmb3ggMTMuMGI2
IGJ1aWxkMS4KCi1idWlsZGJvdAoKCg==
Comment 15•14 years ago
|
||
Return-Path: release@mozilla.com
Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO
zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by
zmmbox3.mail.corp.phx1.mozilla.com with LMTP; Tue, 29 May 2012 21:32:34
-0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 1681C106425;
Tue, 29 May 2012 21:32:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
X-Spam-Flag: YES
X-Spam-Score: 7.198
X-Spam-Level: *******
X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2,
MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no
Received: from buildbot-master34.srv.releng.scl3.mozilla.com (v-1030.fw1.releng.scl3.mozilla.net [63.245.214.82])
by mx2.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id C9804F240B;
Tue, 29 May 2012 21:32:33 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf8"
Date: Tue, 29 May 2012 21:32:32 -0700
subject: ***SPAM*** [release] Thunderbird 13.0b4 build1: ready for releasetest
testing
From: release@mozilla.com
To: Callek@gmail.com, akeybl@mozilla.com, mbanner@mozilla.com,
release@mozilla.com
Message-Id: <20120530043234.1681C106425@zmmta1.mail.corp.phx1.mozilla.com>
TWlycm9ycyBhcmUgYWJzb3JiZWQgZW5vdWdoIGZvciByZWxlYXNldGVzdCB0ZXN0aW5nIG9mIFRo
dW5kZXJiaXJkIDEzLjBiNCBidWlsZDEuCgotYnVpbGRib3QKCg==
Comment 16•14 years ago
|
||
thanks. short-term I'm going to throw these at the spam/ham trainer. Long term I'm seeing a bunch of settings to amavis that'll help, but they're potentially invasive, so I need to get a test server set up to play with first before I touch it on production.
Comment 17•14 years ago
|
||
(In reply to Dave Miller [:justdave] from comment #16)
> thanks. short-term I'm going to throw these at the spam/ham trainer. Long
> term I'm seeing a bunch of settings to amavis that'll help,
(In reply to Dave Miller [:justdave] from comment #8)
> The one that got marked as spam was processed by mx2, the one that wasn't
> was processed by mx1. I *thought* I restarted amavis after copying the
> config over, but maybe I didn't. I'll go kick it again.
We're still seeing different behavior based on whether it's received by mx1 or xm2 first. Silly question - did the mx2 restart really "take" for these settings?
Comment 18•14 years ago
|
||
(In reply to Hal Wine [:hwine] from comment #17)
> We're still seeing different behavior based on whether it's received by mx1
> or xm2 first. Silly question - did the mx2 restart really "take" for these
> settings?
Good question. The configs are auto-synced from mx1, and it's supposed to automatically restart when it picks up config changes. I just manually kicked mx2 just to make sure.
Comment 19•14 years ago
|
||
Alas, this is still occurring:
Return-Path: release@mozilla.com
Received: from zmmta2.mail.corp.phx1.mozilla.com (LHLO
zmmta2.mail.corp.phx1.mozilla.com) (10.20.77.30) by
zmmbox2.mail.corp.phx1.mozilla.com with LMTP; Thu, 31 May 2012 15:43:47
-0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
by zmmta2.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id A299F1020D8
for <release@mozilla.com>; Thu, 31 May 2012 15:43:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
X-Spam-Flag: YES
X-Spam-Score: 7.199
X-Spam-Level: *******
X-Spam-Status: Yes, score=7.199 required=6.31 tests=[BAYES_50=2,
MISSING_MID=1.199, TO_EQ_FM_DIRECT_MX=0.001, TO_NO_BRKTS_DIRECT=3.999]
autolearn=no
Received: from buildbot-master13.build.scl1.mozilla.com (unknown [63.245.222.66])
by mx2.mail.corp.phx1.mozilla.com (Postfix) with SMTP id 46B5DF249F
for <release@mozilla.com>; Thu, 31 May 2012 15:43:47 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf8"
Date: Thu, 31 May 2012 15:43:47 -0700
subject: ***SPAM*** [release] Tagging started for Firefox 13.0b7 build1
From: release@mozilla.com
To: release@mozilla.com
Message-Id: <20120531224347.A299F1020D8@zmmta2.mail.corp.phx1.mozilla.com>
VGFnZ2luZyBzdGFydGVkIGZvciBGaXJlZm94IDEzLjBiNyBidWlsZDEKCi1idWlsZGJvdAoK
-----
Also happens on mx1.
Comment 20•14 years ago
|
||
This is affecting release communication during release crunch time.
Severity: normal → major
Comment 21•14 years ago
|
||
Can we consider using a bigger hammer on this issue?
Are there any significant inbound email flows that can bypass postini? If not, can we disable this level of spam checking?
If it is needed, can we add all machines on the buildvpn to the whitelist? (I'd hope that could be done by subnet)
Comment 22•14 years ago
|
||
This is bypassing postini, it's our internal mail relay that's doing it. The "bigger hammer" is what I alluded to in comment 16 (and includes causing anti-spam being configured to not run on internal mail), but "experimenting" with said features on a production mail server could result in outright *loss* of mail rather than just labeling it. I need at least the weekend to experiment on a VM clone of the mail server to make sure everything is correct before I drop it in production. Or I can completely turn off the spam protection but then you'll be yelling at me about all the spam getting through.
Comment 23•14 years ago
|
||
This tripped us up today with the TB13.0 release. As a short term fix, can we explicitly whitelist certain senders, like "release@m.c" originating in build network, which was in each of the examples so far in this in bug?
-------- Original Message --------
Subject: ***SPAM*** [release] Thunderbird 13.0 build1: ready for
releasetest testing
Date: Tue, 05 Jun 2012 10:54:19 -0700
From: release@mozilla.com
To:
Mirrors are absorbed enough for releasetest testing of Thunderbird 13.0 build1.
-buildbot
Comment 24•14 years ago
|
||
I realize that Dave is on PTO right now but this is really an unacceptable state for us to be in. Release mail is absolutely crucial for hand-offs between RelEng, QA, and Release Management. Since this started happening we have repeatedly had multiple hours of delay during releases because of lost mail. Can we please get this fixed one way or another soon?
Severity: major → critical
Comment 25•14 years ago
|
||
I am not on PTO right now. But you'll notice there were dependencies outside of my jurisdiction on this bug which were just resolved within the last hour. This should be resolved within the next day.
Comment 26•14 years ago
|
||
(In reply to Dave Miller [:justdave] from comment #25)
> there were dependencies outside of my jurisdiction on this bug which were just
> resolved within the last hour.
er, within the hour previous to your comment actually. (I suspect said bugmail about that resolution is what reminded you to comment here)
Comment 27•14 years ago
|
||
(In reply to Dave Miller [:justdave] from comment #26)
> (In reply to Dave Miller [:justdave] from comment #25)
> > there were dependencies outside of my jurisdiction on this bug which were just
> > resolved within the last hour.
>
> er, within the hour previous to your comment actually. (I suspect said
> bugmail about that resolution is what reminded you to comment here)
Thunderbird says I didn't get bugmail about that resolution change :-( However it is nice if the issue is fixed :-)
Comment 28•14 years ago
|
||
bug 761869 is restricted, so we can't see what you were depending on
Comment 29•14 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #28)
> bug 761869 is restricted, so we can't see what you were depending on
Ah, my apologies. Was waiting on a machine to test configuration changes on without having to make them live in production and hope it wouldn't cause all mail to bounce. I have my test machine as of today.
Comment 30•14 years ago
|
||
There were several emails in the junk folder for the releases we have triggered since yesterday.
I have dropped the domain from each email (not sure if necessary but meh)
Here is one example:
-------- Original Message --------
Subject: ***SPAM*** [release] Fennec 14.0b7 build2: completed fennec_source
Date: Wed, 13 Jun 2012 13:52:19 -0700
From: release
To: Callek, akeybl, release
fennec_source complete for Fennec 14.0b7 build2
-buildbot
Comment 31•14 years ago
|
||
(In reply to Dave Miller [:justdave] from comment #25)
> I am not on PTO right now. But you'll notice there were dependencies
> outside of my jurisdiction on this bug which were just resolved within the
> last hour.
We didnt have access to the dep bug, but gtk you are now unblocked.
> This should be resolved within the next day.
Any news? We have 3 different high-urgency releases in flight, and these emails are how we coordinate across groups. When they disappear into spam, it's quite disruptive.
Comment 32•14 years ago
|
||
also, when you rollout fixes, will you need a downtime for this?
Comment 33•14 years ago
|
||
I'm running a little behind because I've been firefighting stuff all day. hopefully I'll have news yet tonight.
It will be fairly instantaneous once I roll it out, just need to make sure on the staging server that it won't break things before I toss it at production.
Comment 34•14 years ago
|
||
this just from one of today's releases (with full headers this time!):
-------- Original Message --------
Return-Path: release@mozilla.com
Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by zmmbox2.mail.corp.phx1.mozilla.com with LMTP; Thu, 14 Jun 2012 17:27:56 -0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200]) by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 0D4AD10678E; Thu, 14 Jun 2012 17:27:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
X-Spam-Flag: YES
X-Spam-Score: 7.198
X-Spam-Level: *******
X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2, MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no
Received: from buildbot-master12.build.scl1.mozilla.com (unknown [63.245.222.66]) by mx2.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id C784AF2707; Thu, 14 Jun 2012 17:27:55 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf8"
Date: Thu, 14 Jun 2012 17:27:45 -0700
subject: ***SPAM*** [release] Firefox 13.0.1 build1: completed repack_1/6 on win32
From: release@mozilla.com
To: Callek@gmail.com, akeybl@mozilla.com, release@mozilla.com
Message-Id: <20120615002756.0D4AD10678E@zmmta1.mail.corp.phx1.mozilla.com>
repack_1/6 complete for Firefox 13.0.1 build1 on win32
-buildbot
Comment 35•14 years ago
|
||
X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2, MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no
Missing Message-ID is 2 points. That'll be enough to get you under 6.31 which will stop it from getting tagged as spam. Have your script add a Message-Id header. The one you pasted was added by the MTA after it got scanned.
TO_NO_BRKTS_DIRECT is 4 points. You can avoid this one by putting names and brackets on the email addresses on the To line. This might be easier to fix than trying to come up with a unique message ID every time.
To: Justin Wood <Callek@gmail.com>, Alex Keybl <akeybl@mozilla.com>, Release Team <release@mozilla.com>
I'm still having fun issues with the bigger config change I was hoping to make getting it to not do spam scanning on internally-originating mail, but I just discovered a way to do spam score modifications by sender address, and release@mozilla.com and bootstrap@mozilla.com both get a -8 score boost in spamassassin now.
Comment 36•14 years ago
|
||
I have a script I run which emails from release-mgmt@mozilla.com to a combination of internal & external email addresses. The bulk of them end up in the Junk folder with this ** SPAM ** marking. Can internal email addresses be exempt from this spam evaluation? Is there a way to get away from being marked as spam without adding a Message-Id header?
Comment 37•14 years ago
|
||
I've added release-mgmt to the -8 score boost.
Comment 38•14 years ago
|
||
Just to validate the suggestion in comment #35 - adding the brackets does drop scores enough, in case the -8 score boost is spoofable.
Return-Path: release@mozilla.com
Received: from zmmta2.mail.corp.phx1.mozilla.com (LHLO
zmmta2.mail.corp.phx1.mozilla.com) (10.20.77.30) by
zmmbox4.mail.corp.phx1.mozilla.com with LMTP; Fri, 15 Jun 2012 12:45:03
-0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
by zmmta2.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 1E38B1020C3
for <hwine@mozilla.com>; Fri, 15 Jun 2012 12:45:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
X-Spam-Flag: NO
X-Spam-Score: -8.301
X-Spam-Level:
X-Spam-Status: No, score=-8.301 required=6.31 tests=[AM:BOOST=-8,
BAYES_00=-1.5, MISSING_MID=1.199] autolearn=no
Received: from dev-master01.build.scl1.mozilla.com (unknown [63.245.222.66])
by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id DF5FBF256D
for <hwine@mozilla.com>; Fri, 15 Jun 2012 12:45:02 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf8"
Date: Fri, 15 Jun 2012 12:45:02 -0700
Subject: [staging-release] Firefox 12.0b3 build1: All win32 builds now
available
From: release@mozilla.com
To: <hwine@mozilla.com>
Message-Id: <20120615194503.1E38B1020C3@zmmta2.mail.corp.phx1.mozilla.com>
QWxsIHdpbjMyIGJ1aWxkcyBub3cgYXZhaWxhYmxlCgpmdHA6Ly9kZXYtc3RhZ2UwMS5zcnYucmVs
ZW5nLnNjbDMubW96aWxsYS5jb20vcHViL21vemlsbGEub3JnL2ZpcmVmb3gvbmlnaHRseS8xMi4w
YjMtY2FuZGlkYXRlcy9idWlsZDEvd2luMzIKCi1idWlsZGJvdAoK
Comment 39•14 years ago
|
||
Anything left to do here?
Comment 40•14 years ago
|
||
Yes, the amavis config still needs to be redone so that internally-originating mail is treated differently as a category of mail rather than just screwing with spamassassin scores for frequent internal sender addresses.
Comment 41•14 years ago
|
||
Dave, I am sure this isn't the only thing on your plate, but I wanted to check in on this bug and get an update, if you have one.
Updated•14 years ago
|
Assignee: justdave → limed
| Assignee | ||
Comment 42•14 years ago
|
||
We are planning on using policy banks[1] in amavis so that we can bypass spam checks if the email is originating from the local network. I just tested this out on mx3.mail.corp.phx1.mozilla.com and it looks like its working. I'm just going to have :justdave take a look at it one more time and then we'll flip the bit on mx1 and mx2
[1]http://www.ijs.si/software/amavisd/amavisd-new-docs.html#pbanks
| Assignee | ||
Comment 43•14 years ago
|
||
Ok the change to amavis has been made to mx1 and mx2, the logs are showing that mail is getting delivered normally and mail from our local network is not getting tagged. I think we are good here
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → FIXED
Updated•13 years ago
|
Component: Server Operations: Infrastructure → Infrastructure: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in
before you can comment on or make changes to this bug.
Description
•