POST emails sent to addons.mozilla.org to API endpoint

RESOLVED FIXED

Status

Cloud Services
Operations: AMO
RESOLVED FIXED
a year ago
9 months ago

People

(Reporter: eviljeff, Assigned: wezhou)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

a year ago
Similar to https://bugzilla.mozilla.org/show_bug.cgi?id=902646, but for AMO this time.

No API endpoint on -dev yet but will likely be something straightforward like "/api/v3/mail/"
(Reporter)

Comment 1

a year ago
The API will be https://addons-dev.allizom.org/api/v3/activity/mail/ and should be available next week.  

Assuming incoming JSON content as per http://www.socketlabs.com/api-reference/inbound-api/ is there anything you need to set it up for -dev?

The proposed code as stands has a whitelist for IP addresses and a header token, both of which I assume would be unused/incompatible with the socketlabs solution.  I will add a setting for the secret, to be loaded from env()
Flags: needinfo?(jthomas)
We will need to configure the Socketlabs Inbound API settings and setup DNS MX records for each environment [1]. The MX records porition is going to be a little bit complicated since we will need to migrate addons-dev.allizom.org, addons.allizom.org and addons.mozilla.org records over to Route53 so that we can make use Alias A records so that we can set the MX records. I had to do this for Marketplace as well.

I am going to pass this over to Wei to handle. Wei let's sync up about this sometime this week.


[1] https://support.socketlabs.com/index.php?/Knowledgebase/Article/View/111/0/inbound-message-processing-and-the-inbound-api
Assignee: nobody → wezhou
Flags: needinfo?(jthomas)
(Assignee)

Comment 3

a year ago
Hi Andrew,

What is the email address people will reply to? Specifically, we'd like to know if it can be at a sub-domain of addons.mozilla.org, for example <name>@<subdomain>.addons.mozilla.org. We think that if a subdomain works for the purpose, then we may not need to do the migration mentioned above, which can save some time.
Status: NEW → ASSIGNED
(Reporter)

Comment 4

a year ago
@reply.addons.mozilla.org is fine.  (@reply.addons.allizom.org @reply.addons-dev.allizom.org - or really any domain you want for non-prod)

settings_base.py defines `INBOUND_EMAIL_DOMAIN = DOMAIN` currently but I can change it to make it an env variable so you can set it.
(Assignee)

Comment 5

a year ago
We have https://bugzilla.mozilla.org/show_bug.cgi?id=1306024 to track the creation the subdomains now.
(Assignee)

Comment 6

a year ago
Hi Andrew,

The doc says,

"Before the settings on this page can be saved, two conditions must be met: the domain(s) to be processed must have their MX records pointing to mx.socketlabs.com; and an Inbound API compatible web application must be running in a location accessible by the On-Demand network. The web application must return the Validation Key from the configuration page in order to pass validation."

Do you know if we have an env variable for the "Validation Key"? Basically, we need to config our app with the validation key so socketlabs can verify us.

The JSON blob socketlabs posted to us will contain a Secret Key, which we can use to verify them, so we need another env variable for that.

Making INBOUND_EMAIL_DOMAIN an env variable is a good idea too.

Also, is https://addons-dev.allizom.org/api/v3/activity/mail/ post only end point? I'm getting 404 when trying to GET it. It's possible the end point is not ready on -dev yet, but let me know if it is.
(Reporter)

Comment 7

a year ago
(In reply to :wezhou from comment #6)
> Do you know if we have an env variable for the "Validation Key"? Basically,
> we need to config our app with the validation key so socketlabs can verify
> us.

We don't yet.  I'll call it `INBOUND_EMAIL_VALIDATION_KEY`.
 
> The JSON blob socketlabs posted to us will contain a Secret Key, which we
> can use to verify them, so we need another env variable for that.

We have an `INBOUND_EMAIL_SECRET_KEY` env variable.

> Making INBOUND_EMAIL_DOMAIN an env variable is a good idea too.

sure.

> Also, is https://addons-dev.allizom.org/api/v3/activity/mail/ post only end
> point? I'm getting 404 when trying to GET it. It's possible the end point is
> not ready on -dev yet, but let me know if it is.

It's POST only.  Does it need to be GET also?

If I read the sample nodejs app correctly, it looks like we need to serve INBOUND_EMAIL_VALIDATION_KEY in the response to every request (even the requests that don't supply a correct secret key).
(Assignee)

Comment 8

a year ago
Hi Andrew,

> It's POST only.  Does it need to be GET also?

I don't think it needs to be GET. I don't know if the end point has been deployed to -dev or not, so I tried to check by sending a GET request to that end point and got 404. maybe I should have tried a POST request.

Couple questions:

1. What does the app use the value of INBOUND_EMAIL_DOMAIN for?
2. `INBOUND_EMAIL_SECRET_KEY` currently has not been made a per env variable, has it (I only find it defined in settings_base.py file)? Can you make it per env?
(Reporter)

Comment 9

a year ago
(In reply to :wezhou from comment #8)
> Hi Andrew,
> 
> > It's POST only.  Does it need to be GET also?
> 
> I don't think it needs to be GET. I don't know if the end point has been
> deployed to -dev or not, so I tried to check by sending a GET request to
> that end point and got 404. maybe I should have tried a POST request.

There was a missing `/` in urls.py so it was `/api/v3/activitymail` rather than `/api/v3/activity/mail` (I will correct in upcoming commit)

> Couple questions:
> 
> 1. What does the app use the value of INBOUND_EMAIL_DOMAIN for?

For the reply-to headers in the emails sent out.

> 2. `INBOUND_EMAIL_SECRET_KEY` currently has not been made a per env
> variable, has it (I only find it defined in settings_base.py file)? Can you
> make it per env?

Yes, will do.

Does the validation key to pass the config check have to come from the same url as the email API, or have to be POST too?  I can do that but the side effect is any other response detail gets suppressed (e.g. the response telling you the request was malformed) and you get the validation key only with every response.
(Reporter)

Comment 10

a year ago
https://github.com/mozilla/addons-server/pull/3650
(Assignee)

Comment 11

a year ago
> Does the validation key to pass the config check have to come from the same
> url as the email API, or have to be POST too?  I can do that but the side
> effect is any other response detail gets suppressed (e.g. the response
> telling you the request was malformed) and you get the validation key only
> with every response.

Yes, I think so (it needs to come from the same url). Because according to the doc [1], it says, "Once these conditions are met, the URL of the endpoint belongs in the "Endpoint URL" field, ..."

Regarding the responses after configuration, I looked at their example implementations in [2] and [3], they seem to only return 200 with the validation key. I think we probably should log the errors on our end.

BTW, below is what they POST to us when I tried to validate our end point in their web console. It matched what the example in [3] is doing.

> POST / HTTP/1.1
> Accept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml
> User-Agent: RestSharp/105.0.1.0
> Content-Type: application/x-www-form-urlencoded
> Host: addons-dev.allizom.org:10009
> Content-Length: 60
> Accept-Encoding: gzip, deflate
> Connection: Keep-Alive

> ServerId=2301&SecretKey=2bjsk2919mm14A9Gmn7W&Type=Validation



[1] https://support.socketlabs.com/index.php?/Knowledgebase/Article/View/111/0/inbound-message-processing-and-the-inbound-api
[2] https://github.com/socketlabs/email-on-demand-examples/blob/master/Node/SampleInboundEndpoint/SampleInboundEndpoint/server.js#L36-L44
[3] https://github.com/socketlabs/email-on-demand-examples/blob/master/DotNet/Inbound%20Parsing%20API/MVC3/Controllers/HomeController.cs#L43-L51
(Reporter)

Comment 12

a year ago
(In reply to :wezhou from comment #11)
> > ServerId=2301&SecretKey=2bjsk2919mm14A9Gmn7W&Type=Validation

Ah! Very useful to know - they send the SecretKey (so we can do auth) and send a specific Type parameter.  Odd that they send form encoded rather than JSON though, but I can make it handle either for that simple case.
(Assignee)

Comment 13

a year ago
> Making INBOUND_EMAIL_DOMAIN an env variable is a good idea too.

Hi Andrew,

Can you set it to DOMAIN actually? Jason and I talked about some idea that may allow us to set the MX records for DOMAIN without having to migrate it to Route53 as we thought before, and we want to try it out on -dev. If it works, we don't need the sub-domains either. Sorry for having you change it back...
(Reporter)

Comment 14

a year ago
(In reply to :wezhou from comment #13)
> > Making INBOUND_EMAIL_DOMAIN an env variable is a good idea too.
> 
> Hi Andrew,
> 
> Can you set it to DOMAIN actually? Jason and I talked about some idea that
> may allow us to set the MX records for DOMAIN without having to migrate it
> to Route53 as we thought before, and we want to try it out on -dev. If it
> works, we don't need the sub-domains either. Sorry for having you change it
> back...

that's what it defaults to, so no change needed.
(Assignee)

Comment 15

a year ago
Do we need to wait https://github.com/mozilla/addons-server/pull/3650/files to be merged before testing it on -dev? Let me know when the end point is ready on -dev so that I can configure it in Socketlabs.
(Reporter)

Comment 16

a year ago
(In reply to :wezhou from comment #15)
> Do we need to wait https://github.com/mozilla/addons-server/pull/3650/files
> to be merged before testing it on -dev? Let me know when the end point is
> ready on -dev so that I can configure it in Socketlabs.

It's just been approved and merged on -dev.
(Assignee)

Comment 17

a year ago
The -dev end point is now configured in Socketlabs. They should start to post to our end point now.

Feel free to test it and let us know if there are issues.
(Assignee)

Comment 18

a year ago
Currently a bug is causing the email being processed twice when it's posted.  tracked in https://github.com/mozilla/addons-server/issues/3725

A workaround is in place at this time: https://github.com/mozilla/addons-server/pull/3763
(Reporter)

Updated

10 months ago
Status: ASSIGNED → RESOLVED
Last Resolved: 10 months ago
Resolution: --- → FIXED
(Reporter)

Comment 19

10 months ago
Premature closing.
Still need got prod (and stage) to do.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 20

9 months ago
-stage end point is now configured in Socketlabs and ready for testing.
(Assignee)

Comment 21

9 months ago
This has gone out to -prod as of today.
(Assignee)

Updated

9 months ago
Status: REOPENED → RESOLVED
Last Resolved: 10 months ago9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.