879412 - Write a cron job to parse raw email message from reviewers/developers

Status: RESOLVED FIXED

(Marketplace Graveyard :: Admin Tools, defect, P3)

Description

[Christopher Van Wiemeersch [:cvan]]

Postfix will dump to a directory and we will just vacuum the emails up with a cron.

The management command goes something like this:

1) Accept a filepath to a directory containing text files of email messages.
2) Parse the `To` address for each email prefixed with `reply+`. Figure out which Thread and User it corresponds to based on the UUID.
3) Save a new Note object for that Comment Thread in the database.

For GitHub's email notifications they use a custom Reply-To token to know which email to parse; they then take the email message and pipe it through this script to strip the signatures, etc. to extract the message: https://github.com/github/email_reply_parser

There's a Python port of this: https://github.com/zapier/email-reply-parser

A sample email looks like this:

--

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: App Approved: Weather Me
From: Chris Van <[e-mail]>
To: Firefox Marketplace <[e-mail]>
Date: Tue, 04 Jun 2013 15:23:11 -0000
Message-ID: <[e-mail]>

Woo hoo! Thanks =)

Unfortunately, my app isn't showing up in the search results yet!! Do you know when that will be?

Thanks,

Chris "seavan" Van
Seavan Studios, Ltd.
https://github.com/cvan

> Congratulations! Your app has been approved and is listed on the Firefox Marketplace.
>
> Follow Mozilla Hacks (http://hacks.mozilla.org, http://twitter.com/mozhacks) to stay updated on when we plan to launch on other platforms.
>
> App: Weather Me
> URL: https://marketplace.firefox.com/app/weather-me/
>
> Reviewed by: schoewilliam
> Comments:
> Found no issue.
> Your App has been approved will be available for installation by Firefox users within a few minutes.
> 
> If you have any questions about this review, please reply to this email or join #app-reviewers on irc.mozilla.org. For general developer support, please contact [e-mail].
>
> --
> Firefox Marketplace
> https://marketplace.firefox.com
>
> ===================================================================
> You received this email because you're subscribed to:
>    * email me when my app is reviewed by an editor
>
>
> You can't unsubscribe from this type of email.
>
> Manage email notifications: https://marketplace.firefox.com/settings#notifications

Comment 1

[Ashish Dubey [:ashish_d]]

So which directory do we find the emails in? Should it be defined as a constant, in settings (conf), or as an argument to the management command?

Comment 2

[Kris Maglione [:kmag]]

A cron job doesn't seem like the best way to do this. Ideally, Postfix should pipe the message to a handler to process it directly, and it should be dumped into a queue if something fails.

I think that doing all processing via a cron job would artificially rate limit conversations more than would be desirable.

Comment 3

[Wil Clouser [:clouserw]]

(In reply to Ashish Dubey from comment #1)
> So which directory do we find the emails in? Should it be defined as a
> constant, in settings (conf), or as an argument to the management command?

A constant please.

(In reply to Kris Maglione [:kmag] from comment #2)
> A cron job doesn't seem like the best way to do this.

Cron is flexible, scalable, fast, and fault-tolerant.  We're talking about a solution which runs very often (probably once a minute).  Cron will be great.

Comment 4

[Kris Maglione [:kmag]]

Well, a lot of things are flexible, scalable, fast, and fault-tolerant, including dispatching the emails as they come in. Doing this from a cron job adds a lot of possible snags, such as attempting to process a message before the mail handler has finished writing it, or two cron jobs running in parallel and processing the same messages twice. They're avoidable, but it still doesn't seem like the ideal solution.

Comment 5

[Ashish Dubey [:ashish_d]]

Can this be done as a background task with celery? Addressing issues like concurrent processing and dispatching the tasks timely will be easier IMO.

Comment 6

[Christopher Van Wiemeersch [:cvan]]

(In reply to Ashish Dubey from comment #5)
> Can this be done as a background task with celery? Addressing issues like
> concurrent processing and dispatching the tasks timely will be easier IMO.

Yes, this should be done as a celery task just like we do with `email_daily_ratings`. Something like https://github.com/mozilla/zamboni/blob/master/scripts/crontab/crontab.tpl#L45

Comment 7

[Christopher Van Wiemeersch [:cvan]]

And in the celery task it'll be trivial to know which emails have already been processed. Let me know if you have more questions - thanks!

Comment 8

[Christopher Van Wiemeersch [:cvan]]

https://github.com/mozilla/zamboni/pull/900

Comment 9

[Ashish Dubey [:ashish_d]]

fixed https://github.com/mozilla/zamboni/commit/a8c80fd