1667308 - Phabricator emails: X- headers are missing, most painfully X-Phabricator-Stamps

Status: NEW

(Conduit :: Phabricator, defect, P3)

Description

[:glob ✱]

Phabricator's native emails contain a ton of X- headers which are invaluable for email filtering, these are completely missing from the new emails, making some filtering tasks impossible.

There's a lot of headers:

X-Phabricator-Sent-This-Message: Yes
X-Phabricator-Mail-Tags: <differential-other>
X-Phabricator-To: <PHID-USER-trei6k7kwz6to4zny7w6>
X-Phabricator-To: <PHID-USER-nnef5qn5u2ambzemxgur>
X-Phabricator-Mail-ID: 2955004
X-Phabricator-Send-Attempt: ekl4ecoxmqxbt7z6
X-Phabricator-Stamps: actor(@zeid) application(Differential) author(@glob) herald(H28) herald(H29) herald(H34) monogram(D88237) object-type(DREV) phid(PHID-DREV-2mokx3nt4plal6wqzndk) reviewer(@zeid) revision-repository(rREVIEW) revision-status(accepted) via(web)

Of these X-Phabricator-Stamps is likely to be the only one that needs to be replicated; although my current filters also use X-Phabricator-Sent-This-Message.

Comment 1

[:Gijs (he/him)]

Is the implementation of these emails in a public or mozilla-org-accessible repo, ie could I contribute a fix to this? This is interfering with my productivity because so many of my emails are for group memberships and I cannot filter them out anymore. :-(

Comment 2

[David Lawrence [:dkl]]

(In reply to :Gijs (he/him) from comment #1)

Is the implementation of these emails in a public or mozilla-org-accessible repo, ie could I contribute a fix to this? This is interfering with my productivity because so many of my emails are for group memberships and I cannot filter them out anymore. :-(

Sorry for this lack of functionality causing issues. As the watchkeeper of the Phabricator emails system, I haven't had much time to do anything with it except to make sure it stays running. I could look at this in the near future and add back some of the missing headers but it might be some time with other goals that need to be complete. In the mean time, if you want to take a look at it, the code lives at:

https://github.com/mozilla-conduit/phabricator-emails

I will be happy to review any changes.

Thanks

Comment 3

[:Gijs (he/him)]

(In reply to David Lawrence [:dkl] from comment #2)

https://github.com/mozilla-conduit/phabricator-emails

I will be happy to review any changes.

Thanks

I've started looking at this using only the existing test coverage as a guide, as I don't have a full local bugzilla + phab install, and was unwilling to invest all the time to obtain both of those. pytest runs fine with just the repo, so I feel like this should work, though it'll need checking against "the real thing" before landing.

Although it seems doable, the big thing I'm running into is I have no idea what data is available from phabricator. I understand that the code slurps JSON "event" blobs. Is there a way to see a sample of such a blob? I have to somehow funnel the data from that JSON into the right place, but because I don't know what the event blob looks like it's hard to work out where it should be sourced, except for information that's already available (e.g. repository identifier, actor).

I've found the old phab extension's https://github.com/mozilla-services/phabricator-extensions/blob/master/moz-extensions/src/email/FeedForEmailQueryAPIMethod.php but that repo is archived (so perhaps not the right thing to look at?) and although I have a basic grasp of PHP, I've not been able to work out exactly what data ends up in the final "event" objects. I haven't been able to find tests in that repo (where I'd assume there's some end-to-end thing that'd help elucidate the JSON end of that pipeline.

Any advice on where to look for this would be really helpful. :-)

Comment 4

[:Gijs (he/him)]

(In reply to :Gijs (he/him) from comment #3)

Although it seems doable, the big thing I'm running into is I have no idea what data is available from phabricator. I understand that the code slurps JSON "event" blobs. Is there a way to see a sample of such a blob? I have to somehow funnel the data from that JSON into the right place, but because I don't know what the event blob looks like it's hard to work out where it should be sourced, except for information that's already available (e.g. repository identifier, actor).

I tried asking in #phabricator, but I guess maybe a needinfo for this is warranted. Sorry to bother you, :dkl, but is it easy for you to answer this bit?

Comment 5

[David Lawrence [:dkl]]

(In reply to :Gijs (he/him) from comment #3)

Although it seems doable, the big thing I'm running into is I have no idea what data is available from phabricator. I understand that the code slurps JSON "event" blobs. Is there a way to see a sample of such a blob? I have to somehow funnel the data from that JSON into the right place, but because I don't know what the event blob looks like it's hard to work out where it should be sourced, except for information that's already available (e.g. repository identifier, actor).

There are two Conduit API queries available that can be used to get the JSON data from Phabricator used by the email Python system.

This one gets the latest feed key that can be stored and then used to get new stories after that point.
https://phabricator.services.mozilla.com/conduit/method/feed.for_email.status/

This one queries Phabricator for the feed stories that have happened since the last stored key from above.
https://phabricator.services.mozilla.com/conduit/method/feed.for_email.query/

Both require membership in a special group in Phabricator to access. I have added you and I to the group for now to figure this out.

So taking the value from the first API endpoint and then plugging it in the second in the 'after' field (may need to wait a while) will give you the JSON data that is used by the Phabricator Email system. There is a table in the email system DB that stores the last known good feed key so when it loops back around it starts from that point.

I've found the old phab extension's https://github.com/mozilla-services/phabricator-extensions/blob/master/moz-extensions/src/email/FeedForEmailQueryAPIMethod.php but that repo is archived (so perhaps not the right thing to look at?) and although I have a basic grasp of PHP, I've not been able to work out exactly what data ends up in the final "event" objects. I haven't been able to find tests in that repo (where I'd assume there's some end-to-end thing that'd help elucidate the JSON end of that pipeline.

Yeah the moz-extension repo was what we used that was layered on top of the upstream Phabricator repository at Docker build time. Now we just have a full fork of the upstream and we have the moz-extensions code as a directory in the fork. Not too different in the end result.

Current repo:
https://github.com/mozilla-conduit/phabricator