1531317 - Add a way to search for email (browser.messages.query)

Status: RESOLVED FIXED

(Thunderbird :: Add-Ons: Extensions API, enhancement)

Description

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

In bug 1488176 comment 27 et al Geoff and I talked about a way to query messages. It looks like we never filed a followup for this.

The search API should be modeled in a way that we can use gloda to speed up searches, but at the same time also make it work when gloda is disabled. an index: false option could allow manually disabling gloda, or true to explicitly request it.

Conceptually, it could be similar to the browser.tabs.query() method, but suitable for mail. The responses should use message lists.

Hermar was looking into writing a gloda based experiment for this, I'd love for us to work with him to make what he is working on suitable for adding to Thunderbird.

Comment 1

[Geoff Lankow (:darktrojan)]

Any progress?

Comment 2

[Marvin]

logic is done for searching with gloda through an API.
I will be adding promises today and then upload a version...
"at the same time also make it work when gloda is disabled" still needs to be made though...

Comment 3

[Marvin]

Facing a problem with Gloda parameters:
"Cannot query on attribute from because its canQuery parameter hasn't been set to true."
otherwise complete

Comment 4

[Geoff Lankow (:darktrojan)]

Please attach what you've done so far. I can't see what's going on (or not) without it.

Comment 5

[Marvin]

Attachment: call to the API after a click on the toolbar

part 1/4

Comment 6

[Marvin]

Attachment: implementation.js is the logic behind the Gloda API

part 2/4

Comment 7

[Marvin]

Attachment: manifest of the gloda API

part 3/4

Comment 8

[Marvin]

Attachment: schema of the Gloda API

some are not working and therefore needs to be checked!
part 4/4

Comment 9

[Geoff Lankow (:darktrojan)]

My understanding of GloDa is very limited, but I don't think you can just pass a string to most of the constraining functions. I got "involves" working for example, but only because I took an existing identity object from an earlier query.

You were looking for a list of things you could query, this might help: https://hg.mozilla.org/comm-central/file/tip/mailnews/db/gloda/modules/fundattr.js#l59. I think you should be able to do most of the ones with objectNoun set to NOUN_FULLTEXT, NOUN_NUMBER, and NOUN_STRING. It looks like authorMatches, subjectMatches, etc. are going to be the most useful ones.

Comment 10

[Marvin]

Well I guess I am done...
I finished the schema based on those: https://hg.mozilla.org/comm-central/file/tip/mailnews/db/gloda/modules/fundattr.js#l59 and
tested the possible options within the Developers Toolbox with Geoffs help.
He confirmed that the "read" option is not working so I removed it from the schema (should it be possible in the near future it would only be adding a couple of rows to the schema so that others may know its possible).

I would like some feedback regarding the "implementation" and the status of "read" though...
(@Geoff: I found that it was slow because I made log entries whenever I added something to the collection, after I removed it I was, in my opinion, working ok)

Comment 11

[Geoff Lankow (:darktrojan)]

Attachment: schema.json

I think your implementation's good in theory but practically it has some problems. I think it's probably best if you unroll that forEach loop and handle each query constraint individually. Some names and types in the schema aren't going to line up with what Gloda expects.

Here's a schema that's roughly what I'd do. I've used some existing types from other built-in schema. Thoughts:

• folder: I think Gloda expects a single nsIMsgFolder, which you can get from FolderManager.get. It would be nice if more than one folder could be used but if it doesn't work let's not attempt that now.
• fromDate and toDate: the type I used should give you a Date object, but I've not used it before so don't quote me on that. If one date is specified but not the other, fill in the other with something sensible.
• author, recipients, subject, and body can use their "*Matches" constraints, though they seem to be case-sensitive which is annoying
• flagged: Gloda calls this starred but we're using flagged for WebExtensions

I've left out read, tags, and attachments for now. Read still seems to be broken, and the others are complicated. Let's get the rest going first.

Comment 12

[Marvin]

Attachment: schema of the experimental Gloda-API

Comment 13

[Marvin]

Attachment: logic of the experimental Gloda-API

Comment 14

[Geoff Lankow (:darktrojan)]

Attachment: 1531317-messages-query-1.diff

Here's Marvin's API (with some minor changes) in the form of a patch. Requesting review from someone else because I've been involved a lot here.

Comment 15

[Geoff Lankow (:darktrojan)]

Attachment: 1531317-messages-query-tests-1.diff

And some tests. I had a major fight with GloDa over this, but I won in the end.

Comment 16

[Geoff Lankow (:darktrojan)]

Well done Marvin. We'll get this into the tree then we can follow up any remaining issues (just read = true/false, I think) in a new bug.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/14390389783d
Message search WebExtension API. r=mkmelin
https://hg.mozilla.org/comm-central/rev/450a66aeeaa8
Message search WebExtension API - tests. r=mkmelin

Comment 18

[Jorg K (CEST = GMT+2)]

You want this on TB 68 beta and later ESR?

Comment 19

[Geoff Lankow (:darktrojan)]

Ultimately yes, I'm hopeful of finishing some remaining things quickly then it can all go at once.

Comment 20

[Geoff Lankow (:darktrojan)]

Attachment: 1531317-messages-query-esr.diff

Comment 21

[Geoff Lankow (:darktrojan)]

Attachment: 1531317-messages-query-tests-esr.diff

Comment 22

[Jorg K (CEST = GMT+2)]

TB 68.2.0 ESR:
https://hg.mozilla.org/releases/comm-esr68/rev/665f90bd4664b70188e0c69a4f6a32d62523dd1c
https://hg.mozilla.org/releases/comm-esr68/rev/9e94413eeda8d3b2e51a1d05a55433ddd9e05928

Comment 23

[Mihai]

author, recipients, subject, and body can use their "*Matches" constraints, though they seem to be case-sensitive which is annoying

Do these support wildcards?

Comment 24

[Marvin]

I am sorry I kind of forgot to answer.

First: I just checked myself... author, reciepients, subject and body do not care for case-sensitivity you may write mArViN and he still finds the mails I send, recieved or those in which my name is in the subject or the body.

Regarding support for wildcards: they do not the reason is the four query options which you mentioned expect a value of string type. "*" would only search for said star in the option which you chose... you may work around that with {"subject":""} or {"author":""} wich will lead to a whole lot of mails but I can not guarantee that those are all available ones...

I hope that this answer somehow helps you.. otherwise ask away I will try to answer timely...

Comment 25

[Mihai]

Thanks Marvin.