Closed Bug 468198 Opened 13 years ago Closed 13 years ago

Fix Spotlight .mozeml generation code

Categories

(Thunderbird :: General, defect)

All
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 3.0b2

People

(Reporter: humph, Assigned: Bienvenu)

References

Details

Attachments

(1 file)

Trying to finish off bug 290057, I'm finding that metadata I'd assume would be set in .mozeml files is not there.  As you'll see below, the indexing and find are working, but he Spotlight UI never shows anything.  You'll note that we're missing a lot of metadata, which I think is causing the Spotlight UI to miss this, even though we are indexing it.

NOTE: I'm using the Thunderbird.mdimporter in the tree with the patches I've recently added to bug 290057 to do proper packaging in the bundle, and add Info.plist data.

John, can you compare this to a .mozeml on your computer, since from your blog I assume you're able to get this working with your manual method?

Steps to Reproduce:

1) cd /Users/dave/Library/Thunderbird/Profiles/bdpf5gyq.Shredder/ImapMail/mercury.senecac.on.ca/INBOX.mozmsgs

2) mdimport -d2 200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml

(Info) Import: Import '/Users/dave/Library/Thunderbird/Profiles/bdpf5gyq.Shredder/ImapMail/mercury.senecac.on.ca/INBOX.mozmsgs/200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml' type 'com.mozilla.thunderbird.mozeml' using '/Applications/ShredderDebug.app/Contents/Library/Spotlight/Thunderbird.mdimporter'
trying to parse date 
got cur date
2008-12-05 20:53:18.839 mdimport[37751:10b] Imported '/Users/dave/Library/Thunderbird/Profiles/bdpf5gyq.Shredder/ImapMail/mercury.senecac.on.ca/INBOX.mozmsgs/200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml' of type 'com.mozilla.thunderbird.mozeml' with plugIn /Applications/ShredderDebug.app/Contents/Library/Spotlight/Thunderbird.mdimporter.
2008-12-05 20:53:18.841 mdimport[37751:10b] Attributes: {
    "_kMDItemFinderLabel" = <null>;
    "com_apple_metadata_modtime" = 250221190;
    kMDItemAlternateNames =     (
        "200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml"
    );
    kMDItemContentCreationDate = 2008-12-05 20:53:10 -0500;
    kMDItemContentModificationDate = 2008-12-05 20:53:10 -0500;
    kMDItemContentType = "com.mozilla.thunderbird.mozeml";
    kMDItemContentTypeTree =     (
        "com.mozilla.thunderbird.mozeml",
        "public.data",
        "public.item",
        "public.content",
        "public.email-message",
        "public.message"
    );
    kMDItemDisplayName = "";
    kMDItemKind =     {
        "" = "Thunderbird Spotlight Metadata";
    };
\nYou are on the CC list for the bug.";ecause: -------prefs.cgi?tab=email 15:11:55 PST ---ac.on.ca>  Do not reply to this email.  You can add comments to this bug at
    kMDItemTitle = "";
}

3) mdls 200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml

kMDItemAlternateNames          = (
    "200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml"
)
kMDItemContentCreationDate     = 2008-12-05 20:53:10 -0500
kMDItemContentModificationDate = 2008-12-05 20:53:10 -0500
kMDItemContentType             = "com.mozilla.thunderbird.mozeml"
kMDItemContentTypeTree         = (
    "com.mozilla.thunderbird.mozeml",
    "public.data",
    "public.item",
    "public.content",
    "public.email-message",
    "public.message"
)
kMDItemDisplayName             = ""
kMDItemFSContentChangeDate     = 2008-12-05 20:53:10 -0500
kMDItemFSCreationDate          = 2008-12-05 20:53:10 -0500
kMDItemFSCreatorCode           = ""
kMDItemFSFinderFlags           = 0
kMDItemFSHasCustomIcon         = 0
kMDItemFSInvisible             = 0
kMDItemFSIsExtensionHidden     = 0
kMDItemFSIsStationery          = 0
kMDItemFSLabel                 = 0
kMDItemFSName                  = "200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml"
kMDItemFSNodeCount             = 0
kMDItemFSOwnerGroupID          = 20
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 1359
kMDItemFSTypeCode              = ""
kMDItemKind                    = "Thunderbird Spotlight Metadata"
kMDItemLastUsedDate            = 2008-12-05 20:53:10 -0500
kMDItemSupportFileType         = (
    MDSystemFile
)
kMDItemTitle                   = ""
kMDItemUsedDates               = (
    2008-12-05 00:00:00 -0500
)

4) mdfind "This patch includes the recommendations by Ted."

/Users/dave/Library/Thunderbird/Profiles/bdpf5gyq.Shredder/ImapMail/mercury.senecac.on.ca/INBOX.mozmsgs/200812052312.mB5NC0p8008279%40mrapp51.mozilla.org.mozeml
Attached patch proposed fixSplinter Review
Sid probably knows this code as well as anyone, but if humph wants to mark it reviewed, I'd take that instead...
Attachment #351721 - Flags: review?(sid1337)
it's actually not the importer, but the code that generates .mozeml files.
Summary: Fix Thunderbird.mdimporter regression → Fix Spotlight .mozeml generation code
Comment on attachment 351721 [details] [diff] [review]
proposed fix

Yeah, this is the regression I'd caused while merging the code in searchCommon.js. Sorry. :(
Attachment #351721 - Flags: review?(sid1337) → review+
Comment on attachment 351721 [details] [diff] [review]
proposed fix

Tested and works, giving me kMDItemDisplayName and kMDItemTitle values.

I'm still not seeing the results in the Spotlight UI at all, though.  I'll look into this more on Monday.
Attachment #351721 - Flags: review+
David: you might want to use mime2DecodedSubject instead of subject, though, as the other code there also uses mime2..., and I think it makes sense here -- you're converting to UTF-8, after all.
I figured out why I'm not seeing any results: Spotlight won't show results for "System Files" which seems to include things in ~/Library.  If I move a .mozeml out of my profile to ~/Desktop, I see it right away in a Spotlight search.

I see lots of noise from people complaining about this on the web (e.g., http://macosx.com/forums/mac-os-x-system-mac-software/296842-spotlight-not-searching-library.html) and some ugly manual hacks to add System Files.  None of them look like responsible things for us to do to someone's setup on install.

I suspect that others are getting this to work with TB because they are dropping the mdimporter into /Library or another path that causes the .mozeml files to get put in a non problematic spot.

John, where are your .mozeml files going?
(In reply to comment #6)
> 
> I suspect that others are getting this to work with TB because they are
> dropping the mdimporter into /Library or another path that causes the .mozeml
> files to get put in a non problematic spot.
> 

The mozeml generation code stores files relative to the profile directory.

Where do applications like Apple Mail store their emails?
moving the mdimporter has no effect on where the .mozeml files are put. They're always in the user's profile dir. Perhaps having the mdimporter in the ~/Library dir makes Spotlight index other files in that dir that are handled by that mdimporter. David, if you put the mdimporter there, does that make it work?

I'm a bit confused because I thought ~/Library was where applications put their data - maybe spotlight is trying to avoid indexing more files than necessary.
fix checked in, using mime2DecodedSubject, thx for catching that, Sid.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Hardware: PC → All
Target Milestone: --- → Thunderbird 3.0b2
(In reply to comment #6)
> I figured out why I'm not seeing any results: Spotlight won't show results for
> "System Files" which seems to include things in ~/Library.  If I move a .mozeml
> out of my profile to ~/Desktop, I see it right away in a Spotlight search.
> 
> I see lots of noise from people complaining about this on the web (e.g.,
> http://macosx.com/forums/mac-os-x-system-mac-software/296842-spotlight-not-searching-library.html)
> and some ugly manual hacks to add System Files.  None of them look like
> responsible things for us to do to someone's setup on install.
> 
> I suspect that others are getting this to work with TB because they are
> dropping the mdimporter into /Library or another path that causes the .mozeml
> files to get put in a non problematic spot.

fyi: I had to change the location of Thunderbird mdimporter during my recent OSX upgrade:
10.4: mdimporter in ~/Library/Spotlight
10.5: mdimporter in /Library after discovering that ~/Library didnt work
anymore.


> John, where are your .mozeml files going?
I dont remember where the mozeml files were in 10.4. In 10.5, my *.mozeml files are within the same directory tree as my Thunderbird email messages. For example, here's what I see for INBOX:
 $ cd ~/email_Mozilla/IMAP-MoCo/
 $ ls -la | grep INBOX
-rw-r--r--@     1 john  john    1842077 Dec  8 00:06 INBOX
drwxr-xr-x  20283 john  john     689622 Dec  3 18:31 INBOX.mozmsgs
-rw-r--r--      1 john  john     245243 Dec  8 00:06 INBOX.msf
drwxr-xr-x      2 john  john         68 Nov 18  2007 INBOX.sbd
 $ cd INBOX.mozmsgs
 $ ls -la | grep mozeml | wc -l 
   20281
 $
You need to log in before you can comment on or make changes to this bug.