1574863 - Cannot open conversation from Gloda search result in some cases

Status: RESOLVED FIXED

(Thunderbird :: Instant Messaging, defect)

Description

[Jorg K (CEST = GMT+2)]

One of the nice things of Gloda is that it indexes chat conversations and can find and open them. However, I'm seeing this weird issue now for the first time.

I was looking for "pike" in my chat history and found 42 hits in IRC. Some can be opened, some can't. For example one from #developers on 09 Jan 2018 opens fine, but one from #maildev on 5 February 2019 doesn't open, in fact, all hits from #maildev don't seem to open.

There's an error on the console:
TypeError: logTime is undefined - logger.js:742:11
which is here:
https://searchfox.org/comm-central/rev/eb044a1bc9fca62c103b25e77dfccb9adcf8a9c4/chat/components/src/logger.js#742
I'm not sure how it can be undefined since it was set a line above:
let [logTime] = getDateFromFilename(OS.Path.basename(path));

I checked on disk and the file is there:

D:\MAIL-THUNDERBIRD\jorgk.default\logs\irc\[e-mail]\#maildev.chat>dir 2019-02-05*
 Volume in drive D has no label.
 Volume Serial Number is 7E0E-9CA4

 Directory of D:\MAIL-THUNDERBIRD\jorgk.default\logs\irc\[e-mail]\#maildev.chat

[stuff deleted]
06 Feb 2019  01:05             8,769 2019-02-05.230651+0100.json
              11 File(s)         28,967 bytes
               0 Dir(s)  158,148,448,256 bytes free

D:\MAIL-THUNDERBIRD\jorgk.default\logs\irc\[e-mail]\#maildev.chat>grep -i pike 2019-02-05*
2019-02-05.230651+0100.json:{"date":"2019-02-05T22:24:39.000Z","who":"Pike","text":"of all the places","flags":["incoming"]}

I've checked in the Gloda database, and it knows the log file:

select path from imConversations where path like "%maildev%2019-02-05%"
[mote hits]
irc/[e-mail]/#maildev.chat/2019-02-05.230651+0100.json

So where is this going wrong?

Comment 1

[Jorg K (CEST = GMT+2)]

Doesn't work in TB 60 either, so not a recent regression.

Comment 2

[Jorg K (CEST = GMT+2)]

OK, found it. I had dropped a file into the logs directory which turned up in the loop here:
https://searchfox.org/comm-central/rev/eb044a1bc9fca62c103b25e77dfccb9adcf8a9c4/chat/components/src/logger.js#736-748
but didn't match the wildcard here:
https://searchfox.org/comm-central/rev/eb044a1bc9fca62c103b25e77dfccb9adcf8a9c4/chat/components/src/logger.js#409

Effect, the entire thing falls apart. Maybe we should be a little more accepting.

Comment 3

[Jorg K (CEST = GMT+2)]

Attachment: 1574863-chat-logs.patch

OK, no one should mess with those logs, but I did some searching and sorting in there and left some files behind. There's no harm in skipping unrecognised files.

I don't know who wants to review this, I'll try Florian first, this only needs visual inspection.

Comment 4

[Florian Quèze [:florian]]

Comment on attachment 9086355 [details] [diff] [review]
1574863-chat-logs.patch

Review of attachment 9086355 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for fixing this edge case. Reading this makes me wonder if we should log the name of unrecognized files to the console, but I don't care enough to request a change to do this.

Comment 5

[Jorg K (CEST = GMT+2)]

Attachment: 1574863-chat-logs.patch

Like this? Console says:
Found log file with name not maching YYYY-MM-DD.HHmmSS+ZZzz.format: all-chat.txt

Comment 6

[Florian Quèze [:florian]]

Comment on attachment 9086376 [details] [diff] [review]
1574863-chat-logs.patch

Review of attachment 9086376 [details] [diff] [review]:
-----------------------------------------------------------------

Seems reasonable, thanks. I hope we don't have massive amounts of unknown files that will spam the console.

Comment 7

[Jorg K (CEST = GMT+2)]

There shouldn't be any, otherwise this bug would have been filed before. I'm likely the only persons processing those log files and leaving undesired files behind in the directory. BTW, I strung all the small log files together to one big file with cat since grepping through a large file is much faster then grepping through 6000+ small files which the file system has to open individually. Also for Gloda to open the log in question takes a few seconds since it's processing many log files in the directory.

Are there any plans to switch form one log file per session to something more compact? Also for backup it's not convenient to have all those small log files.

As for the patch: I considered switching to Services.console.logStringMessage() but that doesn't stand out in the log.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1b4f6fdfb4f0
Handle 'foreign' files in logs directory gracefully. r=florian

Comment 9

[Jorg K (CEST = GMT+2)]

Comment on attachment 9086376 [details] [diff] [review]
1574863-chat-logs.patch

No harm in backporting this.

Comment 10

[Florian Quèze [:florian]]

(In reply to Jorg K (GMT+2) from comment #7)

Are there any plans to switch form one log file per session to something more compact? Also for backup it's not convenient to have all those small log files.

I used to want to automatically compress the old log files into one gzip file per month. But I don't expect to work on this any time soon, if ever.

The fact that you need to grep shows that something is broken with the search. And I happen to sometimes grep my log folder too, so I would probably want search to be fixed (which could involve replacing gloda with something simpler and more efficient) before we compress logs on disk.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c705f6a8345f
Follow-up: remove trailing spaces. rs=white-space-only DONTBUILD

Comment 12

[Jorg K (CEST = GMT+2)]

Sorry about the trailing spaces, they're barely visible in splinter now, filed bug 1575001.

Comment 13

[Jorg K (CEST = GMT+2)]

The fact that you need to grep shows that something is broken with the search.

Well, the other day I was looking for something I thought should have been there. Gloda didn't find it, and neither did grep, so it was just in my head :-(

Comment 14

[Jorg K (CEST = GMT+2)]

TB 69 beta 4:
https://hg.mozilla.org/releases/comm-beta/rev/f513420103a459fa43ee6d3739b4434c52e959b0 (landed without the trailing spaces)

Comment 15

[Jorg K (CEST = GMT+2)]

TB 68.1 ESR:
https://hg.mozilla.org/releases/comm-esr68/rev/6373f14f49e24bc2e4dc614fd3c3017fd7058de4