Closed Bug 1696449 Opened 4 years ago Closed 1 year ago

nsNntpService::StreamHeaders does not work

Categories

(MailNews Core :: Networking: NNTP, defect)

defect

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: manikulin, Unassigned)

References

Details

Attachments

(1 file)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0

Steps to reproduce:

I was trying to write an extension that extracts some information from message headers for my notes. At first, I realized that messages.getFull(id) does not for news (NNTP) messages at all and messages.getRaw(id) works unreliably due to Bug #1644027. Looking for a workaround I have noticed nsIMsgMessageService::streamHeaders() method https://hg.mozilla.org/comm-central/file/61bd1f86844c430fe0ee212c8443dc042e4ab86d/mailnews/base/public/nsIMsgMessageService.idl#l195 that accordingly to its name should suit well for my purposes.

Unfortunately it is broken for nsNntpService class as well. It could be demonstrated using the following extension:

manifest.json

{
	"manifest_version": 2,
	"name": "streamHeaders test",
	"version": "0.1",
	"permissions": [ "messagesRead", "menus" ],
	"background": { "scripts": [ "background.js" ] },
	"experiment_apis": { "tst": {
		"schema": "schema.json",
		"parent": {
			"scopes": [ "addon_parent" ],
			"paths": [ [ "tst" ] ],
			"script": "experiment.js"
		}
	} }
}

schema.json

[ {
	"namespace": "tst",
	"functions": [ {
		"name": "streamHeaders",
		"type": "function",
		"parameters": [ { "name": "messageId", "type": "integer" } ]
	} ]
} ]

experiment.json

"use strict";
var { Services } = ChromeUtils.import("resource://gre/modules/Services.jsm");

var gLoggingUrlListener = {
	OnStartRunningUrl(aUrl) {
		console.log("LoggingUrlListener.OnStartRunningUrl", aUrl);
	},
	OnStopRunningUrl(aUrl, aExitCode) {
		console.log("LoggingUrlListener.OnStopRunningUrl", aUrl, aExitCode);
	},
	QueryInterface: ChromeUtils.generateQI([Ci.nsIUrlListener]),
};

class LoggingStreamListener {
	constructor() {
		const self = this;
		this._stream = null;
	}
	onStartRequest(aRequest) {
		console.log("LoggingStreamListener.onStartRequest", aRequest);
	}
	onStopRequest(aRequest, aStatusCode) {
		console.log("LoggingStreamListener.onStopRequest", aRequest, aStatusCode);
		delete this._stream;
	}
	onDataAvailable(aRequest, aInputStream, aOffset, aCount) {
		console.log("LoggingStreamListener.onDataAvailable", aRequest, aInputStream, aOffset, aCount);
		if (this._stream === null) {
			this._stream = Cc["@mozilla.org/scriptableinputstream;1"].createInstance(
				Ci.nsIScriptableInputStream
			);
			this._stream.init(aInputStream);
		}
		const chunk = this._stream.read(aCount);
		console.log("LoggingStreamListener.read\n", chunk && chunk.substring(0, 256));
	}
}

LoggingStreamListener.prototype.QueryInterface = ChromeUtils.generateQI([Ci.nsIStreamListener]);

var tst = class extends ExtensionCommon.ExtensionAPI {
	getAPI(context) {
		return { tst: {
			streamHeaders(messageId) {
				let messenger = Cc["@mozilla.org/messenger;1"].createInstance(
					Ci.nsIMessenger
				);
				const msgHdr = context.extension.messageManager.get(messageId);
				let msgUri = msgHdr.folder.generateMessageURI(msgHdr.messageKey);
				let service = messenger.messageServiceFromURI(msgUri);
				const streamListener = new LoggingStreamListener();
				const localOnly = false;
				const outUrl = service.streamHeaders(
					msgUri,
					streamListener,
					gLoggingUrlListener,
					localOnly
				);
				console.log("streamHeaders: streamHeaders", outUrl);
			},
		}, };
	}
	onShutdown(isAppShutdown) {
		if (!isAppShutdown) {
			Services.obs.notifyObservers(null, "startupcache-invalidate", null);
		}
	}
};

background.js

"use strict";
function testActionHandler(info, tab) {
	let selectedMessages = info && info.selectedMessages
		&& info.selectedMessages.messages;
	for (const message of selectedMessages) {
		if (info.menuItemId === "TEST_STREAM_HEADERS") {
			console.log("calling streamHeaders() for message %o", message);
			browser.tst.streamHeaders(message.id);
		} else {
			console.error("testActionHandler: unsupported action %o", info.menuItemId);
		}
	}
}
function testMain() {
	browser.runtime.onInstalled.addListener(function testCreateMenu() {
		browser.menus.create({
			contexts: [ "message_list" ],
			id: "TEST_STREAM_HEADERS",
			title: "streamHeaders test",
		});
	});

	browser.menus.onClicked.addListener(function testMenuListener(info, tab) {
		console.debug("testMenuListener", info, tab);
		testActionHandler(info, tab);
	});
}
testMain();
  • create a news account, e.g news.gmane.io that provides archives of various mail lists
  • subscribe to some group, e.g. gmane.comp.mozilla.thunderbird.user
  • invoke extension action using context menu for message list
  • inspect extension and thunderbird console

Actual results:

  • No messages from LoggingStreamListener or LoggingUrlListener for NNTP messages.
  • No exceptions
  • Return value is null not only for news messages but even for local ones.
  • Problems with nsNntpService::streamHeaders implementation are not obvious at the first glance https://hg.mozilla.org/comm-central/file/61bd1f86844c430fe0ee212c8443dc042e4ab86d/mailnews/news/src/nsNntpService.cpp#l1403 implementation certainly is not a stub
  • aUrlListener and aURL arguments are not used in the function body.
  • There is old entry in the changelog file for debian package "Stop using nsIMsgMessageService.streamHeaders, as this doesn't work"
  • There is a comment in mail/extensions/openpgp/content/modules/stdlib/msgHdrUtils.jsm "renders the supposedly-useful streamHeaders function unusable."

Expected results:

  • nsNntpService::StreamHeaders works. Called from JS it returns value.
  • Or at least it throws exception with proper message and it is clearly documented.
  • If streamHeaders method is deprecated than it should be documented in the nsIMsgMessageService.idl file

Lots of issues with header streaming :-( - See bug 1602326 and friends.

Flags: needinfo?(benc)

I should have checked Bugzilla before wasting time trying to find my error, why streamHeaders does not work. Sigh

Well, if you know the history of Enigmail, you'll know that there have always been issues, but more likely for IMAP. See for example the reference in bug 1596036 comment #17.

Hmm. The various StreamHeaders() functions do seem pretty borked. They'll (probably) work fine for locally-stored messages, but not for online ones.
I just hacked up a rough proof-of-concept fix for the NNTP case, which seems to work OK on Max's test extension.
It requests the entire message, then just quits when it detects the end of the headers.

Magnus: I could polish this up and give the IMAP case the same treatment, but a few questions first:

  1. given that no existing code uses this, is there a case for just ditching the streamHeaders() API altogether? It sounds like there's legit use cases (Max's extension, and it looks like Enigmail has had some frustrations with this in the past), but I figure it's a question worth asking.
  2. I vaguely recall that IMAP has some facility for just fetching headers, which I'd guess we make use of somewhere else. I'd also guess that NNTP has something similar. Presumably it'd be worth using these protocol-specific techniques, but I can imagine it'll be a goodly chunk of work to do it. Is it worth just hacking in a cheap-n-nasty version which just starts streaming the whole message and stops after the headers? (ie what I did for this patch)
Flags: needinfo?(benc) → needinfo?(mkmelin+mozilla)

Since streamHeaders is unused, I would just remove it.
I don't know about #2, but for nntp especially seems like tiny issue as messages typically are plain text and tiny, so there is not much to gain from getting just the headers.

Flags: needinfo?(mkmelin+mozilla)

I do not think that a workaround with fetching full message is the proper way. Currently JS code does the same trick.

At the moment when I filed this bug I was trying various approaches to obtain headers and was getting errors. Since that time I managed to find purely JavaScript workaround suitable even for Thunderbird 78, later Bug #1644027 and related issues were fixed (my thanks to the developers).

I may be wrong, but my impression is that streamHeaders is not used due to it was broken. I hope (I am not sure though) that some parts of Thunderbird might benefit from proper implementation by dropping JS hacks. My personal concern is getting all headers from extension, however there is no messages.getAllHeaders method anyway.

There was a discussion whether NNTP messages can behave similarly to other types in respect to ccList field ("Cc" headers), see Bug #1696895 comment #3. So with better NNTP support sometimes it might be possible to gain from extended capabilities of particular server.

(In reply to Magnus Melin [:mkmelin] from comment #6)

I don't know about #2, but for nntp especially seems like tiny issue as messages typically are plain text and tiny, so there is not much to gain from getting just the headers.

Thanks to news.gmane.io that makes a lot of mail list archives available through NNTP protocol. So there are a lot of HTML messages and not so small attachments with images or patches. So the value of proper NNTP support is more than just working with legacy.

@max: What exactly are you referring to as JS hacks? Reading the full message to get the headers? IIRC the NNTP protocol does not support to get just the headers. It only supports a certain set of header fields (which do not include cc for example) which can be queried. There is a protocol extension, which allows querying other header fields, but it must be supported by the server (which many servers do not).

So I do not see how we could do it better. Or can we?

I figured there might be some debate on this, so I'm leaving it open for a bit to see how things lie when the dust settles :-)

A couple of thoughts:

  • I'm not too familar with NNTP. It does look like it supports a HEAD command... but I don't know how NNTP headers are to something expecting email headers. I'd guess that existing NNTP code already does a little munging to make NNTP headers more... email-y.
  • It just occurred to me: you'd probably always be grabbing headers from messages currently showing in folders, right? So the headers would already be there, in the database (even if there's no local/offline copy of the message). So would it be enough for streamHeaders() to just re-synthesize a header block from the nsIMsgDBHdr? Or even to remove streamHeaders() and say that the "offficial" method is to get hold of the nsIMsgDBHdr and read out the headers you're interested in?

(In reply to Ben Campbell from comment #9)

  • I'm not too familar with NNTP. It does look like it supports a HEAD command... but I don't know how NNTP headers are to something expecting email headers. I'd guess that existing NNTP code already does a little munging to make NNTP headers more... email-y.

I have even noticed it as a part of cancelling procedure:
https://searchfox.org/comm-central/source/mailnews/news/src/nsNNTPProtocol.cpp#line-1723
However DFA should be expanded to allow HEAD as a self-sufficient command.

  • It just occurred to me: you'd probably always be grabbing headers from messages currently showing in folders, right? So the headers would already be there, in the database (even if there's no local/offline copy of the message). So would it be enough for streamHeaders() to just re-synthesize a header block from the nsIMsgDBHdr? Or even to remove streamHeaders() and say that the "offficial" method is to get hold of the nsIMsgDBHdr and read out the headers you're interested in?

To show message list only a few headers are obtained. The following logs are obtained using telnet news.gmane.io nntp

200 news.gmane.io InterNetNews NNRP server INN 2.6.3 ready (posting ok)
GROUP gmane.comp.mozilla.thunderbird.user
211 15 1 15 gmane.comp.mozilla.thunderbird.user
XOVER 14-15
224 Overview information for 14-15 follows
14	[SOGo] Calendar is "momentarily not available"	"H" (agents-FcZObrvlYduBUy7/sJONFg@public.gmane.org) <users-XYE0JsqaAHo@public.gmane.org>	Mon, 2 Sep 2019 21:30:14 +0200	<8ef4ae7f-9ae2-8bbd-4160-7f5774e7c782@meddatainc.com>		6591	6	Xref: news.gmane.org gmane.comp.groupware.sogo.user:29215 gmane.comp.mozilla.thunderbird.user:14
15	Re: Thunderbird cannot login to imap.gmail.com	Cy Burnot <cy.burnot@invalid.com>	Wed, 8 Jan 2020 05:55:44 -0500	<-tqdnd4ht7mtKIjDnZ2dnUU7-eHNnZ2d@mozilla.org>	<_rSdnX8fQpIptIjDnZ2dnUU7-RnNnZ2d@mozilla.org> <mailman.419.1578445239.1613.support-thunderbird@lists.mozilla.org> <E5mdnRElhNeetojDnZ2dnUU7-IednZ2d@mozilla.org> <mailman.409.1578446249.1612.support-thunderbird@lists.mozilla.org>	640625	Xref: news.gmane.org gmane.comp.mozilla.thunderbird.user:15
.
quit
205 Bye!

List of headers returned by XOVER

200 news.gmane.io InterNetNews NNRP server INN 2.6.3 ready (posting ok)
GROUP gmane.emacs.orgmode
211 139687 1 139790 gmane.emacs.orgmode
LIST OVERVIEW.FMT
215 Order of fields in overview database
Subject:
From:
Date:
Message-ID:
References:
Bytes:
Lines:
Xref:full
.
quit
205 Bye!

All headers fetched by HEAD

200 news.gmane.io InterNetNews NNRP server INN 2.6.3 ready (posting ok)
GROUP gmane.comp.mozilla.thunderbird.user
211 15 1 15 gmane.comp.mozilla.thunderbird.user
HEAD 15
221 15 <-tqdnd4ht7mtKIjDnZ2dnUU7-eHNnZ2d@mozilla.org> head
Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail
From: Cy Burnot <cy.burnot@invalid.com>
Newsgroups: gmane.comp.mozilla.thunderbird.user
Subject: Re: Thunderbird cannot login to imap.gmail.com
Date: Wed, 8 Jan 2020 05:55:44 -0500
Approved: news@gmane.org
Message-ID: <-tqdnd4ht7mtKIjDnZ2dnUU7-eHNnZ2d@mozilla.org>
References: <_rSdnX8fQpIptIjDnZ2dnUU7-RnNnZ2d@mozilla.org>
 <mailman.419.1578445239.1613.support-thunderbird@lists.mozilla.org>
 <E5mdnRElhNeetojDnZ2dnUU7-IednZ2d@mozilla.org>
 <mailman.409.1578446249.1612.support-thunderbird@lists.mozilla.org>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226";
	logging-data="116797"; mail-complaints-to="usenet@blaine.gmane.org"
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.3.1
To: mozilla-support-thunderbird@lists.mozilla.org
Original-X-From: support-thunderbird-bounces@lists.mozilla.org Wed Jan 08 11:55:55 2020
Return-path: <support-thunderbird-bounces@lists.mozilla.org>
Envelope-to: gcmt-user@m.gmane-mx.org
Original-Received: from lists.mozilla.org ([63.245.210.105] helo=mail.mozilla.org)
	by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.89)
	(envelope-from <support-thunderbird-bounces@lists.mozilla.org>)
	id 1ip90C-000Rlw-Rs
	for gcmt-user@m.gmane-mx.org; Wed, 08 Jan 2020 11:55:53 +0100
Original-Received: from localhost (localhost4.localdomain [127.0.0.1])
	by lists.mozilla.org (Postfix) with ESMTP id F0D65C520D7;
	Wed,  8 Jan 2020 10:55:50 +0000 (UTC)
X-Virus-Scanned: amavisd-new at mozilla.org
Original-Received: from mail.mozilla.org ([127.0.0.1])
	by localhost (mailman2.mail.mdc2.mozilla.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id wYCn5diaMWA6; Wed,  8 Jan 2020 10:55:50 +0000 (UTC)
Original-Received: from lists.mozilla.org (localhost4.localdomain [127.0.0.1])
	by lists.mozilla.org (Postfix) with ESMTP id 9B63FC520D8;
	Wed,  8 Jan 2020 10:55:50 +0000 (UTC)
X-Original-To: mozilla-support-thunderbird@lists.mozilla.org
Delivered-To: mozilla-support-thunderbird@lists.mozilla.org
Original-Received: from localhost (localhost4.localdomain [127.0.0.1])
 by lists.mozilla.org (Postfix) with ESMTP id A1A13C520D5
 for <mozilla-support-thunderbird@lists.mozilla.org>;
 Wed,  8 Jan 2020 10:55:48 +0000 (UTC)
X-Virus-Scanned: amavisd-new at mozilla.org
Original-Received: from mail.mozilla.org ([127.0.0.1])
 by localhost (mailman2.mail.mdc2.mozilla.com [127.0.0.1]) (amavisd-new,
 port 10024) with ESMTP id 9FfeuqXCQvju
 for <mozilla-support-thunderbird@lists.mozilla.org>;
 Wed,  8 Jan 2020 10:55:48 +0000 (UTC)
Received-SPF: pass (giganews.com: Sender is authorized to use
 'poster@giganews.com' in 'mfrom' identity (mechanism
 'include:smtp.phmgmt.com' matched)) receiver=mailman2.mail.mdc2.mozilla.com;
 identity=mailfrom; envelope-from="poster@giganews.com";
 helo=mailstar.serv1.aus.datafoundry.com; client-ip=209.99.125.38
Original-Received: from mailstar.serv1.aus.datafoundry.com
 (mailstar.serv3.aus.datafoundry.com [209.99.125.38])
 by lists.mozilla.org (Postfix) with ESMTP id 40F74C520D4
 for <mozilla-support-thunderbird@lists.mozilla.org>;
 Wed,  8 Jan 2020 10:55:45 +0000 (UTC)
Original-Received: from serv2.dca1.giganews.com (serv2.dca1.giganews.com
 [216.166.98.31])
 by mailstar.serv3.aus.datafoundry.com (Postfix) with ESMTP id 0AC22320164
 for <mozilla-support-thunderbird@lists.mozilla.org>;
 Wed,  8 Jan 2020 04:55:44 -0600 (CST)
Original-Received: by serv2.dca1.giganews.com (Postfix, from userid 9)
 id 622971A0069; Wed,  8 Jan 2020 04:55:44 -0600 (CST)
Original-Path: nntp.mozilla.org!news.mozilla.org.POSTED!not-for-mail
Original-NNTP-Posting-Date: Wed, 08 Jan 2020 04:55:44 -0600
In-Reply-To: <mailman.409.1578446249.1612.support-thunderbird@lists.mozilla.org>
Content-Language: en-US
X-DF-Seen-By: ms
Original-Lines: 19
X-Usenet-Provider: http://www.giganews.com
Original-NNTP-Posting-Host: 174.57.42.182
X-AuthenticatedUsername: NoAuthUser
Original-X-Trace: sv3-vnpSu/VzVo6ixZKQFpcjQ84qPXPlBCEAShMgbKe2hnFF1bcmtQFGr9+pVr05HL0s7Qvl5ytEQqyj1Gd!3zswdpNuK6fGO3yiMmZhozoPP/HhrZfQGr1NJfy+WGhCe7jXaMatMA3u1Q9j6dh/cE84La0jFgI0!yazQP8fwjfo2
Original-X-Complaints-To: abuse@mozilla.org
X-DMCA-Complaints-To: abuse@mozilla.org
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint
 properly
X-Postfilter: 1.3.40
Original-Newsgroups: mozilla.support.thunderbird
X-BeenThere: support-thunderbird@lists.mozilla.org
X-Mailman-Version: 2.1.26
Precedence: list
List-Id: Thunderbird help community <support-thunderbird.lists.mozilla.org>
List-Unsubscribe: <https://lists.mozilla.org/options/support-thunderbird>,
 <mailto:support-thunderbird-request@lists.mozilla.org?subject=unsubscribe>
List-Post: <mailto:support-thunderbird@lists.mozilla.org>
List-Help: <mailto:support-thunderbird-request@lists.mozilla.org?subject=help>
List-Subscribe: <https://lists.mozilla.org/listinfo/support-thunderbird>,
 <mailto:support-thunderbird-request@lists.mozilla.org?subject=subscribe>
Errors-To: support-thunderbird-bounces@lists.mozilla.org
Original-Sender: "support-thunderbird" <support-thunderbird-bounces@lists.mozilla.org>
Xref: news.gmane.org gmane.comp.mozilla.thunderbird.user:15
Archived-At: <http://permalink.gmane.org/gmane.comp.mozilla.thunderbird.user/15>
.
quit
205 Bye!

So attempt to recover StreamHeaders content from stored message list would be too far from being complete.

(In reply to John Bieling (:TbSync) from comment #8)

@max: What exactly are you referring to as JS hacks?

Maybe it is no more than a false impression since I am not familiar with design ideas behind separation between C++ and JS code. Reading discussions on Bug #1644027 and Co, I had feeling that some features that should be implemented in C++ code are actually broken, so JS code (related to message decoding) was added as a kind of workaround. The comment in the enigmail code on StreamHeaders is consistent with such point of view to some extent.

(In reply to John Bieling (:TbSync) from comment #8)

@max: What exactly are you referring to as JS hacks?

A bit more on this question.

Generally I consider C++ code as more efficient and so more friendly to e.g. laptop battery. Maybe I am confused, but my impression is that due to discrepancy of interfaces or incomplete implementation, C++ code (instead of fixing) is replaced by JS. Example: https://bugzilla.mozilla.org/attachment.cgi?id=9208039 "Bug 1696884 - switch getRaw from sync to async streamlistener and no longer use urllistener to work with news as well. r=darktrojan". @mozilla.org/network/sync-stream-listener;1 was replaced by JS implementation inside MsgHdrToRawMessage function.

I have not managed to realize what is supposed way to "parse" and decode messages. My feeling is that due to (relatively) recent fixes fraction of JS code for NNTP handling increased.

(In reply to Magnus Melin [:mkmelin] from comment #6)

Since streamHeaders is unused, I would just remove it.

I do not know the story. Judging from some comments, streamHeaders method was intended for extension and nobody managed to use it due to various problems.

(Kent James (:rkent) in Bug #752768 comment #14, "Ubuntu startup crash in nsMsgMailSession::OnItemAdded (sig: NS_ReadLine)")

AFAICT streamHeaders is not used in the core code, and was originally added
to support extensions (primarily Conversations)

(Jonathan Protzenko [:protz] in Bug #752768 comment #18, I suppose, the author of Conversations)

I disable the call to streamHeaders because that function just doesn't work:
it sometimes returns part of the header, and the first few bytes of the
headers are missing. I once tried to debug this with Bienvenu but we didn't
get far. If you were to find out what's happening, that would be fantastic :)

(Jorg K (CEST = GMT+2) in Bug #1596036 comment #17, "C-C TB: Handling of GetOfflineFileStream return value is wrong.", mentioned in comment #3 above)

StreamHeaders() is defined for IMAP, mailbox and NNTP. It's known to not work, Enigmail had this comment:
https://searchfox.org/comm-central/rev/38c93f8c99cfe10cff5cd1b0ce399ff2334c0003/mail/extensions/openpgp/content/modules/stdlib/msgHdrUtils.jsm#426

I tried it during the development of the pEp add-on and it turned out that for IMAP it only worked if the message was synchronised locally. Here's my PM to the author of Enigmail:

Old changelog entry for the Ubuntu package
http://changelogs.ubuntu.com/changelogs/pool/main/t/thunderbird/thunderbird_78.13.0+build1-0ubuntu0.18.04.1/changelog

thunderbird (15.0.1+build1-0ubuntu2) quantal; urgency=low

  • Update messagingmenu extension to 1.0
    • Stop using nsIMsgMessageService.streamHeaders, as this doesn't work

-- Chris Coulson <chris.coulson@canonical.com> Thu, 13 Sep 2012 12:04:29 +0100

It looks like there was a demand for StreamHeaders in the past. Maybe wider usage of HEADER NNTP command still can help addon developers.

See Also: → 1695235

nsNNTP code no longer exists since nntp-js. If the news behavior is still flawed, please file a new bug report.

Status: UNCONFIRMED → RESOLVED
Closed: 1 year ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: