1656651 - Improve the management and reporting of the pausing of RSS feeds.

Status: RESOLVED FIXED

(MailNews Core :: Feed Reader, enhancement)

Description

[Matt]

+++ This bug was initially created as a clone of Bug #1524241 +++

Bug 1524241 contains a discussion of how the pause is by design however the pausing is not an acceptable way to manage the situation. The inability to connect on this occasion should not lead to a parmantent failure to execute the download of the feed.

I would expect 30 minutes later that the feed would again be checked and the pause would be reset if a successful connection occurred, I leave my Thunderbird running some times for days until I get home to find nothing has downloaded for the past week because an automatic function has determined that it should not be automatic.

This time around I find my feeds have not updated since the 17 July, almost 2 weeks despite a number of restarts which also should have removed this as a startup is basically a get all mail.

Flagging the reason for the error in the error console is also very poor design and needs to be improved. There should be errors entered into the activity manager and the status bar when a failure occurs, not hidden away in a panel that is buried in "developer Tools" on the menu.

Comment 1

[enone]

I Second That! I've Lost Tons Of RSS Articles!
I Must Restart My Feed Manually Every Time!
Thunderbird Must Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry And Retry Not Stopping Without Advice!

Comment 2

[jhibbard]

@enone
Sat in at a few security webinars in the recent past. One of the points that stick with me is that the so-called financial management packages are requiring that the customer divulge their ID/Password which they then use to fire MANY multiple login requests in order to get the data that they then put into some sort of dashboard. In this case, MANY results in somewhere in the vicinity of 65% of all traffic handled by these financial concerns (banks, retirement services, anything having to do with the user's finances). Sloppy, stupid, lazy coding... It just seems to be the landmark of current coding characteristics; especially in Windows.

Comment 3

[gravylake]

There should at least be a way to disable this behavior. I have a spotty internet connection, and I'm having to go in and unpause valid feeds all the time.

Comment 4

[cliffa]

I'm once again chiming in that the way this is currently handled is inappropriate. It flies in the face of some of the foundational principles of computer networking. It turns temporary failures into permanent ones... when all reasonable efforts should be made to accomplish the opposite.

(In reply to jhibbard from comment #2)

@enone
Sat in at a few security webinars in the recent past. One of the points that stick with me is that the so-called financial management packages are requiring that the customer divulge their ID/Password which they then use to fire MANY multiple login requests in order to get the data that they then put into some sort of dashboard. In this case, MANY results in somewhere in the vicinity of 65% of all traffic handled by these financial concerns (banks, retirement services, anything having to do with the user's finances). Sloppy, stupid, lazy coding... It just seems to be the landmark of current coding characteristics; especially in Windows.

I'm sorry but I don't understand how that's related to the current discussion. That some software has been written poorly such that it triggers constant, spurious logins and/or connections does not explain why it's necessary to completely prevent any attempts to retry an rss update, which requires no transition of credentials and is not constant, in Thunderbird. Unless you are simply arguing that because one extreme exists then Thunderbird needs to go to the opposite extreme. If the normal rss update cycle of once every 15 minutes to several hours could somehow be argued to be causing additional or spurious network traffic perhaps you would have a point. But it is very specifically only creating the normal amount of traffic and there is no way to know it is truly spurious until after the update attempt is completed.

Let me reiterate... to say that allowing an rss feed to continue attempting to update because it failed to forge a connection 1, 2, or even 3 times is creating additional connections is to call the successful connections "additional" as well. If after 24 hours of consecutive failures it still hasn't returned to working order... then you might have an argument to make... but there is such a thing as "retry exponential back-off." And in cases where it can reasonably said that a significant failure has occurred and user interaction is needed.. then user interaction is needed*. Making lasting changes to the user's configuration silently is about the single worst way to handle things at that point. It's laughably bad. It should be embarrassing to the developers of Thunderbird. I don't say that in an attempt to offend... but all attempts to press upon the devs what a completely unacceptable state of affairs has been created here have, apparently, not gotten through. Perhaps bugzilla should turn off their email updates and not tell them we have done so... that's what they think should happen when there's a communication failure right? After all, any communication with them after that point is likely to be extra and spurious... right?

The current way of handling this issue is completely unreasonable and seemingly unseasoned. If the Thunderbird devs are not going to correct their mistakes then they should at least let users correct them for them. Bury the option in developer config options if it makes them feel better... just give us the option to make it work in a reasonable way.

Comment 5

[Ben]

I'm sorry but I don't understand how that's related to the current discussion. That some software has been written poorly such that it triggers constant, spurious logins and/or connections does not explain why it's necessary to completely prevent any attempts to retry an rss update, which requires no transition of credentials and is not constant, in Thunderbird. Unless you are simply arguing that because one extreme exists then Thunderbird needs to go to the opposite extreme. If the normal rss update cycle of once every 15 minutes to several hours could somehow be argued to be causing additional or spurious network traffic perhaps you would have a point. But it is very specifically only creating the normal amount of traffic and there is no way to know it is truly spurious until after the update attempt is completed.

I'd even go further and say: yes, there are shitty feed provider out there, but Thunderbird as an integrating app should be as robust to handle these as possible.

But, the question is "how robust". In my case the feed provider randomly throws HTTP-404 as response. If a web resource is not found (anymore?) and how to react to THIS its a debatable thing.

According to the source code a network connection interrupt shouldn't trigger a disabling of the feed (this didn't happen in my scenario, as a HTTP-response is application layer and therefore does not fall into this constraint) - although I haven't tested this behavior in particular.

There should at least be a way to disable this behavior. I have a spotty internet connection, and I'm having to go in and unpause valid feeds all the time.
Is there already a patch "pending"? Does anyone know? Otherwise I'm writing a patch for this.
Although, the thing is: I'm using Thunderbird on three platforms - and I don't wand to patch it for each platform locally -> it should integrate to the codebase.

Comment 6

[Ben]

the last part of my last comment should be (first day on bugzilla - sorry 😁):

(In reply to gravylake from comment #3)

There should at least be a way to disable this behavior. I have a spotty internet connection, and I'm having to go in and unpause valid feeds all the time.

Is there already a patch "pending"? Does anyone know? Otherwise I'm writing a patch for this.
Although, the thing is: I'm using Thunderbird on three platforms - and I don't wand to patch it for each platform locally -> it should integrate to the codebase.

Comment 7

[Ben]

Attachment: Bug 1656651 - adding a prefs flag to disable the 'Pause Updates' on feeds when fetching a feed fails

If no config changes the current behavior of setting feeds to be disabled is not changed.
If a config "rss.disable_feeds_on_failure" is added with the value "true" then feeds won't get disabled when the feed source responds with an error.

Comment 8

[Ben]

So... For anyone who's interested in that matter, here's a patch (to disable that behavior by configuration).
On top of that someone can build retry mechanisms on top of that if needed.

Now I need someone to get it reviewed.

Comment 9

[enone]

(In reply to Ben from comment #8)

So... For anyone who's interested in that matter, here's a patch (to disable that behavior by configuration).
On top of that someone can build retry mechanisms on top of that if needed.

Now I need someone to get it reviewed.

Yes Please!!! I Need It Since Sometimes My PC Loose Internet Connection And All RSS Feed Stop Working Until I Restart Thunderbirth

Comment 10

[Ben]

(In reply to enone from comment #9)

Yes Please!!! I Need It Since Sometimes My PC Loose Internet Connection And All RSS Feed Stop Working Until I Restart Thunderbirth

👍🏻

You know, you can reactivate the updates by just selecting the most top folder and click on "Get Messages"? There shouldn't be the necessity to restart Thunderbird. At least on my installations (Thunderbird 91.1).

But anyways - I admit, it is pretty annoying that you always have to check and then act upon that...

Comment 11

[enone]

(In reply to Ben from comment #10)

(In reply to enone from comment #9)

Yes Please!!! I Need It Since Sometimes My PC Loose Internet Connection And All RSS Feed Stop Working Until I Restart Thunderbirth

👍🏻

You know, you can reactivate the updates by just selecting the most top folder and click on "Get Messages"? There shouldn't be the necessity to restart Thunderbird. At least on my installations (Thunderbird 91.1).

But anyways - I admit, it is pretty annoying that you always have to check and then act upon that...

Yes I Know But It's More Easily For Me To Close And Restart Thunderbirth Instead Of Scrolling Trough Tons Of Accounts And Find The "News" Item And Right Click And Select "Download Messages"...

Comment 12

[Ben]

The current patch attached to this bug only addresses the mitigation of the fact that feeds currently get disabled automatically.
There's no fancy retry mechanism built into this and neither is there logging into the Activity Manger.

How do we want to proceed with this, after the patch gets applied? There are too much "feature requests" for just one filing, in my opinion. Does it make sense to transform this bug into a meta-issue? Is that even possible?

(In reply to Matt from comment #0)

+++ This bug was initially created as a clone of Bug #1524241 +++

Bug 1524241 contains a discussion of how the pause is by design however the pausing is not an acceptable way to manage the situation. The inability to connect on this occasion should not lead to a parmantent failure to execute the download of the feed.

I would expect 30 minutes later that the feed would again be checked and the pause would be reset if a successful connection occurred, I leave my Thunderbird running some times for days until I get home to find nothing has downloaded for the past week because an automatic function has determined that it should not be automatic.

This time around I find my feeds have not updated since the 17 July, almost 2 weeks despite a number of restarts which also should have removed this as a startup is basically a get all mail.

Flagging the reason for the error in the error console is also very poor design and needs to be improved. There should be errors entered into the activity manager and the status bar when a failure occurs, not hidden away in a panel that is buried in "developer Tools" on the menu.

Comment 13

[alta88]

There is no such thing in the protocol as 404 - Hi I'll be back, sometime, maybe, keep trying vs 404 - I'm gone forever. So any comment based on pet feeds or attempting to predict anything about a 404 is uninformed.

A correct balanced solution for temporary vs. permanent 404 errors is something like a doubling time algo, similar to that found when chat attempts to connect to an erroneous server. This patch is unsophisticated and simply defeats preventing useless network traffic, thus adding to rising sea levels and the end of civ. ;)

If a restart is not re-checking all system (vs user) disabled feeds then that's an unintentional bug.

There is plenty of passive information (icon indicator, detailed logs) for the user to maintain their feeds. Feeds die all the time, urls are changed all the time (without sending notice for Tb to autoupdate the url - which it can do) and there is no way around user maintenance of their subscribed feeds.

It would be nice to send selected information to Activity Manager. It is incorrect to send non user initiated biff errors to status bar, or interrupt via a dialog, etc. For a user initiated Get Messages on a feed folder, the error is sent to statusbar already.

f- on this patch.

Comment 14

[Ben]

(In reply to alta88 from comment #13)

There is no such thing in the protocol as 404 - Hi I'll be back, sometime, maybe, keep trying vs 404 - I'm gone forever. So any comment based on pet feeds or attempting to predict anything about a 404 is uninformed.

No one ever said that this is HTTP protocol behavior (I assume devs here know how HTTP works) - this is behavior of shi**y implemented HTTP-services (have a look at nitter.net RSS feeds, for example).

And if the problem is permanent - you still have the passive indicators, even with that patch, so the user can act on it.

"f- on this patch.": You don't have to use this. The config is passive, as in: nothing changes for you (or any other user) if you don't explicitly change this configuration switch.

Nice to meet you, tho...

Comment 15

[Ben]

And yeah, you're right, there are definitely better ways to compensate these service errors. Unfortunately my Thunderbird code base skills are virtually non-existent. But it's an improvement for users - and see, we can even make this improvement even better over time.

Comment 16

[Matt]

I see this patch addressing my personal needs, and probably those of most of the folks afflicted with this terrible design. As pointed out by Alta there is no way to determine if a 404 is permanent, so why are we treating a single occurrence like it is a permanent going off-line. Sure, a more sophisticated result could automatically modify the retry frequency for the individual feed for a much better result, but someone used a nuclear bomb to fix a loose nail to get us here. There were no niceties or consideration. Just forget trying again until a restart. This bug is an attempt to mitigate that appalling lack of respect for the user's choice to leave the feed configured.

Perhaps we should just renounce the users' capacity to know, and just delete the feed on the first failure. Why try again after a restart at all. It was a permanent failure. Apparently someone thought it was not permanent, and that is at the heart of my desire to be able to force Thunderbird to try what I want it to try, not just stop and refuse to continue like a jibing horse.

In recent news to the developer community, support issues would indicate I am not the only one leaving Thunderbird on 24/7 Often enough we see folk taking the time to create a Firefox account, navigate the convoluted user interface that is designed to discourage posting of new questions. To finally post a question asking how to fix things. This is before they even consider restarting their application, let alone the device.

Simply failing to proceed is not an acceptable situation. Waiting for a restart which on my old device took well over 5 minutes with repeated interruptions over passwords was not an acceptable approach to fixing something that I knew was troublesome. My network connection. Perhaps developers would benefit from a defective router or DNS server in the network, so they can get a grip on those working and living in remote areas are putting up with. Basically, the software needs to be fault-tolerant and refusing to continue is not fault tolerance.

If we are having activities that require a restart, we are having activities that are a workaround in the users mind! Our design is not meeting modern user expectations.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5126e0062525
adding a prefs flag to disable the 'Pause Updates' on feeds when fetching a feed fails. r=mkmelin

Comment 18

[enone]

(In reply to Pulsebot from comment #17)

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5126e0062525
adding a prefs flag to disable the 'Pause Updates' on feeds when fetching a feed fails. r=mkmelin

Ok Many Thanks mkmelin!
Now How We Can Use The Fixed Thunderbird Version?
I've Check If There Is Some Updates But On release Channel There Is No New Version Of Thunderbirth To Install...
Thanks Again!

Comment 19

[Magnus Melin [:mkmelin]]

It's only on daily so far. But should be very backportable to 91 - follow this bug for updates.

Comment 20

[Ralf Jung]

[...] 404 [...]

My network connection. Perhaps developers would benefit from a defective router or DNS server in the network, so they can get a grip on those working and living in remote areas are putting up with.

I must be missing something here, since to my knowledge failing DNS or routers will not lead to 404s. They will lead to a totally different kind of error that quite clearly indicates a (potentially transient) network problem. Does Thunderbird currently treat all errors the same? If so, that might be a good starting point for improvements.

Comment 21

[Ben]

As far as I could see (comments in the source code) network connection issues shouldn't cause feeds to disable a feed:

A bad html response code or cert error will cause the url to be disabled, while general network connectivity errors applying to all urls will not.

(In reply to Ralf Jung from comment #20)

[...] 404 [...]

My network connection. Perhaps developers would benefit from a defective router or DNS server in the network, so they can get a grip on those working and living in remote areas are putting up with.

I must be missing something here, since to my knowledge failing DNS or routers will not lead to 404s. They will lead to a totally different kind of error that quite clearly indicates a (potentially transient) network problem. Does Thunderbird currently treat all errors the same? If so, that might be a good starting point for improvements.

Comment 22

[Magnus Melin [:mkmelin]]

Comment on attachment 9240701 [details]
Bug 1656651 - adding a prefs flag to disable the 'Pause Updates' on feeds when fetching a feed fails

[Approval Request Comment]
User impact if declined: it's a new feature to workaround problems with network/server causing feeds to pause when they may not
Testing completed (on c-c, etc.): c-c and beta
Risk to taking this patch (and alternatives if risky): low

Comment 23

[Wayne Mery (:wsmwk)]

Comment on attachment 9240701 [details]
Bug 1656651 - adding a prefs flag to disable the 'Pause Updates' on feeds when fetching a feed fails

[Triage Comment]
Approved for esr91

Comment 24

[Rob Lemley [:rjl]]

Thunderbird 91.2.1:
https://hg.mozilla.org/releases/comm-esr91/rev/1a4d0c7a9c34