957137 - Manually "Updating add-ons" now takes forever (happens often but not every time)

Status: RESOLVED WORKSFORME

(Toolkit :: Add-ons Manager, defect)

Description

[Tony Mechelynck [:tonymec]]

I'm reporting this in Toolkit because AFAIK SeaMonkey's add-on manager comes straight from Toolkit with no changes.

Symptom: When I click "Check for updates" in the rolldown widget near the top of the add-on manager, "Updating add-ons" appears (which is OK) but then it stays there forever (which is not OK) instead of changing, after some "reasonable" length of time, to either "No updates found" or some text which I havent remembered exactly but resembles "View available updates". The available updates tab does appear when the _first_ update (if any) is found, which is OK, but the "Updating add-ons" message does not disappear — apparently it is never decided (or at least after half an hour it is still not known, which is considerably more than before) whether _all_ updates (if any) have been found.

Last good:
seamonkey-2.24a1.en-US.linux-x86_64 (built on Toolkit 27.0a1)
20130919003001
http://hg.mozilla.org/mozilla-central/rev/803189f35921
http://hg.mozilla.org/comm-central/rev/1ea1fc3586db

First bad:
seamonkey-2.26a1.en-US.linux-x86_64 (built on Toolkit 29.0a1)
20140103020054
http://hg.mozilla.org/mozilla-central/rev/325c74addeba
http://hg.mozilla.org/comm-central/rev/cd42f16a4f11

Sorry, between those dates there were no SeaMonkey builds for Linux (not for L32 and not for L64). I haven't been able to reproduce on Firefox but also I don't have a Firefox profile whose number of addons even approches that of my SeaMonkey profile. If someone could reproduce on Firefox (or Thunderbird) or on SeaMonkey-for-Windows it would help.

Next I'll attach the "text to clipboard" copy of my "Troubleshooting Information".

I haven't yet tested this in Safe Mode (I will, and let you know). Of course even if the bug disappears in a virgin profile it would not, in this case, mean that one specific add-on is the culprit. Rather, I suspect that I unwittingly passed the limit of what the add-on subsystem can handle (or, who knows?, that the limit was lowered on me while I wasn't looking: I've been told several times that "nobody has significantly more tyhan 10 add-ons enabled" — well, as you'll see, I do).

I'm CC a few people who seemed to express interest today at the SeaMonkey status meeting. Feel free to remove yourselves if you aren't interested after all.

Notes:
- My add-ons are set to update manually (never check automatically)
- I check them manually about once a day, when I'm ready to install the next SeaMonkey nightly (or, when there wasn't any new nightlies, at the time-of-day when I would be ready to install the next nightly if there were one).

Comment 1

[Tony Mechelynck [:tonymec]]

Attachment: "text" copied to clipboard from Troubleshooting Information page

This is from the current state of my profile, as used with the following app version:

Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26a1 ID:20140106003001 c-c:08b8cae1f27e m-c:14ac61461f2a

It has changed a little, but not much, since SeaMonkey started being built again (mostly by updating by latest versions of the few add-ons which had an update, even if the only way I could notice an update was by browsing https://addons.mozilla.org/en-US/seamonkey/extensions/?sort=updated&unreviewed=on&page=1 ).

Oh-oh, my alarm clock is ringing, I'll soon have to leave home (and this computer is a desktop model, not a laptop, IOW I can't take it to town with me). Sounds like the Safe Mode test will have to wait until after midnight Central European time (where winter time is UTC + 1 hour, or Mozilla Standard Time + 9 hours).

Comment 2

[Tony Mechelynck [:tonymec]]

It's a hang but I'm not setting critical because there are ways to get out of the hang without force-crashing.

Also, not getting all update info is not a "dataloss" since browsing AMO "Recently updated" can be used as a workaround.

And I _really must_ go. Seeya later.

Comment 3

[Tony Mechelynck [:tonymec]]

Problem persists in Safe Mode, as follows:
1. (at about 1:00 AM CET) Under "Edit → Preferences → Appearance → When SeaMonkey starts up, open", untick all checkboxes except "Browser" (normally I also have "Mail & Newsgroups" and "Chatzilla" ticked).
2. Help → Restart with Add-ons Disabled
   (and approve all alerts and dialogs until the restart is done)
3. Wait until all tabs (a little over 100 in my case, I think 105 or so) have been reloaded
4. In the add-on manager tab, unroll the rolldown widget, and select "Check for Updates"
-- The text "Updating add-ons" appears left of the rolldown widget, where there was nothing.
5. Sleep over came me in the computer armchair.
6. Half-conscious, I dragged myself to the bed, and slept some more
7. (at 9:00 AM CET) the alarm-clock function in my cellphone warned me that I had an appointment with my psy. I clothed myself and went, not coming even close to the computer.
8. (at 14:10 CET) I'm back at home, and the add-ons manager still says "Updating add-ons". In the rolldown widget, "Check for Updates" is greyed-out. The "Available Updates" tab has appeared (but is not yet the active tab, of course), and has the number (2) next to its title.
9. Let's select the Available Updates tab. There are updates for the "Lightning 3.1a2nightly" calendar extension and for the "GNOMErunner/GTK Revived 0.3.1" complete theme, both of which are disabled.
10. Let's apply the theme update. Lightning I shall fetch in a few minutes from the FTP site instead, together with the latest gdata-provider.xpi (i.e. "Provider for Google Calendar); the way I fetch them ensures that I only apply the new xpi if its datestamp is newer than the datestamp on the previous copy (usually yesterday's) of the same xpi. I'll also update my application nightlies (in .tar.bz2 + .txt form) from other directories on the same FTP server.
11. Let's check AMO for "recent updates" in categories: Extensions, Complete Themes, (lightweight) Themes, and Search engines... Lightning is absent (but that is expected) in category Extensions, GNOMErunner/GTK Revived is present in category Complete Themes, there is one new lightweight theme which doesn't interest me, and no new search engine. No other updates for add-ons already installed, and no new add-ons that I would like to install; IOW, nothing interesting today that wasn't already found.

Comment 4

[Tony Mechelynck [:tonymec]]

Oh, and I forgot: Before leaving Safe Mode,
12. retick "Mail & Newsgroups" and "ChatZilla" as so they won't be forgotten at next ("unsafe") startup. Hm, ChatZilla is of course absent in Safe Mode so I'll have to start it from its chrome icon instead.

Comment 5

[Frank Wein [:mcsmurf]]

So to summarize this is a bit: The "Updating Add-Ons" text stays there forever in safe-mode for you? I think your Comment 3 provided a bit too many details ;)

Comment 6

[Tony Mechelynck [:tonymec]]

(In reply to Frank Wein [:mcsmurf] from comment #5)
> So to summarize this is a bit: The "Updating Add-Ons" text stays there
> forever in safe-mode for you? I think your Comment 3 provided a bit too many
> details ;)

Both in safe-mode and in non-safe-mode (I observed it first in the latter, then tried it in the former and it was there too). The fact that sleep overcame me and that (how many? thirteen?) hours elapsed before I did something, and it was still there, proves that it is a real hang, not just a too-short wait on my part.

Comment 7

[Tony Mechelynck [:tonymec]]

Still trying to troubleshoot the problem:
- most of my add-ons came from AMO (IIUC, they all have an update URL)
- some are non-AMO add-ons with no update URL
- I might or might not have a few non-AMO add-ons with an update URL

Is there some known way of finding out which is which, and I mean, e.g. by getting a list of all update URLs with the corresponding extension name (getting that list, even from an extension which I probably don't yet have), but definitely NOT by painstakingly opening all install.rdf files one after another to get the update URLs manually?

Comment 8

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26a1 ID:20140107003001 c-c:0c8f6b80f939 m-c:ce917d3dd7c8

IOW the first nightly after the one in comment #1.

For some reason "Updating Add-ons" changed almost immediately (a fraction of a second) to "View Available Updates".

I'm reluctant to set WORKSFORME since I've no idea what may have caused the bug to appear, then not to appear. The only difference AFAIK was that this time I started "only the browser" but in non-safe mode; but for the love of me I can't imagine why it would cause the bug to go away.

Let's wait for a week and set WFM if the bug doesn't reappear (and is seen neither by me nor by anyone else in the meantime), unless…

Unfocused: If something relevant was FIXED on January 6 (between the changesets mentioned in comment #1 and at the start of this comment), please say what.

Comment 9

[Tony Mechelynck [:tonymec]]

P.S. Only Lightning (disabled) was found. Maybe the difference is that this time there were no other updates to be found (and in particular, no updates with maxVersion conflict, but which ought to be installed anyway because of default-to-compatible, except that SeaMonkey support is flaky, e.g. when I try to manually install a default-to-compatible add-on from AMO, I'm told first that it is not compatible, and if I decide to override the incompatibility I'm told that I need Fx 10.0 or later) ??????????

Comment 10

[Blair McBride [:Unfocused] (UNAVAILABLE)]

(In reply to Tony Mechelynck [:tonymec] from comment #8)
> Unfocused: If something relevant was FIXED on January 6 (between the
> changesets mentioned in comment #1 and at the start of this comment), please
> say what.

Nothing landed in the Add-ons Manager in that time.

Comment 11

[Tony Mechelynck [:tonymec]]

Problem has reappeared (in same nightly as in comment #8, no SeaMonkey 2.26a1 builds for linux64 at the moment). Unlike comment #8, this was not at startup. As in comment #8, only Lightning was found, and nothing to be found on the various AMO pages mentioned at comment #3 step 11. Let's see what happens if I restart and retry…

Comment 12

[Tony Mechelynck [:tonymec]]

…Problem still there. :-(

Comment 13

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26a1 ID:20140110130040 c-c:f444666b101c m-c:e89afc241513

I had a case of "bug absent", and not at startup. Still cannot tell what makes it appear or disappear. I'm pushing back the CLOSEME date because of comment #12.

Comment 14

[Philip Chee]

WAG: How many of your extensions have a custom update url (i.e. that doesn't point to a.m.o)? I know that the mozdev server has been having problems recently. There are also orphaned extensions with home sites that are 404.

Comment 15

[Tony Mechelynck [:tonymec]]

(In reply to Philip Chee from comment #14)
> WAG: How many of your extensions have a custom update url (i.e. that doesn't
> point to a.m.o)? I know that the mozdev server has been having problems
> recently. There are also orphaned extensions with home sites that are 404.

I don't know. Is there any way to find that out, other than by viewing all install.rdf's one by one, including those for disabled add-ons of course? Is there even an add-on that would list those update URLs? (AFAIK nothing does it out of the box)


Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26a1 ID:20140114003001 c-c:0456c50549f6 m-c:34dddf6f7ec1

"Updating add-ons" remained forever today, the "Available updates" tab did not appear, IOW nothing was found, and yet, at the least, "Lightweight Themes Manager" (a restartless add-on from AMO) should have been.

Comment 16

[Tony Mechelynck [:tonymec]]

P.S. Just counted the lines with either "Enabled: true" or "Enabled: false" in the "Extensions" part of the clipboard output of about:support. There are 117 of them. I think it is maybe a couple more than in the attachment. I might, someday, open all those 117 install.rdf (plus those for themes, dictionaries and langpacks) manually but it would be a PITA. First I'll do some research about having it done either by SeaMonkey or by some external program.

Comment 17

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26a1 ID:20140116003001 c-c:76d9545cff20 m-c:1f835fe670d7

Now you see it, now you don't.

About to update to the next nightly; "Update add-ons" did become "No updates found" this time.

Comment 18

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 SeaMonkey/2.26a1 ID:20140117003001 c-c:385ea939531d m-c:9bcc52594322

Took more than I would have expected but it finally changed to "No updates found" after all.

Maybe I was just too impatient? The delay mentioned in comment #6 (summarizing comment #3) does seem too long. I can only explain it (if not a real hang) by the add-ons manager being swapped out to do something else (getting mail? getting chat traffic?)

Comment 19

[Tony Mechelynck [:tonymec]]

The following (after the line of dashes) is a list (obtained the hard way) of all my "custom updated" extensions, with their updateURLs. I tried to make it both human- and machine-readable, with one line per add-on for readability.

- This does not include anything not contained in the file named install.rdf at top level of a directory, .xpi or .jar. This also means that "variant file names" like install0.rdf, install.rdf.4, install.old.rdf (all these were found, and more) were not checked.
- .jar files found within an .xpi (for instance, the chrome/console2.jar found inside the .xpi for Console²) have not been checked.
- No mention has been made of add-ons where the tag <updateURL> (possibly with an appropriate namespace) was not found, or was found only in a <!-- comment -->.
- Add-ons were checked regardless of their type or enabled/disabled status.
- For each add-on listed, the ID, name and updateURL have been listed. Nothing else, and in particular, updateKey and updateInfoURL, even if present, were omitted. Of course, additional data can be provided if you ask for it, but remember that all this was obtained manually, by checking all add-ons one by one, so if you need more info about some add-on(s), please be as specific as you can.
- The string &amp; has everywhere been replaced by a single & character; o.t.o.h., %SOMETHING% placeholders have not been altered.

----------------------------------


[{id:'[e-mail]', name:'Firebug', updateURL:'http://getfirebug.com/releases/firebug/1.5/update.rdf'}
,{id:'{0538E3E3-7E9B-4d49-8831-A227C80A7AD3}', name:'Forecastfox', updateURL:'https://static.getforecastfox.com/extension/firefox/update.rdf'}
,{id:'{5ffa002f-7e75-4b01-93aa-9c102f58b0ad}', name:'deviantSkin', updateURL:'http://www.bluespeed9.com/deviantART/deviantSkin/update.rdf'}
,{id:'{e2fda1a4-762b-4020-b5ad-a41df1933103}', name:'Lightning', updateURL:'https://calendar.mozilla.org/update.php?buildID=20140119030203&appABI=%APP_ABI%&appOS=%APP_OS%&locale=%APP_LOCALE%&appVersion=%APP_VERSION%&appID=%APP_ID%'}
,{id:'[e-mail]', name:'Diagnostics for Adblock Plus', updateURL:'https://adblockplus.org/devbuilds/abpwatcher/update.rdf?reqVersion=%REQ_VERSION%&id=%ITEM_ID%&version=%ITEM_VERSION%&maxAppVersion=%ITEM_MAXAPPVERSION%&status=%ITEM_STATUS%&appID=%APP_ID%&appVersion=%APP_VERSION%&appOS=%APP_OS%&appABI=%APP_ABI%&locale=%APP_LOCALE%&currentAppVersion=%CURRENT_APP_VERSION%&updateType=%UPDATE_TYPE%'}
,{id:'jid0-Yia1AkUofBLANOmOYle1V8n713I@jetpack', name:'Facebook Acquaintances', updateURL:'https://secure.toolness.com/xpi/facebook-acquaintances.update.rdf'}
,{id:'keyconfig@dorando', name:'keyconfig', updateURL:'http://mozilla.dorando.at/keyconfig/update.rdf'}
,{id:'{1280606b-2510-4fe0-97ef-9b5a22eafe80}', name:'Console²', updateURL:'http://console2.mozdev.org/update.xml'}
]

Comment 20

[Tony Mechelynck [:tonymec]]

Let's set extensions.{GUID}.update.enabled to false for some of the extensions mentioned in comment #19. At the moment I'm choosing disabled extensions which I suspect might be abandoned.

Comment 21

[Tony Mechelynck [:tonymec]]

At the first run of the first 2.27a1 nightly, the (expected) compatibility-checking dialog came up. Then it got hung near the end (hard to say if there still was a pixel or two to go in the progress dialog), saying "Finished checking Provider for Google Calendar". 37 lines "*** WARN addons.updates: Request timed out" at the (temporary) end of the sysout/syserr log as shown by the tail utility.

Ten I clicked "Cancel" and the program got out of hang. Of course I don't know what (if anything) SeaMonkey "forgot" to check for compatibility. I'll attach the log from that run in case anyone wants to look at it. The "manual cancel" corresponds to line 530 but I notice repeated "too much recursion" errors before that.

Comment 22

[Tony Mechelynck [:tonymec]]

Attachment: sysout/syserr log from first run of first SeaMonkey 2.27a1 nightly

Comment 23

[Tony Mechelynck [:tonymec]]

(Workaround?)
This bug seems not to happen when I update my add-ons this way:
1. Update Lightning only (downloaded by FTP to local disk).
2. Update gdata-provider only (downloaded by FTP to local disk).
3. "Check for updates" by means of the add-ons manager rolldown widget.
Otherwise I see it every time, or almost.

Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0 SeaMonkey/2.27a1 ID:20140214003001 c-c:1ce77c2f9bd0 m-c:d275eebfae04

N.B. This is the latest SeaMonkey L64 trunk nightly.

Comment 24

[Tony Mechelynck [:tonymec]]

P.S. Lightning and gdata-provider are both user-disabled.

Comment 25

[:Irving Reid (No longer working on Firefox)]

The 'too much recursion' error in your log is probably the clue - there are a few cases in the addon manager where we iterate through a list of add-ons by recursively stacking callbacks, and your very large set of add-ons could be blowing our stack space.

Bug 987512, where we hope to convert Add-on Manager's callback APIs to Promises, will help a lot with this, but it might be worth fixing a couple of the worst spots to resolve this bug before we take on the larger project.

Comment 26

[Tony Mechelynck [:tonymec]]

(In reply to :Irving Reid from comment #25)
> The 'too much recursion' error in your log is probably the clue - there are
> a few cases in the addon manager where we iterate through a list of add-ons
> by recursively stacking callbacks, and your very large set of add-ons could
> be blowing our stack space.
> 
I suppose this huge set of add-ons is testing the add-ons manager "beyond its means" — extreme testing, if you will. The fact that upgrading Lightning separately before the rest makes the bug disappear (Lightning takes an unusually long time to install AFAICT) might mean that I'm just at the boundary of what the add-ons manager can handle, what do you think?

> Bug 987512, where we hope to convert Add-on Manager's callback APIs to
> Promises, will help a lot with this, but it might be worth fixing a couple
> of the worst spots to resolve this bug before we take on the larger project.

ah, good news. I'll retest (by not upgrading Lightning first) once that bug gets FIXED… *and* SeaMonkey L64 trunk starts building again.

Comment 27

[Tony Mechelynck [:tonymec]]

(In reply to Tony Mechelynck [:tonymec] from comment #23)
> (Workaround?)
> This bug seems not to happen when I update my add-ons this way:
> 1. Update Lightning only (downloaded by FTP to local disk).
> 2. Update gdata-provider only (downloaded by FTP to local disk).
> 3. "Check for updates" by means of the add-ons manager rolldown widget.
> Otherwise I see it every time, or almost.
> 
> Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0
> SeaMonkey/2.27a1 ID:20140214003001 c-c:1ce77c2f9bd0 m-c:d275eebfae04
> 
> N.B. This is the latest SeaMonkey L64 trunk nightly.

Spoken too fast?

Went to bed this night as Lightning was "installing" so I don't know how long it took; when I came back it had been "installed successfully". Gdata-provider updated as fast as usual. Now SeaMonkey has been "updating add-ons" for more than half an hour. I'm going to my mother's for lunch and I'll see what it says when I come back. Memory is in unusually high use: up to 90% of 3.2 GiB live RAM + slightly less than half of 6 GiB swap (of which half is partition swap, preferred, and the rest is file swap).

SeaMonkey version is still
Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0 SeaMonkey/2.27a1 ID:20140214003001 c-c:1ce77c2f9bd0 m-c:d275eebfae04
since the Linux builds are still perma-red.

Comment 28

[Tony Mechelynck [:tonymec]]

Part of this comment might be off-topic; developers will judge.

When I came back from my mother's around 17:30 my time (CEST, or GMT + DST + 1h, or Mozilla time + 9 hours), keyboard & mouse were so sluggish because of heavy swapping that I finally lost patience and rebooted; but before I did, I noticed that:
- Add-ons Manager was still saying "Updating add-ons"
- The DSL connection had gone down (which is usually not the case when this bug hits me)
- No visible sign of a missing Internet connection appeared in the Add-ons Manager, not even an error banner, and certainly not a big XUL error page as I would get (with the prefs I have) if a normal HTTP or HTTPS URL typed in the Location Bar couldn't be reached. (I didn't think of looking in the error console, but "ifconfig" in a root tty showed that interface dsl0 was not present.)

The latter ("no visible sign") makes me ask: what happens if the Internet connection gets lost while the add-ons manager is looking for add-on updates? This could happen for a number of reasons, both local and remote, e.g. my ISP has the nasty habit of cutting any ADSL connection whose uptime reaches a certain value, which used to be 36 hours but has IIUC been upped to 96 hours (, zero minutes, sometimes one second, according to my connections detail); still well within what can be expected of a typical Linux system where (for the bandwidth used) Internet use is paid per month, not per connection second or per megabyte.

Comment 29

[Tony Mechelynck [:tonymec]]

In reply to comment #23:

AFAICT, I see the bug every time *except* if Lightning has been updated all by itself immediately before (with possibly some updates of other smaller extensions and/or complete themes in between). For this to work, the Lightning update must not be started before all startup tabs have finished being loaded (otherwise the Lightning update itself usually takes forever).

Comment 30

[Tony Mechelynck [:tonymec]]

(In reply to :Irving Reid from comment #25)
> The 'too much recursion' error in your log is probably the clue - there are
> a few cases in the addon manager where we iterate through a list of add-ons
> by recursively stacking callbacks, and your very large set of add-ons could
> be blowing our stack space.
> 
> Bug 987512, where we hope to convert Add-on Manager's callback APIs to
> Promises, will help a lot with this, but it might be worth fixing a couple
> of the worst spots to resolve this bug before we take on the larger project.

I found an unofficial build of SeaMonkey 2.31a1, namely:

Mozilla/5.0 (X11; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0 SeaMonkey/2.31a1 ID:20140723133406 c-c:acb745a4d796 m-c:d0f6259e8446

After upgrading from the 2.27a1 build mentioned in comment #23, the "compatibility check after version change" did *not* take forever (with the 2.27a1 it did): ChatZilla, then Mail And Browser, started up with no need for me to "cancel" the compatibility check.

Then I tried checking for updates *without* reinstalling Lightning immediately before: it went remarkably fast, and found an undate for Lightning with a bogus version number of 3.6a2nightly, which is the Lightning trick to force-update the 3.6a1 nightly without changing its version. (I update Lightning myself from the FTP site once a day, as part of the FTP connection where I check for updates to browser and mailer nightlies.)

I'll try again tomorrow and the day after, and if the bug does not reappear I'll tentatively resolve it WORKSFORME for trunk. Whether to port any possible "worst-spot fixes" to aurora and/or beta (if not already done) is of course your pidgin (and the Add-on Manager team's).

Comment 31

[Tony Mechelynck [:tonymec]]

This build is already "unaffected" by the bug:

Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0 SeaMonkey/2.28 ID:20140721085404 c-c:65d4015ac385 m-c:6befadcaa685

I'm resolving WORKSFORME with Milestone to mean "I don't know which changeset fixed it, or in which bug, but it was at this release".