454108 - Update Hunspell to version 1.2.8

Status: VERIFIED FIXED

(Core :: Spelling checker, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Attachment: Update Hunspell 1.1.12 to 1.2.7

The current version of Hunspell in the tree is version 1.1.12. Hunspell has since been gone through numerous updates, the latest being 1.2.7, with many important fixes going in. Memory usage has improved, suggested replacements have been improved, and many other bugs have been fixed.

This patch applies cleanly to the trunk and compiles fine on both Linux and Windows.

Tryserver builds:
https://build.mozilla.org/tryserver-builds/2008-09-07_13:[e-mail]-1220818647/

Scott, can you rubberstamp this?

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Mike, any word on who's taking over spellchecker? I'd really like to get this in for 3.1b1...

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 337339 [details] [diff] [review]
Update Hunspell 1.1.12 to 1.2.7

Per #developers, changing the sr request.

Comment 3

[Mike Connor [:mconnor]]

Comment on attachment 337339 [details] [diff] [review]
Update Hunspell 1.1.12 to 1.2.7

moving this to dmose, who probably can do this sooner than I can (and owes me some mail on updates for the owner here)

Comment 4

[Giacomo Magnini]

Maybe this will solve also bug 422982?

Comment 5

[Németh László]

(In reply to comment #4)
> Maybe this will solve also bug 422982?

Unfortunately, this is not a spell-checker, but a tokenizer related problem.

Comment 6

[Németh László]

Ryan, I have made an important patch for Hunspell 1.2.7, it removes some incorrect spelling suggestions:

http://sourceforge.net/tracker/index.php?func=detail&aid=2124186&group_id=143754&atid=756395

Best regards, László

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 337339 [details] [diff] [review]
Update Hunspell 1.1.12 to 1.2.7

I'm told that 1.2.8 is coming very shortly, so you're off the hook for now, dmose :)

Comment 8

[Olli Pettay [:smaug][[e-mail]]]

Ryan, is there perhaps a list of changes between 1.1.12 - 1.2.8?
Are there any new things which might require a security review?

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Smaug, detailed changelogs are available on the project's SourceForge page.

1.2 (1.2.1) - http://sourceforge.net/project/shownotes.php?release_id=551113&group_id=143754
1.2.2 - http://sourceforge.net/project/shownotes.php?release_id=569065&group_id=143754
1.2.3 - http://sourceforge.net/project/shownotes.php?release_id=607512&group_id=143754
1.2.4 - http://sourceforge.net/project/shownotes.php?release_id=607577&group_id=143754
1.2.5 - http://sourceforge.net/project/shownotes.php?release_id=612345&group_id=143754
1.2.6 - http://sourceforge.net/project/shownotes.php?release_id=613618&group_id=143754
1.2.7 - http://sourceforge.net/project/shownotes.php?group_id=143754&release_id=619954
1.2.8 - Not released yet. I'll post a link to the changelog when it is.

Some new components have been added, such as the hunzip library, so it may require security review. I really don't know what the criteria are for determining that, though. I suppose the sooner I can get someone to look at the code, the sooner we can get an answer to that.

Comment 10

[Olli Pettay [:smaug][[e-mail]]]

Seem like there are many good things but also something not so nice:
http://sourceforge.net/project/shownotes.php?release_id=551113&group_id=143754
"Hungarian spell checking is 25% slower on most frequent words of Hungarian Webcorpus."

Comment 11

[Németh László]

(In reply to comment #10)
> Seem like there are many good things but also something not so nice:
> http://sourceforge.net/project/shownotes.php?release_id=551113&group_id=143754
> "Hungarian spell checking is 25% slower on most frequent words of Hungarian
> Webcorpus."

25% is indifferent for the normal spell checking. In fact, English spell checking is five times (500%) faster, than Hungarian spell checking (~50 thousand words/sec and ~10 thousand words/sec), thanks to the simple English morphology. Also the suggestion speed depends from the language complexity and the suggestion strategies much better, than the new condition checking algorithm.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Attachment: Fix for comment #6

Here's a followup patch for the issue mentioned in comment #6. This applies cleanly over the first patch.

At this point, I don't know if 1.2.8 will be ready in time for the 3.1b2 freeze. Is there any chance we can go ahead and land 1.2.7 with this fix for now?

Comment 13

[Németh László]

(In reply to comment #12)
> Is there any chance we can go ahead and land 1.2.7 with this fix for
> now?

Hunspell 1.2.8 is out. New features and bug fixes:
https://sourceforge.net/project/shownotes.php?group_id=143754&release_id=637489

Regards, László

Comment 14

[Olli Pettay [:smaug][[e-mail]]]

László, can you estimate how "stable" 1.2.8 is? How likely there will be regressions from 1.1.12? I was told that OOo 3 uses still 1.1.12 by default,
so I guess 1.2.x isn't tested as well as 1.1.x.

I hope we could take 1.2.8. Ryan, will you have time to create the patch?

Comment 15

[Németh László]

1.2.x Hunspell branch is there in several Linux distributions (If I right remember, also in the Hungarian version of OpenOffice.org 2.4) and all detected affix condition checking problems of 1.2.x branch are fixed in Hunspell 1.2.8. There are also 95 other regression test in Hunspell distribution with Valgrind memcheck. I have also successfully checked the spell checking and suggestion on ~100 thousand English words (/usr/share/dict/words). We use Hunspell 1.2 in a full text search database, as indexing engine (the database contains 1 million articles).

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Attachment: Update Hunspell 1.1.12 to 1.2.8

OK, here's a new patch for 1.2.8. I've built this successfully on Windows (and I'm posting this message from the resulting build) and the TryServer built it OK on Linux (the win32 TryServer is on the fritz right now - see bug 462573 for more details)
https://build.mozilla.org/tryserver-builds/2008-11-02_15:[e-mail]-hunspell-update/

Smaug, dmose, mconnor - Let me know if you need anything else.

Comment 17

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 345994 [details] [diff] [review]
Update Hunspell 1.1.12 to 1.2.8

If we want to take this, it must happen now. If there are any significant regressions, we shouldn't be reluctant to backout, IMO.
Beta2 should give good enough baking time...

dmose, mconnor, ok to you?

Comment 18

[Ryan VanderMeulen [:RyanVM]]

mconnor said fine by him in #developers

Comment 19

[Ryan VanderMeulen [:RyanVM]]

http://hg.mozilla.org/mozilla-central/rev/6c4d0ddbe997

Comment 20

[chris hofmann]

possible regression in  Bug 525581