1133363 - Please update en-US Hunspell dictionary to latest version

Status: RESOLVED FIXED

(Core :: Spelling checker, defect)

Comment 1

[Kevin Atkinson]

Please update the en-US Hunspell dictionary to latest upstream version that you can find at http://wordlist.aspell.net/dicts/.

The latest version includes a large number of new words, including many neologisms (newly invented words) such as "selfie" and "smartwatch", and some not so new words that should have been added a long time ago such as "inbox".

It would also be nice if you could review you list of additional words and submit any that are not already added in the latest version using the GitHub issue tracker at https://github.com/kevina/wordlist/issues.

Thanks,
Kevin Atkinson
Upstream Maintainer on en_US Hunspell dictionary.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Ekanan, is this something you'd be interested in working on by chance? :)

Comment 3

[Ekanan Ketunuti]

Yeah, I think I can pull it off.

Comment 4

[Ekanan Ketunuti]

Attachment: Part 1 - add missing /M suffix to en-US dictionary

Comment 5

[Ekanan Ketunuti]

Attachment: Part 2 - add new words to en-US dictioanry

Comment 6

[Ekanan Ketunuti]

Ryan, 

Ehsan is afk. can you review this?

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Unfortunately, I'm not a peer for this code :(. He'll be back this week, though. Thanks for taking this!

Comment 8

[(no longer active)]

Comment on attachment 8565340 [details] [diff] [review]
Part 1 - add missing /M suffix to en-US dictionary

Review of attachment 8565340 [details] [diff] [review]:
-----------------------------------------------------------------

It's a bit sad to lose the current model of storing our local modifications on top of the upstream version.  I originally wanted to ask you to reconstruct a new upstream-hunspell.diff on top of the new baseline, but thinking more about this, that would require some work which you may or may not want to do.  (If you're up to do that, I would really like to review a separate patch, but that can happen in its own bug!)

But can you please add a new hunspell-en_US-20150215.dic file as a snapshot of the new upstream dictionary?

Comment 9

[(no longer active)]

Ekanan, can you please generate a unified version of upstream-hunspell.diff with all of our modifications on top of the new upstream version (which includes the old Chromium modifications that we took years ago as well) and submit it to upstream as per comment 1?  I think Kevin would be interested in having a look at it, and see which ones he would like to take upstream.  That should hopefully help benefit upstream from our work in our fork.  :-)

Thanks!

Comment 10

[Ekanan Ketunuti]

Attachment: upstream-hunspell.diff

diff -ru hunspell-en_US-2015.02.15/en-US.dic extensions/spellcheck/locales/en-US/hunspell/en-US.dic

Comment 11

[Ekanan Ketunuti]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #8)
> Comment on attachment 8565340 [details] [diff] [review]
> Part 1 - add missing /M suffix to en-US dictionary
> 
> Review of attachment 8565340 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> It's a bit sad to lose the current model of storing our local modifications
> on top of the upstream version.  I originally wanted to ask you to
> reconstruct a new upstream-hunspell.diff on top of the new baseline, but
> thinking more about this, that would require some work which you may or may
> not want to do.  (If you're up to do that, I would really like to review a
> separate patch, but that can happen in its own bug!)
> 
> But can you please add a new hunspell-en_US-20150215.dic file as a snapshot
> of the new upstream dictionary?

I tried adding a new hunspell-en_US-20150215.dic file into dictionary-sources and modified edit-dictionary and merge-dictionaries and generated upstream-hunspell.diff by |diff -r hunspell-en_US-20150215.dic ../en-US.dic > upstream-hunspell.diff|.

When I run sh merge-dictionaries, a lot of dupe words are found. Maybe I did something wrong. :(

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/271171d2c620
https://hg.mozilla.org/integration/mozilla-inbound/rev/4f2bdb9b4d5e

Comment 13

[Carsten Book [:Tomcat]]

sorry backed this out for test failures like https://treeherder.mozilla.org/logviewer.html#?job_id=6956225&repo=mozilla-inbound

Comment 14

[Ekanan Ketunuti]

Attachment: Part 3 - Fix tests

Now, lolcat is not an invalid word.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d78c3ed22c93

Comment 15

[(no longer active)]

Comment on attachment 8569138 [details] [diff] [review]
Part 3 - Fix tests

Review of attachment 8569138 [details] [diff] [review]:
-----------------------------------------------------------------

Hahaha :)

Comment 16

[(no longer active)]

(In reply to Ekanan Ketunuti from comment #11)
> (In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from
> comment #8)
> > Comment on attachment 8565340 [details] [diff] [review]
> > Part 1 - add missing /M suffix to en-US dictionary
> > 
> > Review of attachment 8565340 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > It's a bit sad to lose the current model of storing our local modifications
> > on top of the upstream version.  I originally wanted to ask you to
> > reconstruct a new upstream-hunspell.diff on top of the new baseline, but
> > thinking more about this, that would require some work which you may or may
> > not want to do.  (If you're up to do that, I would really like to review a
> > separate patch, but that can happen in its own bug!)
> > 
> > But can you please add a new hunspell-en_US-20150215.dic file as a snapshot
> > of the new upstream dictionary?
> 
> I tried adding a new hunspell-en_US-20150215.dic file into
> dictionary-sources and modified edit-dictionary and merge-dictionaries and
> generated upstream-hunspell.diff by |diff -r hunspell-en_US-20150215.dic
> ../en-US.dic > upstream-hunspell.diff|.
> 
> When I run sh merge-dictionaries, a lot of dupe words are found. Maybe I did
> something wrong. :(

Are they words that appear in <https://dxr.mozilla.org/mozilla-central/source/extensions/spellcheck/locales/en-US/hunspell/dictionary-sources/chromium_en_US.dic_delta>?  If yes, you should remove them from the chromium dictionary.

Comment 17

[Ekanan Ketunuti]

Uh, I think it came from chromium. I will try again tomorrow.

Comment 18

[Ekanan Ketunuti]

I found an error when merge dictionary:

> sh merge-dictionaries Patching Chromium dictionary
> patching file chromium_en_US.dic_delta-patched
> Hunk #1 FAILED at 1.
> 1 out of 1 hunk FAILED -- saving rejects to file chromium_en_US.dic_delta-patched.rej
> Patching Hunspell dictionary
> patching file hunspell-en_US-20150215.dic-patched
> Updating Chromium affixes
> licence//101
> licences/14
> licencing/14
> merge-dictionaries: 51: merge-dictionaries: warn: not found

This is harder than I thought. I rather do not invest time to look into it, so I give up on that solution. :(

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ece86f9a9c3b
https://hg.mozilla.org/integration/mozilla-inbound/rev/d594db3c891b
https://hg.mozilla.org/integration/mozilla-inbound/rev/6a21d167e9f4

Comment 20

[Kevin Atkinson]

I am a bit confused what is going on.

Is this a still a work in progress or have the changes been committed?

I would also really hate to lose the "current model of storing our local modifications on top of the upstream version".  It would be much easier for me to review additions if I knew the reason they were added.

Comment 21

[Carsten Book [:Tomcat]]

(In reply to Kevin Atkinson from comment #20)
> I am a bit confused what is going on.
> 
> Is this a still a work in progress or have the changes been committed?
> 
> I would also really hate to lose the "current model of storing our local
> modifications on top of the upstream version".  It would be much easier for
> me to review additions if I knew the reason they were added.

for the changes in comment #19 those have been commited

Comment 22

[Kevin Atkinson]

I know nothing about Mozilla development.  But if it will help I am willing to investigate some time to fix merge-dictionaries so that you won't have to lose the current model of storing the local modifications on top of the upstream version.  It should not be that difficult for me to get it working again.

Does this sound like something you want?  If so please give be about a week to get this done.

Thanks,
Kevin

Comment 23

[(no longer active)]

(In reply to Ekanan Ketunuti from comment #18)
> I found an error when merge dictionary:
> 
> > sh merge-dictionaries Patching Chromium dictionary
> > patching file chromium_en_US.dic_delta-patched
> > Hunk #1 FAILED at 1.
> > 1 out of 1 hunk FAILED -- saving rejects to file chromium_en_US.dic_delta-patched.rej
> > Patching Hunspell dictionary
> > patching file hunspell-en_US-20150215.dic-patched
> > Updating Chromium affixes
> > licence//101
> > licences/14
> > licencing/14
> > merge-dictionaries: 51: merge-dictionaries: warn: not found
> 
> This is harder than I thought. I rather do not invest time to look into it,
> so I give up on that solution. :(

Can you please attach the chromium_en_US.dic_delta-affix-converted file that is left in the current working directory when you hit that error?  (BTW that error happens because "warn" is not a shell construct (AFAIK) <http://mxr.mozilla.org/mozilla-central/source/extensions/spellcheck/locales/en-US/hunspell/dictionary-sources/merge-dictionaries#51>)  Feel free to move this part to a new bug if needed.

Comment 24

[(no longer active)]

Hi Kevin!

Firstly, the changes have been committed now as Carsten mentioned.

(In reply to Kevin Atkinson from comment #22)
> I know nothing about Mozilla development.  But if it will help I am willing
> to investigate some time to fix merge-dictionaries so that you won't have to
> lose the current model of storing the local modifications on top of the
> upstream version.  It should not be that difficult for me to get it working
> again.
> 
> Does this sound like something you want?  If so please give be about a week
> to get this done.

Yes, this is very helpful!  Right now this diff <https://hg.mozilla.org/integration/mozilla-inbound/raw-file/d594db3c891b/extensions/spellcheck/locales/en-US/hunspell/dictionary-sources/upstream-hunspell.diff> is based on the old hunspell-en_US-20081205.dic dictionary, and I would very much like to have a diff based on the latest en-US dictionary (and then work towards getting that diff to become empty eventually so that we just use the pristine upstream version!).

It seems Ekanan got stuck doing that as per the discussion above, so it would be great if you guys can coordinate so that both of you don't end up doing the same work.  :-)  The main issue is that we have traditionally used two diffs on top of upstream, one from the chromium project (chromium_en_US.dic_delta) and one ours (upstream-hunspell.diff).  As the chromium diff has not been updated in years, perhaps it's best to just merge those words into upstream-hunspell.diff, as there is no real value to maintain both separately.

Please let me know if you need help!

Comment 25

[Kevin Atkinson]

Hi Ehsan,

I will have a look.  As I see it, I would rather not merge chromium_en_US.dic_delta into yours because if I am going to consider words for addition it is very helpful to know where they originated from.

Using diff is not really the best way to manage wordlist changes.  I list of additions and substations (if any) from the upstream dictionary seams like a more reasonable think to do to avoid the problems of conflicts.  Writing a Perl script to manage these lists should not be too hard (I notice the shell script already uses Perl so I assume Perl is an acceptable requirement).  Would that be okay?

In any case I will first try to fix the script and the conflict before I try anything more involved.

Ekanan: Let me know if you are going to put any additional time into this.

Kevin

Comment 26

[Ekanan Ketunuti]

(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #23)
> Feel free to move
> this part to a new bug if needed.

Filed 1137544


(In reply to Kevin Atkinson from comment #25)
> 
> 
> Ekanan: Let me know if you are going to put any additional time into this.
> 
> Kevin

I don't really really know how to handle this Chromium shit. I'd rather spend energy on the other stuff.

Comment 27

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/ece86f9a9c3b
https://hg.mozilla.org/mozilla-central/rev/d594db3c891b
https://hg.mozilla.org/mozilla-central/rev/6a21d167e9f4

Comment 28

[Kevin Atkinson]

Hi Ekanan,

I am very confused on what is going on here and why this bug report was closed and a new one created.

It looks like Ekanan mainly added words and didn't really update the source list.  This is unfortunate as I also removed a number of Junk entries.

I am working on a better solution for you but am now confused where to file it.

BTW: As far as the Chromium list go, I admit it can likely go and at this point is just adding complexity.

Kevin

Comment 29

[Kevin Atkinson]

(In reply to Kevin Atkinson from comment #28)
> Hi Ekanan,

Sorry Ekhanan, I addressed the wrong person I meant to address Ehsan.

Comment 30

[(no longer active)]

(In reply to Kevin Atkinson from comment #25)
> Hi Ehsan,
> 
> I will have a look.  As I see it, I would rather not merge
> chromium_en_US.dic_delta into yours because if I am going to consider words
> for addition it is very helpful to know where they originated from.
> 
> Using diff is not really the best way to manage wordlist changes.  I list of
> additions and substations (if any) from the upstream dictionary seams like a
> more reasonable think to do to avoid the problems of conflicts.  Writing a
> Perl script to manage these lists should not be too hard (I notice the shell
> script already uses Perl so I assume Perl is an acceptable requirement). 
> Would that be okay?

Sure.  We have a slight preference to not use Perl if you can use Python, but if not Perl is fine too as these scripts will only be run by people who want to update the dictionary on their local machines.

(In reply to Kevin Atkinson from comment #28)
> Hi Ekanan,
> 
> I am very confused on what is going on here and why this bug report was
> closed and a new one created.

Just to follow the usual Mozilla process.  We usually use new bugs for new code changes, and close bugs when the corresponding patches for them land.  But I don't want to overburden you with all of the stuff!  Please continue the discussion either here or in bug 1137544 (the latter would be better for us).  I'm watching both places.  :-)

> It looks like Ekanan mainly added words and didn't really update the source
> list.  This is unfortunate as I also removed a number of Junk entries.

Yes, he intentionally left hunspell-en_US-20081205.dic untouched, in order to keep merge-dictionaries working for now.  You should create a new file probably called hunspell-en_US-20150215.dic with the new source list.

> I am working on a better solution for you but am now confused where to file
> it.

Preferably in bug 1137544, but don't worry too much about our process details.  :-)

> BTW: As far as the Chromium list go, I admit it can likely go and at this
> point is just adding complexity.

Sounds great!

Comment 31

[Kevin Atkinson]

For reference here is a link to the upstream version the attached dictionary is created from: http://downloads.sourceforge.net/wordlist/hunspell-en_US-2015.02.15.zip