1932991 - MLSuggest NER misses on some partial cities

Status: RESOLVED FIXED

(Core :: Machine Learning: General, defect, P3)

Description

[cgopal]

example:
{intent:"yelp_intent", location:{city:"##val", state:null}, subject:"coffee near me sunnyval"}
should give city: "Sunnyval"

Comment 1

[Drew Willcoxon :adw]

Here's the nerResult in MLSuggest for this query in case it helps:

[{entity:"I-CITY", score:0.6814061999320984, index:5, word:"##val"}]

Comment 2

[cgopal]

Thanks Drew!

1. Could you please give me some idea as to what level of prefix (partial city name) you want to match.
   E.g. sunnyvale & sunnyval - are they good?
   or do you want that to match sunnyva ?
   is there minimum length of prefix for city name that you would expect to have a match.
   This could be crucial information. As we train this model we could show such examples but at the sametime we don't want the model to learn noise. as in is the weather sunny ? as city prefix.

A good trade off would be you won't get 100% prefix match of sqlite (fts) but at the sametime the number of such partial examples need to be kept minimal otherwise model will pick noise.
Please let me know your thoughts.

Comment 3

[Drew Willcoxon :adw]

Non-ML Yelp and weather suggestions do prefix matching immediately after a keyword is typed, e.g., "ramen n" would match "ramen New York City, NY" if you're near NY. I appreciate the concen about noise, and IMO we should err on the side of reducing false positives. I don't have a strong recommendation for a prefix threshold. In the past, we've used these heuristics for prefix matching:

• A constant 4-5 char threshold, i.e., you have to type 4-5 chars in the word before we try to prefix match it
• For multi-word keywords/subjects, start matching after the first word, e.g., "san" matches "san francisco", "san mateo", "san gabriel", etc.

That second point raises another issue: A prefix can match multiple cities, and ideally MLSuggest would return all matches. That way Firefox can pick the matching city that's closest to the user. If that's hard to do, we can definitely leave it for a v2.

Comment 4

[cgopal]

I kept city with at least 6 characters and with partial keywords. We also want to be mindful about false positive issues.
I have upgraded the model today. If you use the latest nightly and goto about:inference > Models Storage
you should see the latest versions of mozilla/distilbert-uncased-NER-LoRA (v0.1.6) and mozilla/mobilebert-uncased-finetuned-LoRA-intent-classifier (v0.1.5)
Above example works fine for me.
thanks!

Comment 5

[cgopal]

We are closing this issue as initial problem with partial city seems to have been improved. We are keeping now at least 6 characters. We could re-open if this is a blocker or any new issues in the future.