1930144 - Some keywords are wrongly recognized as "subject" in "location" and trigger a ML Yelp Suggestion

Status: VERIFIED FIXED

(Firefox :: Address Bar, defect, P2)

Description

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Attachment: [Yelp] random keywords recognized as subject and location.mp4

[Affected versions]:

• Firefox Nightly 134.0a1 - Build ID: 20241107212807

[Affected Platforms]:

• Windows 10
• macOS 14.4.1
• Linux Mint 20.1

[Prerequisites]:

• Have following prefs set:
  • browser.search.region = US
  • browser.urlbar.yelp.mlEnabled = true
  • browser.urlbar.quicksuggest.mlEnabled = true
  • browser.ml.enable = true

[Steps to reproduce]:

1. Open the browser using the profile from the prerequisites.
2. Type in the addressbar the "footbal" keyword.
3. Observe the results triggered in the "Firefox Suggest" section.

[Expected result]:

• No Yelp suggestion is triggered.

[Actual result]:

• A Yelp suggestion is triggered which is composed of "subject:bal" in "location:foot".

[Notes]:

• This issue is reproducible with the following keywords: top gun maverick, footbal, breaking bad, walm, clean pape, paws scor.
• Attached is a screen recording of the issue.

Comment 1

[cgopal]

footbal, breaking bad, walm, clean pape, paws scor are resolved when I tested on Nightly 134.0a1 (2024-11-14)
top gun maverick issue needs a pre or post filter on city / state lookup

Comment 2

[James Teow [:jteow]]

Hey Drew, sorry for another NI but just wanted to see if there was some additional validation step that was being created to address this.

Comment 3

[Drew Willcoxon :adw]

Bug 1931963 and bug 1931964 will address this.

Comment 4

[Drew Willcoxon :adw]

Of the queries in comment 0, only footbal and clean pape still show Yelp suggestions [1], and the subjects and locations are now correct. Neither are in the original Yelp dataset so I'm not sure why the model matches them, but I don't think we can expect a 100% accuracy rate. Even if we remove those, there are likely other queries that might not really be Yelp intents.

Chidam, do you agree? If so, I think we can close this.

[1] I'm testing on a build with all known bugs fixed, including city validation.

Comment 5

[cgopal]

Model has been trained from open source data and some additional data. It will try to get closer. Latest intent model accuracy rate is around 91%.
Unlike rule based engine, ML learns from the data but if learns too much then it will start memorizing (aka overfitting) so normally it is better to generalize rather than overfit to closed knowledge. As the purpose of the ML Yelp suggestion is to go beyond the standard keywords.
I agree with you and this could be closed.

Comment 6

[Drew Willcoxon :adw]

Thanks Chidam.

Cosmin, the main point here is that we can expect some "false positives," meaning queries that might not seem like Yelp intent, although it's probably debatable in a lot of cases. We should never see invalid cities/locations though.

Comment 7

[Drew Willcoxon :adw]

But if you find any queries that definitely do not seem like Yelp intents at all, like really obviously wrong, please do let us know!

Comment 8

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I have verified this issue on the latest Beta 134.0b1 (Build ID: 20241125200000) and latest Nightly 135.0a1 (Build ID: 20241126093610) on Windows 10 x64, macOS 14.4.1 and Linux Mint 20.1.

• The issue is no longer reproducible with the mentioned keywords in the bug.

We will log new issues if we find other broken results/keywords after all the changes land.