1981787 - Google Lens searches return invalid results

Status: VERIFIED FIXED

(Firefox :: Search, defect, P1)

Description

[Adrian Florinescu [:aflorinescu]]

Attachment: NTP & Visual image.txt

Found in

• Nightly 143.0a1;

Affected versions

• Nightly 143.0a1;

Tested platforms

• Affected platforms: Windows 11, Mac 13, Ubuntu 24

Preconditions

• browser.search.visualSearch.featureGate true

Steps to reproduce

1. Rightclick on any result from the NTP and from context menu go to Search Image with Google Lens
2. From Visual Result page, right click on another image and from context menu go to Search Image with Google Lens

Expected result

• uncertain, see aditional notes

Actual result (see screenshots)

• Something went wrong No image at the URL. Try again with a different URL or image

Additional Notes:

• The url before redirect is a weird one and seems incomplete ?!, I'm attaching a notepad since it has like 14k chars
• Comparing the functionality, seems like Chrome doesn't allow followup Lens searches from images from visual results, option is missing from chrome.
• Replacing the STR with a regular search, then switching to images results tab, then using Search Image with Google Lens on some of the results will return the same behavior Something went wrong No image at the URL. Try again with a different URL or image
• Might also be an additional issue here since it seems that for some visual results(e.g sponsored) initial Search Image with Google Lens doesn't work the first time, then functions propely the 2nd time.

Comment 1

[Drew Willcoxon :adw]

Thanks, if you can find an image URL that is not a data URI and that does not work, please let me know. For now, I'll address this bug by hiding the menu item for images that are encoded as data URIs. And thanks for mentioning that Chrome does not allow follow-up searches from the Lens SERP! That gives me more confidence it's OK for us to do that too, at least for now.

Details:

All the URLs on the Lens SERP seem to be data URIs. I tested with some other data URIs too, and none of them work, not even really short ones, so this seems to be a problem with data URIs in particular.

I pasted a data URI into the Lens web app to see what would happen, and it uses the POST endpoint. For non-data URLs, it uses the GET endpoint, like we're doing. I've started a discussion with Mark about whether we can/should support POST right now, but for the time being, I think it makes sense to hide the menu item for data URIs.

Comment 2

[Drew Willcoxon :adw]

I talked with Mark about this and he agreed that we should hide the menu item for data URIs, at least for now, and that we shouldn't overcomplicate things by adding more search config data. There are other ways we could fix this:

• Go ahead and implement support for the Lens POST endpoint, and use it for data URIs
• Encode some kind of rule in the search config that tells the client the kinds of image URLs that are supported and how to support them (e.g., use POST instead of GET)

Comment 3

[Drew Willcoxon :adw]

Attachment: Bug 1981787 - Hide the visual search content context menu item for data URIs.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/aff0ce4d3a71
https://hg.mozilla.org/integration/autoland/rev/92847f7d7032
Hide the visual search content context menu item for data URIs. r=Standard8

Comment 5

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/92847f7d7032

Comment 6

[Cristian Baica [:cbaica], Release Desktop QA]

The bug is verified fixed using the latest Fx 144.0a1 and Fx143.0b5 on Windows, MacOS and Ubuntu. The context menu option(to "Search image with Google Lens") is no longer available for URIs.
However there was another issue discovered while testing this that results in a similar error. Please see bug 1985563 for the follow-up.