1912768 - [Experiment] The Alt Text generation AI significantly misidentifies some images

Status: NEW

(Firefox :: PDF Viewer, defect)

Description

[Patriciu Pop [:ppop], Ecosystem QA]

Attachment: image used for STR

[Affected versions]:

• Firefox Nightly 131.0a1 (Build ID 20240811212519)
• Firefox Beta 130.0b2 (Build ID 20240807091924)

[Affected Platforms]:

• Windows 10 x64
• macOS 14.4
• Ubuntu 22.04 x64

[Prerequisites]:

• Have a profile enrolled in the "New alt text flow and generation" experiment (https://experimenter.services.mozilla.com/nimbus/new-alt-text-flow-and-generation/summary)
• Have a PDF and the following image saved to the device.

[Steps to reproduce]:

1. Open the browser with the profile from the prerequisites.
2. Open a PDF file in the browser.
3. Click the "Add image" button and use the file picker to add the image.
4. Wait for the Alt Text to be automatically generated.
5. Observe the description.

[Expected result]:

• The description matches the image, identifying the dice.

[Actual result]:

• The AI model misidentifies the dice as balloons with ribbons.

[Notes]:

• Adding the same image of a driver's license provided multiple erroneous descriptions:
  • Person holding a paper showing the contents of an electronic device;
  • Person holding a screen with colorful text;
  • A display of a map with various people in the background.
• The greatest deviation was interpreting a technical drawing as "T.V with a reflection of some people washing clothes in an advertisement on the sideboard"
• Attached an image with multiple identification errors that drastically change the meaning of the image:

Comment 1

[Patriciu Pop [:ppop], Ecosystem QA]

Attachment: Collection of errors