Open Bug 1798543 Opened 3 years ago Updated 1 year ago

Screenshot Attribution and Annotation

Categories

(Firefox :: Screenshots, enhancement, P3)

Product:
Firefox
Component:
Screenshots
Type:
enhancement

Tracking

()

People

(Reporter: chutten, Unassigned)

References

Details

(Keywords: blocked-ux)

This is a feature request.

I don't like that "screenshot of text" is the only interoperable format that the present Web has supplied us with. But that's where we are. So let's improve it.

I can think of two improvements:

  1. Include the title or domain or domain+path (or full URL if we're feeling lucky?) as the EXIF PageName tag (sometimes described as "The name of the page from which this image was scanned") to supply attribution of an image.
    • Then it might be possible for someone else to independently run down the source
    • Could determine how much to share based on whether it's a full-page capture (recommend full url) or just a part of it.
  2. Include (a subset of) the text in the DOM nodes being captured (for node-based captures) in the Description EXIF tag like "Screenshot of a webpage with text '<captured text>'"
    • Could combine with #1 via "Screenshot of <url of webpage> with text '<captured text>'"

Both/either could be optional via checkbox in the screenshot overlay UI ("Include page URL in screenshot metadata", "Include captured text in screenshot metadata") or we could help choose for people either universally or depending on context (in PBM? Capturing full page? Saving vs copying?)

Thanks for filing this, I've been thinking the same thing.

It looks like pagename is a string type, but I'm not sure what we can put in there. We could put e.g. Screenshot of https://example.com/page.html. I'm not sure if there are field length constraints we need to be aware of that might for example truncate a URL making it unusable.

We'll need UI to present this and an opt-in/opt-out? so we don't create surprises for users not expecting the source URL to get captured in the image we save. I'd love to design and implement this to allow for alternative and better output formats. The main value of this browser feature is in providing a way to capture some part of what you see in your browser and then save/share/whatever. The capture step is the right moment to gather context - such as the source URL, text description/alt-text etc. Some output formats won't be able to make full use of this data, but we should facilitate it anyway and work towards better outputs that do.

Severity: -- → N/A
Keywords: blocked-ux
Priority: -- → P3

(In reply to Sam Foster [:sfoster] (he/him) from comment #1)

It looks like pagename is a string type, but I'm not sure what we can put in there. We could put e.g. Screenshot of https://example.com/page.html. I'm not sure if there are field length constraints we need to be aware of that might for example truncate a URL making it unusable.

Agreed. I'm no expert at reading the EXIF spec, and I don't know how much is left up to the tolerances of implementors. I've seen whole novels in some extended fields of an IDv3 tag, so I confess I was assuming we could find similar latitude here.

We'll need UI to present this and an opt-in/opt-out? so we don't create surprises for users not expecting the source URL to get captured in the image we save.

Depends on the meaning of "need". With my Data Steward hat on, this isn't Data Collection so it's clear from that POV. But learning a full URL with query params is inside the screenshot you took would not be a happy sort of surprise for some of our user base. It might be that eTLD+1 is something we could start as opt-out. Sounds like something a Product Owner would have to make a call on.

I'd love to design and implement this to allow for alternative and better output formats. The main value of this browser feature is in providing a way to capture some part of what you see in your browser and then save/share/whatever. The capture step is the right moment to gather context - such as the source URL, text description/alt-text etc. Some output formats won't be able to make full use of this data, but we should facilitate it anyway and work towards better outputs that do.

Yeah. Images are where we appear to be at the moment which is why I leaned towards EXIF. If there were a standard for embedded Image Description as a fallback for non-alt=-having images for screen readers, we could do a lot of a11y good in this arena.

Clarification on the blocked-ux keyword ask here. We need to think through exactly which part of the problem we want to solve, but I think either way we need a way to view/edit metadata for the screenshot at the capture time. That lets us create a intermediary internal format with fields like the source URL, alt/description text, time & date, and potentially some way to represent the capture constraints. There's no obvious place to extend the current UX to include those details, so we'll need to brainstorm some options.

There's an exploration of a related question to this at https://cweiske.de/tagebuch/exif-url.htm. The use case there is "which website was this image downloaded from" which is similar but not exactly the same as "which website was this screenshot captured from". There are two candidate suggestions that might be relevant here:

  • rdf:about "URL of image that was downloaded" is one of the dublin core properties
  • dc:relation "URL of website that linked to the image"

It also kind of depends on how this shows up in common applications which display image metadata, such as MacOS Preview, Adobe Photoshop, , Microsoft Photos etc. Those tend to show common file and photo metadata like created timestamps, pixel dimensions, color depth and maybe the caption/description. Not the more "exotic" fields like rdf:about or dc:relation. If we plant a tree in the woods and no-one can see it, is it even there?

Given the above, I'm actually leaning towards just a caption/description which is something like "Screenshot captured from {eTLD+1} {Page Title}" or something. We can also add the full source url in e.g. dc:relation with the expectation that almost no-one except exiftool users will ever see it.

Duplicate of this bug: 1789900

If there's alt text (or if we can generate some locally?) should we include that as well, someplace?

Yeah. There looks to be reasonable support for both a Description and Caption tag. I imagine one being pre-filled as described in comment 5, and the other being user-editable. I'm still not 100% sure exif is the right place for this, but lets call that an implementation detail for now and we'll attack it from both sides: how to capture and present contextual information in the Screenshots UI, and exactly where and how we write that so it is preserved along with the screenshot itself.

A bit more detail on the current plan:

  • We'll use XMP and the rdf:about keyword to (opt-in) write the source URL into the screenshot PNG we produce. We might also populate the title and descriptions. Those strings will need to get localized.
  • That will happen on the front-end. We'll create a iTXt chunk and insert it right after the IHDR chunk.
  • We'll need a CRC implementation for the chunk.
  • As we are only dealing with newly-created images, we don't need to worry about looking for pre-existing XMP data in there.
  • We'll need new UI to surface this option, which will be opt-in on a case by case basis I think. We can't "forget this site" once the screenshot is saved.
  • Its understood that most software will not surface the rdf:about metadata at this time. The ecosystem is good with location and technical details like the camera used etc, but not so much for this use case. Maybe we can change that.

One more point of clarification: We can only insert the metadata for downloaded screenshots. When you "copy" a screenshot you just get the pixel data on the clipboard so any metadata is lost by definition. I think that is expected, but the UX changes might need to take that into consideration.

Adding 1908205 as a blocking dependency. Before we can do any work in the intermediary data representation which carries the source URL, alt text and other metadata, we have to fix the image data pipeline within the Screenshots code. That starts with bug 1908205.

Depends on: 1908205
You need to log in before you can comment on or make changes to this bug.