1917620 - Support translating PDFs

Status: NEW

(Firefox :: PDF Viewer, enhancement)

Description

[vito.detullio]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0

Steps to reproduce:

It's now possible to open pdf files directly from firefox using pdf.js.
Also it's possible to translate html documents.
It would be nice for firefox to offer the translate capabilities during the visualization of pdf documents.

Actual results:

the pdf is available in the firefox interface, but no translate icon in the url bar

Expected results:

the traslate icon (and the underlying functionality) should be available during the view of a pdf

Comment 1

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Firefox::PDF Viewer' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Marco Castelluccio [:marco]]

This would be an interesting feature, but it is really hard to implement because it requires relaying out the contents of the PDF (as the translated text will have completely different size than the original text). The implementation is kind of similar to what would be required for https://connect.mozilla.org/t5/ideas/reader-mode-for-pdf-files/idi-p/9908.

Starting from Firefox 128, we will support select translation though, so you can at least select specific sentences or paragraphs in PDFs and translate them!

Comment 3

[Rares Doghi, Desktop QA]

Opening this as an enhancement for now, but if we will only support selective translation, maybe we should close it ?

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Rares Doghi, Desktop QA from comment #3)

Opening this as an enhancement for now, but if we will only support selective translation, maybe we should close it ?

We would like to support full translation too, not sure when we'll be able to work on it though.

Comment 6

[Greg Tatum [:gregtatum]]

I'm putting some notes here from some conversations so that they persist.

PDFs are hard to translate because they contain unstructured layout data. The data is stored in presentation order. So for instance taking an arbitrary paper you can see visually distinct paragraphs. But in the layout from pdf.js, it's just a list of spans. There's no way walking the DOM tree to understand what is part of a distinct paragraph.

Each of the following lines are display: block span elements:

• Elias Frantar
• 1
• Dan Alistarh
• 1 2
• Abstract
• We show for the first time that large-scale genera-
• tive pretrained transformer (GPT) family mod-
• els can be pruned to at least 50% sparsity in
• one-shot, without any retraining, at minimal

There's also no guarantee that a two column paper layout will have the text in the correct order, so text from the left column could then have text from the right column next in the DOM list, further creating garbage in, garbage out.

Here our translation models would struggle to translate a segment like tive pretrained transformer (GPT) family mod-. Even concatenating the sentences into one long sentence can product a bad input * Elias Frantar 1 Dan Alistarh 1 2 Abstract We show for the first time that large-scale.

The content of the PDF would need to be translated into some kind of structure that could be reasonably chunked to send in for translation, and then the result applied to the original structure. There could be two approaches here, a "reader mode" where you first convert the the document into a simple markdown-like document. This would be regular HTML, and our TranslationsDocument would do a great job translating it. This reader mode would be useful outside of translations, especially for accessibility reasons. The other approach would be more complicated, but similar to the full page translation approach. We could take the translations, and apply them back to the original layout. This could be a bit tricky as the new text can be either longer or shorter, and the segmentation would be locale-dependent, making it erratic in terms of the final layout.

So the next question is how to go from a document layout to structured data. There are two directions to go, a heuristic approach, or an ML approach. There is probably some sliding scale between those two directions where heuristics and ML could be combined.

We should probably do some more research here, but there is some prior art, especially with LLMs wanting to vacuum up PDF data. This list isn't necessarily advocating for any of these approaches, but just providing a starting place for research:

• Marker - ML based text extraction
• PDF-TREX: An approach for recognizing and extracting tables from PDF documents
• Apache PDFBox
• PDFMiner
• Mistral OCR

Comment 8

[cafxx]

Adding here what I mentioned in bug 1955809:

As, contrary to webpages, PDF contents can't undergo reflow, the translated text should be overlaid in place, on top of the original text (like e.g. happens when translating a picture/screenshot with google lens).

Also worth pointing out that this would also be similar to what would be required to eventually be needed to support translation of text contained inside images (bug 1946998) and, if one can dream, also in videos (e.g. hardcoded subtitles, signs, charts, slides, etc.).

Comment 9

[cafxx]

Starting from Firefox 128, we will support select translation though, so you can at least select specific sentences or paragraphs in PDFs and translate them!

FWIW selection (and therefore select translation) potentially suffers from the same problem highlighted by Greg above: there is no guarantee that text selection will happen following meaningful content flow, and not whatever order the layout spans are being stored in the PDF or rendered by PDF.js.

Furthermore, the whole selection idea also assumes in the first place that:

1. the PDF contains "text", and not e.g. raster images of text (e.g. scanned pages that did not go through OCR)
2. the text that has not been flattened to curves (for whatever reason)
3. selection is not being otherwise hampered (e.g. via PDF restrictions, although I am not sure PDF.js enforces this)

I therefore concur in general with Greg's point that the solution will probably need to rely on some form of OCR.