2062992 - [Listen to Page] Spike: Performance implications of running local models through the AI Runtime on Android

Status: RESOLVED FIXED

(Firefox for Android :: Reader Mode, task)

Description

[Moyin [:moyin]]

This is a spike to understand the performance characteristics of running local models through the Firefox AI Runtime.

Comment 1

[Matt Tighe [:matt-tighe]]

I have left a detailed report at https://docs.google.com/document/d/1Dh0HaLAsYs7osYgAQq0qxjOZKMjzSJPTmXHxaD2Qzuc/edit?tab=t.0#heading=h.nmrvsgnvz8qe

TL;DR - we are missing some compatibilities to make this easy. The platform team is using browser-chrome flavored mochitests in order to understand true E2E performance, and on Android we would need plain flavor or xpcshell tests to start to understand the basics of performance. Mochitests are also broken on Android 17 at the moment (bug 2061900), which prevents us from testing on current flagship devices. Additionally, older devices suffer from a WASM tiering compilation bug (bug 2061875) which significantly harm performance, and current workarounds require a tradeoff in inference performance.

That said, mid-tier device performance was surprisingly capable based on my measurements, but we would still likely want to support hardware acceleration before we integrated it - and fallbacks to off-device methods may always be required if we want to continue serving our lower-tier device segment.