2005145 - Add experimental WebNN component

Status: NEW

(Core :: Machine Learning: On Device, enhancement, P3)

Description

[Tarek Ziadé (:tarek)]

Adds an experimental WebNN Javascript API as a new component - https://www.w3.org/TR/webnn/

• JS API
• Rust binding based on rustnn

Comment 1

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 1/10)

This commit vendors the rustnn crate and all its dependencies for WebNN API support:

• Added rustnn dependency (git revision 5bd4582)

  • Uses ort 2.0.0-rc.10 with load-dynamic feature for runtime ONNX loading
  • Requires smallvec 2.0.0-alpha.10 for API compatibility

• Vendored Rust dependencies and patches:

  • Patched smallvec to 2.0.0-alpha.10 for ort compatibility
  • Downgraded http from 1.4.0 to 0.2.12 for hyper compatibility
  • Removed http_0_2 patch to resolve version conflicts
  • Added cargo vet exemptions for new crate versions

• Updated Cargo.lock with 251+ package changes

• Re-vendored third_party/rust/ directory (5882 files)

The rustnn crate provides the backend implementation for the WebNN API,
using ONNX Runtime for neural network inference.

Comment 2

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 2/10)

Comment 3

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 3/10)

Comment 4

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 4/10)

Comment 5

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 5/10)

Comment 6

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 6/10)

Comment 7

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 7/10)

Comment 8

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 8/10)

Comment 9

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 9/10)

Comment 10

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - vendor RustNN (part 10/10)

Comment 11

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Add WebNN (Web Neural Network) API implementation

This commit adds the initial WebNN API implementation to Firefox:

• Add dom/webnn/ directory with C++ implementation

  • ML: Entry point for WebNN API
  • MLContext: Device context for graph execution
  • MLGraph: Compiled neural network graph
  • MLGraphBuilder: Builder pattern for graph construction
  • MLOperand: Tensor operands in the graph

• Integrate into Firefox build system

  • Add webnn to dom/moz.build DIRS list
  • Add WebNN WebIDL definitions to dom/webidl/moz.build

• WebIDL interfaces: ML.webidl, MLContext.webidl, MLGraph.webidl,
  MLGraphBuilder.webidl, MLOperand.webidl

Comment 12

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Add xpcshell tests for WebNN API

This commit adds comprehensive xpcshell tests for the WebNN implementation:

• Add dom/webnn/test/ directory with test infrastructure

  • test_ml_basic.js: Basic ML interface tests

    • Tests ML object construction
    • Verifies createContext method availability

  • test_ml_context.js: MLContext functionality tests

    • Tests createContext() returns Promise
    • Verifies MLContext methods (createTensor, readTensor, writeTensor, destroy)
    • Tests basic tensor creation with descriptors

  • test_ml_coreml.js: CoreML-specific tests

    • Tests CoreML backend with deviceType: "npu"
    • Verifies CoreML (Neural Engine) availability
    • Tests tensor operations with CoreML backend
    • Validates CoreML context creation and destruction

• Update build configuration

  • Add TEST_DIRS to dom/webnn/moz.build
  • Create dom/webnn/test/moz.build for test registration
  • Add xpcshell.toml test manifest

Comment 13

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Add WebNN image classification demo test

This test demonstrates a complete end-to-end image classification
pipeline using WebNN APIs, similar to the Python example in rustnn:

• Graph construction with MLGraphBuilder:

  • input: Define input tensor (1, 3, 224, 224)
  • constant: Create weight tensors with random initialization
  • conv2d: Convolutional layer (3 -> 32 channels)
  • relu: Activation function
  • reshape: Flatten for fully connected layer
  • matmul + transpose: Fully connected layer (32111111 -> 1000 classes)
  • softmax: Output probability distribution

• Image preprocessing:

  • Synthetic image data generation
  • ImageNet normalization (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

• Inference execution:

  • Tensor creation and data writing
  • Graph execution with compute()
  • Result reading and validation

• Additional tests for binary operations (add, sub, mul, div)
  and activation functions (relu, sigmoid, tanh)

This comprehensive test validates the full WebNN API stack including
graph building, tensor I/O, and computation execution.

Comment 14

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Add rustnn_bridge dependency to gkrust

This adds the rustnn_bridge crate to Firefox's build system,
linking it into the main Rust library (gkrust). The bridge
provides FFI bindings between C++ WebNN code and the rustnn
library for neural network graph operations.

Changes:

• Add rustnn_bridge to gkrust dependencies
• Update Cargo.lock with rustnn_bridge and dependencies
• Export rustnn_bridge module in gkrust

Comment 15

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Implement WebNN API with ONNX Runtime backend

This implements the W3C WebNN (Web Neural Network) API in Firefox using
the rustnn library for graph operations and ONNX Runtime for execution.

Key Features:

• Complete WebNN API implementation (context, builder, graph, operands)
• FFI bridge between C++ and Rust (rustnn_bridge)
• Real ONNX Runtime execution for neural network inference
• Support for basic operations: input, constant, add, sub, mul, div, matmul
• Unary operations: relu, sigmoid, tanh, softmax
• Shape operations: reshape, transpose, concat
• Comprehensive xpcshell test suite (265 tests passing)

Architecture:
JavaScript WebNN API
→ C++ WebNN implementation (dom/webnn/*.cpp)
→ FFI bridge (dom/webnn/rustnn_bridge/)
→ rustnn library (graph operations)
→ ONNX Runtime (execution)

Implementation Details:

• MLContext: Manages device/power preferences, creates builders and graphs
• MLGraphBuilder: Constructs computational graphs with operands and operations
• MLGraph: Compiled graph ready for execution via compute()
• MLOperand: Tensor operands with shape and data type tracking
• RustnnBridge: FFI layer marshaling data between C++ and Rust

Critical Fixes:

• Fixed std::move() bug in MLGraphBuilder::Input() causing shape corruption
• Added output name mapping from JavaScript to ONNX model
• Enabled onnx-runtime feature for actual execution (not stub)

Comment 16

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Implement MLTensor API and WebNN operations

This implements the MLTensor API providing reusable tensor buffers for
efficient neural network operations. Tensors can be created, populated
with data, and read back asynchronously.

Features:

• MLContext::createTensor() - Creates tensors with specified shape and data type
• MLContext::writeTensor() - Asynchronously writes data to tensors
• MLContext::readTensor() - Asynchronously reads data from tensors
• Automatic memory management via RAII

Implementation:

• Added RustnnTensor structure holding tensor data, shape, and data type
• FFI functions: rustnn_tensor_create/destroy/read/write
• Tensor buffers allocated based on shape and data type
• C++ side uses JS_malloc for ArrayBuffer compatibility
• Proper cleanup in MLTensor destructor

Architecture:
JavaScript calls createTensor(descriptor)
→ C++ MLContext allocates RustnnTensor
→ Rust allocates buffer (shape × data_type_size)
→ Returns MLTensor wrapping RustnnTensor pointer

JavaScript calls writeTensor(tensor, data)
→ C++ extracts ArrayBufferView data
→ Rust copies data into tensor buffer
→ Promise resolves

JavaScript calls readTensor(tensor)
→ C++ allocates JS-owned buffer
→ Rust copies tensor data to buffer
→ Creates ArrayBuffer and resolves Promise

Comment 17

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Fix linting issues in vendored smallvec crate

• Fix trailing whitespace in README.md
• Fix empty lines at end of files
• These are auto-fixes from linter for vendored dependency

Comment 18

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Code cleanup and documentation updates for WebNN

This commit includes linting fixes and documentation updates:

• Remove unused RefPtr.h include from MLContext.h
• Fix license header in rustnn_bridge/src/lib.rs (http vs https)
• Add eslint environment declarations to xpcshell test files
• Auto-format JavaScript test files with Prettier
• Update RUSTNN_INTEGRATION_PLAN.md with:
  • Executive summary showing all 7 phases complete
  • Phase 7 (MLTensor API) completion notes
  • Comprehensive next steps for review and enhancement

All WebNN functionality remains unchanged - these are code quality
and documentation improvements only.

Comment 19

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Add WebNN demos, implement min operation, and fix GEMM transpose handling

This commit includes several related WebNN enhancements:

Demo pages:

• webnn_demo.html: Basic WebNN operations demo (linear ops, matrix multiplication)
• mobilenet_complete.html: Full MobileNetV2 image classification with pretrained weights
• README.md: Documentation for the demo pages
• Cross-linked navigation between demos

Min operation implementation:

• Added min() binary operation to WebNN API
• Implemented in WebIDL, C++ MLGraphBuilder, and Rust FFI layer
• Follows same pattern as other binary operations (add, sub, mul, div)

GEMM transpose bug fix:

• Fixed attribute name mismatch in rustnn ONNX converter
• Changed from snake_case (a_transpose, b_transpose) to camelCase (aTranspose, bTranspose)
• This matches attribute names sent by C++ MLGraphBuilder
• Critical for MobileNetV2 classifier: x[1,1280] @ weight[1000,1280].T = [1,1000]

The MobileNetV2 demo successfully classifies images using 106 layers and real
ImageNet pretrained weights, demonstrating full WebNN API functionality.

Comment 20

[Tarek Ziadé (:tarek)]

Attachment: WIP: Bug 2005145 - Add comprehensive WebNN architecture documentation

This commit adds detailed architecture documentation for the WebNN implementation:

Documentation structure:

• architecture.rst: Complete technical documentation with Mermaid diagrams
• README.md: Updated to reference architecture docs and demos
• moz.build: Sphinx integration for documentation building

Architecture documentation includes:

• Overall architecture diagram showing all 6 layers:

  • JavaScript API (navigator.ml, MLContext, MLGraphBuilder, MLGraph, MLTensor)
  • WebIDL Layer (WebNN.webidl)
  • C++ DOM Implementation (dom/webnn/)
  • Rust FFI Bridge (rustnn_bridge/)
  • Rustnn Library (third_party/rust/rustnn/)
  • Backend Layer (ONNX Runtime / CoreML)

• Mermaid diagrams for visualization:

  • Architecture overview showing component relationships
  • Graph building sequence diagram
  • Inference execution sequence diagram

• Multi-backend support documentation:

  • CoreML backend for macOS and iOS (Apple Neural Engine, Metal GPU)
  • ONNX Runtime for Windows, Linux, and Android
  • Automatic backend selection based on platform availability

• Hardware acceleration details:

  • CPU: SIMD on all platforms (SSE/AVX on x86, NEON on ARM)
  • GPU: DirectML (Windows), Metal (macOS), CUDA (Linux with NVIDIA)
  • NPU: Apple Neural Engine (macOS/iOS via CoreML)

• Technical details:

  • Layer descriptions and responsibilities
  • Data flow during graph building and inference
  • Memory management across FFI boundaries
  • Thread safety and error handling
  • Performance considerations

The documentation will be published at:
https://firefox-source-docs.mozilla.org/dom/webnn/architecture.html

Comment 21

[Greg Tatum [:gregtatum]]

This is an experimental implementation of WebNN that's not ready to land. I believe it was heavily LLM assisted for the code generation, but could a useful reference.