839981 - Cannot copy video frames from Camera to Canvas on Unagi

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Julien Wajsberg [:julienw]]

+++ This bug was initially created as a clone of Bug #832745 +++

(patch in Bug 832745 did not fix the bug on Unagi)

Trying to write an application that targets video frames analysis (e.g., QR Code reader, things like this), I have:
 - <video> element, with JS code that sets the source to the camera, .mozSrcObject = stream, which works
 - <canvas> element, to which frames will be copied through Context2D.

The drawImage() call returns an exception:
Exception: [Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: app://heart-rate.gaiamobile.org/js/heart-rate.js :: heartrate_computeFrame :: line 70"  data: no]

It does not happens if the <video> targets a local WebM file.

Tracking the exception, I finally found the code path of the issue.

In |nsCanvasRenderingContext2DAzure::DrawImage()|, there is a call on
|CanvasImageCache::Lookup(element, mCanvasElement, &imgSize);| which fails, hence the next call will be on |nsLayoutUtils::SurfaceFromElement(element, sfeFlags);|.

It calls |nsLayoutUtils::SurfaceFromElement(nsHTMLVideoElement* aElement, uint32_t aSurfaceFlags)|, which will then fail on |nsCOMPtr<nsIPrincipal> principal = aElement->GetCurrentPrincipal();| with principal staying null.

Looking in details at GetCurrentPrincipal(), I had the following:
Breakpoint 10, nsHTMLMediaElement::GetCurrentPrincipal (this=0x41983e00) at /home/alex/codaz/B2G/gecko/content/html/content/src/nsHTMLMediaElement.cpp:3406
3406	  if (mSrcStream) {
(gdb) print mSrcStream
$14 = {mRawPtr = 0x4462b700}
(gdb) n
3407	    nsRefPtr<nsIPrincipal> principal = mSrcStream->GetPrincipal();
(gdb) print mSrcStream
$15 = {mRawPtr = 0x4462b700}
(gdb) print principal
$16 = {mRawPtr = 0xbe99b94e}
(gdb) n
3408	    return principal.forget();
(gdb) print principal
$17 = {mRawPtr = 0x0}
(gdb) print mSrcStream
Cannot access memory at address 0x54
(gdb) info breakpoints 
Num     Type           Disp Enb Address    What
9       breakpoint     keep y   0x406b7728 in nsHTMLMediaElement::GetCurrentPrincipal() at /home/alex/codaz/B2G/gecko/content/html/content/src/nsHTMLMediaElement.cpp:3407
	breakpoint already hit 28 times
10      breakpoint     keep y   0x406b7710 in nsHTMLMediaElement::GetCurrentPrincipal() at /home/alex/codaz/B2G/gecko/content/html/content/src/nsHTMLMediaElement.cpp:3406
	breakpoint already hit 16 times

Comment 1

[Julien Wajsberg [:julienw]]

Basically, this bug prevents any application that needs to analyze a video frame.

This should not be difficult to fix (according to Alexandre, we just need to find the correct value to put in an array) but this needs both access to an Unagi and gdb skills that I don't have.

Comment 2

[Julien Wajsberg [:julienw]]

(In reply to Julien Wajsberg [:julienw] from comment #1)
> Basically, this bug prevents any application that needs to analyze a video
> frame.

(think: a "barcode/flashcode scanner" for example)

Comment 3

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: Adding Unagi pixels constants and color conversion

With this patch and patch from bug 832745 I can draw the camera preview onto a canvas.

Test app: https://github.com/mpizza/gaia/tree/master/apps/camera-test

Comment 4

[:gerard-majax]

That is awesome news! We did a quick review with julienw when I was in Paris office last wednesday, and we thought we had the correct color format at that time. Turns out we were wrong :)

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ea8197d7d588

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ea8197d7d588

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/058788f69b7e