Closed Bug 800223 Opened 12 years ago Closed 10 years ago

Video Lego Blocks - RFC

Categories

(Air Mozilla :: Streaming, defect)

Product:
Air Mozilla
Component:
Streaming
Platform:
ARM
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: richard, Assigned: richard)

Details

This is a proposal for a project based on the work that Dave Richards is doing with the Logitech C920 and the Raspberry Pi.  Please leave comments below.  Comments on feasibility and alternate design ideas are especially welcome. 

The goal is to be able to create inexpensive video feeds that can be used for a wide range of applications.  Camera Blocks can stream to Video Feed Blocks, KRAD instances, or other compatible software.
Note that this document allows for more than one protocol to be specified.  However we initially expect to support only one protocol.

There are three components, a camera block, a video feed block, and a router page.

The Camera Block consists of a Logitech C920 camera or another camera with hardware video encoding, a RaspberryPi computer, and a TP-Link TL-POE10R power splitter to power the RasPi.  This part is identical to what Dave is doing now.  Two changes to the code are required:

1 - The code should read a channel number (0-65535) and a web address from the file /etc/videoblock.conf
2 - On start-up the Camera Block contacts the router page at the web address in /etc/videoblock.conf and supplies it's ip address and channel number.
3 - If the router page responds with an ip address and a protocol, the Camera Block begins streaming the requested protocol to the designated ip address.
4 - if the router page responds with a WAIT message, the Camera Block sleeps for 30(?) seconds and returns to step 2.


The Video Feed Block consists of a RasPi and a power splitter is above.  This block runs code to decode the H.264 stream and output it to the RasPi HDMI connector.  As with the Camera Block it reads a channel number and web address from the file /etc/videoblock.conf.  On start-up it connects with the server at the web address sends it's ip address and channel number.  It then begins listening for a video stream which it outputs to the HDMI connector.


The router page maintains a lookup table sorted by channel number that also contains ip addresses or machine names. 

When a Video Feed Block contacts the router page the page daemon purges the table of any existing entries with the block ip address and then adds the block's channel number and ip address to the table. 

When a Camera Block contacts the router page the page daemon looks up the first occurrence of the channel number and returns the associated ip or web address and requested protocol to the Camera Block.  The Camera Block begins streaming video to the address using the protocol specified.

The router page should contain a web interface to allow specifying non-camera block streaming destinations such as KRAD instances.
Sounds great to me on the whole, I want to note this mostly confirmed hardware technicality right of the bat so we can think of how to best solve it cleanly and repeatedly, the power to the camera needs to be wired separately. Right now I am just using a powered usb hub, and I have been able to record for hours on end, but putting a powered hub into every camera block is probably not the way to go. Well have to strip the usb cord and wire the power into the same thing thats powering the board itself. I haven't looked this up yet, but I know plenty of folks run into this same issue, so there very well may be a ready to go adapter for such a thing.

Here is my first proposal, we expose a usb port in some manner on all blocks, and any kind of usb midi controller can be plugged in. Assigning controls to actions and loading presets would be done thought the web control. This would be in addition to OSC control which is already networked.. what exactly might be done with this I do not yet know ;p
We may not need a hard concept of channel numbers that are set ahead of time, but rather we tag and manage feeds that come in. I also think blocks could possibly be both camera and video display. Just speculating on possible evolution here ;p

I should be able to get you some 'RadioBerry' sd card images tomorrow, I think gotten things to a good place and its backed up too.
(In reply to David Richards from comment #1)
> the power to the camera needs to be wired
> separately. 

I'm curious to see if this is still an issue on the newer boards that don't have fuses on the USB power lines. I'll test as soon as an image becomes available.
We probably need a way to "bump" a block over the net to cause it to redo the handshake with the router page.  (For changing where streams go or which stream to follow).
The current status for H264 passthru is that it works with a bad problem. Using identical krad versions and scripts on both my pc and the raspberry, I get a different result. On the PC the stream is correct, but on the raspberry I am getting something like a unexpected or invalid interframes. I've seen where other people are having this same problem using gstreamer. I'm investigating this and suspect its a difference in userspace or kernel libraries for V4L or UVC. I'm currently getting to the bottom of this, but irregardless I will post an image tonight that can do something, perhaps MJPEG streaming.
Well its a sad story. I spent way way way too many hours on this, but H264 passthru on the raspberry pi is a no go due to usb packet loss. I did not realize the issue was this fatal. I tried every possible thing, but we are now at the mercy of the vendor as is everyone else trying to do the same kinds of things on a $35 computer. I really wanted this to happen, but its not going to at this moment. There is a large number of people upset with the PI because of related issues that essentially make linux look bad.

There is quite a number of plan B's though.

0) There maybe a fix for this issue 5 minutes from now, or never.
1) The raspberry pi folks are releasing a camera add on at some point, that does not communicate via usb so is immune to this problem.
2) The beaglebone, its nearly exactly the same size as the raspberry, has a CPU twice as powerful, and has demonstrated stability and compatibility with H264 passthu using the C920 camera. The downside is it costs 2.3X as much at $89 a pop, and it does not have display output. I've ordered one for testing.
3) The pandaboard, I have two of these, and this works as well, however its $170 so the price point looses interest on scale, and its size is much larger. In return you get builtin wifi N, dual core 1.x ghz cpus.

Of interest to me is if the beaglebone can handle a C920 camera doing passthru, while at the same time a MobilePre (the dual xlr usb soundcard). 

A couple notes: I had a very good experience with Arch Linux ARM vs. Raspbian on the Raspberry, and would recommend it for any projects with the rpi. It's still possible to do frame capture with the rpi, but that's just not as much fun as video.. MJPEG streaming does work, and is somewhat immune to the dropped usb packets due to every frame being an intraframe, however the bitrate is not controllable and is extremely high, over 2M per second for HD, so its quite a clunker. (and also due to the dropped packets, the framerate is not consistent)
Streamed H264 for many hours today using Pandaboard and C920, no doubt it will work on the beaglebone, will be arriving on Wednesday.
Where do we stand on the Video Feed Block?
(In reply to Richard A Milewski[:richard] from comment #8)
> Where do we stand on the Video Feed Block?

My fear is that since we're dropping frames coming in over USB, those frames would be missing on the HDMI out. 

Alternatives:

* There exist DVI-D and HDMI capes for the BeagleBone, but they are limited to XGA (1024x768). 

* Full Beagleboards have HDMI connectors "for DVI-D". Not sure if there's audio on that, but I don't see any resolution limits.

* Pandaboards have full HDMI, and we buy rather a lot of those (860, in Releng to date...)
Sounds like a Pandaboard solution makes sense.  Can we get a few for David and us to test with?
I wasn't sure what exact goals we were shooting for here, ie. something for just Mozilla to use, or a mini platform to share that can do this as cheaply as possible. Either way here is what I can say:

I have had a video feed going for 67 hours strait using the C920 and the pandaboard with krad now. ( http://67.188.0.2:8004/radiopanda.mkv , use vlc or mplayer) (I will probably take that one down tonight though.)

If we have zillions of pandaboards, they can certainly use them, the only downside to them is they just are not tiny, esp once you plug everything into them. They can fit into standard 150x150 mil industrial enclosures, and perhaps down to 130x130 mil if you use angled connectors and half height microsd adaptors, or if you expose ports directly. But this might fit our use, one possibility is to mount the camera on top of the enclosure with pan/tilt servos. (these can be custom built for very cheap, I've used $5 servos and the pololu servo controllers, already have code in krad for this so pan/tilt could be controlled through the Web UX)

A note on R'PI dropped usb packets, is it would not affect hdmi video output from a network stream using TCP, because it would just re-request the data, this does not happen when capturing data from a USB camera however.

I got the beaglebone today and will play with it later. 

Question: Do we want an image for pandaboard? It sounds like we do, but please confirm that I should work on this. My current setup on the pandaboard uses Ubuntu, this works but takes ages to boot and has to much extraneous stuff going on, for nondesktop embedded ARM linux, I'm sold on Arch Linux ARM. Its really fast, and is just as easy as debian/ubuntu to install packages.
Flags: needinfo?
sorry if it's unrelated (maybe a mailing list would be nice?):
How do you encode the stream to be sent by the panda/beagleboard? Is the CPU capable enough to do this realtime? And if yes what resolution and framerate?
Or are you using any kind of hardware/GPU-acceleration? I only know that there's a gstreamer-omx branch to do this on the RaspberryPi.
Flags: needinfo?
Jannis- The Logitech C920 does H.264 in hardware, no encoding on the board required.
Assignee: nobody → drichards
Newsberry Pi

Camera Module Info:

http://www.raspberrypi.org/archives/2555

Strategy: Stay on top of these so we can get a batch for evaluation as soon as available.

Reminder: This does not use USB.

The price point is such a win, I have hopes.
(In reply to David Richards from comment #6)

> Well its a sad story. I spent way way way too many hours on this, but H264
> passthru on the raspberry pi is a no go due to usb packet loss.

Hi,

I tried to do the exact same thing, with the exact same results. While looking for more information, I came upon this Bugzilla page, so I hope I do not intrude.

I know there are lots of discussion about USB and the Raspberry Pi; but I wonder how you came to the conclusion that the C920/H264 problem is due to USB packet loss. Did you managed to check that, and how?

Thank you!
The Raspberry Pi drops USB packets, this is well known. This does not fatally affect TCP network communications because TCP will retransmit lost TCP packets. This does not fatally affect Motion JPEG, because each frame is an intraframe and a lost frame does not affect other frames. H.264 uses inter-frames, so a dropped frame corrupts the stream. At this point you can request a new keyframe, and not include the frame after the detected dropped frame in your stream, but on the RPI it happens every other second, so its not suitable for situations where the goal is actually smooth high quality video.

ODroid U2 is out now and is 3X the cost of a RPI, and 10X as powerful.
Assignee: drichards → richard
Looks like some sort of WebRTC lash-up will do this better.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.