522979 - RFE - Utilize the N95 Accelerometer in a "Marble Madness"-like fashion as a User Interface for Fennec

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, enhancement)

Description

[Rob]

User-Agent:       Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.0.3705; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 4.0.20506; .NET CLR 1.1.4322)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a1pre) Gecko/20091012 Fennec/1.0b5pre

This is a RFE for Fennec on Symbian S60, (specifically the N95 et. al).

The N95 contains an accelerometer (http://www.nokiausers.net/forum/accelerometer/) that can be used to move the mouse and click with hand gestures. That is handy when there is no Touch-Screen (and saves wear on the Keyboard).


A "Marble Madness" style mouse-cursor controlled by the accelerometer would provide an excellent and unique means of surfing the web. This might be useful for persons with large fingers or arthritis.

If the cursor moved like the "scroll-ball" (middle click and gentle drag) on Windows XP you could set it to roll down slowly and tilt sideways back and forth to read web pages that are too wide for the device.

The N95's accelerometer gives control of more directions (all, (360' + 360'_click)) than the N95's scroll-key (4 + click) or touchscreen (360' + click)
so complicated "gestures" can be learned and used to short-cut commands.


Nokia N95 blog
http://n95blog.com/experience-nokia-n95s-accelerometer-yourself/

Demo of the Interface (may need to set phone's date to 2008 to clear cert.):
http://research.nokia.com/files/MovingBall_Example.sisx


Here is the Origonal Game reproduced for the NES Game Console:
Marble Madness NES Review/Walkthrough
http://www.youtube.com/watch?v=fYlZNcXmqus


Thanks,
Rob


Reproducible: Didn't try

Steps to Reproduce:
This is a RFE.

Comment 1

[Rob]

Attachment: MovingBall Source Code

Comment 2

[Rob]

Here is a video that demonstrates what this would be like:
http://www.youtube.com/watch?v=A9DBlolfZd4&feature=related

Rock-N-Scroll
http://www.keynetik.com/

Various software groups are utilizing a "smart-algorithm" in some of these types of User Interfaces to 'learn' gestures as opposed to "hard-wiring" (setting fixed variables for each gesture that the user must figure out).

Rob

Comment 3

[Rob]

--------------------
Other enhancements to this Enhancement are the use of the Camera(s) in combination with the Accelerometer to allow either:


1._ "Greater expressiveness of the Gestures" - You can use a Camera _facing_ the User (for "Eye-Tracking"), for the purpose of enhancing the precision of the Accelerometer and getting a more accurate resute faster.

2. "Eye-Accelerometer Combination Gestures" - You can use a Camera _facing_ the User (for "Eye-Tracking"), for the purpose of letting the User look at an Icon to select it and then "Flicking" the phone to double-click the Icon (etc.).

3. "NonTouch-Hand-Accelerometer Combination Gestures" - You can EITHER Camera to look at the User's Hand. The program would use the 'photo (video) of their finger' to point on the screen (with a simulated stylus pointer). 

Even better if the the person can walk (with some LIMITED shaking) yet still have the 'eye-view of the Pointer' remain "near" where it is intended to be (similar the the N95's motion compensation). 

A "Dampening Setting" could adjust for 'vehicle (wheelchair) shaking', the User's hand shaking due to age, walking, etc. In combination with a _limited_ understand of the web page content (yes, a bit of html interpretation) to favour the pointer moving to where there is something to be chosen.

You could print Icons on a piece of paper and the use the "away-facing Camera" to point at the Icons to navigate the browser.

Also an easy to add hack would be a "semi-tranparent-webpage-background" - where the "away-facing Camera" is used to create an Image to be used as a webpage-background (use the video camera to make the background image simulate the appearance that your phone is see-thru). When your finger is recognized the transparent webpage background would whiten and a stylus would appear where your finger is.

4. Use the Camera(s!!!), Accelerometer, GPS, (more?) to allow the phone's web-browser to "know the User's environment". 

- Have the "User-Facing-Camera" accept finger-pointing and eye-gestures.
- Have the "Regular-Camera" accept finger-pointing and let a Web-Server interact with the "Browser's Knowledge" to allow navigation withing places like "Nokia Image Space" (and any other form of "Augmented virtuality" http://en.wikipedia.org/wiki/Augmented_virtuality_)

You could walk to the Supermarket and look at what you wanted to purchase on the way. You could use your hand to point at (or Grab) a can of Soup and read the label. Then you could put it in your basket and pick it up, or just stay at home and have it delivered. 



Mirage-X Game (Mirage Money)
http://www.youtube.com/watch?v=n2RxcxEPxq0

Eye-tracking on mobile phone (Nokia E71)
http://www.youtube.com/watch?v=KAciDrBuM8k&feature=related

Eyetracking on a mobile device (Ipaq)
http://www.youtube.com/watch?v=P41sGdmov-A

Nokia N95 Accelerometer Camera Shake Detection
http://www.youtube.com/watch?v=DRcRRZ65Fno&feature=related

transparent computer screen
http://www.youtube.com/watch?v=VPd5B29ajWQ&feature=related

Holographic TV in the air - the Future of Television
http://www.youtube.com/watch?v=a_RLhle1bN4&NR=1

LWUIT - Same Interface for all Phones (in Java)
https://lwuit.dev.java.net/

AccelBall 3D
http://www.youtube.com/watch?v=ZsUhfsaNZxg&feature=related or http://sites.google.com/site/mitchsoftware/Home/accelball3d


Make your Phone a HUD with this:
EyeTech Long Distance Eye Gaze Tracking System for Market Research & Museums
http://www.youtube.com/watch?v=TWK0u8nRW2o


And here is a truly great one, "see around things", (3D Menus, _really_ see under Tiles) using real 3D depth by tilting the phone and eye-tracking:

Video: 3D Eye Tracking from TAT, the guys behind the T-Mobile G1 UI
http://www.engadget.com/2009/02/24/video-3d-eye-tracking-from-tat-the-guys-behind-the-t-mobile-g1/#continued



Remember that _some_ Phones (like the N95) have _two_ Processors and failing to use the _second_ (Graphics Processor) causes a huge slowdown in Program operation. Thus, we have enough Processor Power (on newer Phones) for these ideas.


Due to the fact that newer phones have greater Processor power, "fixed and Intergrated Camera(s)" (no need to re-adjust a "regular home webcam" each time you start the Browser (IE: calibration)), and Accelerometers along with the ability to be "fully portable" (as opposed to the "convenience" of a Laptop) we need to allow the Browser Program in the Phone to communicate whatever info it has the abilty to make available (subject to the Browser's Privacy Settings) in order to allow _both_ 

- offloading of the phone's Processing to Interactive Web-Servers and 
- the usage of the Phone's browser without having to open the Keypad to allow both able-bodied and disabled to interact with the Web in a way neber before possible (or Practical). 
- Don't envision that a "Mobile Web Browser" should be as limited as an old-tech Web Browser (like most people are stuck at home with), lets make Fennec _THE_ _Browser_ to run on your phone (due to its use of the Phone's FULL capabilites, in conjunction with (formerly simple) Web Browsing).


The Next generation Nokia S60 3D Touchless Interface
http://www.nclubsoft.com/nclubsoft/blog/?p=552

First glimpse inside Nokia S60 Touch. Going beyond Multi-Touch
http://www.unwiredview.com/2008/01/08/first-glimpse-inside-nokia-s60-touch-going-beyond-multi-touch/


A phone-based web-browser need not be as limited as a home-based web-browser,
Rob

Comment 4

[Rob]

Here is 'the math' (from the Maemo Project):

Accelerometer to pitch and roll
http://tom.pycke.be/mav/69/accelerometer-to-attitude

REF:
Accelerometers
http://wiki.maemo.org/Accelerometers



Idea: Synchronization of the Phone's vibration feature with movements detected (determined) by the accelerometer would provide a tactile feedback to the User that would provide:

A: "web pages that you could 'feel'" in your hand - useful for pushing objects.
B: a Phone that could "nearly jump out of your hand" - useful for Boxing, 'Phone on Table Dancing', 'racing crashes', swiming underwater camera.
C: a Phone that felt like "it had the wind resitance of a 'Cricket Mallet'" - useful for various web-games or Virtual Navigation in 3D.

Rob

Comment 5

[Rob]

"Touchscreen" without Touchscreen - Video Demo

MJS FluidTunes
http://majicjungle.com/fluidtunes.html


1. Use the camera to create a dimmed video image on the background.
2. Analyse image to look for tip of finger (or tipS of fingerS).
3. Use tip of finger to select on-screen Objects _OR_ "Touchscreen menu buttons" on a non-Touchscreen Device by "wafting your hand" (or "swiping").

You do not actually touch the screen (which might be out of camera view in any event) to use this '"Touchscreen" without Touchscreen' method.

Rob

Comment 6

[Rob]

For Reference here is a preliminary plugin for Fennec:

Video Demo - Fennec Gestures and Chromeless browsing
http://vimeo.com/3156495

felipc - "To test the add-on you just need to install it from addons.mozilla.org/en-US/firefox/addon/10675 . 
 You can also see the source code at 
github.com/felipc/fennec-gestures/ 
 and the tracking bug for the Gesture module is bugzilla.mozilla.org/show_bug.cgi?id=476425
 ".

Rob

Comment 7

[Rob]

(In reply to comment #5)
> "Touchscreen" without Touchscreen - Video Demo
> MJS FluidTunes
> http://majicjungle.com/fluidtunes.html
> 1. Use the camera to create a dimmed video image on the background.
> 2. Analyse image to look for tip of finger (or tipS of fingerS).
> 3. Use tip of finger to select on-screen Objects _OR_ "Touchscreen menu
>...
> Rob
The new Samsung "Galaxy S" has a "see through Phone" Application
http://gizmodo.com/5587749/the-samsung-galaxy-s-goes-see+through

It also supports using live video from phone to GoogleMap your location and provide directions.

Since this site's main page says we are dropping support for Symbian I am closing this thread.

Rob