697361 - Sensor API

Status: RESOLVED WONTFIX

(Core :: DOM: Core & HTML, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

We'd like to let apps access sensor data gathered by devices.  We already have orientation events, which are sort of an API to themselves and provide (a weird notion of) gravity values alongside.  It's not 100% clear how the best API approach to adding more sensors; W3C has a proposed API[1].  We at least want ambient light and proximity, and maybe ambient temperature and magnetic field.  There are a few more listed in the w3c doc.  The remaining interesting sensors (orientation, accelerometer, gyroscope) are already summarized or can be inferred by orientation events.  Probably want a good use case for raw data on those.

[1] http://dev.w3.org/2009/dap/system-info/Sensors.html

Comment 1

[Olli Pettay [:smaug][[e-mail]]]

What kind of permission handling will be used for these APIs? or will they be available only
for "trusted" apps?

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Unclear.  The main use case for ambient light and proximity is to implement features like backlight dimming and screen-off, which are definitely privileged operations.  I'm find with treating all sensor data as privileged until we have other use cases that require different APIs.

Comment 3

[Thinker Li [:sinker]]

Attachment: 01-define-nsISensorManager.patch

Definition of nsISensorManager and relative interfaces

Comment 4

[Thinker Li [:sinker]]

Attachment: 02-add-nsISensorManager-to-build.patch

Add nsISensorManager.idl to build system.  Make it being built.

Comment 5

[Thinker Li [:sinker]]

Attachment: 04-nsSensorManager-component.patch

An implementation of nsISensorManager depends on bug 697641.

Comment 6

[Thinker Li [:sinker]]

Attachment: 08-register-nsSensorManager-in-nsLayoutModule.patch

Register nsSensorManager with Gecko in nsLayoutModule.

Comment 7

[Mounir Lamouri (:mounir)]

Comment on attachment 571909 [details] [diff] [review]
01-define-nsISensorManager.patch

Review of attachment 571909 [details] [diff] [review]:
-----------------------------------------------------------------

::: xpcom/system/nsISensorManager.idl
@@ +13,5 @@
> + *
> + * The Original Code is mozilla.org code.
> + *
> + * The Initial Developer of the Original Code is Mozilla Foundation.
> + * Portions created by the Initial Developer are Copyright (C) 2009

2011

Comment 8

[Mounir Lamouri (:mounir)]

Comment on attachment 571910 [details] [diff] [review]
02-add-nsISensorManager-to-build.patch

Review of attachment 571910 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think it's worth having this patch separated from the first one. You should keep them together.

Comment 9

[Thinker Li [:sinker]]

Attachment: 01-define-nsISensorManager.patch

Fix license and merge 02-* patch to 01-* patch.

Comment 10

[Thinker Li [:sinker]]

Attachment: 04-nsSensorManager-component.patch

Fix the license issue

Comment 11

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Jose has been working on this API and (gonk) implementation in parallel.  Let's try to figure out what work can be shared between us.

Comment 12

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Jonas saw Jose's demo too! :)

We spoke briefly about the Sensors API at the WebAPI meeting this morning.  Here are the points that came up
 - new functionality to be provided is proximity and ambient light sensors.  The rest should be covered by deviceorientation.  (Unless compass/magnetic field isn't covered?)
 - there was a security concern raised over ambient light, but not proximity.  I don't fully understand the details there.
 - Jonas proposed a simpler interface for proximity/ambient light
 - ... but also proposed landing the existing Sensors API to unblock
 - Jonas is going to post to the mailing list

Backend support is coming along well.  We need someone to own the DOM API implementation.

Comment 13

[Thinker Li [:sinker]]

Attachment: part 1: Defines nsISensorManger and relative interfaces, v3

Comment 14

[Thinker Li [:sinker]]

Attachment: part 2: nsSensorManager implements nsISesnorManager interface, v3

Comment 15

[Thinker Li [:sinker]]

Attachment: part 3: Registers nsSensorManager component, v3

Comment 16

[Thinker Li [:sinker]]

Attachment: part 2: nsSensorManager implements nsISesnorManager interface, v4

--
Attachment #574261 [details] [diff] - Attachment is obsolete: true

Comment 17

[Thinker Li [:sinker]]

Attachment: part 3: Registers nsSensorManager component, v4

--
Attachment #574262 [details] [diff] - Attachment is obsolete: true

Comment 18

[Thinker Li [:sinker]]

Attachment: part 2: nsSensorManager implements nsISesnorManager interface, v5

--
Attachment #577485 [details] [diff] - Attachment is obsolete: true

Comment 19

[Thinker Li [:sinker]]

Attachment: part 1: Defines nsISensorManger and relative interfaces, v3

--
Attachment #574260 [details] [diff] - Attachment is obsolete: true

Comment 20

[Thinker Li [:sinker]]

Attachment: part 1: Defines nsISensorManger and relative interfaces, v4

--
Attachment #582479 [details] [diff] - Attachment is obsolete: true

Comment 21

[Thinker Li [:sinker]]

Attachment: part 3: Registers nsSensorManager component, v5

--
Attachment #577487 [details] [diff] - Attachment is obsolete: true

Comment 22

[StevenLee[:slee]]

Attachment: Part 1: Define SensorData and related interface proposed by W3C, V5

Comment 23

[StevenLee[:slee]]

Attachment: Part 1: Define SensorData and related interface proposed by W3C, V6

Separate sensor data and sensors into 2 files

Comment 24

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Hmm.. I don't quite understand why we are doing a sensor API for acceleration/compass. Don't we already have orientation events for that sensor?

Comment 25

[StevenLee[:slee]]

Yes, we already have a similar one, nsIDeviceMotionData. But it is a little different from the version of W3C. nsIDeviceMotionData has one more member, type, that indicates the sensor type. And the methods they used are not the same. So, I add new interface for it. Am I right?

Comment 26

[StevenLee[:slee]]

(In reply to StevenLee from comment #25)
> Yes, we already have a similar one, nsIDeviceMotionData. But it is a little
> different from the version of W3C. nsIDeviceMotionData has one more member,
> type, that indicates the sensor type. And the methods they used are not the
> same. So, I add new interface for it. Am I right?

The interface I implemented follows the spec proposed by W3C[1]. It's more general. I think we should obsolete nsIDeviceMotionData in the future.

[1] http://dev.w3.org/2009/dap/system-info/Sensors.html

Comment 27

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Steven, Jonas, how far away are we from being able to land a v0 implementation?  We'd like to use the proximity sensor for the dialer application.

Comment 28

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Temporarily landing to cgjones/mozilla-central, prior to the code being ready to land on upstream mozilla-central, is one path forward.

Comment 29

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Chris: What is the urgency on this? We currently have the orientation events which I *think* implement the same feature set as what this patch implements. I.e. magnetic field and accelerator.

José, makes a good point that the orientation API has some issues, primarily it doesn't let you control how often you want to get event feedback which would affect battery usage.

However fixing that issue doesn't seem urgent enough that we need to rush anything in, is it?

Comment 30

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Proximity sensor support is very nice-to-have for the demo-phone but not a blocker.  Whether we get that through the v0 sensor API here or stuff it into orientation events doesn't matter too much for demo-phone.

Comment 31

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Does this patch implement proximity sensor? Based on the idl interfaces in part 1 that does not appear to be the case.

Comment 32

[StevenLee[:slee]]

Chris, I get stuck in DOM API implementation. I have a simple sample. But it has problem when java script calls the function I implemented. I guess java script cannot find the correct interface. I will fix it ASAP. I will also implement a simple version with proximity sensor first.

Jonas, W3C defines that the interface supports proximity sensor but it does not have the exact definition on its page. So that I did not implement the sensor data interface of proximity sensor.

Comment 33

[StevenLee[:slee]]

(In reply to StevenLee from comment #32)
> Chris, I get stuck in DOM API implementation. I have a simple sample. But it
> has problem when java script calls the function I implemented. I guess java
> script cannot find the correct interface. I will fix it ASAP. I will also
> implement a simple version with proximity sensor first.
> 
Chris, here is the error message.

 [JavaScript Error: "uncaught exception: [Exception... "Component does not have requested interface"  nsresult: "0x80004002 (NS_NOINTERFACE)"  location: "JS frame :: http://localhost:6666/apps/clock/clock.html :: onclick :: line 1"  data: no]"]

Comment 34

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

So I have a few questions on the API itself. Steven wanted me to post here before we bring it to the DAP list.


First of all: The Sensor object has a lot of properties that doesn't seem useful to the page. The 'power', 'vendor', and 'name' properties doesn't seem like the page can do much useful with.

I'm also not sure what the page could do with 'minDelay' and 'resolution'.

This would leave the Sensor interface with very few properties, so I'd suggest we merge it with the SensorConnection interface.


As for the SensorConnection interface I also have a few question:

What is the page expected to do with the oncalibneed event? You could certainly inform the user, but what's the user expected to do? The only sensor where I could think this makes sense is the magnetic compass which at least on the iPhone can be "calibrated" by moving the phone around. Is that the only use case or do you have others in mind?

The 'error' property should use DOMError rather than SensorError. That way we'll also switch to using a string-based error rather than a number based one.

We should expose the latest read sensor data as a property on the SensorConnection interface. This is very useful for for example games which have a main loop which wants to read input data every 60 seconds. If we only have event-based ways of reading the sensor data that means that every page has to remember the result of the last event and read from the remembered value, rather than being able to use the interface directly. Seems nice if we provide this functionality for them. This is input we've gotten for other input related features like gamepad and keyboard. This way we can also get rid of the SensorDataEvent since we'd just need to fire plain events.

What is the purpose of separating the 'new' and 'open' statuses? Seems simpler for the page to just have a 'ready' status or some such.


I'm also not sure that we need to use asynchronous callbacks to get a SensorConnection object. All data from the SensorConnection is ready asynchronously anyway so the implementation won't be required to provide data synchronously just because we return a SensorConnection right away.

So here's what I propose:


interface SensorManager {
  SensorConnection getSensor(DOMString type);
  // returns null if device doesn't have sensor
}

interface SensorConnection {
  readonly attribute DOMString type;
  readonly attribute float? range;
  readonly attribute DOMString status; // ready/watching/error/closed
  readonly attribute DOMError error;

  // returns non-null values when watch is enabled
  readonly attribute float? threashold;
  readonly attribute float? interval;

  readonly attribute any data;
  readonly attribute Date lastUpdate;

  void update(); // formerly 'read()'
  void startWatch(SensorWatchOptions watchOptions);
  void endWatch();
  void close(); // Not sure if this one is needed.

           attribute Function? onerror;
           attribute Function? ondata;
           attribute Function? onstatuschange;
};


Since this is a lot of feedback I'd like to send it to the WHATWG list as soon as possible, but please do take your time to let me know if you have any feedback to these comments.

Comment 35

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Comment on attachment 598160 [details] [diff] [review]
Part 1: Define SensorData and related interface proposed by W3C, V6

Review of attachment 598160 [details] [diff] [review]:
-----------------------------------------------------------------

Removing review request since I think we'll need to do some change to the API here, per comment above.

Oh, and I of course meant the DAP list rather than the WHATWG list.

Comment 36

[Doug Turner (:dougt)]

Note - in the dap mailing list Tran did say he was going to drop the Device Orientation sensor from the spec due to the overlap with the Device Orientation WG work.

Comment 37

[StevenLee[:slee]]

Hi Doug, 
For other sensors except Device Orientation, will they be valid in this spec? Thanks~

Comment 38

[Doug Turner (:dougt)]

StevenLee, yes! :)

Comment 39

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

assinged to me for sec action to schedule a meeting

Comment 40

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Since we've landed the other sensor events, and since there's still not an agreed upon sensor API within W3C, let's mark this as WONTFIX for now. We can reopen or file a new bug if there's agreement on some Sensor API within W3C.

Comment 41

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

removing privacy/sec flags, if a new bug is opened please add them there