Closed Bug 846129 Opened 7 years ago Closed 3 years ago

Find a way to extract "UI model" from UI implementation (HTML) to use in automated tests

Categories

(Core :: General, defect)

x86_64
Linux
defect
Not set

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: cjones, Unassigned)

References

Details

This project is mainly geared at automated testing of gaia functionality at the UI level.  Here's the kind of test I want to write
 1. Launch dialer (somehow)
 2. Request UI model.  (Harness waits for it to stabilize, sends back to test code.)
 3. Get "keypad" region of sceen.  Assert that it exists.
 4. Get clickable objects that represent numbers "1" "2" "3" say.  "Click" those in succession.
 5. Wait for UI model to change.  Harness notifies when it does.
 6. Verify that the new model is an "oncall screen".  (Somehow.)
 7. Verify that it has a text label serving the purpose of listing the dialed number, with text "123".

My understanding is that the current gaia unit tests (being unit tests) are rather lower-level than that.  They reach into app JS and explicitly call functions.  These kinds of tests have a high maintenance burden, because small code refactorings that preserve user-visible functionality can nonetheless break the tests.  I would prefer that we migrate away from those, but they certainly serve a purpose atm.

We some other tests running on gaia that are slightly higher-level, in that they reach into the page DOM directly and manipulate DOM nodes.  These tests are similarly high-burden, because UI "refactorings", or changes to style-class names e.g., can preserve functionality but break tests.

In both cases, the flaw is directly testing the /implementation/, rather than an interface.  If a code change causes a test to fail, there's no way to automatically blame either the test or the implementation.  They may both be "right", just out of sync.

So I propose that we add an interface layer in between, a layer of abstraction.  We should build this interface out of components that we want to change extremely infrequently:
 A. UI interaction models
 B. l10n string identifiers

In that way, if a code change causes a test to fail, and the UI model hasn't changed, we can immediately blame the code change.

To me, this seems like a problem shared with a11y: extract an abstraction of a page that's only the interaction model.

It might also be helpful to write interaction specs in terms of the interaction model that we can test against.  Then UX folks or whoever else can write tests of the model, while developers go to implement it.

Another problem here is that the interface for interacting with that model should be asynchronous.  In b2g for example, the model presented to the user will include interactible elements from several OS processes.  It just makes things easier to build that way from the start.  (Neither directly manipulating JS nor HTML satisfies that.)

We can certainly agree on a set of "model rules" and jury-rig a solution to this problem.  However, I'd like to understand the prior art, and see if we can leverage other features we need in product anyway.

This work merely covers a mechanism for defining and extracting the model.  To interact with it, we'll need harness code; building on marionette seems like the right way to go.  Separate bug! :)
I may be wrong here, but I don't think a generic "UI model" is so useful here.

I agree with the need to have integration tests that does not involve "tapping on x=250, y=140" and rather use "trigger the click event on the element with id 'dialer-1-button'". I think this can be done per app, eventually adding the necessary handles that will not change (through dataset, aria, classes, id, microdata, rdfa) on elements.

I don't think we need another abstraction over this. Merely leverage existing or new semantic data is enough.
I don't entirely follow, but element IDs are the kind of implementation detail I'd want to leave out.
(In reply to Julien Wajsberg [:julienw] from comment #1)
> I may be wrong here, but I don't think a generic "UI model" is so useful
> here.
> 
> I agree with the need to have integration tests that does not involve
> "tapping on x=250, y=140" and rather use "trigger the click event on the
> element with id 'dialer-1-button'". I think this can be done per app,
> eventually adding the necessary handles that will not change (through
> dataset, aria, classes, id, microdata, rdfa) on elements.

We've seen problems with this approach with the automated gaia smoketests. Should the element's id or some other implementation detail change, then its tests fail.t We saw the exact problems that cjones has listed, where if a test fails, then we don't know immediately if the test is revealing a problem with the application, or if the test itself is out of date, and time has to be taken to verify the cause and rectify it.

I think that cjones' suggestion would help distinguish between these two kinds of failures, and would help us write tests that don't require so much maintenance.
What we need from the point of view of platform tests is the ability to extract information like "which application is currently open?" and "is the security dialog open?" (or preferrably "let me know when the security dialog comes up") and take actions like "press the deny button and use the 'remember' feature".

So yeah, I don't think element id's is the right solution since ids change and pieces of UI might even move around between different documents/windows.

I think the solution is to have a JS library which has functions for the various high-level (aka "semantic") operations that we want to perform, and have marionette interact with this JS library. Whenver the UI changes we'd just need to keep the API that the library exposes working.
(In reply to Jonas Sicking (:sicking) from comment #4)
> What we need from the point of view of platform tests is the ability to
> extract information like "which application is currently open?" and "is the
> security dialog open?" (or preferrably "let me know when the security dialog

So, the 11y API's we currently expose to js through the xpidl in accessible/public/ should allow you to answer those questions if you formulate them like

"what is the name / title of the document that is in the forground / focus within."

"I would like to be notified when documents are created and  check if there name is ""application X want's privlidge Y".

now I'm not sure testing that through a11y api's much better than just listening for dom events and poking at the DOM directly.

> comes up") and take actions like "press the deny button and use the
> 'remember' feature".

so, you should be able to do this to formulated something like

find the button whose exposed name / text is "deny" and click it.  Now that seems like a better test than click the element whose id is "button_1_deny" but again I feel like you could write that test just using the DOM directly though it might be somewhat simpler with an a11y api that computes the "exposed name" for you (so you don't have to deal with value="" vs sub tree vs title="" etc).

> So yeah, I don't think element id's is the right solution since ids change
> and pieces of UI might even move around between different documents/windows.

So, I'm not sure how much you can test if you don't assume that UI lives in a particular document / window (though that of course depends on the exact definition of those terms).

> I think the solution is to have a JS library which has functions for the
> various high-level (aka "semantic") operations that we want to perform, and
> have marionette interact with this JS library. Whenver the UI changes we'd
> just need to keep the API that the library exposes working.

I'd tend to agree, but them the question becomes how to implement said library.
As noted in comment 0, keep in mind that the on-screen UI model comprises DOMs from several OS processes composed together.  Poking at the DOM directly is not a route to sane testing IMHO.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #6)
> As noted in comment 0, keep in mind that the on-screen UI model comprises
> DOMs from several OS processes composed together.  Poking at the DOM
> directly is not a route to sane testing IMHO.

err, yeah, forgot about that.  TBH I can't say how a11y works in the multi process case, the platform api's sort of mostly worked on desktop for e10s, but I don't think anyone ever really tried the xpcom one, so I'd sort of expect its broken other than I've never heard Eitan complain.

Eitan may well have tried and know more.
The accessibility APIs probably doesn't help much compared to using the DOM since I believe the accessibility tree is as much per-document and per-process as the DOM is. At least I think that's the case.
> but I don't think anyone ever really tried the xpcom one, so I'd sort of
> expect its broken other than I've never heard Eitan complain.

ok, memories came back from wherever they were cache evicted to, and I suspect it does work if you poke at the accessibility tree for stuff in a content process from within that content process.

(In reply to Jonas Sicking (:sicking) from comment #8)
> The accessibility APIs probably doesn't help much compared to using the DOM
> since I believe the accessibility tree is as much per-document and
> per-process as the DOM is. At least I think that's the case.

yes it is.
(These are yet more reasons why I'd like us to keep a11y in mind for this project, since they need to solve similar problems.)
Ok, I now understand this is especially useful for tests that are not in the gaia code base.

As I understand, you'd like this UI model to be extracted automatically from the DOM tree ? Or would the dev change themselves the model for the app they change ?
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.