824748 - Run robocop tests against phone and 7" tablet UIs

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: Testing, defect, P5)

Description

[Brad Lassey [:blassey] (use needinfo?)]

As I understand it, we only run robocop tests with the tegras and pandas configured with screen sizes that would give us the tablet UI. That mean our phone and 7" tablet UIs aren't being tested and we should fix that.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this might be solved with the autophone project.  I know a lot of work to run mochitests and robocop tests on phones in general was done in Q4.

Comment 2

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Joel Maher (:jmaher) from comment #1)
> this might be solved with the autophone project.  I know a lot of work to
> run mochitests and robocop tests on phones in general was done in Q4.

Autophone is only intended to run once per day on the nightlies, so it still wouldn't be the equivalent of CI. However, we may decide that autophone is enough given the additional load that running the robocop tests on the other 2 UI's would require. I'd like to hear from the front end folks though. Finkle?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

are you implying that we order 300 tablets and 300 phones to put into our CI system?  That seems a bit overkill, I just wanted to call out the end of the spectrum.

Comment 4

[Mark Finkle (:mfinkle) (use needinfo?)]

If we could get some form of emulator working for running tests, that would be enough for me. We don't need to run different screen sizes on physical hardware. Some testing is better than no testing.

There are more variations that affect the front-end:
 device screen: Phone, tablet and phablet
 orientation: portrait and landscape
 hardware keys: hard menu button and soft menu button

I don't think we need to rush to get all configurations in test, but trying to improve our coverage would be nice. We have broken things for only one configuration before and needed to wait for a user to file a bug.

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Joel Maher (:jmaher) from comment #3)
> are you implying that we order 300 tablets and 300 phones to put into our CI
> system?  That seems a bit overkill, I just wanted to call out the end of the
> spectrum.

No, I'm suggesting running the pandas and tegras at different screen sizes.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

changing the resolution on the panda boards is not much of an option.  We have 720p as the resolution now and it seems that trying other resolutions out resulted in the crashes of the kernel or the sutagent.  

The resolution change on the tegras accounted for many of the tegras becoming unpingable.  I am not sure we can change the resolution to be much smaller than 1024x768 on the tegras.

Comment 7

[Adrian Tamas (:AdrianT)]

After Joel investigation for a Robocop bug we found out that the UI for the Tegraboards(Android 2.3) is the phone UI for Gingerbread and the UI for Pandaboards (Android 4.0) is actually the 7" tablet UI. So the tests are actually ran on Gingerbread phone UI and 7" ICS tablets.

Comment 8

[Brad Lassey [:blassey] (use needinfo?)]

filter on [mass-p5]

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Given comment 7, should this bug be closed?

Comment 10

[Adrian Tamas (:AdrianT)]

Unfortunately I have not been working with Robocop since last year and did not keep to date with the current evolution of the framework/test environment. If nothing has changed in the test environments then probably this could be closed but you might want to have a follow-up to ensure everything works also on 10" tablets. Moving the need info to the Softvision Mobile team as they may be able to add more details if there are any other.

Comment 11

[u421692]

Softvision team does not focus on the Robocop side anymore. I think that Joel is the one who can provide more info. From what I see in the logs from https://tbpl.mozilla.org/ Android 2.3 Emulator res: 1024x816 and Android 4.0 Panda res: 1280x672

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

I an not a fan of passing the needinfo flag around, but this is not an area I have worked on much in the last year.  Given the data in the logs and from comment 11, I would vote for closing this bug.

Comment 13

[Geoff Brown [:gbrown]]

Let's close it.

In review: We have Android 4.0 Panda tests at 1280x672/tablet/landscape and Android 2.3 emulator tests at 1024x816/phone/landscape. Also, autophone has come a long way since this bug was opened and regularly runs a few simple tests against various hardware; it does not currently run robocop in continuous integration, but there are current efforts to enable a wider range of tests.

Setting up another emulator configuration is simple in that a minor change to the avd settings is likely all that is required, but running that will require testing and capacity planning. I don't think we can justify the capacity for running another test configuration just for a different resolution (if you disagree -- re-open!).

On the other hand, looking ahead to our vague plans to run another emulator configuration for Android 4.4+, if we are going to run another emulator configuration, let's use different settings in support of the good ideas raised in this bug: I've noted that in bug 1062365.