Autophone - deploy robocop_adobe_flash.html

RESOLVED FIXED

Status

defect
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: bc, Assigned: bc)

Tracking

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox43 affected)

Details

Attachments

(4 attachments)

Assignee

Description

4 years ago
We incidentally test flash via the nytimes page, but need to be more robust in order to test whether in-process flash works on Android.

It appears to me that:

mobile/android/tests/browser/robocop/robocop_adobe_flash.html
mobile/android/tests/browser/robocop/testAdobeFlash.java

will go at least part of the way to getting what we want given the comment:

/**
 * Tests that Flash is working
 * - loads a page containing a Flash plugin
 * - verifies it rendered properly
 */

Since robocop_autophone.ini is not being run in production, it appears that we can add the Flash test to it and enable mochitest-robocop in Autophone production.

snorp: Will this be sufficient?
Assignee

Updated

4 years ago
Flags: needinfo?(snorp)
Yup, this should be enough for a decent smoke test at least.
Flags: needinfo?(snorp)
Assignee

Comment 2

4 years ago
This updates robocop_autophone.ini to only run the flash test. The full set of robocop tests will still be available from the robocop.ini manifest.
Attachment #8658224 - Flags: review?(gbrown)
Assignee

Comment 3

4 years ago
Attachment #8658227 - Flags: review?(gbrown)
Assignee

Comment 4

4 years ago
Attachment #8658229 - Flags: review?(gbrown)
Assignee

Comment 5

4 years ago
enable tests in production, r=self.
Attachment #8658230 - Flags: review+
Assignee

Comment 6

4 years ago
I ran several tests using the default robocop manifest to show the runtestsremote.py vs. runrobocop.py detection across branches:

mozilla-inbound:

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=065f00e9b2fd&filter-searchStr=autophone

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=17acd5b4446f&filter-searchStr=autophone

mozilla-aurora:

https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&revision=84c39fd55917&filter-searchStr=autophone

mozilla-beta:

https://treeherder.allizom.org/#/jobs?repo=mozilla-beta&revision=8d83c88ddc1f&exclusion_profile=false&filter-searchStr=autophone

mozilla-release:

https://treeherder.allizom.org/#/jobs?repo=mozilla-release&revision=24cab2512daa&filter-searchStr=autophone&exclusion_profile=false

I had run local tests of robocop_autophone.ini which appeared ok and then hacked together a test using treeherder for the robocop_autophone manifest

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=f9097a2659c6&filter-searchStr=autophone

Initially, this failed with exceptions due to the inability to pull the mochitest.log. However on a repeated attempt, it worked. I'm not sure what the issue was. It may have just been due to the way I replaced the robocop_autophone.ini manifest. gbrown, do you have any insight?

Once attachment 8658224 [details] [diff] [review] lands, we will have to get approval and land it on aurora, beta and perhaps release before we can enable the flash test on those branches.
Attachment #8658227 - Flags: review?(gbrown) → review+
Attachment #8658224 - Flags: review?(gbrown) → review+
Comment on attachment 8658229 [details] [diff] [review]
bug-1201061-set-pythonpath-v1.patch

Review of attachment 8658229 [details] [diff] [review]:
-----------------------------------------------------------------

It feels like you "shouldn't" do this in runtestsremote, but I suppose it is convenient.
Attachment #8658229 - Flags: review?(gbrown) → review+
(In reply to Bob Clary [:bc:] from comment #6)
> Initially, this failed with exceptions due to the inability to pull the
> mochitest.log. However on a repeated attempt, it worked. I'm not sure what
> the issue was. It may have just been due to the way I replaced the
> robocop_autophone.ini manifest. gbrown, do you have any insight?

I'm not sure what happened. Logcats suggest Firefox was not launched...as though it was not installed?
Assignee

Comment 9

4 years ago
(In reply to Geoff Brown [:gbrown] from comment #7)
> Comment on attachment 8658229 [details] [diff] [review]
> bug-1201061-set-pythonpath-v1.patch
> 
> Review of attachment 8658229 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> It feels like you "shouldn't" do this in runtestsremote, but I suppose it is
> convenient.

Yeah. I first tried to set up a virtualenv and to install the mozbase packages into it, but ran into problems with the lack of pinning on the requirements for each package which caused breakage on the non-nightly branches since newer versions would be installed which weren't supported by the run*test.py for the branch. Setting the PYTHONPATH to point to the versions included in the tests worked well enough I thought. Thanks.

(In reply to Geoff Brown [:gbrown] from comment #8)

> (In reply to Bob Clary [:bc:] from comment #6)
> > Initially, this failed with exceptions due to the inability to pull the
> > mochitest.log. However on a repeated attempt, it worked. I'm not sure what
> > the issue was. It may have just been due to the way I replaced the
> > robocop_autophone.ini manifest. gbrown, do you have any insight?
> 
> I'm not sure what happened. Logcats suggest Firefox was not launched...as
> though it was not installed?

Could be I screwed that up trying to test the tinderbox build and report to treeherder rather than testing my local build with the packaged tests with the patched manifest.
Assignee

Updated

4 years ago
Blocks: 1202826
Assignee

Comment 12

4 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/7793b201f389

waiting for merge to mozilla-central, fx-team, b2g-inbound before deploying.
Assignee

Comment 14

4 years ago
deployed 2015-09-09. will show up under rca jobs.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Whiteboard: [leave-open-until-deployed]
Assignee

Comment 15

4 years ago
There is a problem with the rca flash job:

Traceback (most recent call last):
  File "mochitest/runrobocop.py", line 17, in <module>
    from automation import Automation
  File "/mozilla/projects/autophone/src/autophone/builds/aHR0cDovL2Z0cC5tb3ppbGxhLm9yZy9wdWIvbW96aWxsYS5vcmcvbW9iaWxlL3RpbmRlcmJveC1idWlsZHMvbW96aWxsYS1pbmJvdW5kLWFuZHJvaWQtYXBpLTExLzE0NDE4MDA1NzgvZmVubmVjLTQzLjBhMS5lbi1VUy5hbmRyb2lkLWFybS5hcGs=/tests/mochitest/automation.py", line 26, in <module>
    from mozscreenshot import printstatus, dump_screen
  File "/mozilla/projects/autophone/src/autophone/builds/aHR0cDovL2Z0cC5tb3ppbGxhLm9yZy9wdWIvbW96aWxsYS5vcmcvbW9iaWxlL3RpbmRlcmJveC1idWlsZHMvbW96aWxsYS1pbmJvdW5kLWFuZHJvaWQtYXBpLTExLzE0NDE4MDA1NzgvZmVubmVjLTQzLjBhMS5lbi1VUy5hbmRyb2lkLWFybS5hcGs=/tests/mozbase/mozscreenshot/mozscreenshot/__init__.py", line 9, in <module>
    from mozlog.formatters.process import strstatus
ImportError: No module named process


I'll check on it later this afternoon.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee

Comment 16

4 years ago
The import error appears to have been temporary, but other issues related to the SUTAgent have appeared. I manually installed/re-installed the SUTAgent on the devices where it was failing but was not able to get it working on all devices.

nexus-s-2 ok.
nexus-s-3 execsu perm
nexus-s-4 execsu perm
nexus-s-6 execsu perm
nexus-4-jdq39-1 ok
nexus-4-jdq39-2 ok
nexus-4-jdq39-3 ok
nexus-4-jdq39-4 ok
nexus-5-kot49h-2 execsu perm
nexus-5-kot49h-3 no su
nexus-5-kot49h-4 no su
nexus-5-kot49h-5 no su
nexus-7-jss15q-2 ok

I disabled robocop_autophone on nexus 5, and nexus-s-3,4,6 but enabled it for nexus-s-2 for mozilla-inbound:

https://github.com/mozilla/autophone/commit/b1b12513d809613b4b5fb599c6d84cc49f19a85d

I'll investigate using devicemanagerADB instead of devicemanagerSUT for running the unit tests. Marking this fixed again for now.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.