Bug 1460411 (geckoview_mochitest)

Run mochitests in TestRunnerActivity on packet.net

NEW
Assigned to

Status

()

enhancement
P1
normal
Last year
13 days ago

People

(Reporter: gbrown, Assigned: snorp)

Tracking

(Depends on 10 bugs, {leave-open})

Trunk
Points:
---

Firefox Tracking Flags

(firefox62 affected)

Details

Attachments

(3 attachments)

No description provided.
This switches mochitest/reftest from Fennec to TestRunnerActivity, but just using our regular 4.3 armv7 emulator on aws, for now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0fc44e09d6ed9eb9118a5047db0be18e75025df

To verify we are using TestRunnerActivity, check the logs for something like:

adb launch_application: am start -W -n org.mozilla.geckoview.test/org.mozilla.geckoview.test.TestRunnerActivity
:snorp - You were asking about the current state of TestRunnerActivity for mochitest/reftest. See the try run in comment 1: There's lot of goodness, but also a variety of failures.
Flags: needinfo?(snorp)
Some of these failures look familiar to me from when I initially got things running on GeckoView. We probably have some work to do in order to green these up.
Flags: needinfo?(snorp)
Depends on: 1291387
Will initially push some configuration changes with everything disabled...
Keywords: leave-open
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a097fbd2403
Add tooltool manifest for androidx86_7_0 avd archive; r=me,a=test-only
https://hg.mozilla.org/integration/mozilla-inbound/rev/958824009b6c
Add mozharness configuration for androidx86_7_0 tests; r=me,a=test-only
https://hg.mozilla.org/integration/mozilla-inbound/rev/06f48abf78dc
Add taskcluster configuration for android-x86-7.0 tests; r=me,a=test-only
This is looking pretty wonderful:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=62ae20bad95904555e783dee1c423b2dd7c18fb0

Notice that all of these are running at least 10x faster than 4.3/armv7.
There are also challenges, with lots of failing tests, like comments 1-3:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=23571fe13a3974dd800c9e844f347dbe6fc1bb6a
I do not expect these changes to have any effect on existing test platforms: Only the new android x86 emulator platform, which will run with docker --privileged.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9475a169daff94a761d251ba25782dc732d6fe1
Attachment #8975842 - Flags: review?(jmaher)
Comment on attachment 8975842 [details] [diff] [review]
add kvm to desktop1604-test docker image

Review of attachment 8975842 [details] [diff] [review]:
-----------------------------------------------------------------

does this increase the size of the docker image?  I suspect it is insignificant, but want to double check.
Attachment #8975842 - Flags: review?(jmaher) → review+
There is a size increase of about 10%.

Existing task logs show:

Downloaded 1418.856 mb

From the try push, we now have:

Downloaded 1559.251 mb


I suppose some of that may just be from re-building the image (ubuntu updates, etc).
now we have it documented :)  I think the larger image is better than creating a second image which is harder to maintain.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8a820882e927
Add kvm to desktop1604-test image; r=jmaher
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/12bfd7a50a79
Follow-up - diffoscope repo moved; updated url; r=me on a CLOSED TREE
It would be nice to have artifacts soon.

On aws, Android test tasks use:

"artifacts": {
    "public/logs/": {
      "expires": "2018-05-30T23:44:14.575Z",
      "path": "/builds/worker/workspace/build/upload/logs/",
      "type": "directory"
    },
    "public/test": {
      "expires": "2018-05-30T23:44:14.575Z",
      "path": "/builds/worker/artifacts/",
      "type": "directory"
    },
    "public/test_info/": {
      "expires": "2018-05-30T23:44:14.575Z",
      "path": "/builds/worker/workspace/build/blobber_upload_dir/",
      "type": "directory"
    }
  },

Notice they are all "directory" types: That would be convenient for packet.net also, but I can't get directory artifacts to work.

https://tools.taskcluster.net/groups/HF-RiXDdQey2h9B79d0iaA/tasks/cgdxwrUMQkyQYS8L4v7eNg/details used:

  "artifacts": [
    {
      "name": "public/logs/",
      "path": "/builds/worker/workspace/build/upload/logs/",
      "type": "directory"
    },
    {
      "name": "public/test",
      "path": "/builds/worker/artifacts/",
      "type": "directory"
    },
    {
      "name": "public/test_info/",
      "path": "/builds/worker/workspace/build/blobber_upload_dir/",
      "type": "directory"
    }
  ],

and failed:

[taskcluster:error]  String 'public/logs/' at task.payload.artifacts[0].name doesn't match regular expression '^([\x20-\x2e\x30-\x7e][\x20-\x7e]*)[\x20-\x2e\x30-\x7e]$'
[taskcluster:error]  String '/builds/worker/workspace/build/upload/logs/' at task.payload.artifacts[0].path doesn't match regular expression '^.*[^/]$'
[taskcluster:error]  String '/builds/worker/artifacts/' at task.payload.artifacts[1].path doesn't match regular expression '^.*[^/]$'
[taskcluster:error]  String '/builds/worker/workspace/build/blobber_upload_dir/' at task.payload.artifacts[2].path doesn't match regular expression '^.*[^/]$'
[taskcluster:error]  String 'public/test_info/' at task.payload.artifacts[2].name doesn't match regular expression '^([\x20-\x2e\x30-\x7e][\x20-\x7e]*)[\x20-\x2e\x30-\x7e]$'


If I remove the trailing /'s:

https://tools.taskcluster.net/groups/ezF8WLo2RFewYu3tC3HkIw/tasks/bUqw4PeFQxCOsKTqHNLOUA/details

  "artifacts": [
    {
      "name": "public/logs",
      "path": "/builds/worker/workspace/build/upload/logs",
      "type": "directory"
    },
    {
      "name": "public/test",
      "path": "/builds/worker/artifacts",
      "type": "directory"
    },
    {
      "name": "public/test_info",
      "path": "/builds/worker/workspace/build/blobber_upload_dir",
      "type": "directory"
    }
  ],

the task fails with an unspecified exception (I see no error message, nor any log at all).


Are you guys aware of these problems? Fix coming soon??
Flags: needinfo?(wcosta)
Flags: needinfo?(jopsen)
Quick update before I take off for a week...

Taskcluster improvements needed:
 - relengapi or another way of accessing internal tooltool downloads
 - directory artifacts (comment 16)
 - scalable provisioning
 - it would be nice if the payload schema was approximately the same for all docker-worker, to avoid this kind of complication: https://hg.mozilla.org/try/rev/d1c86d9227a1f2a89c781f94a2ed0f961eba88ed#l2.13
 - it would be nice if we could use in-tree images

I need to compare performance vs cost for at least t1.small.x86 vs c1.small.x86.

Failing tests:
 - mochitest-chrome appears to be busted
 - I've started to identify failing mochitests: https://hg.mozilla.org/try/rev/438d12e39c9e2513a10e9c19c8385c5c5c514a09
 - I am concerned about plain reftests, but they may be doing better with swiftshader_indirect

https://treeherder.mozilla.org/#/jobs?repo=try&revision=015a24454c07fb6037586e304cc1163f66a9a287
I tried directory artifacts again: They still fail.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f198b6f6d82d32834be4607b8cf5e6d5556d8b0b

https://tools.taskcluster.net/groups/DW0uuKW6S6aB5kSA1sq4xQ/tasks/Byh3NHDIQ4aUTnnLYEb3CA/runs/3/logs/public%2Flogs%2Flive.log

"artifacts": [
    {
      "name": "public/logs",
      "path": "/builds/worker/workspace/build/upload/logs",
      "type": "directory"
    },
    {
      "name": "public/test",
      "path": "/builds/worker/artifacts",
      "type": "directory"
    },
    {
      "name": "public/test_info",
      "path": "/builds/worker/workspace/build/blobber_upload_dir",
      "type": "directory"
    }
  ],
Flags: needinfo?(wcosta)
An initial trial of the worker variants:

tc-worker-docker-v1-A1 (t1.small.x86: 4 atom, 4g ram; concurrency: 1)
tc-worker-docker-v1-A4 (t1.small.x86: 4 atom, 4g ram; concurrency: 4)
tc-worker-docker-v1-B1 (c1.small.x86: 4 xeon, 32g ram ; concurrency: 1)
tc-worker-docker-v1-B4 (c1.small.x86: 4 xeon, 32g ram ; concurrency: 4)

https://treeherder.mozilla.org/#/jobs?repo=try&author=gbrown@mozilla.com&tochange=867da42edfac08fd0a9454f4bc3814401fdae282&fromchange=7b836283b6f849a5090ade28bfe2a1c6d9ad9753

I'll clean up some of the easy test failures, then repeat with --rebuild for some more solid data.
Reminder of taskcluster improvements needed:
 - relengapi or another way of accessing internal tooltool downloads (just needs to be deployed?)
 - folder artifacts (reportedly supported, but doesn't work for me: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f198b6f6d82d32834be4607b8cf5e6d5556d8b0b)
 - in-tree images (reportedly supported, but doesn't work for me: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3ca1f383e276ea21a86f2c87701dc91a1a4de450)


I tried repeating the worker variants experiment with --rebuild:

https://treeherder.mozilla.org/#/jobs?repo=try&author=gbrown@mozilla.com&tochange=f1ca3d458035c9b101a55a94fdd809784ba70262&fromchange=b331ee6f709ed31625864c2365c7d9739d21a947

The B1 and B4 runs are okay (lots of test failures -- :jmaher is helping to "green" them up).
A1 was running well for many hours, then starting failing -- I'll repeat.
A4 instances were initially okay but now report that all disk space is used -- I'm trying to find where it has gone to.
(In reply to Geoff Brown [:gbrown] from comment #21)
> A4 instances were initially okay but now report that all disk space is used
> -- I'm trying to find where it has gone to.

df showed / was 100% full but 'du -sx /*' found nothing unusual (same directory sizes as on A1, where there is no lack of free disk space). I noticed that df also showed /etc/hosts as mounted (expected?) and full, but before I could investigate further, I lost all access to the A4:

https://tools.taskcluster.net/groups/Y13ZaKn4R2u6pzZdndTVVg/tasks/Y13ZaKn4R2u6pzZdndTVVg/runs/0/logs/public%2Flogs%2Flive.log

  Unhandled worker error encountered incidentID= 27f81395-0a71-4c79-87ac-662326e78814

Jonas - can you get the A4s unstuck?
Depends on: 1466657
I've given up on the A4s. I think that's asking too much of the t1.small.x86; even A1 is pretty slow. Jonas might configure an A2 for comparison.
Approximate times and costs per test type (at fixed, quoted rates of $0.07/hour and $0.40/hour for t1.small.x86 and c1.small.x86):

                   A1 minutes    A1 cost    B1 minutes     B1 cost      B4 minutes      B4 cost     aws minutes
geckoview-junit         6        $0.007          3         $0.020            3          $0.005           25
mochitest-plain       140        $0.163         45         $0.300           55          $0.092          840
mochitest-clipboard     7        $0.008          3         $0.020            3          $0.005           20
mochitest-gpu	        7        $0.008          3         $0.020            3          $0.005           10
mochitest-media	       35        $0.041         15         $0.100           15          $0.025           80
crashtest	       30        $0.035          8         $0.053           10          $0.017          140
jsreftest	      400        $0.467         95         $0.633          150          $0.250         2000
reftest-plain         225        $0.263         50         $0.333           75          $0.125         1260

Interpretation: The mochitest-plain tests take about 140 minutes to run (perhaps in several chunks) on the A1 type. See comment 20 for the A1/B1/B4 configurations. 140 / 60 x $0.07 = $0.163: It costs about $0.163 to run mochitest-plain on an A1. It costs about $0.300 to run mochitest-plain on B1 and about $0.092 to run mochitest-plain on B4. (Costs on the B4 are $0.40/hour / 4 = $0.10/hour...but jobs take a little longer than on the B1.) "aws minutes" are included for time comparison to our current test on the arm emulator on aws, but I'm not sure how to estimate cost for that.

In general, tests run slowest on the A1, fastest on the B1, and cheapest on the B4.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d22112a18625
Minor changes to test configs for android x86 7.0; r=me,a=test-only
(In reply to Geoff Brown [:gbrown] from comment #25)
> In general, tests run slowest on the A1, fastest on the B1, and cheapest on the B4.

We discussed this in yesterday's mobile testing meeting. B4 seems to be the preferred choice: Almost as fast as B1 and the cheapest option. There was some concern about the possibility of one task affected another task running on the same hardware at the same time, but at this time we don't see any evidence of that sort of problem; should it arise, we'd want to be able to run in a B1 configuration, possibly on short notice.
Depends on: 1468532
It seems like tasks are still not starting (trouble started just before the All Hands, when we were trying to deploy fixes, I think):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c4ce0de68e0e8df6163cc17f2aea01ee74bf02ae

Hopefully Jonas and Wander are working on this...?
I was experimenting with running using TestRunnerActivity on Bitbar.

Normal Fennec tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2255314a96b2bedf3469c5643aed31b3643d9c2&filter-tier=1&filter-tier=2&filter-tier=3

TestRunnerActivity tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=977823485e411bc18845cf55e78b7b65a1f82581&filter-tier=1&filter-tier=2&filter-tier=3

There are new failures for TestRunnerActivity which do not appear in Fennec though some of the errors are the same I believe.

I see cases where the task fails with the TestRunnerActivity with a claim-expired exception which I do not see in Fennec. I also see cases where it appears the task hangs with TestRunnerActivity which results in Bitbar killing it after 2 hours.

I see similar results for some limited tests I ran on mda and jit today.
(In reply to Geoff Brown [:gbrown] from comment #30)
> It seems like tasks are still not starting (trouble started just before the
> All Hands, when we were trying to deploy fixes, I think):

Jonas got this working again.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=29b3da069801326332178fdcd791edd7e09acf1e
Flags: needinfo?(jopsen)
Folder artifacts are still working.
tooltool public downloads from relengapi are working.
Priority: -- → P1
tooltool INTERNAL downloads from relengapi are working.

https://treeherder.mozilla.org/logviewer.html#?job_id=184244540&repo=try&lineNumber=666-689

[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO - Contents:
[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO -  
[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO -          [
[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO -          {
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "size": 193383673,
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "digest": "6609e8b95db59c6a3ad60fc3dcfc358b2c8ec8b4dda4c2780eb439e1c5dcc5d550f2e47ce56ba14309363070078d09b5287e372f6e95686110ff8a2ef1838221",
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "algorithm": "sha512",
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "filename": "android-sdk18_0.r18moz1.orig.tar.gz",
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "unpack": "True"
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          }
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          ] 
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - retry: Calling run_command with args: (['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/tooltool.tt', '-o', '-c', '/builds/worker/tooltool_cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f200969f178>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f200a011c48>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7f2009fa4648>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/builds/worker/workspace/build', 'privileged': False}, attempt #1
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - Running command: ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/tooltool.tt', '-o', '-c', '/builds/worker/tooltool_cache'] in /builds/worker/workspace/build
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - Copy/paste: /usr/bin/python2.7 -u /builds/worker/workspace/mozharness/external_tools/tooltool.py --url http://taskcluster/relengapi/tooltool/ fetch -m /builds/worker/workspace/build/tooltool.tt -o -c /builds/worker/tooltool_cache
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - Calling ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/tooltool.tt', '-o', '-c', '/builds/worker/tooltool_cache'] with output_timeout 600
[taskcluster]  GEThttps://api.pub.build.mozilla.org/tooltool/sha512/6609e8b95db59c6a3ad60fc3dcfc358b2c8ec8b4dda4c2780eb439e1c5dcc5d550f2e47ce56ba14309363070078d09b5287e372f6e95686110ff8a2ef1838221
[task 2018-06-21T19:43:23.816Z] 19:43:23     INFO -  INFO - File android-sdk18_0.r18moz1.orig.tar.gz not present in local cache folder /builds/worker/tooltool_cache
[task 2018-06-21T19:43:23.816Z] 19:43:23     INFO -  INFO - Attempting to fetch from 'http://taskcluster/relengapi/tooltool/'...
[task 2018-06-21T19:43:34.086Z] 19:43:34     INFO -  INFO - File android-sdk18_0.r18moz1.orig.tar.gz fetched from http://taskcluster/relengapi/tooltool/ as /builds/worker/workspace/build/tmplRAmmD
[task 2018-06-21T19:43:34.448Z] 19:43:34     INFO -  INFO - File integrity verified, renaming tmplRAmmD to android-sdk18_0.r18moz1.orig.tar.gz
[task 2018-06-21T19:43:34.448Z] 19:43:34     INFO -  INFO - Updating local cache /builds/worker/tooltool_cache...
[task 2018-06-21T19:43:34.550Z] 19:43:34     INFO -  INFO - Local cache /builds/worker/tooltool_cache updated with android-sdk18_0.r18moz1.orig.tar.gz
[task 2018-06-21T19:43:34.551Z] 19:43:34     INFO -  INFO - untarring "android-sdk18_0.r18moz1.orig.tar.gz"
[task 2018-06-21T19:43:36.791Z] 19:43:36     INFO - Return code: 0


(I have been having trouble downloading the file I actually need...but it must be some other issue...)
(In reply to Geoff Brown [:gbrown] from comment #35)
> (I have been having trouble downloading the file I actually need...but it
> must be some other issue...)

...must have been some problem in tooltool. I uploaded a new copy of the sdk with 27.1.12 x86 emulator and all is well.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bbeaea4044cf302bf144aaa6a2dce2a5e0d45986
Now, with internal tooltool downloads OK, I could start using the normal in-tree image, but when I try that, the task fails:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec25df5b9dc4e1fd4b52b5b86b6938496a8efdbb

https://tools.taskcluster.net/groups/LPomFkB-Ti6aRctjdlI39A/tasks/ElmgQ2TyRs2L-JnBCThhuQ/runs/0/logs/public%2Flogs%2Flive.log

[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 0 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 7 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 18 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 27 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 39 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 53 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 71 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 79 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 99 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster:error]  Unhandled worker error encountered incidentID= 23ee419f-b3b7-4703-8ddb-28b45d2bb74c
:snorp - You were asking about running tests on packet.net. Here's a new example:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1279e55fe97874ce9a60b7248fbe92449a837a60

Just include that patch and you should be able to schedule tests with 'mach try fuzzy' and a query like 'android x86 7.0'. Capacity is limited currently, but I hope to get more soon.
Flags: needinfo?(snorp)
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/89ccff5672f2
Update configuration for androidx86_7_0 tests; r=me,a=test-only
(In reply to Geoff Brown [:gbrown] from comment #38)
> :snorp - You were asking about running tests on packet.net. Here's a new example:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e9685046bdc500b83dac698ba45adcbed338f95f

(The patch keeps evolving -- I'll keep posting the latest here.)
(In reply to Geoff Brown [:gbrown] from comment #38)
> :snorp - You were asking about running tests on packet.net. Here's a new
> example:
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=1279e55fe97874ce9a60b7248fbe92449a837a60
> 
> Just include that patch and you should be able to schedule tests with 'mach
> try fuzzy' and a query like 'android x86 7.0'. Capacity is limited
> currently, but I hope to get more soon.

Thanks!
Flags: needinfo?(snorp)
Depends on: 1473632
Depends on: 1473635
packet.net tasks are currently not starting. :wcosta is investigating...
(In reply to Geoff Brown [:gbrown] from comment #43)
> packet.net tasks are currently not starting. :wcosta is investigating...

Fixed today: packet.net tasks are running again. Also, there's more capacity: up to 20 tasks running concurrently.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e979a4bf967bed885a98143372da2bf8cfadb314
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.
Priority: P1 → P5
Priority: P5 → P1
wcosta reports that docker-worker for packet.net is ready. He ran this task manually:

https://tools.taskcluster.net/groups/Vct_vCm8RXWEJdUjTQVVvA/tasks/JeIGMqcuSV2r9pKkhWAAmQ/details
Depends on: 1479584
This works great for me on try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=45d7316337f6f2f5042834a64b610d7dce99effb

(the geckoview team is working on greening up tests, so I'm not actually enabling/running any new tests yet -- just getting ready for that.)

I feel like I am struggling a little with setting 'privileged' in a proper/tidy fashion. I think it is better to do this than to push it out to the yml with yet another set of by-test-platform's. 'privileged' is required for all android-em-7.0-x86 tests: all the android emulator tests run on packet.net.
Attachment #8996848 - Flags: review?(dustin)
Comment on attachment 8996848 [details] [diff] [review]
support android-em-7.0-x86 tests on docker-worker

Oops, looks like :dustin is not available, and I am eager to move forward here...
Attachment #8996848 - Flags: review?(dustin) → review?(jmaher)
Comment on attachment 8996848 [details] [diff] [review]
support android-em-7.0-x86 tests on docker-worker

Review of attachment 8996848 [details] [diff] [review]:
-----------------------------------------------------------------

this is a simple set of changes.

::: taskcluster/taskgraph/transforms/tests.py
@@ +571,5 @@
>              test['treeherder-machine-platform'] = test['test-platform']
>          elif 'android-hw' in test['test-platform']:
>              test['treeherder-machine-platform'] = test['test-platform']
> +        elif 'android-em-7.0-x86' in test['test-platform']:
> +            test['treeherder-machine-platform'] = 'android-em-7-0-x86/opt'

will we do debug as well?
Attachment #8996848 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher ) (UTC+2) from comment #51)
> will we do debug as well?

We may eventually, but there are no concrete plans today. A more complex transform may be required in future.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4af8de99d321
Add taskcluster config support for Android x86 7.0 tests on packet.net in docker-worker; r=jmaher
Depends on: 1481587
Depends on: 1481945
Tests can be run by explicit selection in 'try fuzzy':

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9cd1ad5225081416b71a0ed4580f9ca65aa1cbd&filter-tier=1&filter-tier=2&filter-tier=3

but do not run with -u all:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=926fc18e63469d7ee96ede8a688e7b0ee38c7140&filter-tier=1&filter-tier=2&filter-tier=3

mochitest-plain, reftest, and mochitest-media still have lots of failing tests, so I'm not attempting to enable those yet.
Attachment #8999015 - Flags: review?(jmaher)
Comment on attachment 8999015 [details] [diff] [review]
run some geckoview tests on mozilla-central, tier 3

Review of attachment 8999015 [details] [diff] [review]:
-----------------------------------------------------------------

just curious about running tests in duplicate- I think as these are tier3 there is nothing actionable

::: taskcluster/ci/test/misc.yml
@@ +37,5 @@
> +            default: default
> +    run-on-projects:
> +        by-test-platform:
> +            android-em-7.0-x86/opt: ['mozilla-central']
> +            default: built-projects

as these are running on arm emulators, when could we turn them off there and only run on x86?
Attachment #8999015 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher ) (UTC+2) from comment #57)
> as these are running on arm emulators, when could we turn them off there and
> only run on x86?

I think we should promote the x86 ones to tier 1 first...hopefully not too far off.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/79cb6b81ad3e
Enable some Android x86 7.0 geckoview tests, tier 3, mozilla-central only; r=jmaher
This failed when it merged to master, due to missing scopes for `queue:create-task:highest:terraform-packet/gecko-t-linux`. I've gone ahead and add `queue:create-task:highest:terraform-packet/gecko-t-*` to `moz-tree:level:3:gecko`.
(In reply to Tom Prince [:tomprince] from comment #60)
> This failed when it merged to master, due to missing scopes for
> `queue:create-task:highest:terraform-packet/gecko-t-linux`. I've gone ahead
> and add `queue:create-task:highest:terraform-packet/gecko-t-*` to
> `moz-tree:level:3:gecko`.

Oh! I hadn't thought of that. Thanks Tom!
Several tests are running on Android x86 7.0, in TestRunnerActivity, on mozilla-central now. They are tier 3: hidden by default, so select tier 3 in the treeherder "Tiers" menu to see them.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=android%207.0&filter-tier=1&filter-tier=2&filter-tier=3

These are not running on inbound, autoland, or anywhere else yet.

They can be run on try, but only by explicitly selecting them in 'mach try fuzzy'.
Plain mochitests, media mochitests, and plain reftests are excluded, since they still have many failures. To run these on try, use a patch like:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=403d6c087dbb98aeadde8c1c36efaded7e3b832f&filter-tier=1&filter-tier=2&filter-tier=3
Depends on: 1494388
Alias: geckoview_mochitest
Depends on: 1507157
Depends on: 1507158
Depends on: 1507159
Depends on: 1507160
Depends on: 1507162
Depends on: 1507164
Depends on: 1507166
Depends on: 1507167
Depends on: 1507170
Depends on: 1510448
Depends on: 1511205
Depends on: 1513607
Depends on: 1514283
Depends on: 1514289
Depends on: 1514349
Depends on: 1514360

:snorp's driving all the hard work here these days.

Assignee: gbrown → snorp
Depends on: 1523625
Depends on: 1523674
Depends on: 1523675
No longer blocks: 1523692
Depends on: 1523692

With bug 1498298, tier 1 tests move to x86_64 builds. It might be best to do the same for the mochitests and reftests contemplated in this bug. The taskcluster patch only shifts slightly, as seen here:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=a50af786c7dfe77cb535e2ab698c4efde81f23e5

I think all the current work here is on mochitests. reftests also have failures on packet.net, but let's pursue those in a separate bug - bug 1501582.

Summary: Run mochitests and reftests in TestRunnerActivity on packet.net → Run mochitests in TestRunnerActivity on packet.net
Duplicate of this bug: 1494388
Depends on: 1525925
Depends on: 1527355
Depends on: 1530464
No longer depends on: 1507167
You need to log in before you can comment on or make changes to this bug.