1460411 - (geckoview_mochitest) Run mochitests in TestRunnerActivity on packet.net

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, enhancement, P2)

Comment 1

[Geoff Brown [:gbrown]]

This switches mochitest/reftest from Fennec to TestRunnerActivity, but just using our regular 4.3 armv7 emulator on aws, for now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0fc44e09d6ed9eb9118a5047db0be18e75025df

To verify we are using TestRunnerActivity, check the logs for something like:

adb launch_application: am start -W -n org.mozilla.geckoview.test/org.mozilla.geckoview.test.TestRunnerActivity

Comment 2

[Geoff Brown [:gbrown]]

:snorp - You were asking about the current state of TestRunnerActivity for mochitest/reftest. See the try run in comment 1: There's lot of goodness, but also a variety of failures.

Comment 3

[James Willcox (:snorp) ([e-mail]) (he/him)]

Some of these failures look familiar to me from when I initially got things running on GeckoView. We probably have some work to do in order to green these up.

Comment 4

[Geoff Brown [:gbrown]]

Will initially push some configuration changes with everything disabled...

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a097fbd2403
Add tooltool manifest for androidx86_7_0 avd archive; r=me,a=test-only
https://hg.mozilla.org/integration/mozilla-inbound/rev/958824009b6c
Add mozharness configuration for androidx86_7_0 tests; r=me,a=test-only
https://hg.mozilla.org/integration/mozilla-inbound/rev/06f48abf78dc
Add taskcluster configuration for android-x86-7.0 tests; r=me,a=test-only

Comment 6

[Geoff Brown [:gbrown]]

This is looking pretty wonderful:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=62ae20bad95904555e783dee1c423b2dd7c18fb0

Notice that all of these are running at least 10x faster than 4.3/armv7.

Comment 7

[Geoff Brown [:gbrown]]

There are also challenges, with lots of failing tests, like comments 1-3:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=23571fe13a3974dd800c9e844f347dbe6fc1bb6a

Comment 8

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/2a097fbd2403
https://hg.mozilla.org/mozilla-central/rev/958824009b6c
https://hg.mozilla.org/mozilla-central/rev/06f48abf78dc

Comment 9

[Geoff Brown [:gbrown]]

Attachment: add kvm to desktop1604-test docker image

I do not expect these changes to have any effect on existing test platforms: Only the new android x86 emulator platform, which will run with docker --privileged.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9475a169daff94a761d251ba25782dc732d6fe1

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8975842 [details] [diff] [review]
add kvm to desktop1604-test docker image

Review of attachment 8975842 [details] [diff] [review]:
-----------------------------------------------------------------

does this increase the size of the docker image?  I suspect it is insignificant, but want to double check.

Comment 11

[Geoff Brown [:gbrown]]

There is a size increase of about 10%.

Existing task logs show:

Downloaded 1418.856 mb

From the try push, we now have:

Downloaded 1559.251 mb


I suppose some of that may just be from re-building the image (ubuntu updates, etc).

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

now we have it documented :)  I think the larger image is better than creating a second image which is harder to maintain.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8a820882e927
Add kvm to desktop1604-test image; r=jmaher

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/12bfd7a50a79
Follow-up - diffoscope repo moved; updated url; r=me on a CLOSED TREE

Comment 15

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/8a820882e927
https://hg.mozilla.org/mozilla-central/rev/12bfd7a50a79

Comment 16

[Geoff Brown [:gbrown]]

It would be nice to have artifacts soon.

On aws, Android test tasks use:

"artifacts": {
    "public/logs/": {
      "expires": "2018-05-30T23:44:14.575Z",
      "path": "/builds/worker/workspace/build/upload/logs/",
      "type": "directory"
    },
    "public/test": {
      "expires": "2018-05-30T23:44:14.575Z",
      "path": "/builds/worker/artifacts/",
      "type": "directory"
    },
    "public/test_info/": {
      "expires": "2018-05-30T23:44:14.575Z",
      "path": "/builds/worker/workspace/build/blobber_upload_dir/",
      "type": "directory"
    }
  },

Notice they are all "directory" types: That would be convenient for packet.net also, but I can't get directory artifacts to work.

https://tools.taskcluster.net/groups/HF-RiXDdQey2h9B79d0iaA/tasks/cgdxwrUMQkyQYS8L4v7eNg/details used:

  "artifacts": [
    {
      "name": "public/logs/",
      "path": "/builds/worker/workspace/build/upload/logs/",
      "type": "directory"
    },
    {
      "name": "public/test",
      "path": "/builds/worker/artifacts/",
      "type": "directory"
    },
    {
      "name": "public/test_info/",
      "path": "/builds/worker/workspace/build/blobber_upload_dir/",
      "type": "directory"
    }
  ],

and failed:

[taskcluster:error]  String 'public/logs/' at task.payload.artifacts[0].name doesn't match regular expression '^([\x20-\x2e\x30-\x7e][\x20-\x7e]*)[\x20-\x2e\x30-\x7e]$'
[taskcluster:error]  String '/builds/worker/workspace/build/upload/logs/' at task.payload.artifacts[0].path doesn't match regular expression '^.*[^/]$'
[taskcluster:error]  String '/builds/worker/artifacts/' at task.payload.artifacts[1].path doesn't match regular expression '^.*[^/]$'
[taskcluster:error]  String '/builds/worker/workspace/build/blobber_upload_dir/' at task.payload.artifacts[2].path doesn't match regular expression '^.*[^/]$'
[taskcluster:error]  String 'public/test_info/' at task.payload.artifacts[2].name doesn't match regular expression '^([\x20-\x2e\x30-\x7e][\x20-\x7e]*)[\x20-\x2e\x30-\x7e]$'


If I remove the trailing /'s:

https://tools.taskcluster.net/groups/ezF8WLo2RFewYu3tC3HkIw/tasks/bUqw4PeFQxCOsKTqHNLOUA/details

  "artifacts": [
    {
      "name": "public/logs",
      "path": "/builds/worker/workspace/build/upload/logs",
      "type": "directory"
    },
    {
      "name": "public/test",
      "path": "/builds/worker/artifacts",
      "type": "directory"
    },
    {
      "name": "public/test_info",
      "path": "/builds/worker/workspace/build/blobber_upload_dir",
      "type": "directory"
    }
  ],

the task fails with an unspecified exception (I see no error message, nor any log at all).


Are you guys aware of these problems? Fix coming soon??

Comment 17

[Geoff Brown [:gbrown]]

Quick update before I take off for a week...

Taskcluster improvements needed:
 - relengapi or another way of accessing internal tooltool downloads
 - directory artifacts (comment 16)
 - scalable provisioning
 - it would be nice if the payload schema was approximately the same for all docker-worker, to avoid this kind of complication: https://hg.mozilla.org/try/rev/d1c86d9227a1f2a89c781f94a2ed0f961eba88ed#l2.13
 - it would be nice if we could use in-tree images

I need to compare performance vs cost for at least t1.small.x86 vs c1.small.x86.

Failing tests:
 - mochitest-chrome appears to be busted
 - I've started to identify failing mochitests: https://hg.mozilla.org/try/rev/438d12e39c9e2513a10e9c19c8385c5c5c514a09
 - I am concerned about plain reftests, but they may be doing better with swiftshader_indirect

https://treeherder.mozilla.org/#/jobs?repo=try&revision=015a24454c07fb6037586e304cc1163f66a9a287

Comment 18

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=03f75634cdf18a2bcf1e0e6df79fac656dc6ef0f

Comment 19

[Geoff Brown [:gbrown]]

I tried directory artifacts again: They still fail.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f198b6f6d82d32834be4607b8cf5e6d5556d8b0b

https://tools.taskcluster.net/groups/DW0uuKW6S6aB5kSA1sq4xQ/tasks/Byh3NHDIQ4aUTnnLYEb3CA/runs/3/logs/public%2Flogs%2Flive.log

"artifacts": [
    {
      "name": "public/logs",
      "path": "/builds/worker/workspace/build/upload/logs",
      "type": "directory"
    },
    {
      "name": "public/test",
      "path": "/builds/worker/artifacts",
      "type": "directory"
    },
    {
      "name": "public/test_info",
      "path": "/builds/worker/workspace/build/blobber_upload_dir",
      "type": "directory"
    }
  ],

Comment 20

[Geoff Brown [:gbrown]]

An initial trial of the worker variants:

tc-worker-docker-v1-A1 (t1.small.x86: 4 atom, 4g ram; concurrency: 1)
tc-worker-docker-v1-A4 (t1.small.x86: 4 atom, 4g ram; concurrency: 4)
tc-worker-docker-v1-B1 (c1.small.x86: 4 xeon, 32g ram ; concurrency: 1)
tc-worker-docker-v1-B4 (c1.small.x86: 4 xeon, 32g ram ; concurrency: 4)

https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&tochange=867da42edfac08fd0a9454f4bc3814401fdae282&fromchange=7b836283b6f849a5090ade28bfe2a1c6d9ad9753

I'll clean up some of the easy test failures, then repeat with --rebuild for some more solid data.

Comment 21

[Geoff Brown [:gbrown]]

Reminder of taskcluster improvements needed:
 - relengapi or another way of accessing internal tooltool downloads (just needs to be deployed?)
 - folder artifacts (reportedly supported, but doesn't work for me: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f198b6f6d82d32834be4607b8cf5e6d5556d8b0b)
 - in-tree images (reportedly supported, but doesn't work for me: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3ca1f383e276ea21a86f2c87701dc91a1a4de450)


I tried repeating the worker variants experiment with --rebuild:

https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&tochange=f1ca3d458035c9b101a55a94fdd809784ba70262&fromchange=b331ee6f709ed31625864c2365c7d9739d21a947

The B1 and B4 runs are okay (lots of test failures -- :jmaher is helping to "green" them up).
A1 was running well for many hours, then starting failing -- I'll repeat.
A4 instances were initially okay but now report that all disk space is used -- I'm trying to find where it has gone to.

Comment 22

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #21)
> A4 instances were initially okay but now report that all disk space is used
> -- I'm trying to find where it has gone to.

df showed / was 100% full but 'du -sx /*' found nothing unusual (same directory sizes as on A1, where there is no lack of free disk space). I noticed that df also showed /etc/hosts as mounted (expected?) and full, but before I could investigate further, I lost all access to the A4:

https://tools.taskcluster.net/groups/Y13ZaKn4R2u6pzZdndTVVg/tasks/Y13ZaKn4R2u6pzZdndTVVg/runs/0/logs/public%2Flogs%2Flive.log

  Unhandled worker error encountered incidentID= 27f81395-0a71-4c79-87ac-662326e78814

Jonas - can you get the A4s unstuck?

Comment 23

[Geoff Brown [:gbrown]]

I've given up on the A4s. I think that's asking too much of the t1.small.x86; even A1 is pretty slow. Jonas might configure an A2 for comparison.

Comment 24

[Geoff Brown [:gbrown]]

Some additional experiments:

A1: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0a26f10f2135d0f1a6361cc09f5cb0681d13f56e
B1: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4a386adb8c521c0a8727ee909b3088c28e282c9d
B4: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5770ee5f5e63decf3aa964964669be9aa5c520a1

Comment 25

[Geoff Brown [:gbrown]]

Approximate times and costs per test type (at fixed, quoted rates of $0.07/hour and $0.40/hour for t1.small.x86 and c1.small.x86):

                   A1 minutes    A1 cost    B1 minutes     B1 cost      B4 minutes      B4 cost     aws minutes
geckoview-junit         6        $0.007          3         $0.020            3          $0.005           25
mochitest-plain       140        $0.163         45         $0.300           55          $0.092          840
mochitest-clipboard     7        $0.008          3         $0.020            3          $0.005           20
mochitest-gpu	        7        $0.008          3         $0.020            3          $0.005           10
mochitest-media	       35        $0.041         15         $0.100           15          $0.025           80
crashtest	       30        $0.035          8         $0.053           10          $0.017          140
jsreftest	      400        $0.467         95         $0.633          150          $0.250         2000
reftest-plain         225        $0.263         50         $0.333           75          $0.125         1260

Interpretation: The mochitest-plain tests take about 140 minutes to run (perhaps in several chunks) on the A1 type. See comment 20 for the A1/B1/B4 configurations. 140 / 60 x $0.07 = $0.163: It costs about $0.163 to run mochitest-plain on an A1. It costs about $0.300 to run mochitest-plain on B1 and about $0.092 to run mochitest-plain on B4. (Costs on the B4 are $0.40/hour / 4 = $0.10/hour...but jobs take a little longer than on the B1.) "aws minutes" are included for time comparison to our current test on the arm emulator on aws, but I'm not sure how to estimate cost for that.

In general, tests run slowest on the A1, fastest on the B1, and cheapest on the B4.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d22112a18625
Minor changes to test configs for android x86 7.0; r=me,a=test-only

Comment 27

[Geoff Brown [:gbrown]]

Folder artifacts work now: 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a832329da64b86654600e405776e62e6d1aa9fa0

Comment 28

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #25)
> In general, tests run slowest on the A1, fastest on the B1, and cheapest on the B4.

We discussed this in yesterday's mobile testing meeting. B4 seems to be the preferred choice: Almost as fast as B1 and the cheapest option. There was some concern about the possibility of one task affected another task running on the same hardware at the same time, but at this time we don't see any evidence of that sort of problem; should it arise, we'd want to be able to run in a B1 configuration, possibly on short notice.

Comment 29

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/d22112a18625

Comment 30

[Geoff Brown [:gbrown]]

It seems like tasks are still not starting (trouble started just before the All Hands, when we were trying to deploy fixes, I think):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c4ce0de68e0e8df6163cc17f2aea01ee74bf02ae

Hopefully Jonas and Wander are working on this...?

Comment 31

[Bob Clary [:bc] (inactive)]

I was experimenting with running using TestRunnerActivity on Bitbar.

Normal Fennec tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2255314a96b2bedf3469c5643aed31b3643d9c2&filter-tier=1&filter-tier=2&filter-tier=3

TestRunnerActivity tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=977823485e411bc18845cf55e78b7b65a1f82581&filter-tier=1&filter-tier=2&filter-tier=3

There are new failures for TestRunnerActivity which do not appear in Fennec though some of the errors are the same I believe.

I see cases where the task fails with the TestRunnerActivity with a claim-expired exception which I do not see in Fennec. I also see cases where it appears the task hangs with TestRunnerActivity which results in Bitbar killing it after 2 hours.

I see similar results for some limited tests I ran on mda and jit today.

Comment 32

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #30)
> It seems like tasks are still not starting (trouble started just before the
> All Hands, when we were trying to deploy fixes, I think):

Jonas got this working again.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=29b3da069801326332178fdcd791edd7e09acf1e

Comment 33

[Geoff Brown [:gbrown]]

Folder artifacts are still working.

Comment 34

[Geoff Brown [:gbrown]]

tooltool public downloads from relengapi are working.

Comment 35

[Geoff Brown [:gbrown]]

tooltool INTERNAL downloads from relengapi are working.

https://treeherder.mozilla.org/logviewer.html#?job_id=184244540&repo=try&lineNumber=666-689

[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO - Contents:
[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO -  
[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO -          [
[task 2018-06-21T19:43:23.785Z] 19:43:23     INFO -          {
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "size": 193383673,
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "digest": "6609e8b95db59c6a3ad60fc3dcfc358b2c8ec8b4dda4c2780eb439e1c5dcc5d550f2e47ce56ba14309363070078d09b5287e372f6e95686110ff8a2ef1838221",
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "algorithm": "sha512",
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "filename": "android-sdk18_0.r18moz1.orig.tar.gz",
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          "unpack": "True"
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          }
[task 2018-06-21T19:43:23.786Z] 19:43:23     INFO -          ] 
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - retry: Calling run_command with args: (['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/tooltool.tt', '-o', '-c', '/builds/worker/tooltool_cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f200969f178>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f200a011c48>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7f2009fa4648>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/builds/worker/workspace/build', 'privileged': False}, attempt #1
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - Running command: ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/tooltool.tt', '-o', '-c', '/builds/worker/tooltool_cache'] in /builds/worker/workspace/build
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - Copy/paste: /usr/bin/python2.7 -u /builds/worker/workspace/mozharness/external_tools/tooltool.py --url http://taskcluster/relengapi/tooltool/ fetch -m /builds/worker/workspace/build/tooltool.tt -o -c /builds/worker/tooltool_cache
[task 2018-06-21T19:43:23.787Z] 19:43:23     INFO - Calling ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/relengapi/tooltool/', 'fetch', '-m', '/builds/worker/workspace/build/tooltool.tt', '-o', '-c', '/builds/worker/tooltool_cache'] with output_timeout 600
[taskcluster]  GEThttps://api.pub.build.mozilla.org/tooltool/sha512/6609e8b95db59c6a3ad60fc3dcfc358b2c8ec8b4dda4c2780eb439e1c5dcc5d550f2e47ce56ba14309363070078d09b5287e372f6e95686110ff8a2ef1838221
[task 2018-06-21T19:43:23.816Z] 19:43:23     INFO -  INFO - File android-sdk18_0.r18moz1.orig.tar.gz not present in local cache folder /builds/worker/tooltool_cache
[task 2018-06-21T19:43:23.816Z] 19:43:23     INFO -  INFO - Attempting to fetch from 'http://taskcluster/relengapi/tooltool/'...
[task 2018-06-21T19:43:34.086Z] 19:43:34     INFO -  INFO - File android-sdk18_0.r18moz1.orig.tar.gz fetched from http://taskcluster/relengapi/tooltool/ as /builds/worker/workspace/build/tmplRAmmD
[task 2018-06-21T19:43:34.448Z] 19:43:34     INFO -  INFO - File integrity verified, renaming tmplRAmmD to android-sdk18_0.r18moz1.orig.tar.gz
[task 2018-06-21T19:43:34.448Z] 19:43:34     INFO -  INFO - Updating local cache /builds/worker/tooltool_cache...
[task 2018-06-21T19:43:34.550Z] 19:43:34     INFO -  INFO - Local cache /builds/worker/tooltool_cache updated with android-sdk18_0.r18moz1.orig.tar.gz
[task 2018-06-21T19:43:34.551Z] 19:43:34     INFO -  INFO - untarring "android-sdk18_0.r18moz1.orig.tar.gz"
[task 2018-06-21T19:43:36.791Z] 19:43:36     INFO - Return code: 0


(I have been having trouble downloading the file I actually need...but it must be some other issue...)

Comment 36

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #35)
> (I have been having trouble downloading the file I actually need...but it
> must be some other issue...)

...must have been some problem in tooltool. I uploaded a new copy of the sdk with 27.1.12 x86 emulator and all is well.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bbeaea4044cf302bf144aaa6a2dce2a5e0d45986

Comment 37

[Geoff Brown [:gbrown]]

Now, with internal tooltool downloads OK, I could start using the normal in-tree image, but when I try that, the task fails:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec25df5b9dc4e1fd4b52b5b86b6938496a8efdbb

https://tools.taskcluster.net/groups/LPomFkB-Ti6aRctjdlI39A/tasks/ElmgQ2TyRs2L-JnBCThhuQ/runs/0/logs/public%2Flogs%2Flive.log

[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 0 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 7 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 18 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 27 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 39 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 53 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 71 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 79 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 99 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster]  Fetching image: artifact public/image.tar.zst from DL6HrWWdSbi8UCGfcg84Gw/0 100 %
[taskcluster:error]  Unhandled worker error encountered incidentID= 23ee419f-b3b7-4703-8ddb-28b45d2bb74c

Comment 38

[Geoff Brown [:gbrown]]

:snorp - You were asking about running tests on packet.net. Here's a new example:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1279e55fe97874ce9a60b7248fbe92449a837a60

Just include that patch and you should be able to schedule tests with 'mach try fuzzy' and a query like 'android x86 7.0'. Capacity is limited currently, but I hope to get more soon.

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/89ccff5672f2
Update configuration for androidx86_7_0 tests; r=me,a=test-only

Comment 40

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/89ccff5672f2

Comment 41

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #38)
> :snorp - You were asking about running tests on packet.net. Here's a new example:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e9685046bdc500b83dac698ba45adcbed338f95f

(The patch keeps evolving -- I'll keep posting the latest here.)

Comment 42

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Geoff Brown [:gbrown] from comment #38)
> :snorp - You were asking about running tests on packet.net. Here's a new
> example:
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=1279e55fe97874ce9a60b7248fbe92449a837a60
> 
> Just include that patch and you should be able to schedule tests with 'mach
> try fuzzy' and a query like 'android x86 7.0'. Capacity is limited
> currently, but I hope to get more soon.

Thanks!

Comment 43

[Geoff Brown [:gbrown]]

packet.net tasks are currently not starting. :wcosta is investigating...

Comment 44

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #43)
> packet.net tasks are currently not starting. :wcosta is investigating...

Fixed today: packet.net tasks are running again. Also, there's more capacity: up to 20 tasks running concurrently.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e979a4bf967bed885a98143372da2bf8cfadb314

Comment 45

[Firefox Bug Husbandry Bot]

Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.

Comment 46

[Geoff Brown [:gbrown]]

wcosta reports that docker-worker for packet.net is ready. He ran this task manually:

https://tools.taskcluster.net/groups/Vct_vCm8RXWEJdUjTQVVvA/tasks/JeIGMqcuSV2r9pKkhWAAmQ/details

Comment 47

[Geoff Brown [:gbrown]]

With docker-worker: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce593980c529b8f3e595aac03610bc40833e9dbd

Comment 48

[Geoff Brown [:gbrown]]

Looking good!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=300d11a54c31ccb6f8f9101012fc3b4f8d9b0af8

Comment 49

[Geoff Brown [:gbrown]]

Attachment: support android-em-7.0-x86 tests on docker-worker

This works great for me on try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=45d7316337f6f2f5042834a64b610d7dce99effb

(the geckoview team is working on greening up tests, so I'm not actually enabling/running any new tests yet -- just getting ready for that.)

I feel like I am struggling a little with setting 'privileged' in a proper/tidy fashion. I think it is better to do this than to push it out to the yml with yet another set of by-test-platform's. 'privileged' is required for all android-em-7.0-x86 tests: all the android emulator tests run on packet.net.

Comment 50

[Geoff Brown [:gbrown]]

Comment on attachment 8996848 [details] [diff] [review]
support android-em-7.0-x86 tests on docker-worker

Oops, looks like :dustin is not available, and I am eager to move forward here...

Comment 51

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8996848 [details] [diff] [review]
support android-em-7.0-x86 tests on docker-worker

Review of attachment 8996848 [details] [diff] [review]:
-----------------------------------------------------------------

this is a simple set of changes.

::: taskcluster/taskgraph/transforms/tests.py
@@ +571,5 @@
>              test['treeherder-machine-platform'] = test['test-platform']
>          elif 'android-hw' in test['test-platform']:
>              test['treeherder-machine-platform'] = test['test-platform']
> +        elif 'android-em-7.0-x86' in test['test-platform']:
> +            test['treeherder-machine-platform'] = 'android-em-7-0-x86/opt'

will we do debug as well?

Comment 52

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher ) (UTC+2) from comment #51)
> will we do debug as well?

We may eventually, but there are no concrete plans today. A more complex transform may be required in future.

Comment 53

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4af8de99d321
Add taskcluster config support for Android x86 7.0 tests on packet.net in docker-worker; r=jmaher

Comment 54

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/4af8de99d321

Comment 55

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c8b5bfdb914c30c17ff45fcaaec52ed8b9234d27

Comment 56

[Geoff Brown [:gbrown]]

Attachment: run some geckoview tests on mozilla-central, tier 3

Tests can be run by explicit selection in 'try fuzzy':

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9cd1ad5225081416b71a0ed4580f9ca65aa1cbd&filter-tier=1&filter-tier=2&filter-tier=3

but do not run with -u all:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=926fc18e63469d7ee96ede8a688e7b0ee38c7140&filter-tier=1&filter-tier=2&filter-tier=3

mochitest-plain, reftest, and mochitest-media still have lots of failing tests, so I'm not attempting to enable those yet.

Comment 57

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8999015 [details] [diff] [review]
run some geckoview tests on mozilla-central, tier 3

Review of attachment 8999015 [details] [diff] [review]:
-----------------------------------------------------------------

just curious about running tests in duplicate- I think as these are tier3 there is nothing actionable

::: taskcluster/ci/test/misc.yml
@@ +37,5 @@
> +            default: default
> +    run-on-projects:
> +        by-test-platform:
> +            android-em-7.0-x86/opt: ['mozilla-central']
> +            default: built-projects

as these are running on arm emulators, when could we turn them off there and only run on x86?

Comment 58

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher ) (UTC+2) from comment #57)
> as these are running on arm emulators, when could we turn them off there and
> only run on x86?

I think we should promote the x86 ones to tier 1 first...hopefully not too far off.

Comment 59

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/79cb6b81ad3e
Enable some Android x86 7.0 geckoview tests, tier 3, mozilla-central only; r=jmaher

Comment 60

[Tom Prince [:tomprince]]

This failed when it merged to master, due to missing scopes for `queue:create-task:highest:terraform-packet/gecko-t-linux`. I've gone ahead and add `queue:create-task:highest:terraform-packet/gecko-t-*` to `moz-tree:level:3:gecko`.

Comment 61

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/79cb6b81ad3e

Comment 62

[Geoff Brown [:gbrown]]

(In reply to Tom Prince [:tomprince] from comment #60)
> This failed when it merged to master, due to missing scopes for
> `queue:create-task:highest:terraform-packet/gecko-t-linux`. I've gone ahead
> and add `queue:create-task:highest:terraform-packet/gecko-t-*` to
> `moz-tree:level:3:gecko`.

Oh! I hadn't thought of that. Thanks Tom!

Comment 63

[Geoff Brown [:gbrown]]

Several tests are running on Android x86 7.0, in TestRunnerActivity, on mozilla-central now. They are tier 3: hidden by default, so select tier 3 in the treeherder "Tiers" menu to see them.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=android%207.0&filter-tier=1&filter-tier=2&filter-tier=3

These are not running on inbound, autoland, or anywhere else yet.

They can be run on try, but only by explicitly selecting them in 'mach try fuzzy'.

Comment 64

[Geoff Brown [:gbrown]]

Plain mochitests, media mochitests, and plain reftests are excluded, since they still have many failures. To run these on try, use a patch like:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=403d6c087dbb98aeadde8c1c36efaded7e3b832f&filter-tier=1&filter-tier=2&filter-tier=3

Comment 65

[Geoff Brown [:gbrown]]

:snorp's driving all the hard work here these days.

Comment 66

[Geoff Brown [:gbrown]]

With bug 1498298, tier 1 tests move to x86_64 builds. It might be best to do the same for the mochitests and reftests contemplated in this bug. The taskcluster patch only shifts slightly, as seen here:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=a50af786c7dfe77cb535e2ab698c4efde81f23e5

Comment 67

[Geoff Brown [:gbrown]]

I think all the current work here is on mochitests. reftests also have failures on packet.net, but let's pursue those in a separate bug - bug 1501582.

Comment 69

[Chris Peterson [:cpeterson]]

Downgrading priority from P1 to P2

Comment 70

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:snorp, maybe it's time to close this bug?

Comment 71

[James Willcox (:snorp) ([e-mail]) (he/him)]

Seems ok to me!