873179 - Add support for JSReftests to B2G

Status: RESOLVED INCOMPLETE

(Testing :: Reftest, defect)

Description

[Jonathan Griffin (:jgriffin)]

We currently don't run jsreftests on B2G, and because we don't, we run them Android 2.2 noion.  If we get them running on B2G, we could stop the noion runs and builds.

Since we already have crashtests and reftests running on B2G, theoretically this shouldn't be much work.  Additionally, since I don't think the jsreftests are GL-dependent, we could probably run these on VMs.

Comment 1

[Andrew Halberstadt [:ahal]]

I'd recommend doing the following:
1) Get them running locally, ignore test failures
2) Write (or adapt) a mozharness script, get it running locally
3) Test mozharness script on Ash to make sure it doesn't break other builds
4) Land mozharness changes
5) File bugs to get JSReftests running on Cedar
6) Do quick triage of failures, disable if solution isn't obvious

Comment 2

[Mihnea Dobrescu-Balaur (:mihneadb)]

I got the tests running locally after creating a symlink for test-package-stage/jsreftest into test-package-stage/reftest.

Can we find how is the setup done for android noion's reftests or for b2g's reftests? I expect we can use the scripts for one of those.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

the symlink is done inside of remotereftest.py.  Otherwise, we don't do any mozharness work for jsreftests on noion, but  Ibelieve we do it for panda android:)

Comment 4

[Andrew Halberstadt [:ahal]]

We set the manifest for the suites in the mozharness script: https://hg.mozilla.org/build/mozharness/file/bcb2c3b49c36/scripts/b2g_emulator_unittest.py#l301

If android jsreftests are symlinking, then maybe that's the best thing to do here as well. (There's a restriction on the reftest server that the tests need to be served from a child directory which is why this is necessary).

So I think your next steps are:
1) clone mozharness (https://hg.mozilla.org/build/mozharness)
2) familiarize yourself with it and set up a local testing environment. To invoke the mozharness script you simply run it and pass in a config file, e.g 'python scripts/b2g_unittest.py --cfg configs/b2g/emulator_test_config.py --test-suite jsreftests'. To use the emulator_test_config.py you'll need to setup a local webserver, serve the files it's looking for (e.g tests.zip etc) and change the values to match your system.
3) get jsreftests working (use crashtests as a guideline as they are also run using the reftest harness)
4) file bugs to get it running on ash and cedar

Comment 5

[Mihnea Dobrescu-Balaur (:mihneadb)]

I have a few questions:

* I need to setup a local http server that can serve the stuff that's defined with URLs in the config file, right?

* The strings like "%(adbpath)" are interpolated in the python files later on, is that correct?

* Basically, to set up jsreftests, I need to add a new entry in emulator_automation_config.py. Is there something else? [After I set up the environment and the correct URLs]


Thanks

Comment 6

[Andrew Halberstadt [:ahal]]

(In reply to Mihnea Dobrescu-Balaur (:mihneadb) from comment #5)
> I have a few questions:
> 
> * I need to setup a local http server that can serve the stuff that's
> defined with URLs in the config file, right?
> 
> * The strings like "%(adbpath)" are interpolated in the python files later
> on, is that correct?
> 
> * Basically, to set up jsreftests, I need to add a new entry in
> emulator_automation_config.py. Is there something else? [After I set up the
> environment and the correct URLs]
> 
> 
> Thanks

You are correct on all three accounts :). You could also just copy stuff to your people account if you'd rather do that instead of setting up a local webserver.

Comment 7

[Mihnea Dobrescu-Balaur (:mihneadb)]

Right, I see. I was trying to set up a virtual machine recipe with vagrant and puppet for this but now I don't really know what else I could do other than setting up the webserver.

Comment 8

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: mozharness changes

Comment 9

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: changes to runtestsb2g

Need to do the same symlink hack as the android harness does.

Comment 10

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: mozharness changes

Reuploading because of patch name collision

Comment 11

[Mihnea Dobrescu-Balaur (:mihneadb)]

Ok, so I was able to start up the process, just that I cannot get xpcshell to work properly (it just crashes). I'm not sure if I need to add anything else to the script.

Comment 12

[Andrew Halberstadt [:ahal]]

Are you using the one on my people account? Is there a log/error message?

Comment 13

[Mihnea Dobrescu-Balaur (:mihneadb)]

That one is 32bit. I made my own 64bit xre package. No log/error msg, just xpcshell crashes and gives back return code 1.

I'm setting up an ubuntu 32bit VM and trying there.

Comment 14

[Mihnea Dobrescu-Balaur (:mihneadb)]

(In reply to Mihnea Dobrescu-Balaur (:mihneadb) from comment #13)
> That one is 32bit. I made my own 64bit xre package. No log/error msg, just
> xpcshell crashes and gives back return code 1.
> 
> I'm setting up an ubuntu 32bit VM and trying there.

Ok, it works on the 32bit Ubuntu VM. I noticed I forgot some stuff in one of the diff files, will clean up asap.

Comment 15

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: update runtestsb2g

Performing the same symlink 'hack' that the Android harness does.

Comment 16

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: mozharness changes

Added in the extra profile file parameter.

Comment 17

[Andrew Halberstadt [:ahal]]

Comment on attachment 782301 [details] [diff] [review]
update runtestsb2g

Review of attachment 782301 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, you should use an extra os.path.join though.

::: layout/tools/reftest/runreftestb2g.py
@@ +540,5 @@
>      auto.setRemoteLog(options.remoteLogFile)
>      auto.setServerInfo(options.webServer, options.httpPort, options.sslPort)
>  
> +    # Hack in a symbolic link for jsreftest
> +    os.system("ln -s ../jsreftest " + str(os.path.join(SCRIPT_DIRECTORY, "jsreftest")))

"ln -s %s %s" % (os.path.join('..', 'jsreftest'), os.path.join(SCRIPT_DIRECTORY, 'jsreftest'))

Comment 18

[Andrew Halberstadt [:ahal]]

Comment on attachment 782306 [details] [diff] [review]
mozharness changes

Review of attachment 782306 [details] [diff] [review]:
-----------------------------------------------------------------

Great! I'll file a bug to get these enabled on cedar

Comment 19

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: update runtestsb2g

Comment 20

[Mihnea Dobrescu-Balaur (:mihneadb)]

Comment on attachment 782756 [details] [diff] [review]
update runtestsb2g

keeping r+

Comment 21

[Andrew Halberstadt [:ahal]]

I'm cool with leaving the symlink for now as that's what android is doing and it will keep you unblocked, but I filed bug 899283 to fix the root of the problem properly.

Comment 22

[Andrew Halberstadt [:ahal]]

Pushed mozharness patch:
https://hg.mozilla.org/build/mozharness/rev/e70f9920171d

This was tested on Ash just in case and looks good.

Comment 23

[John Hopkins (:jhopkins)]

in production

Comment 24

[Mihnea Dobrescu-Balaur (:mihneadb)]

The "update runtestsb2g" patch needs to be landed for the mozharness changes to work.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/b2g-inbound/rev/b6811a3ea649

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/b6811a3ea649

Comment 27

[Mihnea Dobrescu-Balaur (:mihneadb)]

Good news, they are running [1].

Now we need to split them into chunks, probably like we do in the Android-noion case.

[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=26612537&tree=Cedar&full=1

Comment 28

[Mihnea Dobrescu-Balaur (:mihneadb)]

Reopening to add the buildbot cfg patch.

Comment 29

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: Split B2G JSReftests into chunks in buildbot-configs

Comment 30

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: Split B2G JSReftests into chunks in buildbot-configs

Forgot to change suite: to jsreftest

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 791346 [details] [diff] [review]
Split B2G JSReftests into chunks in buildbot-configs

Looks good!

> JSREFTEST = [
>-    ('jsreftest', {'suite': 'reftest',
>+    ('jsreftest-1', {'suite': 'jsreftest',
>+                   'use_mozharness': True,
>+                   'script_path': 'scripts/b2g_emulator_unittest.py',
>+                   },
>+     ),
>+    ('jsreftest-2', {'suite': 'jsreftest',
>+                   'use_mozharness': True,
>+                   'script_path': 'scripts/b2g_emulator_unittest.py',
>+                   },
>+     ),
>+    ('jsreftest-3', {'suite': 'jsreftest',
>                    'use_mozharness': True,
>                    'script_path': 'scripts/b2g_emulator_unittest.py',
>                    },
>      ),
> ]

Nit: Could you get these to line up?
I think that's just a matter of adding 2 spaces in front of each of the lines ('use_mozharness', 'script_path', and the closing curly brace).

Thanks for the patch!

Comment 32

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: Split B2G JSReftests into chunks in buildbot-configs

Whoops, fixed.

Comment 33

[Mihnea Dobrescu-Balaur (:mihneadb)]

Comment on attachment 791376 [details] [diff] [review]
Split B2G JSReftests into chunks in buildbot-configs

keeping r+

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 791376 [details] [diff] [review]
Split B2G JSReftests into chunks in buildbot-configs

https://hg.mozilla.org/build/buildbot-configs/rev/a15f51b20bce

Comment 35

[Armen [:armenzg]]

Change merged to production branch and live on the buildbot masters.