Closed Bug 873179 Opened 7 years ago Closed 4 years ago

Add support for JSReftests to B2G

Categories

(Testing :: Reftest, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED INCOMPLETE
mozilla26

People

(Reporter: jgriffin, Assigned: mihneadb)

References

Details

(Whiteboard: [mozharness])

Attachments

(2 files, 7 obsolete files)

We currently don't run jsreftests on B2G, and because we don't, we run them Android 2.2 noion.  If we get them running on B2G, we could stop the noion runs and builds.

Since we already have crashtests and reftests running on B2G, theoretically this shouldn't be much work.  Additionally, since I don't think the jsreftests are GL-dependent, we could probably run these on VMs.
I'd recommend doing the following:
1) Get them running locally, ignore test failures
2) Write (or adapt) a mozharness script, get it running locally
3) Test mozharness script on Ash to make sure it doesn't break other builds
4) Land mozharness changes
5) File bugs to get JSReftests running on Cedar
6) Do quick triage of failures, disable if solution isn't obvious
I got the tests running locally after creating a symlink for test-package-stage/jsreftest into test-package-stage/reftest.

Can we find how is the setup done for android noion's reftests or for b2g's reftests? I expect we can use the scripts for one of those.
the symlink is done inside of remotereftest.py.  Otherwise, we don't do any mozharness work for jsreftests on noion, but  Ibelieve we do it for panda android:)
We set the manifest for the suites in the mozharness script: https://hg.mozilla.org/build/mozharness/file/bcb2c3b49c36/scripts/b2g_emulator_unittest.py#l301

If android jsreftests are symlinking, then maybe that's the best thing to do here as well. (There's a restriction on the reftest server that the tests need to be served from a child directory which is why this is necessary).

So I think your next steps are:
1) clone mozharness (https://hg.mozilla.org/build/mozharness)
2) familiarize yourself with it and set up a local testing environment. To invoke the mozharness script you simply run it and pass in a config file, e.g 'python scripts/b2g_unittest.py --cfg configs/b2g/emulator_test_config.py --test-suite jsreftests'. To use the emulator_test_config.py you'll need to setup a local webserver, serve the files it's looking for (e.g tests.zip etc) and change the values to match your system.
3) get jsreftests working (use crashtests as a guideline as they are also run using the reftest harness)
4) file bugs to get it running on ash and cedar
I have a few questions:

* I need to setup a local http server that can serve the stuff that's defined with URLs in the config file, right?

* The strings like "%(adbpath)" are interpolated in the python files later on, is that correct?

* Basically, to set up jsreftests, I need to add a new entry in emulator_automation_config.py. Is there something else? [After I set up the environment and the correct URLs]


Thanks
(In reply to Mihnea Dobrescu-Balaur (:mihneadb) from comment #5)
> I have a few questions:
> 
> * I need to setup a local http server that can serve the stuff that's
> defined with URLs in the config file, right?
> 
> * The strings like "%(adbpath)" are interpolated in the python files later
> on, is that correct?
> 
> * Basically, to set up jsreftests, I need to add a new entry in
> emulator_automation_config.py. Is there something else? [After I set up the
> environment and the correct URLs]
> 
> 
> Thanks

You are correct on all three accounts :). You could also just copy stuff to your people account if you'd rather do that instead of setting up a local webserver.
Right, I see. I was trying to set up a virtual machine recipe with vagrant and puppet for this but now I don't really know what else I could do other than setting up the webserver.
Attached patch mozharness changes (obsolete) — Splinter Review
Assignee: nobody → mihneadb
Attached patch changes to runtestsb2g (obsolete) — Splinter Review
Need to do the same symlink hack as the android harness does.
Attachment #781361 - Attachment is obsolete: true
Attached patch mozharness changes (obsolete) — Splinter Review
Reuploading because of patch name collision
Ok, so I was able to start up the process, just that I cannot get xpcshell to work properly (it just crashes). I'm not sure if I need to add anything else to the script.
Flags: needinfo?(ahalberstadt)
Are you using the one on my people account? Is there a log/error message?
Flags: needinfo?(ahalberstadt)
That one is 32bit. I made my own 64bit xre package. No log/error msg, just xpcshell crashes and gives back return code 1.

I'm setting up an ubuntu 32bit VM and trying there.
(In reply to Mihnea Dobrescu-Balaur (:mihneadb) from comment #13)
> That one is 32bit. I made my own 64bit xre package. No log/error msg, just
> xpcshell crashes and gives back return code 1.
> 
> I'm setting up an ubuntu 32bit VM and trying there.

Ok, it works on the 32bit Ubuntu VM. I noticed I forgot some stuff in one of the diff files, will clean up asap.
Attached patch update runtestsb2g (obsolete) — Splinter Review
Performing the same symlink 'hack' that the Android harness does.
Attachment #782301 - Flags: review?(ahalberstadt)
Attachment #781362 - Attachment is obsolete: true
Added in the extra profile file parameter.
Attachment #782306 - Flags: review?(ahalberstadt)
Attachment #781363 - Attachment is obsolete: true
Comment on attachment 782301 [details] [diff] [review]
update runtestsb2g

Review of attachment 782301 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, you should use an extra os.path.join though.

::: layout/tools/reftest/runreftestb2g.py
@@ +540,5 @@
>      auto.setRemoteLog(options.remoteLogFile)
>      auto.setServerInfo(options.webServer, options.httpPort, options.sslPort)
>  
> +    # Hack in a symbolic link for jsreftest
> +    os.system("ln -s ../jsreftest " + str(os.path.join(SCRIPT_DIRECTORY, "jsreftest")))

"ln -s %s %s" % (os.path.join('..', 'jsreftest'), os.path.join(SCRIPT_DIRECTORY, 'jsreftest'))
Attachment #782301 - Flags: review?(ahalberstadt) → review+
Comment on attachment 782306 [details] [diff] [review]
mozharness changes

Review of attachment 782306 [details] [diff] [review]:
-----------------------------------------------------------------

Great! I'll file a bug to get these enabled on cedar
Attachment #782306 - Flags: review?(ahalberstadt) → review+
Attached patch update runtestsb2g (obsolete) — Splinter Review
Attachment #782301 - Attachment is obsolete: true
Comment on attachment 782756 [details] [diff] [review]
update runtestsb2g

keeping r+
Attachment #782756 - Flags: review+
I'm cool with leaving the symlink for now as that's what android is doing and it will keep you unblocked, but I filed bug 899283 to fix the root of the problem properly.
Pushed mozharness patch:
https://hg.mozilla.org/build/mozharness/rev/e70f9920171d

This was tested on Ash just in case and looks good.
Whiteboard: [mozharness]
Depends on: 900014
in production
The "update runtestsb2g" patch needs to be landed for the mozharness changes to work.
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/b6811a3ea649
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla26
Good news, they are running [1].

Now we need to split them into chunks, probably like we do in the Android-noion case.

[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=26612537&tree=Cedar&full=1
Reopening to add the buildbot cfg patch.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attachment #782756 - Attachment is obsolete: true
Forgot to change suite: to jsreftest
Attachment #791346 - Flags: review?(aki)
Attachment #791343 - Attachment is obsolete: true
Attachment #791343 - Flags: review?(aki)
Comment on attachment 791346 [details] [diff] [review]
Split B2G JSReftests into chunks in buildbot-configs

Looks good!

> JSREFTEST = [
>-    ('jsreftest', {'suite': 'reftest',
>+    ('jsreftest-1', {'suite': 'jsreftest',
>+                   'use_mozharness': True,
>+                   'script_path': 'scripts/b2g_emulator_unittest.py',
>+                   },
>+     ),
>+    ('jsreftest-2', {'suite': 'jsreftest',
>+                   'use_mozharness': True,
>+                   'script_path': 'scripts/b2g_emulator_unittest.py',
>+                   },
>+     ),
>+    ('jsreftest-3', {'suite': 'jsreftest',
>                    'use_mozharness': True,
>                    'script_path': 'scripts/b2g_emulator_unittest.py',
>                    },
>      ),
> ]

Nit: Could you get these to line up?
I think that's just a matter of adding 2 spaces in front of each of the lines ('use_mozharness', 'script_path', and the closing curly brace).

Thanks for the patch!
Attachment #791346 - Flags: review?(aki) → review+
Attachment #791346 - Attachment is obsolete: true
Comment on attachment 791376 [details] [diff] [review]
Split B2G JSReftests into chunks in buildbot-configs

keeping r+
Attachment #791376 - Flags: review+
Change merged to production branch and live on the buildbot masters.
Status: REOPENED → RESOLVED
Closed: 7 years ago4 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.