1768092 - Local run of `mach raptor -t sunspider` fails

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Steve Fink [:sfink] [:s:]]

How do you run a raptor test locally? When I do mach raptor -t sunspider, for example, it fails with CRITICAL - raptor-perftest Critical: Failed to run sunspider: NetworkError when attempting to fetch resource.. This is preceded by many error messages saying raptor-control-server Info: received request for unknown file: /json/sunspider.json.

There are two problems with that: first, why is it trying to fetch that file from the control server instead of the content server or whatever it's called? Second, even if I try manually making an HTTP request to the content server, it fails because it looks for $objdir/testing/raptor/benchmarks/json/sunspider.json which does not exist. The relevant file appears to be $srcdir/testing/raptor/raptor/tests/json/sunspider.json.

I will note that while raptor is running (stalled in a pdb prompt, actually), $srcdir/testing/raptor/webext/raptor/auto_gen_test_config.js contains:

// this file is auto-generated by raptor, do not edit directly
function getTestConfig() {
  return {
    "cs_port": "56091",
    "test_name": "sunspider",
    "test_settings_url": "http://127.0.0.1:56091/json/sunspider.json",
    "post_startup_delay": "30000",
    "benchmark_port": "35091",
    "host": "127.0.0.1",
    "debug_mode": "0",
    "browser_cycle": "1"
  };
}

Those port numbers are correct; 56091 does seem to be the control port, and I can get content back from port 35091 if I do symlink tricks to put the data where it's looking.

Anyway, STR:

• make a local build
• run mach raptor -t sunspider

Comment 1

[Steve Fink [:sfink] [:s:]]

Hm, maybe that is supposed to be coming from the control server? I just noticed the test_settings_url, which has the control port in it. So perhaps the control server isn't serving from the right place? But I got to here because do_GET() in control_server.py insists:

            if tail.startswith("raptor") and tail.endswith(".json"):

and "sunspider.json" does not start with "raptor". It looks like it wants something like "/raptor/sunspider.json" that it'll turn into "json/sunspider.json"? So confused.

Comment 2

[Steve Fink [:sfink] [:s:]]

I don't know how that would work given gen_test_config.py has:

    "test_settings_url": "http://{host}:{control_server_port}/json/{test}.json",

No "raptor" in sight.

Comment 3

[Steve Fink [:sfink] [:s:]]

It all works if I remove the tail.startswith("raptor") clause.

Comment 4

[Greg Mierzwinski [:sparky]]

:sfink, can you try this command: ./mach raptor -t sunspider --browsertime

I think the issue here is that raptor thinks you want to run the test through the webextension but that is no longer supported for the most part.

Comment 5

[Greg Mierzwinski [:sparky]]

I've got a patch here to deal with this issue: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=513d08c4a2b7cbf514019a635a6cf3acdce4e6e5

We should be running browsertime by default now and not the webextension.

Comment 6

[Steve Fink [:sfink] [:s:]]

(In reply to Greg Mierzwinski [:sparky] from comment #4)

:sfink, can you try this command: ./mach raptor -t sunspider --browsertime

I think the issue here is that raptor thinks you want to run the test through the webextension but that is no longer supported for the most part.

That's what I ended up doing. I naively thought that I should start with the seemingly most basic form of the command, but for what I want to do, I need --browsertime anyway.

Should the webext mechanism be removed, or at least the local run portion of it? Or fail unless you pass --yes-i-know-its-deprecated-run-it-anyway?

Comment 7

[Steve Fink [:sfink] [:s:]]

Ah, it looks like it's still used for a few tests.

Comment 8

[Greg Mierzwinski [:sparky]]

(In reply to Steve Fink [:sfink] [:s:] from comment #6)

(In reply to Greg Mierzwinski [:sparky] from comment #4)

:sfink, can you try this command: ./mach raptor -t sunspider --browsertime

I think the issue here is that raptor thinks you want to run the test through the webextension but that is no longer supported for the most part.

That's what I ended up doing. I naively thought that I should start with the seemingly most basic form of the command, but for what I want to do, I need --browsertime anyway.

Should the webext mechanism be removed, or at least the local run portion of it? Or fail unless you pass --yes-i-know-its-deprecated-run-it-anyway?

As you noticed, there are still some tests which use it unfortunately. There's still a couple outstanding issues we need to work out before we can turn off or port those tests. I've made a patch that resolves the issue for me locally but we'll likely have some discussions before going further with it. It's possible that we get rid of those cpu/memory/power tests in the very near future allowing us to make a patch to fully disable webext.

Comment 9

[Kash Shampur [:kshampur] ⌚EST]

I believe now browsertime is now default, so --browsertime should not be needed.

:sfink could you confirm if this works as expected now?

Comment 10

[Steve Fink [:sfink] [:s:]]

Yes, I don't need the unsupported configuration, and the change of default means I don't need to worry about this anymore. Thanks!

Comment 11

[Kash Shampur [:kshampur] ⌚EST]

Great to hear, thanks for confirming!