unable to run mach raptor-test due to not finding ares-6 benchmark

RESOLVED FIXED in Firefox 63

Status

defect
RESOLVED FIXED
Last year
11 months ago

People

(Reporter: jmaher, Assigned: ahal)

Tracking

unspecified
mozilla63
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox63 fixed)

Details

Attachments

(1 attachment)

I am not sure what landed in the last 24 hours, but I am not able to run |mach raptor-test -t raptor-wasm-misc| anymore which I was developing.

$ ./mach raptor-test -t raptor-wasm-misc
Updating external benchmarks from https://github.com/mozilla/perf-automation
Cloning into 'c:/Users/elvis/.mozbuild\performance-tests'...
remote: Counting objects: 281, done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 281 (delta 6), reused 15 (delta 2), pack-reused 258
Receiving objects: 100% (281/281), 40.68 MiB | 2.75 MiB/s, done.
Resolving deltas: 100% (18/18), done.
Checking out files: 100% (254/254), done.
Note: checking out '6beb3d3e22abce8cf8e2e89bc45acd4152258f12'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 6beb3d3 Use window.postMessage to send results back to Raptor's control server
[Error 2] The system cannot find the file specified: u'c:\\Users\\elvis\\mozilla-inbound\\objdir-frontend\\testing\\raptor\\benchmarks\\ARES-6'


Possibly we need to adjust our cloning?
here is what I have in the performance-benchmarks repo locally:
$ ls ~/.mozbuild/performance-tests/
LICENSE  benchmarks  pagesets

elvis@JMAHER-WIN10 ~/mozilla-inbound
$ ls ~/.mozbuild/performance-tests/benchmarks/
unity-webgl


this is odd since I recently added code to performance-tests for the web-tooling-benchmark, that seems to have disappeared.
backing out bug 1472800 locally seemed to solve the above mentioned problem, but then when I run benchmarks it appears that I cannot copy data from third_part/webkit/PerformanceTests -> objdir/testing/raptor/benchmarks;  so something odd is going on.
:ahal, do you have any additional data you could provide here?
Flags: needinfo?(ahal)
That bug pins the repository to a revision (to make sure everyone is running the same version of the benchmarks locally):
https://searchfox.org/mozilla-central/source/testing/raptor/mach_commands.py#25

You'll just need to update that to something that includes your changes.

One gotcha to this sytem is that if the new revision contains unrelated changes to other benchmarks (for example if I had changed unity-webgl but then not created a new github release for those changes), doing this would update unity-webgl for people locally to a version newer than the one running in CI. This probably isn't a huge deal for now, but we may want to make it better in the future.
Flags: needinfo?(ahal)
I think the problem is that we have benchmarks in third_party/webkit/PerformanceTests which are not getting copied over- ares-6 is not in the github repo.
ares-6 is a jsshell-bench test so shouldn't have anything to do with raptor or the perf-automation repo. Do you still have problems after updating the revision I linked to in comment 4?

Fwiw I can run raptor benchmarks that live in both ~/.mozbuild/performance-tests and third_party/webkit/PerformanceTests just fine.
I get this same error on any benchmark I try to run even if it lives in third_party/ in m-c.
even rebooting my computer and working on a fresh branch didn't seem to help.  Basically right now I cannot run |mach raptor-test|.

I am on windows 10- otherwise I don't know of anything odd or custom I am doing.
Blocks: 1473078
This is a bug in how I was using the FileCopier.
Assignee: nobody → ahal
Status: NEW → ASSIGNED
This fixes a bug where the benchmarks weren't being copied properly on Windows.
While it is slightly slower to add every file to the manifest (~0.2 seconds), it
still outperforms shutil.copytree and is more robust (will use links on Linux/Mac
while using copy on Windows).
Comment on attachment 8999699 [details]
Bug 1482057 - [raptor] Fix benchmark setup on Windows, r=jmaher

Joel Maher ( :jmaher ) (UTC+2) has approved the revision.
Attachment #8999699 - Flags: review+
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f88a04dbc4b1
[raptor] Fix benchmark setup on Windows, r=jmaher
https://hg.mozilla.org/mozilla-central/rev/f88a04dbc4b1
Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
You need to log in before you can comment on or make changes to this bug.