1494678 - [BinAST] Benchmark for BinAST

Status: RESOLVED DUPLICATE of bug 1499463

(Core :: JavaScript Engine, enhancement)

Description

[David Teller [:Yoric] - still alive but not very active]

We want to benchmark BinAST speedups on a variety of site.

Protocol:
- snapshot a site;
- instrument load times;
- convert JS to BinAST;
- compare load times.

We need to determine the list of websites.

Comment 1

[David Teller [:Yoric] - still alive but not very active]

Florian, do we already have tooling that would work for instrumenting/comparing load times?

Comment 2

[David Teller [:Yoric] - still alive but not very active]

Vladan, do you have a framework for such measures that we could use to instrument other websites?

Comment 3

[Florian Quèze [:florian]]

(In reply to David Teller [:Yoric] (please use "needinfo") from comment #1)
> Florian, do we already have tooling that would work for
> instrumenting/comparing load times?

I don't have a good answer to this question unfortunately. The closest I can think of would be talos, where you would have a test that loads the page several times in a row without BinAST (record the average load time), and then with BinAST (also record the average load time), substracts the second value from the first one, and report that. This doesn't feel like a great solution, but I have no other idea at the moment.

Comment 4

[David Teller [:Yoric] - still alive but not very active]

Yeah, that's not a great solution, but if it's the only path, that's the one we'll take.

I thought I had heard about someone on your team (or a contractor?) working on a new generation benchmark for websites, but I apparently misheard/misunderstood.

Comment 5

[Florian Quèze [:florian]]

(In reply to David Teller [:Yoric] (please use "needinfo") from comment #4)

> I thought I had heard about someone on your team (or a contractor?) working
> on a new generation benchmark for websites, but I apparently
> misheard/misunderstood.

This may be refering to raptor. I don't know much details, but I think Joel knows enough to help you decide if raptor can be helpful for your use case.

Comment 6

[David Teller [:Yoric] - still alive but not very active]

Ah, yes, this was raptor. I have no details about it, though.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

We have raptor to handle the use case of AWFY where we can run a benchmark or a webpage (via mitmproxy) and compare it over time.  This is designed to run on firefox and chrome.  I assume you would record a webpage normally, then either convert it to binast, or run the pageload test with different preferences.  In the end you would have two data points and you could compare those.  Raptor is similar to Talos except that it is more lightweight and focuses on specific scenarios.

If you want to load a page normally, then load it with different prefs, then compare the data, that doesn't exist today and we would have to determine how we want to do that and add that scenario to an existing benchmark.

If we were to add features to a harness is this something that will live for a long time, or is this a short term project?

Comment 9

[Vladan Djeric (:vladan)]

> Vladan, do you have a framework for such measures that we could use to instrument other websites?

No I didn't have a general-purpose framework. Andrew Comminos (he's on Bugzilla, currently works at FB) did write a proxy that was general purpose