1310917 - Add a project "Chrome" under framework "Hasal"

Status: RESOLVED WONTFIX

(Tree Management :: Perfherder, defect)

Description

[Bobby Chien]

As nightly testing, we would like to keep track Chrome testing history to againt Firefox. Also, compare result between Chrome and Firefox per Hasal tests.

Comment 1

[William Lachance (:wlach)]

Could you provide a PR adding the repository to the existing set of fixtures? http://treeherder.readthedocs.io/common_tasks.html#add-a-new-repository

How are you planning to track revision/push information for Chrome?

Comment 2

[Bobby Chien]

(In reply to William Lachance (:wlach) from comment #1)
> Could you provide a PR adding the repository to the existing set of
> fixtures?
> http://treeherder.readthedocs.io/common_tasks.html#add-a-new-repository
> 
> How are you planning to track revision/push information for Chrome?

The idea of testing is when we got a firefox nightly build, the we will trigger Hasal tests on the nightly build and a reference Chrome build. At the end of testing, we will post test results to perfherher. The 2 result records, firefox and chrome, all based on same firefox revision hash. Because of these tests were tested at the same time. 

Does this make sense to you?

Askeing, could you help to add comment if I have anything wrong? Thanks.

Comment 3

[William Lachance (:wlach)]

(In reply to Bobby Chien [:bchien] from comment #2)
> (In reply to William Lachance (:wlach) from comment #1)
> > Could you provide a PR adding the repository to the existing set of
> > fixtures?
> > http://treeherder.readthedocs.io/common_tasks.html#add-a-new-repository
> > 
> > How are you planning to track revision/push information for Chrome?
> 
> The idea of testing is when we got a firefox nightly build, the we will
> trigger Hasal tests on the nightly build and a reference Chrome build. At
> the end of testing, we will post test results to perfherher. The 2 result
> records, firefox and chrome, all based on same firefox revision hash.
> Because of these tests were tested at the same time. 
> 
> Does this make sense to you?

Well, that approach has the problem that you won't know which version of Chrome you're testing against.

I think it would be better to actually reference the Chromium git repository: https://chromium.googlesource.com/chromium/src.git/+refs

You can look up the revision from that repository by matching the version tag against a revision: http://serverfault.com/a/646402

http://serverfault.com/a/646402

This way at least changes in Chrome's numbers will be displayed on Perfherder.

Comment 4

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #3)
> This way at least changes in Chrome's numbers will be displayed on
> Perfherder.

Sorry, I meant *an explanation* for changes in Chrome's numbers.

Comment 5

[Askeing Yen[:askeing][:fyen]]

The idea is adding two result data with the same commit hash, and adding Chrome's version information into job details. (https://chromium.googlesource.com/chromium/src.git/+/<VERSION>)
So that we can compare the result easier by selecting different project and the same hash.

Comment 6

[Askeing Yen[:askeing][:fyen]]

(In reply to Askeing Yen[:askeing][:fyen] from comment #5)
> The idea is adding two result data with the same commit hash, and adding
> Chrome's version information into job details.
> (https://chromium.googlesource.com/chromium/src.git/+/<VERSION>)
> So that we can compare the result easier by selecting different project and
> the same hash.

After have discussion with Bobby, we plan to add new project with "chrome-release" name, and put into "other" group.
ex:
    "fields": {
        "dvcs_type": "hg",
        "name": "chrome-release",
        "url": "https://hg.mozilla.org/mozilla-central",
        "active_status": "active",
        "codebase": "gecko",
        "repository_group": 7,
        "description": "For tracking Chrome testing history to against Firefox"
    }

Does this make sense to you?
Thank you.

Comment 7

[William Lachance (:wlach)]

(In reply to Askeing Yen[:askeing][:fyen] from comment #6)
> (In reply to Askeing Yen[:askeing][:fyen] from comment #5)
> > The idea is adding two result data with the same commit hash, and adding
> > Chrome's version information into job details.
> > (https://chromium.googlesource.com/chromium/src.git/+/<VERSION>)
> > So that we can compare the result easier by selecting different project and
> > the same hash.
> 
> After have discussion with Bobby, we plan to add new project with
> "chrome-release" name, and put into "other" group.
> ex:
>     "fields": {
>         "dvcs_type": "hg",
>         "name": "chrome-release",
>         "url": "https://hg.mozilla.org/mozilla-central",
>         "active_status": "active",
>         "codebase": "gecko",
>         "repository_group": 7,
>         "description": "For tracking Chrome testing history to against
> Firefox"
>     }
> 
> Does this make sense to you?
> Thank you.

I don't really like this solution as it means the revision box will give you a link to Firefox checkins when you click on a Chrome datapoint in links like this one: https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-inbound,4ac072004ea185659e3a65ce4a3403f891a35c56,1,1%5D&selected=%5Bmozilla-inbound,4ac072004ea185659e3a65ce4a3403f891a35c56,37676,37622423,1%5D

As I said before, Perfherder isn't designed to do this type of comparison (the focus has been exclusively on viewing one product compared to itself) so anything we do is likely to be a hack. My advice is still to create a grafana dashboard (or whatever) for showing the comparison, and just use Perfherder for generating alerts when Firefox performance changes.

But if you really want to use Perfherder for everything, the approach in comment 5 seems like a better idea. Each Hasal job could submit two datums for each test. One could be `firefox-test-x`, another could be `firefox-test-chrome-x`. That would I guess let you visualize Firefox vs. a Chrome baseline in the same time range on the graph (even though Chrome's commits were likely made at a much different time).

Comment 8

[Shako Ho]

Hi Will,

  I can fully understand and respect your opinion. We all know Perfherder is not designed for this type of comparison. However Perfherder seems to be the only official performance dashboard for Mozilla (please correct me if I'm wrong), and we don't really want to create another dashboard for this. So upload the result to Perfherder become the only option for us, but when we come out the Hasal project, the major goal is to compare the Firefox with other browser on different webapps. It's not only focus on Firefox itself, but also the difference between other browser. That's why we need the result to compare with other browser.

  So back to the solution, the approach of comment 5 is doable for us, but the only problem is we can't use the compare feature of Perfherder to compare the data between tests (test for firefox and other browser) (please correct me if I'm wrong). So what if we upload the difference between browsers as an independent test results, for example:
we will upload the 3 test data as below to perfherder, we don't really need to care about the data of test_for_chrome, but once we observe the diff become higher we might need to reference that data.
test_for_firefox
test_for_chrome
test_for_diff  
Do you think this approach is okay for you?

Comment 9

[William Lachance (:wlach)]

(In reply to Shako Ho from comment #8)
> Hi Will,
> 
>   I can fully understand and respect your opinion. We all know Perfherder is
> not designed for this type of comparison. However Perfherder seems to be the
> only official performance dashboard for Mozilla (please correct me if I'm
> wrong), and we don't really want to create another dashboard for this. So
> upload the result to Perfherder become the only option for us, but when we
> come out the Hasal project, the major goal is to compare the Firefox with
> other browser on different webapps. It's not only focus on Firefox itself,
> but also the difference between other browser. That's why we need the result
> to compare with other browser.

I'm not aware of any other dashboard for tracking performance test data, no. 

I guess as long as you're aware Perfherder isn't really designed for this use case and that some compromises might be necessary, I think we'll be ok.

>   So back to the solution, the approach of comment 5 is doable for us, but
> the only problem is we can't use the compare feature of Perfherder to
> compare the data between tests (test for firefox and other browser) (please
> correct me if I'm wrong). So what if we upload the difference between
> browsers as an independent test results, for example:
> we will upload the 3 test data as below to perfherder, we don't really need
> to care about the data of test_for_chrome, but once we observe the diff
> become higher we might need to reference that data.
> test_for_firefox
> test_for_chrome
> test_for_diff  
> Do you think this approach is okay for you?

Well, if we have alerting enable we should notice if either Firefox or Chrome's numbers improve or regress, so I don't think we need to explicitly track the difference.

One thing to consider might be modifying the existing e10s dashboard views (https://treeherder.mozilla.org/perf.html#/e10s) to do a comparison between Chrome and Firefox. In this case, you'd just add an extra option called "chrome" to the Chrome data per test (instead of labelling the test "chrome"): https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json#L76. I've been thinking about making these dashboards more modular since there are other internal measures we'd want to compare against (e.g. Firefox with stylo enabled vs. not: bug 1288857)

Comment 10

[Shako Ho]

Hi Will,

  Thanks for your help. 

  Based on your suggestion, We will add the chrome and firefox separately into extra option. And the e10s dashboard is pretty much we need, except we might need to consider how we compare the data between revision. We don't see there is such option on dashboard (please correct me if I'm wrong). 

  Back to the topic, so will you create another dashboard like e10 for Hasal? Or we need to create a new bug like bug 1288857 for this?  

   And we just finish the testing in our local environment and plan to upload the data to perfherder stage next week. As I know, we already have the credential for stage, is there any other setting we need to setup before uploading the data?

  Thanks,

Comment 11

[William Lachance (:wlach)]

(In reply to Shako Ho from comment #10)
> Hi Will,
> 
>   Thanks for your help. 
> 
>   Based on your suggestion, We will add the chrome and firefox separately
> into extra option. And the e10s dashboard is pretty much we need, except we
> might need to consider how we compare the data between revision. We don't
> see there is such option on dashboard (please correct me if I'm wrong). 

There isn't at present, no. You could look at the graphs for an individual series to look at fine-grained difference though. I assume you mainly want to look at the Chrome data to observe a trend, no?

>   Back to the topic, so will you create another dashboard like e10 for
> Hasal? Or we need to create a new bug like bug 1288857 for this?  

I think we will need to modify the existing e10s dashboard to do this. I'm anticipating we'll need to do this for a bunch of different projects, so we might do this in the context of generalizing the dashboard code. Feel free to file a bug to add a dashboard for tracking hasal results and we'll triage as appropriate. 

>    And we just finish the testing in our local environment and plan to
> upload the data to perfherder stage next week. As I know, we already have
> the credential for stage, is there any other setting we need to setup before
> uploading the data?

I don't think so. If you run into troubles let me know!

Comment 12

[Shako Ho]

Hi Will,

  Yes, that is our major goal, but when we think of having some alert mechanism or triage the gap between two browsers, the revision comparison might help. (or there is some other mechanism already work for this instead of having revision comparison on dashboard?)

  Ok, the bug is created. see Bug 1314223

Thanks for your help

Comment 13

[William Lachance (:wlach)]

(In reply to Shako Ho from comment #12)
> Hi Will,
> 
>   Yes, that is our major goal, but when we think of having some alert
> mechanism or triage the gap between two browsers, the revision comparison
> might help. (or there is some other mechanism already work for this instead
> of having revision comparison on dashboard?)
> 
>   Ok, the bug is created. see Bug 1314223
> 
> Thanks for your help

If you want details on a specific test, you can always click through any line on the e10s dashboard for subtest information, or hover and click graph to see how things are changing over time. I think this should be good for your needs, if you want help using the dashboard feel free to reach out to either me, jmaher, or the e10s group (who have used the dashboard for tracking the readiness of that feature)