790955 - Should have startup tests which measure time from start to about:home

Status: RESOLVED FIXED

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

Eideticker is already set up to do this, but we want to hook this up to the dashboard as well. This is a great first step to what we want to do eventually (Eideticker version of S1S2, webapps startup-time measurement), so let's get it done!

Comment 1

[William Lachance (:wlach)]

Attachment: Add basic test to measure time from cold start to about:home to dashboard

Let me know if this makes sense. If it does hopefully we can add similar tests for B2G-type stuff later. I know the code to automatically extract things like measures from dashboard data is kind of gross, but the performance seems to be ok in practice. I'd rather put off a larger-scale refactoring of our data schema until later.

Comment 2

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 660834 [details] [diff] [review]
Add basic test to measure time from cold start to about:home to dashboard

Review of attachment 660834 [details] [diff] [review]:
-----------------------------------------------------------------

Yup, the changes make sense, and I'm looking forward to the data schema changes :)

Comment 3

[William Lachance (:wlach)]

First code for this landed, tests in production.

There's two more bits of work here before I think this can be said to be truly useful:

1. For the cold start case, we should reboot the phone before running the test.
2. We should run the browser once to initialize the profile fully before starting.

Comment 4

[William Lachance (:wlach)]

For (2) I just created two variants of the tests, one which runs fennec once to fully initialize the profile, one which doesn't. For (2), I'm not sure if rebooting the phone would actually generate much more interesting results, and it would interfere with the reliability of the whole system, so I'm going to punt on doing that for now.