Bug 1364235 Comment 24 Edit History

Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.

(Hopefully a needinfo is sufficient without review flags. Should https://wiki.mozilla.org/Firefox/Data_Collection#Requesting_Approval be updated?)

1) What questions will you answer with this data?

- To what extent are our caches improving startup performance?
- How often do we see startup caches not aligning with our expectations? I.e., do we see any users who consistently encounter mostly misses w/ their startup caches (accounting for invalidations from app updates)
- Ongoing: have startup caches regressed in some way?

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements?

- Establish baselines or measure changes in product or platform quality or performance.

- We have a bug report (in this bug) indicating that this was at one time a problem. We want to try to assess if it is still a problem.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

- Eyeballing the code / trying to reproduce cache problems ourselves. Neither of these can show conclusively that there's not a significant problem in the wild.

4) Can current instrumentation answer these questions?

No.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.

<table>
  <tr>
    <td>Measurement Description</td>
    <td>Data Collection Category</td>
    <td>Tracking Bug #</td>
  </tr>
  <tr>
    <td>Number of hits / misses to the StartupCache</td>
    <td>Category 1</td>
    <td>1364235</td>
    <td>Number of hits / misses to the ScriptPreloader</td>
    <td>Category 1</td>
    <td>1364235</td>
    <td>Distribution of amount of time spent waiting for off-thread compiles in the ScriptPreloader</td>
    <td>Category 1</td>
    <td>1364235</td>
    <td>How many times we ended up recompiling a script from the script preloader on the main thread.</td>
    <td>Category 1</td>
    <td>1364235</td>
  </tr>
</table>


6) How long will this data be collected?  Choose one of the following:

* I want this data to be collected for 6 months initially (potentially renewable).

7) What populations will you measure?

* Which release channels?

All

* Which countries?

All

* Which locales?

All

* Any other filters?  Please describe in detail below.

8) If this data collection is default on, what is the opt-out mechanism for users?

The usual technical data opt-out in about:preferences

9) Please provide a general description of how you will analyze this data.

Viewing the histograms via TMO, and deeper analysis via SQL or Spark

10) Where do you intend to share the results of your analysis?

Likely in this bug or a follow-up.

11) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:

No.
(Hopefully a needinfo is sufficient without review flags. Should https://wiki.mozilla.org/Firefox/Data_Collection#Requesting_Approval be updated?)

1) What questions will you answer with this data?

- To what extent are our caches improving startup performance?
- How often do we see startup caches not aligning with our expectations? I.e., do we see any users who consistently encounter mostly misses w/ their startup caches (accounting for invalidations from app updates)
- Ongoing: have startup caches regressed in some way?

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements?

- Establish baselines or measure changes in product or platform quality or performance.

- We have a bug report (in this bug) indicating that this was at one time a problem. We want to try to assess if it is still a problem.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

- Eyeballing the code / trying to reproduce cache problems ourselves. Neither of these can show conclusively that there's not a significant problem in the wild.

4) Can current instrumentation answer these questions?

No.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.

- Number of hits / misses to the StartupCache
  - Category 1
  - Bug 1364235

- Number of hits / misses to the ScriptPreloader
  - Category 1
  - Bug 1364235

- Distribution of amount of time spent waiting for off-thread compiles in the ScriptPreloader
  - Category 1
  - Bug 1364235

- How many times we ended up recompiling a script from the script preloader on the main thread.
  - Category 1
  - Bug 1364235

6) How long will this data be collected?  Choose one of the following:

* I want this data to be collected for 6 months initially (potentially renewable).

7) What populations will you measure?

* Which release channels?

All

* Which countries?

All

* Which locales?

All

* Any other filters?  Please describe in detail below.

8) If this data collection is default on, what is the opt-out mechanism for users?

The usual technical data opt-out in about:preferences

9) Please provide a general description of how you will analyze this data.

Viewing the histograms via TMO, and deeper analysis via SQL or Spark

10) Where do you intend to share the results of your analysis?

Likely in this bug or a follow-up.

11) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:

No.
(Hopefully a needinfo is sufficient without review flags. Should https://wiki.mozilla.org/Firefox/Data_Collection#Requesting_Approval be updated?)

1) **What questions will you answer with this data?**

- To what extent are our caches improving startup performance?
- How often do we see startup caches not aligning with our expectations? I.e., do we see any users who consistently encounter mostly misses w/ their startup caches (accounting for invalidations from app updates)
- Ongoing: have startup caches regressed in some way?

2) **Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements?**

- Establish baselines or measure changes in product or platform quality or performance.

- We have a bug report (in this bug) indicating that this was at one time a problem. We want to try to assess if it is still a problem.

3) **What alternative methods did you consider to answer these questions? Why were they not sufficient?**

- Eyeballing the code / trying to reproduce cache problems ourselves. Neither of these can show conclusively that there's not a significant problem in the wild.

4) **Can current instrumentation answer these questions?**

No.

5) **List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.**

- Number of hits / misses to the StartupCache
  - Category 1
  - Bug 1364235

- Number of hits / misses to the ScriptPreloader
  - Category 1
  - Bug 1364235

- Distribution of amount of time spent waiting for off-thread compiles in the ScriptPreloader
  - Category 1
  - Bug 1364235

- How many times we ended up recompiling a script from the script preloader on the main thread.
  - Category 1
  - Bug 1364235

6) **How long will this data be collected?  Choose one of the following:**

* I want this data to be collected for 6 months initially (potentially renewable).

7) **What populations will you measure?**

* Which release channels?

All

* Which countries?

All

* Which locales?

All

* Any other filters?  Please describe in detail below.

8) **If this data collection is default on, what is the opt-out mechanism for users?**

The usual technical data opt-out in about:preferences

9) **Please provide a general description of how you will analyze this data.**

Viewing the histograms via TMO, and deeper analysis via SQL or Spark

10) **Where do you intend to share the results of your analysis?**

Likely in this bug or a follow-up.

11) **Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:**

No.

Back to Bug 1364235 Comment 24