Closed Bug 1615432 Opened 6 years ago Closed 5 years ago

[SPOC V2] Performance report & greenlight

Categories

(Firefox :: New Tab Page, task, P1)

Product:
Firefox
Component:
New Tab Page
Type:
task

Tracking

()

Status:
VERIFIED FIXED
Iteration:
76.2 - Mar 23 - Apr 5

People

(Reporter: jdavis, Assigned: thecount)

Details

Run performance numbers w/ SPOC V2 enabled and get performance greenlight with Fx 74, and Fx 75, before we can run experimetn with Fx 74 on March 10.

Timeline:
Performance numbers by 2/21.
Review and approval (or mitigations ID'd) by 2/28

RISK: IF we don't pass Fx 74 (or Fx 75) - there's likely a lot of work to make this performance green. Scott to run performance numbers as soon as Nightly 75 things land.

Assignee: nobody → sdowne
Iteration: --- → 75.2 - Feb 24 - Mar 8
Priority: -- → P1
No longer blocks: pocket-newtab

Setup steps, do this for all following test cases:

  1. Create a new profile, but doing it on an older profile with some history can produce more accurate results.
  2. Install the profiler extension on that profile. https://profiler.firefox.com/
  3. Set browser.newtabpage.activity-stream.asrouter.devtoolsEnabled to true
  4. Open about:newtab#devtools-ds
  5. Ensure "show_spocs" is checked

Steps to test performance of v2 download fresh attachments:

  1. Click "refresh cache"
  2. Click ctrl+shift+1 to start recording.
  3. Scroll down a bit and click "Enable V2 Personalization"
  4. Scroll up and click "Trigger Idle Daily"
  5. You should now have all the attachments from remote settings downloaded.
  6. Scroll up and click "Expire Cache"
  7. Scroll up and click "Trigger System Tick"
  8. Ctrl+shift+2 to stop recording.
  9. You should have a new profile, but you probably want to click "publish" to save it or share it.

Steps to test performance of v2 with existing attachments:

  1. Do the no cache steps.
  2. Click "refresh cache"
  3. Click ctrl+shift+1 to start recording.
  4. Scroll up and click "Expire Cache"
  5. Scroll up and click "Trigger System Tick"
  6. Ctrl+shift+2 to stop recording.
  7. You should have a new profile, but you probably want to click "publish" to save it or share it.
Iteration: 75.2 - Feb 24 - Mar 8 → 76.1 - Mar 9 - Mar 22
Iteration: 76.1 - Mar 9 - Mar 22 → 76.2 - Mar 23 - Apr 5
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

Verified this using Firefox Nightly 77.0a1 (Build ID 20200407093520) on Windows 10 x64 and obtained the following results:

@Gavin: Could you please take a look at these links and let me know what do you mean when saying "green performance", at what should I be looking? At the "Running Time"? Thanks!

Flags: needinfo?(gsuntop)

The main bit of work that cleared up this issue and got us into the green is: https://bugzilla.mozilla.org/show_bug.cgi?id=1621018

Sorry I didn't link that earlier, but I moved most all of my work into there once I knew the strategy I was going to use.

Clicking on those links doesn't take me to them, I think you need to publish them first.

Flags: needinfo?(gsuntop)

Sorry about the links! I think they are ok, now.

Those don't show any issues, which is good.

But it also looks like from when I made these steps to test, and when we finished that work and tested it, some things changed, and it's not showing everything in the profiles.

I also think I did an overkill on the steps to test by providing two ways to test, only 1 is needed at this point. I'll update it with some better steps, but no rush on redoing them. Doing the following steps is the most up to date and is going to get the most accurate results.

  1. Create a new profile, but doing it on an older profile with some history can produce more accurate results.
  2. Install the profiler extension on that profile. https://profiler.firefox.com/
  3. Set browser.newtabpage.activity-stream.asrouter.devtoolsEnabled to true
  4. Set browser.newtabpage.activity-stream.discoverystream.spocs-endpoint to https://spocs.getpocket.com/spocs?country=US&region=CA
  5. Click the profiler button in the top right (looks like a little stop watch) and select "Firefox Front-End" from the settings dropdown.
  6. Open about:newtab#devtools-ds, most of the next steps button clicks are done from within this page.
  7. Ensure "show_spocs" is checked
  8. Ensure you can see spocs
  9. Click "refresh cache"
  10. Click ctrl+shift+1 to start recording.
  11. Scroll down a bit and click "Enable V2 Personalization"
  12. Scroll up and click "Trigger Idle Daily"
  13. You should now have all the attachments from remote settings downloaded.
  14. Scroll up and click "Expire Cache"
  15. Scroll up and click "Trigger System Tick"
  16. Ctrl+shift+2 to stop recording.
  17. You should have a new profile, but you probably want to click "publish" to save it or share it.

Hey @Scott,

Valentina is currently on PTO so I’ll fill-in for her on this bug verification.

I’ve tested the performance by using the latest Firefox Nightly 77.0a1 (Build ID: 20200408033650) and the steps you’ve provided in comment 6 on Windows 10 x64, macOS 10.15, and Ubuntu Linux. The results of my testing can be found here:
Windows 10: https://perfht.ml/2UPaR3o
MacOS: https://perfht.ml/3e7TKRW
Ubuntu Linux: https://perfht.ml/2XhGJ2o

Given the above results and the fact that bug 1621018 is marked as VERIFIED, do you think that this is enough verification from our side in order to mark this bug as VERIFIED?

Flags: needinfo?(sdowne)

Those profiles look good to me!

What I'm looking for with those, is any red bars caused by personality provider files, in the parent process.

In the case of both windows and mac, there is no red in the parent process.

In the case of linux, there is some. I couldn't find any related to personality provider, nor was any of it "high" I suspect it was just a different machine with different hardware specs, so simple operations became more of an issue. This is normal, for when I ran profiles on lower end machines.

I also checked all three profiles you provided in the DOM worker thread for the personality provider work, and found it. This is to verify we didn't have a false positive. To find the dom worker threads, you need to turn them on inside the "tracks visible" dropdown. But it's cool, I don't expect you to do that. I think we can mark this verified.

I've configured on of your examples to show the red bars in the worker thread, and not the parent process, which is what we're looking for https://perfht.ml/39Swlkq

Flags: needinfo?(sdowne)

Thank you very much, Scott! Now that you’ve explained what we are looking for, it starts to make more sense.
Given the verification done in comment 7 and the confirmation from Scott in comment 8, I’m marking this bug as VERIFIED.

Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.