1863130 - Gather LargestContentfulPaint (LCP) metric in browsertime pageload tests

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for gathering the LCP metric in all of our pageload tests.

Comment 1

[Alex Finder]

Would we need to sheriff this metric once we add it ?

Comment 2

[Greg Mierzwinski [:sparky]]

Yes, we'll be sheriffing this metric. I'm not sure if it's available on android browsers yet, but on desktop it definitely is.

Comment 3

[Greg Mierzwinski [:sparky]]

PR for it in browsertime: https://github.com/sitespeedio/browsertime/pull/2018

Comment 4

[Greg Mierzwinski [:sparky]]

PR was merged. Running try runs now: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=74ffdce755c1b6fbfb8b8e458c8f414616297afa

Note that this metric is surfacing some pre-existing permas. For instance, on twitch, the LCP is never produced intermittently as the page doesn't finish loading before we finish the cycle. The same issue is happening with imgur. I'm going to try to re-record them to see if it fixes it.

Comment 5

[Greg Mierzwinski [:sparky]]

Twitch is a permafail for LCP. We can't gather it from there as it doesn't get produced. I'm going to disable it for that page. Imgur seems fine locally after a re-recording though.

Comment 6

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1863130 - Gather LargestContentfulPaint (LCP) in browsertime pageload tests. r?#perftest

This patch adds the ability to gather LargestContentfulPaint in browsertime pageload tests.

Comment 7

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1863130 - Re-record twitch, and imgur to fix LCP issues. r?#perftest

Depends on D193645

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f487d5a07ab5
Gather LargestContentfulPaint (LCP) in browsertime pageload tests. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/fd253d27e64e
Re-record twitch, and imgur to fix LCP issues. r=perftest-reviewers,afinder

Comment 9

[Natalia Csoregi [:nataliaCs]]

Backed out for causing largestContentfulPaint browsertime failures.

• backout: https://hg.mozilla.org/integration/autoland/rev/65a0553b296ae1432e5d18c3d2d607cacbb0a35e
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=CiBPFLrXQSmbVl7h7oOMzA.0&revision=fd253d27e64ec2f0a321ef83bffbcf28392e9f88
• failure log: https://treeherder.mozilla.org/logviewer?job_id=438211884&repo=autoland&lineNumber=2379

[task 2023-11-29T22:47:42.192Z] 22:47:42     INFO -  raptor-perftest Info: Removing temporary directory: /tmp/tmpq5bnex19
[task 2023-11-29T22:47:42.220Z] 22:47:42    ERROR -  Traceback (most recent call last):
[task 2023-11-29T22:47:42.220Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/raptor.py", line 186, in <module>
[task 2023-11-29T22:47:42.221Z] 22:47:42     INFO -      main()
[task 2023-11-29T22:47:42.221Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/raptor.py", line 135, in main
[task 2023-11-29T22:47:42.221Z] 22:47:42     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2023-11-29T22:47:42.222Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/perftest.py", line 488, in run_tests
[task 2023-11-29T22:47:42.222Z] 22:47:42     INFO -      return self.process_results(tests, test_names)
[task 2023-11-29T22:47:42.222Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/perftest.py", line 510, in process_results
[task 2023-11-29T22:47:42.223Z] 22:47:42     INFO -      res = self.results_handler.summarize_and_output(self.config, tests, test_names)
[task 2023-11-29T22:47:42.223Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/results.py", line 1089, in summarize_and_output
[task 2023-11-29T22:47:42.223Z] 22:47:42     INFO -      for new_result in self.parse_browsertime_json(
[task 2023-11-29T22:47:42.224Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/results.py", line 842, in parse_browsertime_json
[task 2023-11-29T22:47:42.224Z] 22:47:42     INFO -      raise MissingResultsError(
[task 2023-11-29T22:47:42.224Z] 22:47:42     INFO -  results.MissingResultsError: Browsertime cycle missing ['largestContentfulPaint', 'renderTime'] measurement
[task 2023-11-29T22:47:42.269Z] 22:47:42     INFO - Return code: 1
[task 2023-11-29T22:47:42.269Z] 22:47:42  WARNING - setting return code to 1
[task 2023-11-29T22:47:42.269Z] 22:47:42     INFO - Copying Raptor results to upload dir:
[task 2023-11-29T22:47:42.270Z] 22:47:42     INFO - /home/cltbld/tasks/task_170129487746103/build/blobber_upload_dir/perfherder-data.json
[task 2023-11-29T22:47:42.270Z] 22:47:42     INFO - Copying raptor results from /home/cltbld/tasks/task_170129487746103/build/raptor.json to /home/cltbld/tasks/task_170129487746103/build/blobber_upload_dir/perfherder-data.json
[task 2023-11-29T22:47:42.270Z] 22:47:42 CRITICAL - Error copying results /home/cltbld/tasks/task_170129487746103/build/raptor.json to upload dir /home/cltbld/tasks/task_170129487746103/build/blobber_upload_dir/perfherder-data.json

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/41233474de74
Gather LargestContentfulPaint (LCP) in browsertime pageload tests. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/faeaf5bd4445
Re-record twitch, and imgur to fix LCP issues. r=perftest-reviewers,afinder

Comment 11

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/41233474de74
https://hg.mozilla.org/mozilla-central/rev/faeaf5bd4445

Comment 12

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1863130 - Gather LargestContentfulPaint (LCP) in browsertime pageload tests. r?#perftest

This patch adds the ability to gather LargestContentfulPaint in browsertime pageload tests.

Original Revision: https://phabricator.services.mozilla.com/D193645

Comment 13

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1863130 - Gather LargestContentfulPaint (LCP) in browsertime pageload tests. r?#perftest

This patch adds the ability to gather LargestContentfulPaint in browsertime pageload tests.

Original Revision: https://phabricator.services.mozilla.com/D193645

Comment 14

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1863130 - Re-record twitch, and imgur to fix LCP issues. r?#perftest

Original Revision: https://phabricator.services.mozilla.com/D193646

Comment 15

[Phabricator Automation]

Uplift Approval Request

• String changes made/needed: None
• Is Android affected?: no
• Fix verified in Nightly: yes
• Steps to reproduce for manual QE testing: None
• Risk associated with taking this patch: None
• User impact if declined: None
• Needs manual QE test: no
• Explanation of risk level: This patch updates the recording used for a perma failing test.
• Code covered by automated testing: no

Comment 16

[Greg Mierzwinski [:sparky]]

I've abandoned the uplift approvals because there are merge conflicts in the perfdocs files (which would require a few other uplifts).

Comment 17

[Alex Finder]

(In reply to Natalia Csoregi [:nataliaCs] from comment #9)

Backed out for causing largestContentfulPaint browsertime failures.

• backout: https://hg.mozilla.org/integration/autoland/rev/65a0553b296ae1432e5d18c3d2d607cacbb0a35e
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=CiBPFLrXQSmbVl7h7oOMzA.0&revision=fd253d27e64ec2f0a321ef83bffbcf28392e9f88
• failure log: https://treeherder.mozilla.org/logviewer?job_id=438211884&repo=autoland&lineNumber=2379

[task 2023-11-29T22:47:42.192Z] 22:47:42     INFO -  raptor-perftest Info: Removing temporary directory: /tmp/tmpq5bnex19
[task 2023-11-29T22:47:42.220Z] 22:47:42    ERROR -  Traceback (most recent call last):
[task 2023-11-29T22:47:42.220Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/raptor.py", line 186, in <module>
[task 2023-11-29T22:47:42.221Z] 22:47:42     INFO -      main()
[task 2023-11-29T22:47:42.221Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/raptor.py", line 135, in main
[task 2023-11-29T22:47:42.221Z] 22:47:42     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2023-11-29T22:47:42.222Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/perftest.py", line 488, in run_tests
[task 2023-11-29T22:47:42.222Z] 22:47:42     INFO -      return self.process_results(tests, test_names)
[task 2023-11-29T22:47:42.222Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/perftest.py", line 510, in process_results
[task 2023-11-29T22:47:42.223Z] 22:47:42     INFO -      res = self.results_handler.summarize_and_output(self.config, tests, test_names)
[task 2023-11-29T22:47:42.223Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/results.py", line 1089, in summarize_and_output
[task 2023-11-29T22:47:42.223Z] 22:47:42     INFO -      for new_result in self.parse_browsertime_json(
[task 2023-11-29T22:47:42.224Z] 22:47:42     INFO -    File "/home/cltbld/tasks/task_170129487746103/build/tests/raptor/raptor/results.py", line 842, in parse_browsertime_json
[task 2023-11-29T22:47:42.224Z] 22:47:42     INFO -      raise MissingResultsError(
[task 2023-11-29T22:47:42.224Z] 22:47:42     INFO -  results.MissingResultsError: Browsertime cycle missing ['largestContentfulPaint', 'renderTime'] measurement
[task 2023-11-29T22:47:42.269Z] 22:47:42     INFO - Return code: 1
[task 2023-11-29T22:47:42.269Z] 22:47:42  WARNING - setting return code to 1
[task 2023-11-29T22:47:42.269Z] 22:47:42     INFO - Copying Raptor results to upload dir:
[task 2023-11-29T22:47:42.270Z] 22:47:42     INFO - /home/cltbld/tasks/task_170129487746103/build/blobber_upload_dir/perfherder-data.json
[task 2023-11-29T22:47:42.270Z] 22:47:42     INFO - Copying raptor results from /home/cltbld/tasks/task_170129487746103/build/raptor.json to /home/cltbld/tasks/task_170129487746103/build/blobber_upload_dir/perfherder-data.json
[task 2023-11-29T22:47:42.270Z] 22:47:42 CRITICAL - Error copying results /home/cltbld/tasks/task_170129487746103/build/raptor.json to upload dir /home/cltbld/tasks/task_170129487746103/build/blobber_upload_dir/perfherder-data.json

== Change summary for alert #40650 (as of Tue, 12 Dec 2023 12:27:38 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
6%	twitch SpeedIndex	windows10-64-shippable-qr	fission warm webrender	443.05 -> 414.93	Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=40650