Open Bug 1869513 Opened 5 months ago Updated 5 months ago

Deprecate talos real webextensions test once we confirmed the new tp6 webextensions job is fully replacing it

Categories

(WebExtensions :: General, task, P3)

Product:
WebExtensions
Component:
General
Type:
task

Tracking

(Not tracked)

People

(Reporter: rpl, Unassigned)

References

(Depends on 1 open bug)

Details

In Bug 1859549 we have introduced new tp6 and tp6m webextensions taskcluster jobs that runs these tp6 and tp6m perftests on Firefox Desktop and Firefox for Android instances with real webextensions installed.

As part of the previous Talos perftests framework we also have a talos-realworld-webextensions (introduced from Bug 1575089), see talos docs here.

Talos perftests framework is now in maintenance mode and expected to be fully replaced by the new mozperftests and raptor perftests frameworks, and so once we have promoted the new tp6 / tp6m webextensions jobs to tier-2 (Bug 1867781) and confirmed that the new perftests are fully replacing the talos-realworld-webextensions job, we will be removing the talos job in favor of the new ones.

The following are a few of the improvements that we expect the new tp6 / tp6m webextensions jobs to be providing compared to the previous talos-realworld-webextensions job:

  • the new tp6m job runs on Firefox for Android, while the talos-realworld-webextensions job runs only on Firefox Desktop

  • the new tp6/tp6m jobs use more recent dumps of top real websites to run the perftests on (compared to the one used by talos tests)

  • the new tp6/tp6m jobs replay the HTTP requests using the real websites domain, while talos does replay the HTTTP requests using fake talos domains, and so the new job are able to trigger more realistic adblockers behaviors (e.g. ublock is suspending and cancelling ads requests while running the tp6/tp6m amazon perf-test)

  • the talos-realworld-webextensions job installs all the extensions included in a given talos-real-webextensions zip file, whereas the new jobs are using a condprof customization file which is falling back to download additonal xpi files from a given url (unless the expected xpi is already part of the firefox-addons.tar file fetched through a taskcluster fetch task dependency), which will allow to run these perftests on additional arbirary third party xpi files in a try push by just adding the new extension to the condprof customization file in try push itself

As the original author of talos-realworld-webextension, I fully support this. Exciting!

You need to log in before you can comment on or make changes to this bug.