Create a bug form for incoming CI Automation requests
Categories
(bugzilla.mozilla.org :: Custom Bug Entry Forms, task)
Tracking
()
People
(Reporter: jmaher, Assigned: dkl)
References
Details
We are looking to formalize our process for requests that come in for CI automation changes. Having a form would be a good start.
Upon completion, bugs would end up in Testing::General.
Here are the fields we would like (with description below each question). Each question would be a free form text field. Possibly later we could add select boxes, but I think that would require too much maintenance.
Type of Change
Hosting/Cloud provider change: (move locations, adjust contracts)
Hardware change: add/change/remove specific hardware (perf, android phones, osx)
OS change: add tools to OS, upgrade OS to different version
Build Type change: add linux64-tsan in CI, ccov for linux and win64, mingw32 builds and tests
Variant Addition: runtime variant via preference, duplicating 1+ test suites on 1+ build configs
Example: Migrate Windows 10 tests from AWS -> Azure / migrate builds + tests on linux from AWS -> GCP / upgrade windows 10 to windows 11 / add Mac arm64 builds and tests, add fission variant by default on all tests and configurations
Team Requesting work
Responsible party for priority decisions, budget signoff, and triaging issues
Example: Graphics team sees a need for testing Wayland support with Linux in CI
Risk of not completing work
Example: We will not be running automation on the OS that the majority of users run on, will miss opportunities to catch bugs.
Which test suites will this change apply to
Select all test suites which will need to verify run on this new request
Example: xpcshell, browser-chrome, talos / ALL / all unittests / all perftests
Which configs will be affected by this
In order to plan for capacity and timelines, we need to know which platforms and build configs will be affected.
Example: only adding a new build type to linux / all windows: shippable, debug, asan, ccov / desktop only
|
Assignee | |
Comment 1•4 years ago
|
||
As for creating a complete custom bug entry form, it will be probably the beginning of next year (H122) before I can start on this due to some projects that must be completed before the end of the year.
In the meantime, maybe we can make this work as a new bug comment template similar to what we do not for other products/components. For an example see:
https://bugzilla.mozilla.org/enter_bug.cgi?product=bugzilla.mozilla.org&component=Administration
https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Applied+Machine+Learning
Let me know if the template works and what you would like the text to be.
|
Reporter | |
Comment 2•4 years ago
|
||
I like this idea- I will work on a template and do a dry run with 2 different scenarios that we have a use for this month; if all is well, we can leave it in place, possibly not needing a custom form.
|
|
Reporter | |
Comment 3•4 years ago
|
||
Type of Change: Choose One
Hosting or Cloud Provider Change
Example: change machines used at cloud provider for more CPU; move load from AWS -> Azure; add capacity at Bitbar
Hardware Change (add/change/remove specific hardware)
Example: upgrade mac minis to new hardware revision; replace Pixel2 phones with Galaxy S7; add arm64 windows machines to CI
OS Change (add tools to OS, upgrade OS to different version)
Example: upgrade windows 10 -> Windows 11; upgrade OSX 11 to latest security release; add codec/font to CI test machines
Build Type Change
Example: Add android-lite builds+tests to CI; add mingw32 builds and tests to CI; remove osx ccov builds and tests from CI
Test Variant (ci coverage via runtime variant set by preference)
*Example: add fission variant by default on all tests and configuration; *
Team Requesting Work (Responsible party for priority decisions, budget signoff, and triaging issues)
Risk of not completing work
Example: We will not be running automation on the OS that the majority of users run on, will miss opportunities to catch bugs
Which test suites will this change apply to (please be verbose)
Example: xpcshell, browser-chrome, talos; ALL; all unittests; all perftests
Which configs will be affected by this (To capacity plan and set timelines, which platforms and build configs will be affected)
Example: only adding a new build type to linux / all windows: shippable, debug, asan, ccov / desktop only
Other useful info
|
|
Assignee | |
Comment 4•4 years ago
|
||
Added. Let me know if looks ok.
https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=General
|
|
Reporter | |
Comment 5•4 years ago
|
||
I think this looks good, the one change that needs to happen is this shouldn't be on testing::general. Maybe we create a new component Testing::CI Requests, and make this default template for that.
|
|
Assignee | |
Comment 6•4 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #5)
I think this looks good, the one change that needs to happen is this shouldn't be on testing::general. Maybe we create a new component Testing::CI Requests, and make this default template for that.
Ah I see. I will remove the template for now from General. Please file a BMO Administration bug for adding the new component and then I will put the template there.
|
|
Assignee | |
Comment 8•4 years ago
|
||
This can be closed now.
Description
•