2028315 - High frequency [tier 2] Cannot find Google Chrome at C:\Progra~2\Google\Chrome\Application\chrome.exe

Status: NEW

(Testing :: Performance, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: agoloman [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=557153196&repo=mozilla-central&task=XUuoYiIITRGW5vdbGIJYhw.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/XUuoYiIITRGW5vdbGIJYhw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2026-04-01T00:09:29.153+00:00] 00:09:29     INFO - Return code: 0
[task 2026-04-01T00:09:29.156+00:00] 00:09:29     INFO - Running post-action listener: _resource_record_post_action
[task 2026-04-01T00:09:29.156+00:00] 00:09:29     INFO - Running post-action listener: _start_resource_monitoring
[task 2026-04-01T00:09:29.156+00:00] 00:09:29     INFO - Starting resource monitoring.
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - [mozharness: 2026-04-01 00:09:29.181178Z] Finished create-virtualenv step (success)
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - [mozharness: 2026-04-01 00:09:29.181178Z] Skipping install-chrome-android step.
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - [mozharness: 2026-04-01 00:09:29.181178Z] Skipping install-chromium-android step.
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - [mozharness: 2026-04-01 00:09:29.181178Z] Running install-chromium-distribution step.
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - Running pre-action listener: _resource_record_pre_action
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - Running main action method: install_chromium_distribution
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - Chrome should be preinstalled.
[task 2026-04-01T00:09:29.181+00:00] 00:09:29    ERROR - Cannot find Google Chrome at C:\Progra~2\Google\Chrome\Application\chrome.exe
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - Running post-action listener: _resource_record_post_action
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - [mozharness: 2026-04-01 00:09:29.181178Z] Finished install-chromium-distribution step (success)
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - [mozharness: 2026-04-01 00:09:29.181178Z] Skipping install-safari-technology-preview step.
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - [mozharness: 2026-04-01 00:09:29.181178Z] Running install step.
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - Running pre-action listener: _resource_record_pre_action
[task 2026-04-01T00:09:29.181+00:00] 00:09:29     INFO - Running main action method: install
[task 2026-04-01T00:09:29.191+00:00] 00:09:29     INFO - Getting output from command: ['C:\\task_177500164320532\\build\\venv\\Scripts\\python', '-m', 'pip', 'list', '--format', 'freeze', '--no-index']
[task 2026-04-01T00:09:29.191+00:00] 00:09:29     INFO - Copy/paste: C:\task_177500164320532\build\venv\Scripts\python -m pip list --format freeze --no-index
[task 2026-04-01T00:09:29.574+00:00] 00:09:29     INFO - Reading from file C:\Users\task_177500164320532\AppData\Local\Temp\tmpfile_xb8ctm_e_stdout
[task 2026-04-01T00:09:29.584+00:00] 00:09:29     INFO - Using _rmtree_windows ...
[task 2026-04-01T00:09:29.584+00:00] 00:09:29     INFO - Running command: del /F /Q "C:\Users\task_177500164320532\AppData\Local\Temp\tmpfile_j0jttcv8_stderr"
[task 2026-04-01T00:09:29.605+00:00] 00:09:29     INFO - Return code: 0
[task 2026-04-01T00:09:29.605+00:00] 00:09:29     INFO - Using _rmtree_windows ...
[task 2026-04-01T00:09:29.605+00:00] 00:09:29     INFO - Running command: del /F /Q "C:\Users\task_177500164320532\AppData\Local\Temp\tmpfile_xb8ctm_e_stdout"
[task 2026-04-01T00:09:29.623+00:00] 00:09:29     INFO - Return code: 0
[task 2026-04-01T00:09:29.624+00:00] 00:09:29     INFO - Return code: 0
[task 2026-04-01T00:09:29.624+00:00] 00:09:29     INFO - mkdir: C:\task_177500164320532\build\application
[task 2026-04-01T00:09:29.624+00:00] 00:09:29     INFO - Getting output from command: ['C:\\task_177500164320532\\build\\venv\\Scripts\\mozinstall', 'C:\\task_177500164320532\\build\\target.zip', '--destination', 'C:\\task_177500164320532\\build\\application']
[task 2026-04-01T00:09:29.624+00:00] 00:09:29     INFO - Copy/paste: C:\task_177500164320532\build\venv\Scripts\mozinstall C:\task_177500164320532\build\target.zip --destination C:\task_177500164320532\build\application
[task 2026-04-01T00:09:32.211+00:00] 00:09:32     INFO - Reading from file C:\Users\task_177500164320532\AppData\Local\Temp\tmpfile_kiu_gmq1_stdout
[task 2026-04-01T00:09:32.221+00:00] 00:09:32     INFO - Output received:
[task 2026-04-01T00:09:32.221+00:00] 00:09:32     INFO -  C:\task_177500164320532\build\application\firefox\firefox.exe
[task 2026-04-01T00:09:32.221+00:00] 00:09:32     INFO - Using _rmtree_windows ...
[task 2026-04-01T00:09:32.221+00:00] 00:09:32     INFO - Running command: del /F /Q "C:\Users\task_177500164320532\AppData\Local\Temp\tmpfile_x25j1pj4_stderr"
[task 2026-04-01T00:09:32.229+00:00] 00:09:32     INFO - Return code: 0
[task 2026-04-01T00:09:32.229+00:00] 00:09:32     INFO - Using _rmtree_windows ...
[task 2026-04-01T00:09:32.232+00:00] 00:09:32     INFO - Running command: del /F /Q "C:\Users\task_177500164320532\AppData\Local\Temp\tmpfile_kiu_gmq1_stdout"
[task 2026-04-01T00:09:32.240+00:00] 00:09:32     INFO - Return code: 0
[task 2026-04-01T00:09:32.241+00:00] 00:09:32     INFO - Return code: 0

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 2024992

Comment 2

[Daniel Holbert [:dholbert]]

This was filed in the wrong component - it has nothing to do with CSS, and I don't think the regressor makes sense either.

This looks like a test-harness environment issue of some sort, during this performance test suite (which is "Windows 11 24H2 Shippable opt Browsertime performance tests on Google Chrome Release test-windows11-64-24h2-shippable")

(The regressor could not have impacted that; its commit just added a new test in a completely different test suite, WPT.)

Moving to Testing|Performance, which also might not be right but is at least closer.

Comment 3

[Daniel Holbert [:dholbert]]

The last green run I can find for this platform/task is here:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=sp3%2Cwindows%2C11%2C24h2%2Cshippable&revision=381df99d5fa440f09960e020224cc87918380c4c
https://treeherder.mozilla.org/logviewer?job_id=556654267&repo=mozilla-central&task=UzjlA2KQS_uapPoyncbEJw.0&lineNumber=2158

(That log does have some ConnectionResetError: issues, but it's apparently still judged to be "good"/green.)

And the first bad run is here:
https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=sp3%2Cwindows%2C11%2C24h2%2Cshippable&revision=0b76dd137a61420640d9276c563698c3b3eff72a
https://treeherder.mozilla.org/logviewer?job_id=556859461&repo=mozilla-central&task=I3D8T65LT2yBNG8G_4Pssw.0&lineNumber=927

Treating this as a regression between those^ pushes, that would mean we'd have this regression range:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=381df99d5fa440f09960e020224cc87918380c4c&tochange=0b76dd137a61420640d9276c563698c3b3eff72a

...but I'm not seeing anything obvious jumping out in there.

Comparing the last-good vs. first-bad log, the main difference I'm seeing is ~1 vs. ~2 in the file path that we're checking for Chrome. The "good" log has:

Google Chrome found in expected location C:\Progra~1\Google\Chrome\Application\chrome.exe

whereas the "bad" log has:

 Cannot find Google Chrome at C:\Progra~2\Google\Chrome\Application\chrome.exe

Maybe we there was some change to our Windows environment around then, such that we're checking the wrong Program Files directory now? (I think some systems have both "Program Files" and "Program Files (x86)", and presumably one of those is ~1 and the other is ~2.)

jmaher or ahal, maybe one of you know whether there could have been a config change like that around that point in history? (or have other ideas about what to do or who to poke)

Comment 4

[Daniel Holbert [:dholbert]]

(redirecting needinfo to sparky who I'm seeing in commit logs as having recently worked with infra related to browsertime/speedometer3.)

Comment 5

[Greg Mierzwinski [:sparky]]

Thanks for the ping :dholbert!

:kshampur, it looks like this started on around March 30th. Do you think this lines up with any work that's been done on windows for the noise stuff? It looks intermittent right now based on the orangefactor and mozilla-central pushes.

Comment 6

[Daniel Holbert [:dholbert]]

(In reply to Greg Mierzwinski [:sparky] from comment #5)

It looks intermittent right now based on the orangefactor and mozilla-central pushes.

Ah, true - lemme fix that in the bug title.

Specifically, here are two recent "good" runs (again, ignore the "Traceback (most recent call last)/ ConnectionResetError" issues, those apparently are just noise and don't make it orange):
https://treeherder.mozilla.org/logviewer?job_id=559868398&repo=mozilla-central&task=LVrcyQSDSdysLsHiYL-jZg.0
https://treeherder.mozilla.org/logviewer?job_id=559674034&repo=mozilla-central&task=aFAl0byEQQ2_8gh_lS5BgA.0

And here's a recent "bad" run:
https://treeherder.mozilla.org/logviewer?job_id=559616882&repo=mozilla-central&task=I6NRemVNT7eXmOznGOJ91w.0

The pattern that I observed in comment 3 holds up here (where the "good" runs use C:\Progra~1\ while the "bad" run uses C:\Progra~2\).

Comment 7

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Greg Mierzwinski [:sparky] from comment #5)

Thanks for the ping :dholbert!

:kshampur, it looks like this started on around March 30th. Do you think this lines up with any work that's been done on windows for the noise stuff? It looks intermittent right now based on the orangefactor and mozilla-central pushes.

that would be weird if that stuff affected it, but the timing is suspicious for sure.

looping in :mcornmesser for thoughts...

Comment 8

[Mark Cornmesser [:markco]]

It works out to:

• Progra~1 = C:\Program Files
• Progra~2 = C:\Program Files (x86)

Chrome should be in the `1:

PS C:\Program Files\Google\Chrome\Application> dir


    Directory: C:\Program Files\Google\Chrome\Application


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          4/8/2026   2:42 PM                147.0.7727.56
d-----         4/10/2026  10:38 AM                SetupMetrics
-a----          4/6/2026   9:23 PM        4040344 chrome.exe
-a----          4/8/2026   2:42 PM            410 chrome.VisualElementsManifest.xml
-a----          4/6/2026   9:23 PM        2266776 chrome_proxy.exe
-a----         3/30/2026   6:00 PM            155 initial_preferences

Do we know where C:\Progra~2\ is comoing from? Are we getting it from an envrioment variable?

Comment 9

[Intermittent Failures Robot]

31 failures were associated with this bug in the last 7 days.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 31

Table

	no_variant
windows11-x86_64/opt	31

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2028315&startday=2026-04-08&endday=2026-04-14&tree=all

Comment 10

[Intermittent Failures Robot]

26 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 26

Table

	no_variant
windows11-x86_64/opt	26

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2028315&startday=2026-04-13&endday=2026-04-19&tree=all

Comment 11

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Mark Cornmesser [:markco] from comment #8)

It works out to:

• Progra~1 = C:\Program Files
• Progra~2 = C:\Program Files (x86)

Chrome should be in the `1:

PS C:\Program Files\Google\Chrome\Application> dir


    Directory: C:\Program Files\Google\Chrome\Application


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          4/8/2026   2:42 PM                147.0.7727.56
d-----         4/10/2026  10:38 AM                SetupMetrics
-a----          4/6/2026   9:23 PM        4040344 chrome.exe
-a----          4/8/2026   2:42 PM            410 chrome.VisualElementsManifest.xml
-a----          4/6/2026   9:23 PM        2266776 chrome_proxy.exe
-a----         3/30/2026   6:00 PM            155 initial_preferences

Do we know where C:\Progra~2\ is comoing from? Are we getting it from an envrioment variable?

thanks for taking a look Mark.

so looks like Progra~2 is coming from here

https://searchfox.org/firefox-main/rev/bd4dbb8971753f7082ad484540007c164cf112bc/testing/mozharness/mozharness/mozilla/testing/raptor.py#914-919

because apparently the Progra~1 path does not exist?

Comment 12

[Kash Shampur [:kshampur] ⌚EST]

perhaps the progra alias is not reliable. I'll make a try push trying with the full path

Comment 13

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #12)

perhaps the progra alias is not reliable. I'll make a try push trying with the full path

https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=Cd8BJM8MR92rsqmqQUwjRw.0&tier=1%2C2%2C3&revision=37e3fa5df0cf9ae521766818dced434d4caba202

Comment 14

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #13)

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #12)

perhaps the progra alias is not reliable. I'll make a try push trying with the full path

https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=Cd8BJM8MR92rsqmqQUwjRw.0&tier=1%2C2%2C3&revision=37e3fa5df0cf9ae521766818dced434d4caba202

it's looking for x86 chrome because it seems some machines just don't have x64 chrome

some pass, like this js3 one

[task 2026-04-20T17:37:52.888+00:00] 17:37:52 INFO - Google Chrome found in expected location C:\Program Files\Google\Chrome\Application\chrome.exe which is correct

but it is otherwise failing to find it in Program Files and attempting to fall back to x86 (which btw i think we shouldn't do anymore since we don't run 32-bit perf tests anymore, but that's another issue altogether)
Cannot find Google Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

:mcornmesser is it possible Chrome is missing from C:\Program Files\Google\Chrome\Application\ on some machines?

Comment 15

[Mark Cornmesser [:markco]]

On random spot checks it is there. I wonder if we are hitting a file lock because of updating.

We start with a base MSI then let the machine update Chrome on the next boot after deployment. We may have drifted to far back. I will update the MSI and do a deployment this week.

Comment 16

[Kash Shampur [:kshampur] ⌚EST]

Thanks for checking!

Comment 17

[Mark Cornmesser [:markco]]

(In reply to Mark Cornmesser [:markco] from comment #15)

On random spot checks it is there. I wonder if we are hitting a file lock because of updating.

We start with a base MSI then let the machine update Chrome on the next boot after deployment. We may have drifted to far back. I will update the MSI and do a deployment this week.

The deployment may be delayed till next week. I need to leave this pool intact as I am doing the CPU throttling investigation.

Comment 18

[Intermittent Failures Robot]

30 failures were associated with this bug in the last 7 days.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 30

Table

	no_variant
windows11-x86_64/opt	30

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2028315&startday=2026-04-20&endday=2026-04-26&tree=all

Comment 19

[Intermittent Failures Robot]

16 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Table

	no_variant
windows11-x86_64/opt	16

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2028315&startday=2026-04-27&endday=2026-05-03&tree=all

Comment 20

[Mark Cornmesser [:markco]]

An updated base MSI had been included in a deployment last week. Please let me know if we continue to see this issue.

Comment 21

[Kash Shampur [:kshampur] ⌚EST]

Thanks a lot! I see no failures since april 29th