Closed Bug 1703134 Opened 3 years ago Closed 27 days ago

Intermittent [tier2] raptor-browsertime Critical: [chrome] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters

Categories

(Testing :: Raptor, defect, P5)

defect

Tracking

(firefox96 wontfix)

RESOLVED INCOMPLETE
96 Branch
Tracking Status
firefox96 --- wontfix

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Attachments

(2 files)

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=335507079&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Yd8bNRlbT-CLitFo8MSnog/runs/0/artifacts/public/logs/live_backing.log


[task 2021-04-06T00:39:11.314Z] 00:39:11     INFO -  raptor-perftest Info: Merging profile: /opt/worker/tasks/task_161766945739418/build/tests/raptor/raptor/profile_data/raptor
[task 2021-04-06T00:39:11.315Z] 00:39:11     INFO -  raptor-perftest Info: Local browser profile: /var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default
[task 2021-04-06T00:39:11.315Z] 00:39:11     INFO -  raptor-browsertime Info: Removing mozprofile delimiters from browser profile
[task 2021-04-06T00:39:11.315Z] 00:39:11 CRITICAL -  raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters
[task 2021-04-06T00:39:11.316Z] 00:39:11     INFO -  raptor-browsertime Info: cwd: '/opt/worker/tasks/task_161766945739418/build'
[task 2021-04-06T00:39:11.316Z] 00:39:11     INFO -  raptor-browsertime Info: browsertime_node: /opt/worker/tasks/task_161766945739418/fetches/node/bin/node
[task 2021-04-06T00:39:11.316Z] 00:39:11     INFO -  raptor-browsertime Info: browsertime_node: os.stat_result(st_mode=33261, st_ino=31538611, st_dev=16777221, st_nlink=1, st_uid=36, st_gid=20, st_size=42289088, st_atime=1617669464, st_mtime=1609766268, st_ctime=1617669466)
[task 2021-04-06T00:39:11.317Z] 00:39:11     INFO -  raptor-browsertime Info: browsertime_browsertimejs: /opt/worker/tasks/task_161766945739418/fetches/browsertime/node_modules/browsertime/bin/browsertime.js
[task 2021-04-06T00:39:11.317Z] 00:39:11     INFO -  raptor-browsertime Info: browsertime_browsertimejs: os.stat_result(st_mode=33261, st_ino=31541327, st_dev=16777221, st_nlink=1, st_uid=36, st_gid=20, st_size=4376, st_atime=1617669463, st_mtime=1610746082, st_ctime=1617669466)
[task 2021-04-06T00:39:11.317Z] 00:39:11     INFO -  raptor-browsertime Info: browsertime_geckodriver: /opt/worker/tasks/task_161766945739418/fetches/geckodriver```

This looks to be osx 10.15 specific, backfilling does not show any conclusive results, so it might've been failing on osx 10.15 since it was enabled intree.

Sparky, what is your take on this?

Flags: needinfo?(gmierz2)
Summary: Intermittent raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters → Intermittent OSX 10.15 raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters

Oh those tests shouldn't be running there. :bebe/:alexandrui can either of you disable these benchmark tests on autoland?

Flags: needinfo?(gmierz2)
Flags: needinfo?(fstrugariu)
Flags: needinfo?(aionescu)
Assignee: nobody → aionescu
Status: NEW → ASSIGNED
Flags: needinfo?(aionescu)

alex took the bug

Flags: needinfo?(fstrugariu)
Pushed by aionescu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3eb017d9f3fa
Disable chrome/chroium browsertime benchmarks on autoland r=perftest-reviewers,sparky
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 89 Branch

Alexandru, this is happening again.

Should we reopen this bug to keep track of this issue, or make a new one?

Flags: needinfo?(aionescu)

Bebe, can you take a look?

Flags: needinfo?(aionescu) → needinfo?(fstrugariu)

Looks like this is not reproducing anymore.
I will take a look if we see this again

Flags: needinfo?(fstrugariu)

:Sparky can you take a look over this please

Flags: needinfo?(fstrugariu) → needinfo?(gmierz2)

No I don't have time, you'll have to look into it. We could decrease this test to tier 3 since it isn't very important.

Flags: needinfo?(gmierz2) → needinfo?(fstrugariu)

Tested this locally and it works as expected on my setup.
Note: I have a Linux setup and this issue is mac specific.

Also from the logs it looks like this fails on a random cycle. So the browser and chromedriver should work but at some point a error is triggered in browser.

UnexpectedAlertOpenError: unexpected alert open: {Alert text : WebGL context lost, please reload the page}

This error looks something browser/WebGL specific

Can we get someone with a MAC setup to run this locally to see what happens.

Flags: needinfo?(gmierz2)
Flags: needinfo?(fstrugariu)
Flags: needinfo?(dave.hunt)

:bebe, before trying to look into more locally, can you check to see if an older version of chromium works? Alternatively, if there isn't a version which passes, we could drop it to a tier-3 test.

Flags: needinfo?(gmierz2) → needinfo?(fstrugariu)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #17)

Tested this locally and it works as expected on my setup.
Note: I have a Linux setup and this issue is mac specific.

Also from the logs it looks like this fails on a random cycle. So the browser and chromedriver should work but at some point a error is triggered in browser.

UnexpectedAlertOpenError: unexpected alert open: {Alert text : WebGL context lost, please reload the page}

This error looks something browser/WebGL specific

Can we get someone with a MAC setup to run this locally to see what happens.

I'm happy to try to run something on my MacBook Pro if you can provide a command line for me, and details of any other prerequisites.

Flags: needinfo?(dave.hunt)

It's looking good now, I think a chromium update fixed it. :bebe, no need for looking into this for now: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=chromium

Flags: needinfo?(fstrugariu)
Assignee: aionescu → nobody
Summary: Intermittent OSX 10.15 raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters → High frequency OSX 10.15 raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters
Target Milestone: 89 Branch → ---

The last of these browsertime tests with Chrome Release failed.

Flags: needinfo?(fstrugariu)

:Sparky can you take a look over this?

Flags: needinfo?(fstrugariu) → needinfo?(gmierz2)

It looks like Chrome Release isn't updating on MacOSX 10.15 Shippable. :dividehex, would you able to look into this issue? On these machines, chrome should autoupdate and the version should be at 91 but it's currently at 88:

[task 2021-05-28T10:44:57.767Z] 10:44:57     INFO -  raptor-perftest Info: Attempting to get version through fallback method...
[task 2021-05-28T10:44:57.802Z] 10:44:57     INFO -  Google Chrome 88.0.4324.192
[task 2021-05-28T10:44:57.894Z] 10:44:57     INFO -  raptor-perftest Info: Browser name: chrome
[task 2021-05-28T10:44:57.894Z] 10:44:57     INFO -  raptor-perftest Info: Browser version: 88.0.4324.192
Flags: needinfo?(gmierz2) → needinfo?(jwatkins)

:dhouse/:aerickson could you look into this issue?

Flags: needinfo?(dhouse)
Flags: needinfo?(aerickson)

I'll check these. Our auto-update of chrome may have broken when we upgraded to MacOS Catalina, on the R8 macminis.

Assignee: nobody → dhouse
Flags: needinfo?(jwatkins)
Flags: needinfo?(aerickson)

the macminis are running Chrome 91.0.4472.77 now. I re-triggered keystone and reset the permissions on /Applications/Google.

I think because chrome was installed on the R8's under root, the file permissions blocked the user execution of keystone (gets set as by chrome as a launchagent. so it runs when the user logs into the display console) from overwriting the chrome binaries and resources.
keystone was giving errors like:

Exit code: 8. Standard error output: \"rsync: failed to set times on \\\"/Applications/Google Chrome.app/.\\\": Operation not permitted (1)\\nrsync: failed to set times on \\\"/Applications/Google Chrome.app/Contents\\\"
Flags: needinfo?(dhouse)

Thanks :dhouse!

Whiteboard: [perftest:triage]

No need to do anything here, the failures were caused by an outdated chrome version on the macs.

Whiteboard: [perftest:triage]

One worker macmini had not updated (macmini-r8-257 was on chrome v88). I manually triggered the upgrade on it and it pulled v91

Summary: High frequency OSX 10.15 raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters → High frequency OSX raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters

:dhouse, it looks like some machines are having issues updating chrome to the latest version (should be 91). Could you take a look? I'm seeing some machines with version 81, and others with version 88.

Flags: needinfo?(dhouse)
See Also: → 1722017
Depends on: 1722015
Summary: High frequency OSX raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters → High frequency raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters
Summary: High frequency raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters → Perma [tier2] raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters

The real error here is that chromedriver can't be located:

Exception: Cannot find the chromedriver for the chrome version being tested: /home/cltbld/tasks/task_163229241112151/fetches/94chromedriver

This could be due to the version of Chrome updating (v94 appears to have been released today). Cristian: please can you open a separate bug for this issue?

:davehunt Hi, as per your request, logged https://bugzilla.mozilla.org/show_bug.cgi?id=1732226

There have been 79 failures in the last 7 days:

  • 35 failures on linux1804-64-shippable-qr opt
  • 20 failures on macosx1014-64-shippable-qr opt
  • 9 failures on macosx1015-64-shippable-qr opt
  • 15 failures on windows10-64-shippable-qr opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=358595498&repo=mozilla-central&lineNumber=2306

[task 2021-11-19T11:59:56.720Z] 11:59:56     INFO -  raptor-perftest Info: Local browser profile: /tmp/tmpt74z27ri.mozrunner/Default
[task 2021-11-19T11:59:56.721Z] 11:59:56     INFO -  raptor-browsertime Info: Removing mozprofile delimiters from browser profile
[task 2021-11-19T11:59:56.721Z] 11:59:56 CRITICAL -  raptor-browsertime Critical: [chrome] Exception [Errno 2] No such file or directory: '/tmp/tmpt74z27ri.mozrunner/Default/user.js' while removing mozprofile delimiters
[task 2021-11-19T11:59:56.721Z] 11:59:56     INFO -  raptor-browsertime Info: cwd: '/home/cltbld/tasks/task_163732313428545/build'
[task 2021-11-19T11:59:56.721Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_node: /home/cltbld/tasks/task_163732313428545/fetches/node/bin/node
[task 2021-11-19T11:59:56.722Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_node: os.stat_result(st_mode=33261, st_ino=7379503, st_dev=64768, st_nlink=1, st_uid=1006, st_gid=1007, st_size=48928552, st_atime=1637323136, st_mtime=1617721616, st_ctime=1637323136)
[task 2021-11-19T11:59:56.722Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_browsertimejs: /home/cltbld/tasks/task_163732313428545/fetches/browsertime/node_modules/browsertime/bin/browsertime.js
[task 2021-11-19T11:59:56.722Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_browsertimejs: os.stat_result(st_mode=33261, st_ino=10497085, st_dev=64768, st_nlink=1, st_uid=1006, st_gid=1007, st_size=4988, st_atime=1637323136, st_mtime=1633607361, st_ctime=1637323136)
[task 2021-11-19T11:59:56.723Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_geckodriver: /home/cltbld/tasks/task_163732313428545/fetches/geckodriver
[task 2021-11-19T11:59:56.723Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_geckodriver: os.stat_result(st_mode=33261, st_ino=6988416, st_dev=64768, st_nlink=1, st_uid=1006, st_gid=1007, st_size=8106856, st_atime=1637323135, st_mtime=1636390340, st_ctime=1637323135)
[task 2021-11-19T11:59:56.723Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_chromedriver: /home/cltbld/tasks/task_163732313428545/fetches/{}chromedriver
[task 2021-11-19T11:59:56.724Z] 11:59:56     INFO -  raptor-browsertime Info: browsertime_chromedriver: [Errno 2] No such file or directory: '/home/cltbld/tasks/task_163732313428545/fetches/{}chromedriver'

Greg, as the owner of the component could you help us assign this to someone?

Flags: needinfo?(gmierz2)
Whiteboard: [stockwell needswork:owner]

This issue will be fixed when the patch under review lands.

Flags: needinfo?(gmierz2)
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/6c90d21f400c
Add chromedriver for v96. r=perftest-reviewers,AlexandruIonescu
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch

That failure bump yesterday was expected as the new chromedrive hadn't landed yet.

This has started appearing again on central, Greg, should we make a new bug for it?

Flags: needinfo?(gmierz2)

I've reopened this one, thanks!

:dhouse, it looks like the mac machines aren't updating correctly, can you take a look?

Status: RESOLVED → REOPENED
Flags: needinfo?(gmierz2)
Resolution: FIXED → ---
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Flags: needinfo?(dhouse)

Sorry for the mass ni? but could either of you take a look at what's happening with the macs please? They aren't updating to the new Chrome version (96) for some reason. The versions seem to be stuck at 88 and 92.

Flags: needinfo?(jmaher)
Flags: needinfo?(dhouse)
Flags: needinfo?(aerickson)

here is a task from 2 days ago that shows fetch-chromium:
https://firefox-ci-tc.services.mozilla.com/tasks/Ptd7eftwQgKh6wTOcjvWVA (ends up with revision: 945744)

I am not able to easily figure out a chromium version. at least 88/92/96. I did see in the original task referenced here:
https://firefox-ci-tc.services.mozilla.com/tasks/fBwFXYN1TC6wz1NZsjKzMg/runs/0/logs/public/logs/live.log

this:
{'name': 'chromium', 'version': '98.0.4735.0'}

The script calls fetch-chromium.py --platform mac, and we have had some recent changes to fetch-chromium:
https://hg.mozilla.org/mozilla-central/log/tip/taskcluster/scripts/misc/fetch-chromium.py

when the script runs locally it finds a revision via:
https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Mac%2FLAST_CHANGE?alt=media

this results in revision=946511.

if I unpack and ls -laR | grep -i Version, I see files like:
Chromium.app/Contents/Frameworks/Chromium Framework.framework/Versions/98.0.4740.0/Resources/

so this indicates that we are using version 98- should we be using a different version?

possibly this is resolved?

Flags: needinfo?(jmaher)

Sorry, this isssue is unrelated to chromium now. It's a chrome failure. Here's a link to one of the most recent failures: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedJob=359548232&revision=d03f875556391582e06abbf647835af8ca59f94b&searchStr=mac%2Cchrome&selectedTaskRun=bQ_MlsMoS9mrmRHMCPdGDw.0

Note the error line that suggests chrome 92 is installed on the mac:

Exception: Cannot find the chromedriver for the chrome version being tested: /opt/worker/tasks/task_163818342129411/fetches/92chromedriver

I think we should probably rename this bug or make a new one.

Summary: Perma [tier2] raptor-browsertime Critical: [chromium] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters → Perma [tier2] raptor-browsertime Critical: [chrome] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters

could chromedriver be different from chromium version?

odd, that specific task linked:
https://firefox-ci-tc.services.mozilla.com/tasks/bQ_MlsMoS9mrmRHMCPdGDw

depends on task fetch chromedriver 96:
https://firefox-ci-tc.services.mozilla.com/tasks/AZdzONqnQ6u6EGdW4ModaQ

and in the log of the failing task it shows chromedriver 96 being downloaded and unpacked:
https://firefox-ci-tc.services.mozilla.com/tasks/bQ_MlsMoS9mrmRHMCPdGDw/runs/0/logs/public/logs/live.log

it also shows chromedriver 94 and 95, but not 92. Maybe there is a hardcoded version on a path somewhere?

given the error message, we must have chrome v.92 if it is looking for 92chromedriver (I am making an assumption).

looking for chrome.app in the failing task, I see:
https://firefox-ci-tc.services.mozilla.com/tasks/bQ_MlsMoS9mrmRHMCPdGDw/runs/0/logs/public/logs/live.log

/Applications/Google Chrome.app/Contents/MacOS/Google Chrome

one would assume this is hardcoded to v.92? do we need to update our preinstalled version of google chrome on osx 10.15 mac minis?

For chromium, we get the chromedriver it needs through the fetch-task. But for chrome, we have these fetch-tasks for the chromedrivers: https://searchfox.org/mozilla-central/source/taskcluster/ci/fetch/browsertime.yml#32-138

The way we have the chromedriver thing setup in raptor is that we find the current chrome version and then use the correct chromedriver for it in the moz_fetches directory (assuming it's there). We leave the last 3 versions of chrome in-tree because the machines don't update instantaneously.

I thought we had auto-updating on these macs but maybe it's been hardcoded to v92 and if so, then we'd need to update it.

I think :dhouse could determine if the R8 minis have the preinstalled google chrome set to auto update?

I have forced an update across the mac workers (r8's), and I'm investigating why they got stuck (again, this happened in https://bugzilla.mozilla.org/show_bug.cgi?id=1722015 when we needed v92).

I'll re-check their chrome version this afternoon and manually fix ones that don't get it this morning.

Flags: needinfo?(aerickson)

I found that the r7 macs were auto-updating (so the autoupdate was fixed there) but only some of the r8's were (I think ones that had been new/reimaged).
They're all updated to chrome v96 now (I quarantined the ones I am manually updating, and I will un-quarantine them when they're updated), and I plan to reimage a set of r8's to see if a new install fixes the auto-update.

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There have been 37 total failures in the last 7 days (recent failure log).
Affected platforms are:

  • linux1804-64-shippable-qr
  • macosx1014-64-shippable-qr
  • macosx1015-64-shippable-qr
  • windows10-64-shippable-qr

Greg, could you please take a look?

Flags: needinfo?(gmierz2)

This should be resolved by https://phabricator.services.mozilla.com/D135251 which landed yesterday

Flags: needinfo?(gmierz2)
Flags: needinfo?(dhouse)
Status: REOPENED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Assignee: dhouse → nobody
See Also: → 1828417
Status: REOPENED → RESOLVED
Closed: 2 years ago1 year ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 1 year ago9 months ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 9 months ago7 months ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Summary: Perma [tier2] raptor-browsertime Critical: [chrome] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters → Intermittent [tier2] raptor-browsertime Critical: [chrome] Exception [Errno 2] No such file or directory: '/var/folders/0y/lyz4b4sx50dg5sbmmsj0tn3r000014/T/tmpzv56t98k.mozrunner/Default/user.js' while removing mozprofile delimiters
Status: REOPENED → RESOLVED
Closed: 7 months ago27 days ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: