1759084 - Perma systemsymbols tasks: Exception: The "common" site is out-of-date, even though it has already been activated. Was it modified while this Mach process was running?

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Narcis Beleuzu [:NarcisB]]

Push with failure:
https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&fromchange=446c2fcf0c0b317aeca92bd4ee2182543e722c31&selectedTaskRun=Dn2DqBH0RuuvqRZXQdYSMA.0

Log link:
https://treeherder.mozilla.org/logviewer?job_id=370621931&repo=mozilla-central&lineNumber=159

Log snippet:

[task 2022-03-11T00:24:09.718Z] created virtual environment CPython3.6.9.final.0-64 in 23ms
[task 2022-03-11T00:24:09.718Z]   creator CPython3Posix(dest=/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/common, clear=False, no_vcs_ignore=False, global=False)
[task 2022-03-11T00:24:09.718Z]   activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
[task 2022-03-11T00:24:10.023Z] Traceback (most recent call last):
[task 2022-03-11T00:24:10.023Z]   File "toolkit/crashreporter/tools/upload_symbols.py", line 260, in <module>
[task 2022-03-11T00:24:10.023Z]     sys.exit(main())
[task 2022-03-11T00:24:10.023Z]   File "toolkit/crashreporter/tools/upload_symbols.py", line 61, in main
[task 2022-03-11T00:24:10.023Z]     config.activate_virtualenv()
[task 2022-03-11T00:24:10.023Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 863, in activate_virtualenv
[task 2022-03-11T00:24:10.023Z]     self.virtualenv_manager.activate()
[task 2022-03-11T00:24:10.023Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 552, in activate
[task 2022-03-11T00:24:10.023Z]     self.ensure()
[task 2022-03-11T00:24:10.023Z]   File "/builds/worker/checkouts/gecko/python/mach/mach/site.py", line 530, in ensure
[task 2022-03-11T00:24:10.023Z]     f'The "{self._site_name}" site is out-of-date, even though it has '
[task 2022-03-11T00:24:10.023Z] Exception: The "common" site is out-of-date, even though it has already been activated. Was it modified while this Mach process was running?
[task 2022-03-11T00:24:10.053Z] Creating default state directory: /builds/worker/.mozbuild
[taskcluster 2022-03-11 00:24:10.396Z] === Task Finished ===
[taskcluster 2022-03-11 00:24:10.396Z] Unsuccessful task run with exit code: 1 completed in 153.653 seconds

Jon, could this be caused by Bug 1757956 ?

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

I highly doubt that this is a gsvelto-related issue :)
I'll dig in later today.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1759084: Tighten `MachSiteManager` API surface

The Mach site management has been significantly simplified
in mach_initialize.py, so less of the MachSiteManager
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1759084: Print reason for site being out-of-date

There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.

Return a reason string from _is_venv_up_to_date(), and print it in
cases where a virtualenv being out-of-date is erroneous.

Depends on D140854

Comment 4

[Gabriele Svelto [:gsvelto]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #1)

I highly doubt that this is a gsvelto-related issue :)
I'll dig in later today.

Thanks! I'm in the process of making a significant overhaul of those tasks so I was super-puzzled that they broke even before I landed my changes :)

Comment 5

[Intermittent Failures Robot]

18 failures in 2837 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 18

Platform and build breakdown:

• symbols: 17
  • opt: 17
• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1759084&startday=2022-03-07&endday=2022-03-13&tree=all

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1759084: Don't redundantly activate site in upload_symbols.py

upload_symbols.py is always executed within a Mach site (otherwise
activating the common command site would fail), therefore
re-activating common is redundant.

Depends on D140855

Comment 7

[Intermittent Failures Robot]

16 failures in 3304 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Platform and build breakdown:

• symbols: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1759084&startday=2022-03-14&endday=2022-03-20&tree=all

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/13aef63ad6a5
Tighten `MachSiteManager` API surface r=ahal
https://hg.mozilla.org/integration/autoland/rev/5f18914637a3
Print reason for site being out-of-date r=ahal
https://hg.mozilla.org/integration/autoland/rev/412197c555c6
Don't redundantly activate site in upload_symbols.py r=ahal

Comment 9

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/13aef63ad6a5
https://hg.mozilla.org/mozilla-central/rev/5f18914637a3
https://hg.mozilla.org/mozilla-central/rev/412197c555c6

Comment 10

[Cristian Tuns]

Backed out for causing symbol bustages.

• Backout link
• Push with failures
• Failure Log
• Failure line: ModuleNotFoundError: No module named 'zstandard'

Comment 11

[Mitchell Hentges [:mhentges] 🦀]

Patch updated and re-landed.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d254d42ed23
Tighten `MachSiteManager` API surface r=ahal
https://hg.mozilla.org/integration/autoland/rev/c74d2b2bd792
Print reason for site being out-of-date r=ahal
https://hg.mozilla.org/integration/autoland/rev/43f7556db2e1
Don't redundantly activate site in upload_symbols.py r=ahal

Comment 13

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/3d254d42ed23
https://hg.mozilla.org/mozilla-central/rev/c74d2b2bd792
https://hg.mozilla.org/mozilla-central/rev/43f7556db2e1

Comment 14

[Intermittent Failures Robot]

4 failures in 3341 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• symbols: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1759084&startday=2022-03-21&endday=2022-03-27&tree=all