1828751 - when running mozbuild source tests on windows 11 (not windows 10), we get a perma failure in test_build.py::TestBuild::test_faster_make

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

as seen on this try push, the mbu job is perma failing:

[task 2023-04-18T19:20:22.267Z] ============================= test session starts =============================
[task 2023-04-18T19:20:22.267Z] platform win32 -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0 -- Z:\task_168184305659392\build\src\obj-x86_64-pc-windows-msvc\_virtualenvs\python-test\Scripts\python.exe
[task 2023-04-18T19:20:22.267Z] rootdir: z:\task_168184305659392\build\src, configfile: config\mozunit\mozunit\pytest.ini
[task 2023-04-18T19:20:22.267Z] collecting ... collected 3 items
[task 2023-04-18T19:20:22.267Z] 
[task 2023-04-18T19:20:22.267Z] python/mozbuild/mozbuild/test/backend/test_build.py::TestBuild::test_faster_make TEST-UNEXPECTED-FAIL
[task 2023-04-18T19:20:22.267Z] python/mozbuild/mozbuild/test/backend/test_build.py::TestBuild::test_faster_recursive_make TEST-UNEXPECTED-FAIL
[task 2023-04-18T19:20:22.267Z] python/mozbuild/mozbuild/test/backend/test_build.py::TestBuild::test_recursive_make TEST-UNEXPECTED-FAIL
[task 2023-04-18T19:20:22.267Z] 
[task 2023-04-18T19:20:22.268Z] ================================== FAILURES ===================================
[task 2023-04-18T19:20:22.268Z] _________________________ TestBuild.test_faster_make __________________________
[task 2023-04-18T19:20:22.268Z] python\mozbuild\mozbuild\test\backend\test_build.py:164: in test_faster_make
[task 2023-04-18T19:20:22.268Z]     build._run_make(
[task 2023-04-18T19:20:22.268Z] python\mozbuild\mozbuild\base.py:801: in _run_make
[task 2023-04-18T19:20:22.268Z]     return fn(**params)
[task 2023-04-18T19:20:22.268Z] python\mozbuild\mozbuild\base.py:807: in _run_command_in_objdir
[task 2023-04-18T19:20:22.268Z]     return self.run_process(cwd=self.topobjdir, **args)
[task 2023-04-18T19:20:22.268Z] python\mach\mach\mixin\process.py:110: in run_process
[task 2023-04-18T19:20:22.268Z]     self.log(logging.INFO, "new_process", {"args": " ".join(args)}, "{args}")
[task 2023-04-18T19:20:22.268Z] E   TypeError: sequence item 0: expected str instance, NoneType found
[task 2023-04-18T19:20:22.268Z] ---------------------------- Captured stdout call -----------------------------
[task 2023-04-18T19:20:22.268Z] 
[task 2023-04-18T19:20:22.268Z] ____________________ TestBuild.test_faster_recursive_make _____________________
[task 2023-04-18T19:20:22.268Z] python\mozbuild\mozbuild\test\backend\test_build.py:136: in test_faster_recursive_make
[task 2023-04-18T19:20:22.268Z]     build._run_make(
[task 2023-04-18T19:20:22.268Z] python\mozbuild\mozbuild\base.py:801: in _run_make
[task 2023-04-18T19:20:22.268Z]     return fn(**params)
[task 2023-04-18T19:20:22.268Z] python\mozbuild\mozbuild\base.py:807: in _run_command_in_objdir
[task 2023-04-18T19:20:22.268Z]     return self.run_process(cwd=self.topobjdir, **args)
[task 2023-04-18T19:20:22.268Z] python\mach\mach\mixin\process.py:110: in run_process
[task 2023-04-18T19:20:22.268Z]     self.log(logging.INFO, "new_process", {"args": " ".join(args)}, "{args}")
[task 2023-04-18T19:20:22.268Z] E   TypeError: sequence item 0: expected str instance, NoneType found
[task 2023-04-18T19:20:22.268Z] ---------------------------- Captured stdout call -----------------------------
[task 2023-04-18T19:20:22.268Z] 
[task 2023-04-18T19:20:22.268Z] ________________________ TestBuild.test_recursive_make ________________________
[task 2023-04-18T19:20:22.268Z] python\mozbuild\mozbuild\test\backend\test_build.py:107: in test_recursive_make
[task 2023-04-18T19:20:22.268Z]     build._run_make(
[task 2023-04-18T19:20:22.269Z] python\mozbuild\mozbuild\base.py:801: in _run_make
[task 2023-04-18T19:20:22.269Z]     return fn(**params)
[task 2023-04-18T19:20:22.269Z] python\mozbuild\mozbuild\base.py:807: in _run_command_in_objdir
[task 2023-04-18T19:20:22.269Z]     return self.run_process(cwd=self.topobjdir, **args)
[task 2023-04-18T19:20:22.269Z] python\mach\mach\mixin\process.py:110: in run_process
[task 2023-04-18T19:20:22.269Z]     self.log(logging.INFO, "new_process", {"args": " ".join(args)}, "{args}")
[task 2023-04-18T19:20:22.269Z] E   TypeError: sequence item 0: expected str instance, NoneType found
[task 2023-04-18T19:20:22.269Z] ---------------------------- Captured stdout call -----------------------------

there was a similar intermittent bug 1790968 in the past, but that usually failed for other errors (like rmtree issues).

What is different here:

• current tests run on windows10-64-2004
• failing tests are trying to upgrade to windows11-64-2009

I am not familiar with this test and would prefer the test owner chime in on what this specific set of 3 tests are doing and what is expected.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Alex, do you know who would be a good contact for looking into this bug? specifically python/mozbuild/mozbuild/test/backend/test_build.py

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1828751 - Use mozmake toolchain artifact for backend.test_build on automation.

Comment 3

[Mike Hommey [:glandium]]

Can you test the attached patch?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I get an error pushing to try, but here is the try syntax as well as the error:

./mach try fuzzy --worker-override='win10-64-2004=gecko-t/win11-64-2009-source-alpha' -q 'source-test windows mozbuild'
estimates: Runs 4 tasks (1 selected, 3 dependencies)
estimates: Total task duration 0:40:03
estimates: In the top 97% of durations
estimates: Should take about 0:44:19 (Finished around 2023-04-20 07:29)
Creating temporary commit for remote...
A try_task_config.json
pushing to ssh://hg.mozilla.org/try
searching for changes
remote: waiting for lock on working directory of /repo/hg/mozilla/try held by process '18106' on host 'hgssh1.dmz.mdc1.mozilla.com/effffffc'
remote: abort: working directory of /repo/hg/mozilla/try: timed out waiting for lock held by 'hgssh1.dmz.mdc1.mozilla.com/effffffc:24242'
temporary commit removed, repository restored
abort: stream ended unexpectedly (got 0 bytes, expected 4)
Error running mach:

    ['try', 'fuzzy', '--worker-override=win10-64-2004=gecko-t/win11-64-2009-source-alpha', '-q', 'source-test windows mozbuild']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file try`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '('C:\\mozilla-build\\python3\\Scripts\\hg.EXE', 'push-to-try', '-m', 'Fuzzy query=source-test windows mozbuild\n\nPushed via `mach try fuzzy`')' returned non-zero exit status 255.

  File "C:\Users\elvis\mozilla-central\tools\tryselect\mach_commands.py", line 338, in try_fuzzy
    return run(command_context, **kwargs)
  File "C:\Users\elvis\mozilla-central\tools\tryselect\mach_commands.py", line 196, in run
    return mod.run(**kwargs)
  File "c:\users\elvis\mozilla-central\tools\tryselect\selectors\fuzzy.py", line 248, in run
    return push_to_try(
  File "c:\users\elvis\mozilla-central\tools\tryselect\push.py", line 228, in push_to_try
    vcs.push_to_try(commit_message, allow_log_capture=allow_log_capture)
  File "c:\users\elvis\mozilla-central\python\mozversioncontrol\mozversioncontrol\__init__.py", line 555, in push_to_try
    subprocess.check_call(
  File "C:\mozilla-build\python3\lib\subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 5

[Mike Hommey [:glandium]]

sounds like a problem with the try server. Try again.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

ah, this works!

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=19ccf0177f1848c3b31fecc4b99dcccc874de655&searchStr=mozbuild&selectedTaskRun=RdG0jajbTwC_inlFu5F9ug.0

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a05ee20f3d3
Use mozmake toolchain artifact for backend.test_build on automation. r=firefox-build-system-reviewers,ahochheiden

Comment 8

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/2a05ee20f3d3