1653909 - icupkg not found

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Dan Horák]

After https://hg.mozilla.org/integration/autoland/rev/d3004626988d landed in trunk on Friday (July 17th) we see a build failure on Fedora/s390x (big endian) system.

...
gmake[4]: Nothing to be done for 'target-objects'.
gmake[4]: Leaving directory '/home/sharkcz/projects/firefox/obj-s390x-ibm-linux-gnu/config/external/icu/common'
./icudt67b.dat.stub
/home/sharkcz/projects/firefox/obj-s390x-ibm-linux-gnu/_virtualenvs/init_py3/bin/python -m mozbuild.action.file_generate /home/sharkcz/projects/firefox/config/external/icu/data/convert_icudata.py main config/external/icu/data/icudt67b.dat config/external/icu/data/.deps/icudt67b.dat.pp config/external/icu/data/.deps/icudt67b.dat.stub /home/sharkcz/projects/firefox/config/external/icu/data/icudt67l.dat
Error opening file "/home/sharkcz/projects/firefox/obj-s390x-ibm-linux-gnu/dist/host/bin/icupkg"
Traceback (most recent call last):
  File "/home/sharkcz/projects/firefox/python/mozbuild/mozbuild/action/file_generate.py", line 72, in main
    ret = module.__dict__[method](output, *args.additional_arguments, **kwargs)
  File "/home/sharkcz/projects/firefox/config/external/icu/data/convert_icudata.py", line 13, in main
    subprocess.run([
  File "/usr/lib64/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib64/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/sharkcz/projects/firefox/obj-s390x-ibm-linux-gnu/dist/host/bin/icupkg'
gmake[3]: *** [backend.mk:261: config/external/icu/data/.deps/icudt67b.dat.stub] Error 1

Comment 1

[Dan Horák]

It looks to me it's a build order problem. When icupkg tries to run only obj-s390x-ibm-linux-gnu/config/external/icu/common contains compiled files (*.o) and make in obj-s390x-ibm-linux-gnu/config/external/icu/icupkg fails. After I run make manually in i18n and toolutil dirs, everything went OK (for ICU).

Comment 2

[Makoto Kato [:m_kato]]

Dan, https://hg.mozilla.org/integration/autoland/rev/d3004626988d l is backed out, then it is re-landed as https://hg.mozilla.org/integration/autoland/rev/65418a2fb344. So could you test this on the latest mozilla-central?

Comment 3

[Dan Horák]

I'm now at https://hg.mozilla.org/mozilla-central/rev/f0ed5585b6adac269e39dc3241fba9c4f52aa4c4 (resp. at https://github.com/mozilla/gecko-dev/commit/b58a9a8eca78bc6a8543ba263816c271d4b62ce1), but the problem is still there. I'm using a sequential build with mach build -j1, so it's not a parallel make issue.

Comment 4

[Mike Hommey [:glandium]]

Can you do the following:

• mach clobber
• mach configure
• mach build pre-export export

Then attach the output for:

• mach build -C . config/external/icu/data/icudt67b.dat

Please also attach $objdir/root.mk.

Comment 5

[Dan Horák]

Done, I run the 3 steps (no .mozconfig) and then

sharkcz@rock-kvmlp-fedora firefox]$ ./mach build -j1 -C . config/external/icu/data/icudt67b.dat
 0:00.24 /usr/bin/gmake -C /home/sharkcz/projects/firefox/obj-s390x-ibm-linux-gnu/. -j1 -s config/external/icu/data/icudt67b.dat
 0:00.29 ./icudt67b.dat.stub
 0:00.37 Error opening file "/home/sharkcz/projects/firefox/obj-s390x-ibm-linux-gnu/dist/host/bin/icupkg"
 0:00.37 Traceback (most recent call last):
 0:00.37   File "/home/sharkcz/projects/firefox/python/mozbuild/mozbuild/action/file_generate.py", line 72, in main
 0:00.37     ret = module.__dict__[method](output, *args.additional_arguments, **kwargs)
 0:00.37   File "/home/sharkcz/projects/firefox/config/external/icu/data/convert_icudata.py", line 13, in main
 0:00.37     subprocess.run([
 0:00.37   File "/usr/lib64/python3.8/subprocess.py", line 489, in run
 0:00.37     with Popen(*popenargs, **kwargs) as process:
 0:00.37   File "/usr/lib64/python3.8/subprocess.py", line 854, in __init__
 0:00.37     self._execute_child(args, executable, preexec_fn, close_fds,
 0:00.37   File "/usr/lib64/python3.8/subprocess.py", line 1702, in _execute_child
 0:00.37     raise child_exception_type(errno_num, err_msg, err_filename)
 0:00.37 FileNotFoundError: [Errno 2] No such file or directory: '/home/sharkcz/projects/firefox/obj-s390x-ibm-linux-gnu/dist/host/bin/icupkg'
 0:00.38 gmake: *** [backend.mk:261: config/external/icu/data/.deps/icudt67b.dat.stub] Error 1
 0:00.40 109 compiler warnings present.
Hey! Builds initiated with `mach build $A_SPECIFIC_TARGET` may not always work, even if the code being built is correct. Consider doing a bare `mach build` instead.

Comment 6

[Dan Horák]

Attachment: root.mk

Comment 7

[Mike Hommey [:glandium]]

Err, sorry, I wanted $objdir/backend.mk.

Comment 8

[Dan Horák]

Attachment: backend.mk

no problem :-)

Comment 9

[Mike Hommey [:glandium]]

Attachment: Bug 1653909 - Use the stub file for the icu data file dependencies.

The rules we create in the backend for generated files use a stub file
for the real file generation, and we do need dependencies on that
specific target for the changes in bug 1264836 to work without races.

Comment 10

[Mike Hommey [:glandium]]

Can you check the attached patch works?

Comment 11

[Dan Horák]

(In reply to Mike Hommey [:glandium] from comment #10)

Can you check the attached patch works?

Yes, seems to work OK now, thanks.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1264836

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7f344a59f74a
Use the stub file for the icu data file dependencies. r=froydnj

Comment 14

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/7f344a59f74a