1629337 - `make_dafsa.py` is slow

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement, P3)

Description

[Nick Alexander :nalexander [he/him]]

I'm running mach build export a lot right now, and I notice that it's much slower than I remember. Ctrl-C breaking in the "big pause" reveals consistently a 100% CPU Python 3.7 process like:

/Users/nalexander/Mozilla/objdirs/objdir-gecko-android/_virtualenvs/init_py3/bin/python -m mozbuild.action.file_generate /Users/nalexander/Mozilla/gecko/xpcom/ds/tools/make_dafsa.py main nsSTSPreloadList.h .deps/nsSTSPreloadList.h.pp .deps/nsSTSPreloadList.h.stub /Users/nalexander/Mozilla/gecko/security/manager/ssl/nsSTSPreloadList.inc

Is it possible that this is significantly slower with Python 3 (or 3.7) than it is in Python 2?

Comment 1

[Nick Alexander :nalexander [he/him]]

rstewart: can you mark this as possibly regressed by some Python 3 conversion ticket? Thanks!

Comment 2

[Ricky Stewart]

I'll look into it in a bit. (For what it's worth the transition from Py2 to Py3 to this script happened 2 months ago, and I don't know if that tracks with your experience or if it's unrelated.)

Comment 3

[Ricky Stewart]

There isn't a meaningful difference between Py2 and Py3 here, this script just takes an extremely long time to run for some reason.

rickystewart-a5lvdq:mozilla-unified rickystewart$ rm obj-x86_64-apple-darwin19.2.0/security/manager/ssl/.deps/nsSTSPreloadList.h.*
rickystewart-a5lvdq:mozilla-unified rickystewart$ time make -C obj-x86_64-apple-darwin19.2.0/security/manager/ssl/ nsSTSPreloadList.h
security/manager/ssl/nsSTSPreloadList.h.stub
/Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init_py3/bin/python -m mozbuild.action.file_generate /Users/rickystewart/src/mozilla-unified/xpcom/ds/tools/make_dafsa.py main nsSTSPreloadList.h .deps/nsSTSPreloadList.h.pp .deps/nsSTSPreloadList.h.stub /Users/rickystewart/src/mozilla-unified/security/manager/ssl/nsSTSPreloadList.inc

real	0m27.952s
user	0m27.473s
sys	0m0.404s
rickystewart-a5lvdq:mozilla-unified rickystewart$ emacs security/manager/ssl/moz.build   # Use py2 instead
rickystewart-a5lvdq:mozilla-unified rickystewart$ ./mach build-backend
 0:00.72 /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init_py3/bin/python /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/config.status
Reticulating splines...
 0:01.49 File already read. Skipping: /Users/rickystewart/src/mozilla-unified/gfx/angle/targets/angle_common/moz.build
Finished reading 1778 moz.build files in 2.68s
Read 61 gyp files in parallel contributing 1.23s to total wall time
Processed into 10905 build config descriptors in 2.86s
RecursiveMake backend executed in 4.31s
  3858 total backend files; 0 created; 4 updated; 3854 unchanged; 0 deleted; 30 -> 1421 Makefile
FasterMake backend executed in 0.30s
  11 total backend files; 0 created; 2 updated; 9 unchanged; 0 deleted
Total wall time: 11.71s; CPU time: 9.48s; Efficiency: 81%; Untracked: 1.55s
rickystewart-a5lvdq:mozilla-unified rickystewart$ rm obj-x86_64-apple-darwin19.2.0/security/manager/ssl/.deps/nsSTSPreloadList.h.*
rickystewart-a5lvdq:mozilla-unified rickystewart$ time make -C obj-x86_64-apple-darwin19.2.0/security/manager/ssl/ nsSTSPreloadList.h
security/manager/ssl/nsSTSPreloadList.h.stub
/Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init/bin/python -m mozbuild.action.file_generate /Users/rickystewart/src/mozilla-unified/xpcom/ds/tools/make_dafsa.py main nsSTSPreloadList.h .deps/nsSTSPreloadList.h.pp .deps/nsSTSPreloadList.h.stub /Users/rickystewart/src/mozilla-unified/security/manager/ssl/nsSTSPreloadList.inc

real	0m26.835s
user	0m26.221s
sys	0m0.535s
rickystewart-a5lvdq:mozilla-unified rickystewart$

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1629337: Implements incremental dafsa algorithm r?rstewart

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1720818b9015
Implements incremental dafsa algorithm r=rstewart

Comment 6

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/1720818b9015