Closed Bug 1629337 Opened 4 years ago Closed 4 years ago

`make_dafsa.py` is slow

Categories

(Firefox Build System :: General, enhancement, P3)

enhancement

Tracking

(firefox80 fixed)

RESOLVED FIXED
mozilla80
Tracking Status
firefox80 --- fixed

People

(Reporter: nalexander, Assigned: mhentges)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

I'm running mach build export a lot right now, and I notice that it's much slower than I remember. Ctrl-C breaking in the "big pause" reveals consistently a 100% CPU Python 3.7 process like:

/Users/nalexander/Mozilla/objdirs/objdir-gecko-android/_virtualenvs/init_py3/bin/python -m mozbuild.action.file_generate /Users/nalexander/Mozilla/gecko/xpcom/ds/tools/make_dafsa.py main nsSTSPreloadList.h .deps/nsSTSPreloadList.h.pp .deps/nsSTSPreloadList.h.stub /Users/nalexander/Mozilla/gecko/security/manager/ssl/nsSTSPreloadList.inc

Is it possible that this is significantly slower with Python 3 (or 3.7) than it is in Python 2?

rstewart: can you mark this as possibly regressed by some Python 3 conversion ticket? Thanks!

Flags: needinfo?(rstewart)
Product: Core → Firefox Build System

I'll look into it in a bit. (For what it's worth the transition from Py2 to Py3 to this script happened 2 months ago, and I don't know if that tracks with your experience or if it's unrelated.)

There isn't a meaningful difference between Py2 and Py3 here, this script just takes an extremely long time to run for some reason.

rickystewart-a5lvdq:mozilla-unified rickystewart$ rm obj-x86_64-apple-darwin19.2.0/security/manager/ssl/.deps/nsSTSPreloadList.h.*
rickystewart-a5lvdq:mozilla-unified rickystewart$ time make -C obj-x86_64-apple-darwin19.2.0/security/manager/ssl/ nsSTSPreloadList.h
security/manager/ssl/nsSTSPreloadList.h.stub
/Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init_py3/bin/python -m mozbuild.action.file_generate /Users/rickystewart/src/mozilla-unified/xpcom/ds/tools/make_dafsa.py main nsSTSPreloadList.h .deps/nsSTSPreloadList.h.pp .deps/nsSTSPreloadList.h.stub /Users/rickystewart/src/mozilla-unified/security/manager/ssl/nsSTSPreloadList.inc

real	0m27.952s
user	0m27.473s
sys	0m0.404s
rickystewart-a5lvdq:mozilla-unified rickystewart$ emacs security/manager/ssl/moz.build   # Use py2 instead
rickystewart-a5lvdq:mozilla-unified rickystewart$ ./mach build-backend
 0:00.72 /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init_py3/bin/python /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/config.status
Reticulating splines...
 0:01.49 File already read. Skipping: /Users/rickystewart/src/mozilla-unified/gfx/angle/targets/angle_common/moz.build
Finished reading 1778 moz.build files in 2.68s
Read 61 gyp files in parallel contributing 1.23s to total wall time
Processed into 10905 build config descriptors in 2.86s
RecursiveMake backend executed in 4.31s
  3858 total backend files; 0 created; 4 updated; 3854 unchanged; 0 deleted; 30 -> 1421 Makefile
FasterMake backend executed in 0.30s
  11 total backend files; 0 created; 2 updated; 9 unchanged; 0 deleted
Total wall time: 11.71s; CPU time: 9.48s; Efficiency: 81%; Untracked: 1.55s
rickystewart-a5lvdq:mozilla-unified rickystewart$ rm obj-x86_64-apple-darwin19.2.0/security/manager/ssl/.deps/nsSTSPreloadList.h.*
rickystewart-a5lvdq:mozilla-unified rickystewart$ time make -C obj-x86_64-apple-darwin19.2.0/security/manager/ssl/ nsSTSPreloadList.h
security/manager/ssl/nsSTSPreloadList.h.stub
/Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init/bin/python -m mozbuild.action.file_generate /Users/rickystewart/src/mozilla-unified/xpcom/ds/tools/make_dafsa.py main nsSTSPreloadList.h .deps/nsSTSPreloadList.h.pp .deps/nsSTSPreloadList.h.stub /Users/rickystewart/src/mozilla-unified/security/manager/ssl/nsSTSPreloadList.inc

real	0m26.835s
user	0m26.221s
sys	0m0.535s
rickystewart-a5lvdq:mozilla-unified rickystewart$
Flags: needinfo?(rstewart)
Summary: `make_dafsa.py` under Python 3.7 appears to be slow → `make_dafsa.py` is slow
Priority: -- → P3
Assignee: nobody → mhentges
Status: NEW → ASSIGNED
Blocks: 1646939
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1720818b9015
Implements incremental dafsa algorithm r=rstewart
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla80
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: