Closed Bug 1062668 Opened 10 years ago Closed 5 years ago

no-op mach build binaries takes around 3 seconds

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bzbarsky, Assigned: glandium)

Details

Attachments

(2 files, 1 obsolete file)

Only process the "include" install manifest when invoking the binaries target
3.67 KB, patch
gps
: review+
bzbarsky
: feedback+
Details | Diff | Splinter Review
Add a timestamp to the message mach prints to indicate the end of the build
1.35 KB, patch
gps
: review+
Details | Diff | Splinter Review 
Recent mbp with an SSD.

Timestamped log:

 0:00.22 /usr/bin/make -C /Users/bzbarsky/mozilla/vanilla/obj-firefox -j8 -s backend.RecursiveMakeBackend
 0:00.40 /usr/bin/make -j8 -s binaries
 0:00.76 From dist/sdk: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.77 From dist/public: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.78 From dist/private: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.82 From dist/idl: Kept 1101 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.88 From dist/bin: Kept 94 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.96 From dist/include: Kept 4218 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:02.31 From _tests: Kept 19508 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:02.43 make[1]: Circular memory/replace/dummy/target <- mozglue/build/target dependency dropped.
 0:02.96 make[3]: warning: -jN forced in submake: disabling jobserver mode.
 0:02.96 Note: rebuild with "/Applications/Xcode.app/Contents/Developer/usr/bin/make VERBOSE=1 " to show all compiler parameters.
 0:02.97 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `stubdata'
 0:02.98 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `common'
 0:03.03 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `i18n'
 0:03.09 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `tools'
 0:03.10 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `toolutil'
 0:03.11 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `ctestfw'
 0:03.12 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `makeconv'
 0:03.13 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `genrb'
 0:03.13 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `genbrk'
 0:03.14 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gencnval'
 0:03.14 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gensprep'
 0:03.15 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `icuinfo'
 0:03.16 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `genccode'
 0:03.16 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gencmn'
 0:03.17 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `icupkg'
 0:03.17 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `pkgdata'
 0:03.18 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gentest'
 0:03.19 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gennorm2'
 0:03.19 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gencfu'
 0:03.20 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gendict'
 0:03.21 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `data'
 0:03.46 Note: rebuild with "/Applications/Xcode.app/Contents/Developer/usr/bin/make VERBOSE=1 all-local" to show all compiler parameters.
 0:03.99 cd include; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:04.01 cd src; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:04.01 mkdir: /Users/bzbarsky/mozilla/vanilla/obj-firefox/security/nss/lib/freebl: File exists
 0:04.02 cd builtins; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:04.04 cd mangle; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:04.06 cd legacydb; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs

Note that for that run "time" claimed:

  12.514u 3.364s 0:04.63 342.7%   0+0k 0+97io 0pf+0w

so there's another 0.6 seconds hiding somewhere outside of the timestamps.
(In reply to Boris Zbarsky [:bz] from comment #0)
> Recent mbp with an SSD.
> 
> Timestamped log:
> 
>  0:00.22 /usr/bin/make -C /Users/bzbarsky/mozilla/vanilla/obj-firefox -j8 -s
> backend.RecursiveMakeBackend
>  0:00.40 /usr/bin/make -j8 -s binaries
>  0:00.76 From dist/sdk: Kept 0 existing; Added/updated 0; Removed 0 files
> and 0 directories.
>  0:00.77 From dist/public: Kept 0 existing; Added/updated 0; Removed 0 files
> and 0 directories.
>  0:00.78 From dist/private: Kept 0 existing; Added/updated 0; Removed 0
> files and 0 directories.
>  0:00.82 From dist/idl: Kept 1101 existing; Added/updated 0; Removed 0 files
> and 0 directories.
>  0:00.88 From dist/bin: Kept 94 existing; Added/updated 0; Removed 0 files
> and 0 directories.
>  0:00.96 From dist/include: Kept 4218 existing; Added/updated 0; Removed 0
> files and 0 directories.
>  0:02.31 From _tests: Kept 19508 existing; Added/updated 0; Removed 0 files
> and 0 directories.

We should at the very least skip _tests during mach build binaries.

>  0:02.43 make[1]: Circular memory/replace/dummy/target <-
> mozglue/build/target dependency dropped.
>  0:02.96 make[3]: warning: -jN forced in submake: disabling jobserver mode.
>  0:02.96 Note: rebuild with
> "/Applications/Xcode.app/Contents/Developer/usr/bin/make VERBOSE=1 " to show
> all compiler parameters.
>  0:02.97 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making
> `all' in `stubdata'
>  0:02.98 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making
> `all' in `common'
>  0:03.03 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making
> `all' in `i18n'
>  0:03.09 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making
> `all' in `tools'
>  0:03.10 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `toolutil'
>  0:03.11 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `ctestfw'
>  0:03.12 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `makeconv'
>  0:03.13 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `genrb'
>  0:03.13 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `genbrk'
>  0:03.14 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `gencnval'
>  0:03.14 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `gensprep'
>  0:03.15 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `icuinfo'
>  0:03.16 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `genccode'
>  0:03.16 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `gencmn'
>  0:03.17 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `icupkg'
>  0:03.17 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `pkgdata'
>  0:03.18 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `gentest'
>  0:03.19 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `gennorm2'
>  0:03.19 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `gencfu'
>  0:03.20 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making
> `all' in `gendict'
>  0:03.21 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making
> `all' in `data'
>  0:03.46 Note: rebuild with
> "/Applications/Xcode.app/Contents/Developer/usr/bin/make VERBOSE=1
> all-local" to show all compiler parameters.
>  0:03.99 cd include; /Applications/Xcode.app/Contents/Developer/usr/bin/make
> libs
>  0:04.01 cd src; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
>  0:04.01 mkdir:
> /Users/bzbarsky/mozilla/vanilla/obj-firefox/security/nss/lib/freebl: File
> exists
>  0:04.02 cd builtins;
> /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
>  0:04.04 cd mangle; /Applications/Xcode.app/Contents/Developer/usr/bin/make
> libs
>  0:04.06 cd legacydb;
> /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
>
> Note that for that run "time" claimed:
> 
>   12.514u 3.364s 0:04.63 342.7%   0+0k 0+97io 0pf+0w
> 
> so there's another 0.6 seconds hiding somewhere outside of the timestamps.

That probably comes from the fact that there is no "build is finished" message when doing mach build binaries.

So, part of this is _tests, as mentioned above, another part is a conscious regression from bug 1043862. Eventually, we'll come back to using the same trick as before. And yet another part is from the fact that mach build binaries now also recurses icu and nss, which it didn't do before.

The first is actionable now.
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Comment on attachment 8483927 [details] [diff] [review]
Only process the "include" install manifest when invoking the binaries target

Boris, can you paste a new timestamped log with this patch applied?
Attachment #8483927 - Flags: feedback?(bzbarsky)
Comment on attachment 8483927 [details] [diff] [review]
Only process the "include" install manifest when invoking the binaries target

Mmmm in fact forget it, that doesn't remove _tests.
Attachment #8483927 - Flags: review?(gps)
Attachment #8483927 - Flags: feedback?(bzbarsky)
> That probably comes from the fact that there is no "build is finished" message
> when doing mach build binaries.

Actually, there is, but it's not timestamped.
Attachment #8483927 - Attachment is obsolete: true
Attachment #8483930 - Flags: feedback?(bzbarsky)
Comment on attachment 8483930 [details] [diff] [review]
Only process the "include" install manifest when invoking the binaries target

With this patch:

 0:00.21 /usr/bin/make -C /Users/bzbarsky/mozilla/vanilla/obj-firefox -j8 -s backend.RecursiveMakeBackend
 0:00.40 /usr/bin/make -j8 -s binaries
 0:00.87 From dist/include: Kept 4218 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.99 make[1]: Circular memory/replace/dummy/target <- mozglue/build/target dependency dropped.
 0:01.51 make[3]: warning: -jN forced in submake: disabling jobserver mode.
 0:01.52 Note: rebuild with "/Applications/Xcode.app/Contents/Developer/usr/bin/make VERBOSE=1 " to show all compiler parameters.
 0:01.53 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `stubdata'
 0:01.53 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `common'
 0:01.59 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `i18n'
 0:01.65 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `tools'
 0:01.66 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `toolutil'
 0:01.67 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `ctestfw'
 0:01.68 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `makeconv'
 0:01.68 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `genrb'
 0:01.69 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `genbrk'
 0:01.70 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gencnval'
 0:01.71 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gensprep'
 0:01.71 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `icuinfo'
 0:01.72 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `genccode'
 0:01.72 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gencmn'
 0:01.73 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `icupkg'
 0:01.73 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `pkgdata'
 0:01.74 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gentest'
 0:01.74 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gennorm2'
 0:01.75 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gencfu'
 0:01.76 /Applications/Xcode.app/Contents/Developer/usr/bin/make[4]: Making `all' in `gendict'
 0:01.77 /Applications/Xcode.app/Contents/Developer/usr/bin/make[3]: Making `all' in `data'
 0:01.99 Note: rebuild with "/Applications/Xcode.app/Contents/Developer/usr/bin/make VERBOSE=1 all-local" to show all compiler parameters.
 0:02.50 cd include; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:02.52 mkdir: /Users/bzbarsky/mozilla/vanilla/obj-firefox/security/nss/lib/freebl: File exists
 0:02.52 cd src; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:02.52 cd builtins; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:02.53 cd mangle; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
 0:02.57 cd legacydb; /Applications/Xcode.app/Contents/Developer/usr/bin/make libs
Your build was successful!
10.530u 2.892s 0:03.13 428.7%   0+0k 11+1286io 0pf+0w

which is a good bit better!
Attachment #8483930 - Flags: feedback?(bzbarsky) → feedback+
Attachment #8483936 - Flags: review?(gps) → review+
Attachment #8483930 - Flags: review?(gps) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/7dac4acfe611
https://hg.mozilla.org/integration/mozilla-inbound/rev/4ec0be60b1cc
Keywords: leave-open
Summary: no-op mach build binaries takes around 5 seconds → no-op mach build binaries takes around 3 seconds
Product: Core → Firefox Build System
The leave-open keyword is there and there is no activity for 6 months.
:glandium, maybe it's time to close this bug?
Flags: needinfo?(mh+mozilla)
What do you think, Greg?
Flags: needinfo?(mh+mozilla) → needinfo?(gps)

Quite a few improvements and changes have landed since the patches in this bug, we can track new work in new bugs.

Flags: needinfo?(gps)
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: