1627075 - Cache more things from omnijars with the StartupCache

Status: RESOLVED FIXED

(Core :: XPCOM, enhancement)

Description

[Alex Thayer [:alexical] (she/her)]

During a startup where the StartupCache has been populated, we still read about 2MB (Edit: 4MB) jumping across the entire space of our two omnijars. If we cached more in the StartupCache by allowing caching of things loaded in background threads and child processes, we could reduce this to nearly 0.

Comment 1

[Mike Hommey [:glandium]]

Please note that we reorder omnijars during the build and preload parts of them on startup, but we only do so on PGO builds, so you won't see it on local builds.

Comment 2

[Alex Thayer [:alexical] (she/her)]

Yes, I've been doing all my testing on PGO builds for that reason. (Though we generally read far more during startup than is actually included in that section.)

Comment 3

[Alex Thayer [:alexical] (she/her)]

Also until bug 1627040 lands, we actually read the entire combined 75MB of both omnijars during startup.

Comment 4

[Mike Hommey [:glandium]]

(In reply to Doug Thayer [:dthayer] from comment #2)

(Though we generally read far more during startup than is actually included in that section.)

If that's the case, we should fix that.

Comment 5

[:Gijs (he/him)]

Tentatively marking as p2 for now.

Comment 6

[Alex Thayer [:alexical] (she/her)]

Attachment: Bug 1627075 - Build Omnijar file list from startup cache r?froydnj

Comment 7

[Alex Thayer [:alexical] (she/her)]

Attachment: Bug 1627075 - Init StartupCache before Omnijar r?froydnj

Depends on D77632

Comment 8

[Alex Thayer [:alexical] (she/her)]

Attachment: Bug 1627075 - Route Omnijar requests through StartupCache r?froydnj

Depends on D77633

Comment 9

[Alex Thayer [:alexical] (she/her)]

Attachment: Bug 1627075 - OMT and OMP StartupCache access r?froydnj

Depends on D77634

Comment 10

[Alex Thayer [:alexical] (she/her)]

Attachment: Bug 1627075 - Allow lazily initializing nsZipArchives r?froydnj

Opening our Omnijars can be expensive, and it should be deferrable until after
startup is completed, provided we have a startup cache. In a previous patch in this
stack, we implemented caching of the zip central directory for omnijars, but we
still have to open the file in order to hand the object off to various omnijar
consumers. In a later patch, we will wrap nsZipArchive access in a class which
will allow us to transparently cache nsZipArchive results. These two get us
most of the way to not needing to read from the underlying omnijar files during
startup, but there are still nontrivial pieces, like nsZipFind for instance,
which we don't want to just duplicate inside of a wrapper class, so we would
like to sort out a way in which we can use an nsZipArchive class, but not
actually back it up with the real underlying file until we really need data
from it which we can't find in a cache.

Depends on D77633

Comment 11

[Alex Thayer [:alexical] (she/her)]

Attachment: Bug 1627075 - Include WebGLParent.h in WebGLChild.h r?handyman

Depends on D77635

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/763a5a7b43a0
Build Omnijar file list from startup cache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/0f0d1bd2a8ac
Init StartupCache before Omnijar r=froydnj
https://hg.mozilla.org/integration/autoland/rev/6cd31f17a931
Allow lazily initializing nsZipArchives r=froydnj
https://hg.mozilla.org/integration/autoland/rev/65c0e6790a33
OMT and OMP StartupCache access r=froydnj
https://hg.mozilla.org/integration/autoland/rev/b07887474f51
Route Omnijar requests through StartupCache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/cf23b456ba12
Include WebGLParent.h in WebGLChild.h r=handyman,jgilbert

Comment 13

[Alexandru Michis [:malexandru]]

Backed out 6 changesets for causing failures regarding startupcache.

Backout link: https://hg.mozilla.org/integration/autoland/rev/e1c28fef32cdd9b6717e4faa186139d558adeb95

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&searchStr=bc&fromchange=597ca7dea6877156e934d52668d4ac6e60bb0826&tochange=a3e2109a3207c86113e5637b4f694a28a54a3c10&selectedTaskRun=Jz2MLbtURtOcropVPzH9Gw.0

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308229242&repo=autoland&lineNumber=47252

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b6029c7c0016
Build Omnijar file list from startup cache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/f5106898239f
Init StartupCache before Omnijar r=froydnj
https://hg.mozilla.org/integration/autoland/rev/6c4eedaa0d04
Allow lazily initializing nsZipArchives r=froydnj
https://hg.mozilla.org/integration/autoland/rev/ece9a4223349
OMT and OMP StartupCache access r=froydnj
https://hg.mozilla.org/integration/autoland/rev/7ae4df10749c
Route Omnijar requests through StartupCache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/699212a443c3
Include WebGLParent.h in WebGLChild.h r=handyman,jgilbert

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9705b2759d45
Remove unused 'start' variable from startupcache/StartupCache.cpp. a=Bb-fix CLOSED TREE

Comment 16

[Noemi Erli[:noemi_erli]]

Backed out 7 changesets (Bug 1627075) for causing @nsZipArchive crashes CLOSED TREE

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=699212a443c35f37bc998f21cad29cb9fc2e8c95

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308295093&repo=autoland&lineNumber=666

Backout: https://hg.mozilla.org/integration/autoland/rev/4c73ffd9cf3141bb98f321b059465e32a9d52834

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ebdcd96a9d20
Build Omnijar file list from startup cache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/920398d1c3d3
Init StartupCache before Omnijar r=froydnj
https://hg.mozilla.org/integration/autoland/rev/2cb021a493d8
Allow lazily initializing nsZipArchives r=froydnj
https://hg.mozilla.org/integration/autoland/rev/288a67aed661
OMT and OMP StartupCache access r=froydnj
https://hg.mozilla.org/integration/autoland/rev/a345e05df151
Route Omnijar requests through StartupCache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/fc144caf5d06
Include WebGLParent.h in WebGLChild.h r=handyman,jgilbert

Comment 18

[Alexandru Michis [:malexandru]]

Backed out 6 changesets for causing bustages in StartupCache.cpp

Backout link: https://hg.mozilla.org/integration/autoland/rev/232921af2e09cccce37ca4378c9d18068551bb5c

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=IQmO8phoR4yr6Gh0iY5hhA.0&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=fc144caf5d065041df3359c5cb1d7d9e997c93b8

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308793702&repo=autoland&lineNumber=29203

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/333d035afe92
Build Omnijar file list from startup cache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/13ecd68b3c0d
Init StartupCache before Omnijar r=froydnj
https://hg.mozilla.org/integration/autoland/rev/dfd71f4ecb81
Allow lazily initializing nsZipArchives r=froydnj
https://hg.mozilla.org/integration/autoland/rev/eb5265addd5e
OMT and OMP StartupCache access r=froydnj
https://hg.mozilla.org/integration/autoland/rev/e29b15980da2
Route Omnijar requests through StartupCache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/21605186687e
Include WebGLParent.h in WebGLChild.h r=handyman,jgilbert

Comment 20

[Narcis Beleuzu [:NarcisB]]

Backed out for bustages on startupcache/StartupCache.cpp

Backout link: https://hg.mozilla.org/integration/autoland/rev/560ca8fba0a1591b00730db8e6252379bf9ab38d
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308871117&repo=autoland&lineNumber=23150

Comment 21

[Alex Thayer [:alexical] (she/her)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=47e403d91b24d252906ba3d0a8dab9a78462f0e4

Nth time's the charm...

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4208ed5fc3b0
Build Omnijar file list from startup cache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/1d838c1c4669
Init StartupCache before Omnijar r=froydnj
https://hg.mozilla.org/integration/autoland/rev/d431b24bebf6
Allow lazily initializing nsZipArchives r=froydnj
https://hg.mozilla.org/integration/autoland/rev/d2e40366118c
OMT and OMP StartupCache access r=froydnj
https://hg.mozilla.org/integration/autoland/rev/aca7cd850cc9
Route Omnijar requests through StartupCache r=froydnj
https://hg.mozilla.org/integration/autoland/rev/4d3569a5cd9f
Include WebGLParent.h in WebGLChild.h r=handyman,jgilbert

Comment 23

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/4208ed5fc3b0
https://hg.mozilla.org/mozilla-central/rev/1d838c1c4669
https://hg.mozilla.org/mozilla-central/rev/d431b24bebf6
https://hg.mozilla.org/mozilla-central/rev/d2e40366118c
https://hg.mozilla.org/mozilla-central/rev/aca7cd850cc9
https://hg.mozilla.org/mozilla-central/rev/4d3569a5cd9f