1503923 - Experiment to measure the page load impact of assets

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: Performance, defect, P2)

Description

[John Whitlock [:jwhitlock]]

Browser-based testing shows that blocking our mdn.css and wiki.css has a 50% effect on load time (25%) each, but that layout is still 80% of time spent.

Ship some experimental pages that will allow synthetic testing of different scenarios, to determine where to focus the next performance changes:

* Normal page, but served as static HTML (normal.html)
* CSS is replaced by inline, minimized CSS (unused-css.html)
* Remove inline JS (no-inline-js.html)
* Remove inline and external JS (no-js.html)
* Remove CSS, inline, and external JS (raw-html.html)

Robots should be discouraged from indexing these pages.

This work is already done:

* https://github.com/mdn/sprints/issues/527
* https://github.com/mozilla/kuma/pull/5062

Comment 1

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/0c93bff262799815ed2d820137758e2ed221b4f0
bug 1503923: Static pages for testing layout perf (#5062)

* Static pages for testing layout perf
* No JS test added
* Tweaking no-inline test
* Make sure SEO robots don't index

Comment 2

[John Whitlock [:jwhitlock]]

Pages are deployed:

* https://developer.mozilla.org/static/layout-tests/normal.html - static version of [forEach](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach)
* https://developer.mozilla.org/static/layout-tests/unused-css.html - External CSS replaced with inline, minimized styles
* https://developer.mozilla.org/static/layout-tests/no-inline-js.html - Inline JS removed
* https://developer.mozilla.org/static/layout-tests/no-js.html - Inline and external JS removed
* https://developer.mozilla.org/static/layout-tests/raw-html.html - CSS, inline JS, and external JS removed

Comment 3

[Tim Kadlec]

There are some path issues on the unused CSS page that I missed. I'll open another PR on Kuma to resolve.

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/6a281b3584fa13a461c19ac39df4867ee44b0562
Merge pull request #5070 from tkadlec/fix/paths-in-unused-css

bug 1503923: Updating paths for unused CSS test

Comment 5

[John Whitlock [:jwhitlock]]

New page is in production, unfortunately there are still 404s.

Comment 6

[Tim Kadlec]

Hrm. The page source is identical to the older version. Could we just be seeing some sort of cache issue?

Comment 7

[John Whitlock [:jwhitlock]]

Yes, there's a potential caching issue. Everything in static is served with a long cache timeout of 10 years:

cache-control: max-age=315360000, public, immutable

If you request the file with the same name, then there's a chance the CDN will serve the older version. f you request the hashed version, you'll get the latest:

https://developer.mozilla.org/static/layout-tests/unused-css.11e8b28b3682.html

To figure out the hashed version, it is easiest to go to the docker image. Find the latest tagged kuma image:

https://hub.docker.com/r/mdnwebdocs/kuma/tags/

At the time I wrote this, it is "630203c". You can also get the current shipped revision (first 7 characters are the docker tag) with:

https://developer.mozilla.org/media/revision.txt

Then you can run this to see what the hashed name is:

docker run -it mdnwebdocs/kuma:6d0203c ls /app/static/layout-tests/

Alternatively, we can rename the unused-css.html so that we get an unpolluted filename.

Comment 8

[John Whitlock [:jwhitlock]]

Or, I can invalidate the CDN cache for that path. I've started an invalidation, the CDNs should start serving the new version soon.

Comment 9

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/8af75f1408c2e504f93f5a70d152969d2c72f0dc
bug 1503923: Add static test of inlined example plus unused css (#5098)

Comment 10

[John Whitlock [:jwhitlock]]

De-framed example is now at https://developer.mozilla.org/static/layout-tests/iex-unused-css.html

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/2590b5e4f59b6190949f24ab74e49154cb2a6d2a
Additional page variations for preformance experiments (#5132)

bug 1503923: Additional page variations for performance experiments

Comment 12

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/5046a2d87f5aa42dbf0eab816780f6c1fcc44962
Bug 1503923, additional page variations for performance experiment (#5144)

* Bug 1503923, additional page variations for performance experiment

* Changes based on review feedback

Comment 13

[Kadir Topal [:atopal]]

This is late, but here we go: https://docs.google.com/spreadsheets/d/1yYyU54sI3bAJG4aICOkM2L23y4c2Mkq6gU3Xu29U8iY/edit#gid=0

Short summary:

• Removing unused CSS is significantly better for time to interactive (-50%) and start render (-30%)
• Removing unused CSS and inlining interactive examples is significantly better at Speedindex (-50%), start render (-50%) and time to interactive example (-50%)
• removing the BCD, contributors list and side bar is significantly better at Speedindex (-40%) and start render (-40%)

From these results, it still seems worthwhile to shoot for inlining of interactive examples, but even just removing unused CSS should have a large impact.

I'd like to run one more experiment to determine what was responsible for the large drop in start render time when we removed BCD, the contributors list and the side bar.

Comment 14

[BMO Automation]

MDN Web Docs' bug reporting has now moved to GitHub. From now on, please file content bugs at https://github.com/mdn/sprints/issues/ and platform bugs at https://github.com/mdn/kuma/issues/.