stylo: Stylo is slower than Gecko on etc/ci/performance TP5 tests

RESOLVED WORKSFORME

Status

()

Core
CSS Parsing and Computation
P1
normal
RESOLVED WORKSFORME
7 months ago
16 days ago

People

(Reporter: shinglyu, Assigned: bholley)

Tracking

(Depends on: 1 bug, Blocks: 3 bugs)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

7 months ago
I ran the performance test in Servo's etc/ci/performance on TP5 test suite. Here are the result for comparing Stylo (release build) with Gecko 52

=================
 595.0  774.0 +179.0  +30.08%   http://localhost:8000/page_load_test/tp5n/globo.com/www.globo.com/index.html.
 360.0  792.0 +432.0 +120.00%   http://localhost:8000/page_load_test/tp5n/56.com/www.56.com/index.html.
 287.0  524.0 +237.0  +82.58%   http://localhost:8000/page_load_test/tp5n/rakuten.co.jp/www.rakuten.co.jp/index.html.
 135.0  228.0  +93.0  +68.89%   http://localhost:8000/page_load_test/tp5n/chemistry.about.com/chemistry.about.com/index.html.
 293.0  397.0 +104.0  +35.49%   http://localhost:8000/page_load_test/tp5n/huffingtonpost.com/www.huffingtonpost.com/index.html.
 203.0  275.0  +72.0  +35.47%   http://localhost:8000/page_load_test/tp5n/imgur.com/imgur.com/gallery/index.html.
 135.0  212.0  +77.0  +57.04%   http://localhost:8000/page_load_test/tp5n/ezinearticles.com/ezinearticles.com/index.html@Migraine-Ocular---The-Eye-Migraines&id=4684133.html.
 259.0  661.0 +402.0 +155.21%   http://localhost:8000/page_load_test/tp5n/myspace.com/www.myspace.com/albumart.html.
 160.0  310.0 +150.0  +93.75%   http://localhost:8000/page_load_test/tp5n/mail.ru/mail.ru/index.html.
 230.0  345.0 +115.0  +50.00%   http://localhost:8000/page_load_test/tp5n/tudou.com/www.tudou.com/index.html.
  91.0  198.0 +107.0 +117.58%   http://localhost:8000/page_load_test/tp5n/goo.ne.jp/goo.ne.jp/index.html.
 198.0  353.0 +155.0  +78.28%   http://localhost:8000/page_load_test/tp5n/bbc.co.uk/www.bbc.co.uk/news/index.html.
 381.0  564.0 +183.0  +48.03%   http://localhost:8000/page_load_test/tp5n/youku.com/www.youku.com/index.html.
 104.0  150.0  +46.0  +44.23%   http://localhost:8000/page_load_test/tp5n/w3.org/www.w3.org/standards/webdesign/htmlcss.html.
 136.0  218.0  +82.0  +60.29%   http://localhost:8000/page_load_test/tp5n/beatonna.livejournal.com/beatonna.livejournal.com/index.html.
 170.0  362.0 +192.0 +112.94%   http://localhost:8000/page_load_test/tp5n/dailymotion.com/www.dailymotion.com/us.html.
 190.0  307.0 +117.0  +61.58%   http://localhost:8000/page_load_test/tp5n/cgi.ebay.com/cgi.ebay.com/ALL-NEW-KINDLE-3-eBOOK-WIRELESS-READING-DEVICE-W-WIFI-/130496077314@pt=LH_DefaultDomain_0&hash=item1e622c1e02.html.
 172.0  281.0 +109.0  +63.37%   http://localhost:8000/page_load_test/tp5n/reddit.com/www.reddit.com/index.html.
 231.0  362.0 +131.0  +56.71%   http://localhost:8000/page_load_test/tp5n/stackoverflow.com/stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered.html.
 455.0  599.0 +144.0  +31.65%   http://localhost:8000/page_load_test/tp5n/dailymail.co.uk/www.dailymail.co.uk/ushome/index.html.
 249.0  373.0 +124.0  +49.80%   http://localhost:8000/page_load_test/tp5n/yelp.com/www.yelp.com/biz/alexanders-steakhouse-cupertino.html.
 143.0  280.0 +137.0  +95.80%   http://localhost:8000/page_load_test/tp5n/youtube.com/www.youtube.com/music.html.
 407.0  647.0 +240.0  +58.97%   http://localhost:8000/page_load_test/tp5n/amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/507846.html.
  56.0   75.0  +19.0  +33.93%   http://localhost:8000/page_load_test/tp5n/google.com/www.google.com/search@q=mozilla.html.
 214.0  444.0 +230.0 +107.48%   http://localhost:8000/page_load_test/tp5n/spiegel.de/www.spiegel.de/index.html.
 173.0  283.0 +110.0  +63.58%   http://localhost:8000/page_load_test/tp5n/page.renren.com/page.renren.com/index.html.
======================

The columns means (from left to right)
* Gecko page load time
* Stylo page load time
* (Stylo - Gecko) page load time difference
* page load time difference in percentage
* Test case

All the page load time for Stylo is slower then Gecko, ranging from 30% to 240% slower.

You can find instructions on how to run this yourself here: https://github.com/servo/servo/tree/master/etc/ci/performance
(Reporter)

Comment 1

7 months ago
cc Chris
Flags: needinfo?(cpeterson)
Yeah, this wouldn't surprise me a lot right now. We'd be faster in a full restyle I think, but I suspect these results may mean one of two things:

 * That our incremental restyle isn't quite there yet (and that's known).
 * That our CSS parser is slower (see https://github.com/servo/servo/issues/13778) for an example.

CCing Bobby and Simon for that.

Shing, could you run an instrumented build against these test cases to see how many elements are we restyling? That'd mean adding a counter in recalc_style_at in Servo, and in ElementRestyler::Restyle in Gecko, and check the difference.

It'd be nice to know if the page load time drop from stylo comes from CSS parsing, or restyling (or both).
Blocks: 1289864
No longer blocks: 1243581
Shing, great work getting Talos running with Stylo!

Blocking the stylo-nightly milestone bug 1330412 because we don't want to enable Stylo in Nightly until it is faster than Gecko.
Blocks: 1330412
Flags: needinfo?(cpeterson)
Summary: [Stylo] Stylo is slow → [Stylo] Stylo is slower than Gecko on tp5 Talos tests
Depends on: 1331843
Depends on: 1331848
(Reporter)

Comment 4

7 months ago
Emilio: Is the instrumented build code available? Or are you suggesting me writing one myself?

Chris: It's not Talos yet, I'm using Servo's performance test, but running the same test case as Talos.
(Reporter)

Updated

7 months ago
Summary: [Stylo] Stylo is slower than Gecko on tp5 Talos tests → [Stylo] Stylo is slower than Gecko on etc/ci/performance Talos tests
(Reporter)

Updated

7 months ago
Summary: [Stylo] Stylo is slower than Gecko on etc/ci/performance Talos tests → [Stylo] Stylo is slower than Gecko on etc/ci/performance TP5 tests
Depends on: 1331856
(In reply to Shing Lyu [:shinglyu] from comment #4)
> Emilio: Is the instrumented build code available? Or are you suggesting me
> writing one myself?

I think he may have been suggesting the latter, but it might be a bit involved. I filed bug 1331856 to do this.
Assignee: nobody → bobbyholley
Priority: -- → P1
Summary: [Stylo] Stylo is slower than Gecko on etc/ci/performance TP5 tests → stylo: Stylo is slower than Gecko on etc/ci/performance TP5 tests
Blocks: 1243581
(Reporter)

Comment 6

4 months ago
Some new results, to summarize: 

1. Stylo (parallel, 6 threads) is slightly slow than Gecko in general, but we do have some big wins like amazon.com & hoffintonpost.com. On the other hand, some sites are extremely slow Stylo like myspace and bild.de

2. Parallel stylo is not faster than sequential stylo. (see below.) The difference is around -5% to 5%, centered around 0%. This might indicate a big risk in our claim that parallelism is useful for the style system. 
 

Gecko vs Stylo (parallel)

test                     gecko     stylo	difference
----------------------------------------------------------------
tudou.com                315.36    402.52	-27.64%
youtube.com              267.44    308.92	-15.51%
sohu.com                 462.68    506.72	-9.52%
store.apple.com          354.28    509.2	-43.73%
people.com.cn            380.96    410.08	-7.64%
rakuten.co.jp            440.92    539.76	-22.42%
myspace.com              447.4     700.92	-56.67%
stackoverflow.com        403.92    475.68	-17.77%
ifeng.com                494.32    486.68	1.55%
cgi.ebay.com             301.12    329.44	-9.40%
ezinearticles.com        209.64    214.6	-2.37%
thepiratebay.org         298.8     323.32	-8.21%
media.photobucket.com    221.64    245.76	-10.88%
digg.com                 367.64    421.28	-14.59%
huffingtonpost.com       586.4     505.36	13.82%
noimpactman.typepad.com  328.8     319.2	2.92%
163.com                  609.32    673.36	-10.51%
dailymotion.com          304.84    385.84	-26.57%
chinaz.com               278.16    305.16	-9.71%
w3.org                   179.6     192.52	-7.19%
chemistry.about.com      183.52    205.64	-12.05%
mashable.com             575.52    649.84	-12.91%
aljazeera.net            462.64    465.44	-0.61%
youku.com                475.4     506.36	-6.51%
reuters.com              444.76    477.36	-7.33%
reddit.com               286.68    311.88	-8.79%
dailymail.co.uk          579.56    592.48	-2.23%
uol.com.br               455.08    486.32	-6.86%
naver.com                414.52    462.16	-11.49%
page.renren.com          251.64    267.76	-6.41%
xinhuanet.com            826.96    966.04	-16.82%
cnn.com                  432.28    495.96	-14.73%
amazon.com               813.76    652.44	19.82%
slideshare.net           321.88    373.72	-16.11%
56.com                   601.2     680.52	-13.19%
imgur.com                254.4     268.12	-5.39%
wsj.com                  496.04    601.92	-21.35%
bbc.co.uk                312.96    290.28	7.25%
guardian.co.uk           365.96    399.12	-9.06%
beatonna.livejournal.com 158.24    163.52	-3.34%
globo.com                492.56    678.64	-37.78%
google.com               129.92    136.0	-4.68%
xunlei.com               773.4     860.76	-11.30%
mail.ru                  349.24    422.04	-20.85%
homeway.com.cn           436.16    517.52	-18.65%
bild.de                  1042.68   1470.52	-41.03%
spiegel.de               397.92    431.92	-8.54%
imdb.com                 413.68    439.2	-6.17%
goo.ne.jp                156.88    178.56	-13.82%
indiatimes.com           433.2     466.36	-7.65%
yelp.com                 501.36    522.6	-4.24%


Sequential Stylo vs Parallel Stylo (6 threads)

test                     sequential parallel	difference
----------------------------------------------------------------------
tudou.com                424.6     402.52	5.20%
youtube.com              320.28    308.92	3.55%
sohu.com                 514.12    506.72	1.44%
store.apple.com          510.28    509.2	0.21%
people.com.cn            408.52    410.08	-0.38%
rakuten.co.jp            511.32    539.76	-5.56%
myspace.com              673.28    700.92	-4.11%
stackoverflow.com        500.6     475.68	4.98%
ifeng.com                496.2     486.68	1.92%
cgi.ebay.com             318.36    329.44	-3.48%
ezinearticles.com        204.44    214.6	-4.97%
thepiratebay.org         339.24    323.32	4.69%
media.photobucket.com    226.2     245.76	-8.65%
digg.com                 391.04    421.28	-7.73%
huffingtonpost.com       531.44    505.36	4.91%
noimpactman.typepad.com  306.84    319.2	-4.03%
163.com                  636.52    673.36	-5.79%
dailymotion.com          378.4     385.84	-1.97%
chinaz.com               293.24    305.16	-4.06%
w3.org                   175.96    192.52	-9.41%
chemistry.about.com      195.32    205.64	-5.28%
mashable.com             634.48    649.84	-2.42%
aljazeera.net            452.84    465.44	-2.78%
youku.com                537.72    506.36	5.83%
reuters.com              461.76    477.36	-3.38%
reddit.com               312.36    311.88	0.15%
dailymail.co.uk          639.04    592.48	7.29%
uol.com.br               474.48    486.32	-2.50%
naver.com                439.12    462.16	-5.25%
page.renren.com          261.28    267.76	-2.48%
xinhuanet.com            940.16    966.04	-2.75%
cnn.com                  483.56    495.96	-2.56%
amazon.com               677.68    652.44	3.72%
slideshare.net           356.6     373.72	-4.80%
56.com                   665.84    680.52	-2.20%
imgur.com                251.76    268.12	-6.50%
wsj.com                  609.56    601.92	1.25%
bbc.co.uk                278.64    290.28	-4.18%
guardian.co.uk           398.4     399.12	-0.18%
beatonna.livejournal.com 161.84    163.52	-1.04%
globo.com                701.16    678.64	3.21%
google.com               130.08    136.0	-4.55%
xunlei.com               803.32    860.76	-7.15%
mail.ru                  412.08    422.04	-2.42%
homeway.com.cn           509.76    517.52	-1.52%
bild.de                  1613.04   1470.52	8.84%
spiegel.de               436.52    431.92	1.05%
imdb.com                 416.88    439.2	-5.35%
goo.ne.jp                167.12    178.56	-6.85%
indiatimes.com           443.96    466.36	-5.05%
yelp.com                 533.56    522.6	2.05%
(Reporter)

Comment 7

4 months ago
Created attachment 8857282 [details]
Histogram of performance improvements

The blue one is Stylo vs Gecko comparison. The orange is parallel stylo vs sequential stylo.
Looked at this a bit last week, there's a bunch of dumb stuff we're doing. We haven't spent much time optimizing the actual styling yet, which I plan to do over the next couple of weeks. We should watch what that does to the numbers here, since I expect them to improve considerably.
Depends on: 1357973
Depends on: 1358693
Depends on: 1358694
Depends on: 1358806
Depends on: 1361843
Depends on: 1360088
The graphs are pretty much all at parity at this point, mostly because it turns out that the 2011-era sites don't have that much styling in them. We'll keep an eye on these numbers, but this bug as-filed is fixed.
Status: NEW → RESOLVED
Last Resolved: 16 days ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.