68570 - RFE: Set XSUNTRANPORT/XSUNSMESIZE on Solaris to improve rendering speed...

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Roland Mainz]

Xsun in Solaris has an unique feature which may be used to speed-up client
requests - a shared-memory-based _transport_.
This would be quite usefull for Xlib toolkit which (AFAIK) does not use the
MIT-SHM extension (GTK+ toolkit should be accerlated, too - assuming GDK does
not do any crazy hacks with shared memory... =:-)

To quote the Xsun(1) manual page:
-- snip --
NETWORK CONNECTIONS
 In  addition  to  the  network  connections   described   in
 Xserver(1), Xsun provides the following connection:

 Shared Memory
     Sun provides a shared memory transport mechanism via the
     SUN_SME extension.  This extension provides the capabil-
     ity of sending client requests to the server via  shared
     memory.  Shared memory is used for client requests only.
     Replies from the server and  events  are  sent  via  the
     default  transport  mechanism.  To enable this transport
     mechanism, one has to set the DISPLAY environment  vari-
     able to :x.y (where x is the display number and y is the
     screen number), and the environment  variable  XSUNTRAN-
     SPORT to "shmem".  The size of the segment can be set by
     setting enviroment variable XSUNSMESIZE.  By default, it
     is set to "64" which implies that the size of the shared
     memory segment is 64K.
-- snip --

I would vote to add this to the "mozilla" startup script, e.g. add
-- snip --
export XSUNTRANPORT="shmem" XSUNSMESIZE="64"
-- snip --

That's all... :-)
(non-Xsun Xservers/Xlib-versions would simply ignore the XSUN* env. vars...)

Comment 1

[Asa Dotzler [:asa]]

pav, any chance you know who should get this?

Comment 2

[Roland Mainz]

Uhm... why isn't it possible to get this _in_ for Mozilla 0.9 ? The changes for
the dist/bin/mozilla script look stupid...
... should I file a patch !? =:-)

Comment 3

[Roland Mainz]

Attachment: patch for mozilla/build/unix/run-mozilla.sh to set XSUNTRANSPORT&co.

Comment 4

[Roland Mainz]

Pavlov: 
Can I "overtake" this bug, please ?

Comment 5

[Stuart Parmenter]

if you want it, go ahead

Comment 6

[Roland Mainz]

OK, thanks. Overtaking...

richb/pavlov: Wanna r= that patch, please ?

Comment 7

[Roland Mainz]

Accepting bug...

Adding milestone 0.9.1 - this bug is more than easy... :-)

Comment 8

[Rich Burridge]

Roland, this looks like a good win, and thanks for adding me to
the cc, but can we get some quantative data behind it? How about
running the benchmarks at http://i-bench.zdnet.com/ibench/i-bench.htm
both before and after making this change and reporting back the results?
Note also that I have no authority to r= anything, but I'd added cls to
the cc list. Hopefully this falls under his domain.

Comment 9

[Roland Mainz]

> How about
> running the benchmarks at http://i-bench.zdnet.com/ibench/i-bench.htm
> both before and after making this change and reporting back the results?

1. the test doesn't work here - at always wants to do some PDF plugin stuff -
which isn't installed on my box. Looking...
... but http://i-bench.zdnet.com/ibench/testlist/home_js.php#pOverallScore works
- after disabling the PDF stuff...
2. I am sitting on a totally wrong machine for such a test. My assumtion is that
machines with more than one CPU and _accerlated_ cards like Creator/Elite/Expert
will benefit more from this enhancement than my dumb m64 framebuffer (Ultra5)
does...
3. I assume that all not-so-optimized Unix-toolkits which do not fiddle around
with shared-memory (Xlib/Qt/Xprint) will benefit more from this "magic switch"
than those which eat-up all available shared memory segments and are
tuned-to-h*ll-and-back...

> Note also that I have no authority to r= anything, but I'd added cls to
> the cc list. Hopefully this falls under his domain.

Please correct me... AFAIK anyone can r= stuff, right ?

Test results (2001-05-11-08-trunk with --enable-optimize (but not
--disable-debug - optimisation for nsPressShell.o has been turned off due
crashes in optimized version), build with Sun Workshop 6 Update 2 EA2):
a) with XSUNTRANSPORT enabled:
Load Pages       HTML
All iterations    2441.18              
First iteration (downloaded)    596.28              
Subsequent iteration (cached)    263.56              
b) without XSUNTRANSPORT:
Load Pages       HTML
All iterations    4494.19              
First iteration (downloaded)    832.88              
Subsequent iteration (cached)    523.04

richb: 
1. Can you confirm these results, please ? I never thought that XSUNTRANSPORT
can have such an improvement to browser performance. Either these results are 
totally wrong or enabling this should get highest priority...
2. Can you ask the Xsun people (Alan!?) for comments ?

Comment 10

[Rich Burridge]

I've just tried running the HTML and XML tests at
http://i-bench.zdnet.com/ibench/i-bench.htm

Here are my results:

                               Before        After
HTML Load Speed  All            413.07       418.34
                 First           56.21        55.10
                 Subsequent      50.98        51.89

XML/CSS Load Speed All           59.39        61.09
                   First          3.53         3.16
                   Subsequrnt     2.94         3.05

This was on a 2 processor Ultra 60 running Solaris 8
with GNOME 1.4 (512 Mybyes of memory, local disk, ffb0 frame buffer).
It was with a version of Mozilla compiled with
--disable-debug, --enable-optimize (build id 2001050210).

I'm not seeing the large differential that you are seeing, which 
suggests that the simple patch attachment is either not enough, or
I didn't apply it correctly.

I've asked Alan Coopersmith in the Xserver group to comment.

Also, if anybody can do an r=, can you point me to a web page where
this is documented please? I've been constantly told that only the bug
owner or module owner, or one of his/her peers can give r=.

Comment 11

[drapeau]

Hey, is this assuming a build with the "--enable-X11-shm" .mozconfig option set
before doing your Mozilla build?  I'm not sure that flag is set by default, but
it seems like maybe that needs to happen before these environment variables are
going to do you any good.

Can you confirm this please, Roland?

Comment 12

[Rich Burridge]

Comments from Alan Coopersmith in the X server group here at Sun:

"The shared memory transport should mostly just work - there have been some
bug fixes to make it work with multithreaded programs (it used to cause 
Microsoft IE to hang forever if you used the SME transport), but I don't know
of any problems with non-mt programs.

As for the question in the bug report from George Drapeau, the MIT-SHM extension
is seperate from the SME transport.  MIT-SHM adds calls to explicitly put some
pixmaps in shared memory - the SME transport transparently uses shared memory
whenever possible for transporting large amounts of data.  To use MIT-SHM on
24-bit frame buffers, you need to install an Xsun patch with the fix for
bug 4305597."

Comment 13

[Roland Mainz]

> I've just tried running the HTML and XML tests at
> http://i-bench.zdnet.com/ibench/i-bench.htm
> 
> Here are my results:
> 
>                                Before        After
> HTML Load Speed  All            413.07       418.34
>                  First           56.21        55.10
>                  Subsequent      50.98        51.89
> 
> XML/CSS Load Speed All           59.39        61.09
>                    First          3.53         3.16
>                    Subsequrnt     2.94         3.05

"Before" = before applying this patch, "After" = after applying this patch ?

> This was on a 2 processor Ultra 60 running Solaris 8
> with GNOME 1.4 (512 Mybyes of memory, local disk, ffb0 frame buffer).
> It was with a version of Mozilla compiled with
> --disable-debug, --enable-optimize (build id 2001050210).

I was testing this under Solaris 7 on a Ultra5/333MHz/2MB 2ndLCache, 384MB,
local disk, build with Sun Workshop 6 Update 2 EA2 using the build-in m64
framebuffer with the 24bit visual (the machine has a 2nd m64 card which runs in
8bit mode ("traditional" dual head - no Xinerama))... is it possible that SME
wasn't used for some reason (see below) ?

> I'm not seeing the large differential that you are seeing, which 
> suggests that the simple patch attachment is either not enough, or
> I didn't apply it correctly.

Mhhh...
1. I didn't run the XML test because they render nothing
2. ...it can be possible that my test are simple wrong - and differences in
network speed may also cause some (small) differences
3. ...maybe the SME transport wasn't enabled because something eat-up all the
shared memory for that process - or there was no big chunk of memory left for
SME
4. ...maybe SME only speeds-up machines with one CPU
5. ...maybe the optimisations in Mozilla's GTK+ toolkit support hide the
benefits of SME... use of shared memory, the GC cache and other stuff may reduce
the traffic between mozilla<-->Xsun below the level where SME start to be
usefull.
6. Which window size and visual (depth!!) were you using to test Mozilla ?
Running these test on a 8bit visual may not be very usefull... =:-))))
7. My patch only enables SME if XSUNTRANPORT env var is _not_ set yet. If it is
set my patch simply does not touch these vars.

Questions:
- is there a guranteed way how to check if SME is really used or not ? can
/usr/proc/bin/pmap <pid-of-mozilla-bin> | egrep "bla-bla" # help here ?
- SME has a setting (XSUNSMESIZE env var, see Xsun(1) for details...) to select
the size of the shared memory segment - maybe it's worth to "play" with this
value and check the results (the SME stuff looks little bit "underdocumented"...
;-((((
- can you check whether your Xsun has the SME transport ?
  % xdpyinfo | grep SUN_SME # should return "SUN_SME"...

[snip]
> Also, if anybody can do an r=, can you point me to a web page where
> this is documented please? I've been constantly told that only the bug
> owner or module owner, or one of his/her peers can give r=.

I was told on IRC #mozilla that this rule is history. Maybe brendan give a
definite answer for that...

----

> Comments from Alan Coopersmith in the X server group here at Sun:
>
> "The shared memory transport should mostly just work - there have been some
> bug fixes to make it work with multithreaded programs (it used to cause 
> Microsoft IE to hang forever if you used the SME transport), but I don't know
> of any problems with non-mt programs.

What about the bug that enabling SME transport crashes "showmetv" ? I think I
have reported that multiple times...

Comment 14

[Rich Burridge]

stard:/home/richb% xdpyinfo | grep SUN_SME
    SUN_SME

This is on a 24 bit deep display? I've no idea how to determine if SUN_SME
is being used. I've asked Alan to comment again on your last comments.

Comment 15

[chris hofmann]

let's land this in 0.9.2. let drivers know if this creates problems. thanks.

Comment 16

[Roland Mainz]

Any news here ?

Comment 17

[Roland Mainz]

No news... ;-((

Retargeting to milestone 0.9.3 per asa's email to all bug owners...

Comment 18

[Roland Mainz]

(per comment by Uros Bizjak <[e-mail]> - thanks !)
There is a typo in the patch which may explain the performance difference:
s/XSUNTRANPORT/XSUNTRANSPORT/

Comment 19

[Roland Mainz]

Attachment: Fixed patch

Comment 20

[Roland Mainz]

Requesting r=/sr= for that patch...

Comment 21

[Roland Mainz]

imoT: Wanna r= this patch, please ? Thanks!

Comment 22

[Christopher Blizzard (:blizzard)]

r/sr=blizzard  I don't see the harm, I guess.

Comment 23

[Roland Mainz]

No... it does not harm. If SME transport fails to initalize it uses the "normal"
transport.
Other platforms are not affected as this is a Solaris/Xsun-only feature.

Comment 24

[Axel Hecht]

I have some stability concerns on this one.
I tried this on the 0.9.2 branch (I have no other build right now), and the
100x100 table stress test. Just scrolling happily. It might get a bit faster,
(the ibench stuff was .exe for download, so no numbers), but I had
buserrors and segfaults on exit at some test runs. They were not really 
reproducable, but I didn't have any without the patch, but two or three
segfaults and one bus error with.

Axel

Comment 25

[Roland Mainz]

pike:
This crash_at_exit issue is something different. I even get this without using
SME. Try to run the Zilla with export LD_PRELOAD=watchmalloc.so.1 and you'll see
that something is trashing the heap in current builds (man watchmalloc(1)). I
assume this is not related to SME; SME may cause that the memory allocations may
happen in a different order. Fun... ;-(

We're running SME stuff since ~1 year here without problems (in CDE startup
script) - except one: Sun's "showmetv" video/audio playback tool crashes. But
that looks like bad code in showmetv...

Finally... which Xsun patch are you using ?

Comment 26

[Axel Hecht]

I was at first using the environment variables.
And of course, the one with "Fixed" was the one I was looking at.
I know that the default maximum size of shared mem can be set, what if that
is too low? Did you test that?
And even if using shared mem just triggers a bug more often than without, it
shouldn't get in. $.02.
oh, and /me won't learn about watchmalloc in the near future, sorry.

Axel

Comment 27

[Roland Mainz]

alex:
AFAIK the SME stuff cannot be configured wrong. If you do that it will simply
fall-back to the "normal" transport.

> And even if using shared mem just triggers a bug more often than without, it
> shouldn't get in.

As I said... there is something else broken in current builds. AFAIK it does not
affect this stuff. If we would block each enhancement until all other bugs have
been fixed first we would still trying to get "ready" for M2, not M0.9.3 ... :-)

Comment 28

[John "AntiTuX" Dee]

I'll give an r= if I'm allowed to. Tested on an  Ultra2 2x440mhz, 768m ram,
Solaris 8 (1/01), Sun's Gnome 1.4 (since ximian doesn't want to develop it
anymore, it seems).

Comment 29

[Roland Mainz]

timeless:
Wanna check this patch in, please ? "trunk" only, please...

Comment 30

[timeless]

fixed on trunk (will be there for 094 and beyond)

Comment 31

[Roland Mainz]

patch has been checked 
(http://bonsai.mozilla.org/cvsquery.cgi?module=MozillaTinderboxAll&branch=HEAD&cvsroot=/cvsroot&date=explicit&mindate=996811680&maxdate=996811980&who=timeless%25mac.com).
Thanks !

Marking bug as FIXED.

Comment 32

[Joshua Xia]

Adding XSUNTRANSPORT and XSUNSMESIZE lead to some Java Applet hang when use
SunRay server, and it seems slower than normal situation (without setting
XSUNTRANSPORT or XSUNSMESIZE)

Comment 33

[Roland Mainz]

Joshua Xia wrote:
> Adding XSUNTRANSPORT and XSUNSMESIZE lead to some Java Applet hang when use
> SunRay server, and it seems slower than normal situation (without setting
> XSUNTRANSPORT or XSUNSMESIZE)

This sounds like a SunRay software bug...

... can you file a seperate bug in bugzilla and CC: me that we can investigate
the issue, please ?

Comment 34

[Joshua Xia]

It is RFE, so deleting these two environment variable will not lead to any
critical problem? right?
I think it is improper to set XSUNSMESIZE to be constant, because it should be
dynamically changed according to System's environment.

There are two SUN internal Applet-base application hang on SunRay when set these
two variable.

Comment 35

[Roland Mainz]

Joshua Xia wrote:
> It is RFE, so deleting these two environment variable will not lead to any
> critical problem? right?

Well, Mozilla would be slower on local displays, especially for large image
blits, alpha blending, DHTML rendering, etc.

> I think it is improper to set XSUNSMESIZE to be constant, because it should be
> dynamically changed according to System's environment.

Well, for machines which have more memory and allow larger shared memory
segments a higher value may be better, however if I recall the issue correctly
that requires some /etc/system hacking which is far far out of of abilities of
the mozilla startup script (the system admin would have to do the adjustment).

Comment 36

[Roland Mainz]

Joshua Xia wrote:
> There are two SUN internal Applet-base application hang on SunRay when set 
> these two variable.

IMHO the issue should be fixed from both sides:
- Solaris/Xsun-side: A sunsolve bug should be filed about the interaction
between SunRay, JAVA and the shared-memory transport (just curious: Is it
possible that JAVA uses the "SUN_DGA"-extension ? I could imagine that - in
theory - may not like that someone uses "SUN_DGA" and the shared-memory
transport at the same time)
- Mozilla-side: AFAIK there are some SunRay-specific ENV vars set when the
application is displaying itself to a SunRay terminal so a _WORKAROUND_ may be
to |ifdef| the XSUNTRANSPORT stuff out if those vars are set (however I don't
like that idea because the issue likely sits in either the SunRay software
and/or JAVA and should be fixed there instead of stuffing more and more
workarounds into Mozilla) ...

... but again - please file a new bug that we can discuss and handle the issue
there...