Closed Bug 721634 Opened 12 years ago Closed 12 years ago

Compare Startup times for Multi-Locale vs. Single Locale Builds

Categories

(Firefox for Android Graveyard :: General, defect, P1)

Product:
Firefox for Android Graveyard
Component:
General
Version:
11 Branch
Platform:
ARM
Android
Type:
defect

Tracking

(firefox11 fixed, blocking-fennec1.0 beta+)

Status:
RESOLVED FIXED
Milestone:
Firefox 11
Tracking Flags:
Tracking Status
firefox11 --- fixed
blocking-fennec1.0 --- beta+

People

(Reporter: elan, Assigned: nhirata)

References

Details

We need to understand delta in start up time (% faster or slower) for single-locale compared to multi-locale apks.
Yesterday, did not have Galaxy S to try to compare the data with.  
- Ran on G2 and Galaxy SII with the English local versus Multi pack; result : no major time difference (yesterday)
- Noticed that English local is 1 meg smaller than all localized builds in the l10n dir which is in turn 1 meg smaller than Multipack (today)
- Received Galaxy S Captivate today; issue with trying to get into the phone.

- Currently working on trying to root Galaxy S Captivate to make the video shooting easier/faster.  [It helps shave time and also tends to have more consistent numbers if it's automated]
Priority: -- → P1
Details were talked about in the Browser Tech Meeting.

Update Status : 
- Galaxy S rooted.
- starting video capture and going to compare english vs spanish vs multi on Galaxy S

- note : twitter web page changed to output to log, logcat being captured for review purposes for the video; this allows for faster processing as I don't have to bother looking for the onload number.
Ran into bug 721644 for the spanish build.
Video capture for multi and en done; other single locale languages are blocked by bug 721644.  Importing videos.
Data posted : https://docs.google.com/spreadsheet/ccc?key=0Arku3jleCA0UdE5OTGZLdXJCR3gwWjdqV3RTcmNvMGc&hl=en_US#gid=4

About a 200 ms difference or so.  2 crashes occurred during video testing which need to be tracked down.  logcat of the run still available; on PTO today (house inspections), need to take care of some personal things so I'll try to get them when I get back home.  Also need to investigate if I had builds flipflopped for the G2 at some point.

Do we care about the numbers for the other single locale builds?
Yes, we should care about startup time of localized builds.
(In reply to Naoki Hirata :nhirata from comment #5)
> Data posted :
> https://docs.google.com/spreadsheet/
> ccc?key=0Arku3jleCA0UdE5OTGZLdXJCR3gwWjdqV3RTcmNvMGc&hl=en_US#gid=4
> 
> About a 200 ms difference or so.  2 crashes occurred during video testing
> which need to be tracked down.  logcat of the run still available; on PTO
> today (house inspections), need to take care of some personal things so I'll
> try to get them when I get back home.  Also need to investigate if I had
> builds flipflopped for the G2 at some point.
> 
> Do we care about the numbers for the other single locale builds?

The data shows the G2 single-locale build is overall slower than the G2 Multi-locale build.  is this really true? 

Also, is there a reason you didn't label the builds on both devices consistent in their graph color theme?
right tony... this data doesn't make sense.

Also, why are you collecting data via video.  We added instrumentation to the builds that will produce the numbers you want.
(In reply to Doug Turner (:dougt) from comment #8)
> right tony... this data doesn't make sense.
> 
> Also, why are you collecting data via video.  We added instrumentation to
> the builds that will produce the numbers you want.

Since there are some questions around the results of the G2, Naoki will double, and TRIPLE! check his results when he gets home later tonite.   Additionally, there's been engineering request to have results from automated performance runs, which he'll check against a rooted Motorola Droid Pro and a Nexus S.  

Stay tuned for more results posted in here and the comment 5 spreadsheet.
(In reply to Tony Chung [:tchung] from comment #9)
> Stay tuned for more results posted in here and the comment 5 spreadsheet.

Any updates here?
I need to borrow the G2 from kbrosnan; it does look like I had the builds switched.
Based on today's build using the instrumentation:
https://docs.google.com/spreadsheet/ccc?key=0Arku3jleCA0UdE5OTGZLdXJCR3gwWjdqV3RTcmNvMGc&hl=en_US#gid=4

I had the builds reversed in my video footage originally.
Note: the throbber time is calculated from throbber start to throbber stop, so it may very but should be roughly the same.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Thanks Naoki for confirming those results.  Certainly makes more sense this way.   Doug, Brad, do you guys have what you need to resolve this issue?
(In reply to Naoki Hirata :nhirata from comment #12)
> Based on today's build using the instrumentation:
> https://docs.google.com/spreadsheet/
> ccc?key=0Arku3jleCA0UdE5OTGZLdXJCR3gwWjdqV3RTcmNvMGc&hl=en_US#gid=4
> 
> I had the builds reversed in my video footage originally.
> Note: the throbber time is calculated from throbber start to throbber stop,
> so it may very but should be roughly the same.

can you just report the numbers in the bug please?
(identifier)	Onload	Throbber	Paint
G2 Multi Native Nightly 1/30/2012	4730.82352941176	1779.11764705882	6094.47058823529
G2 Single-En Native Nightly 1/30/2012	4637.64705882353	1725.52941176471	6056.17647058824
G2 Single-ES Native Nightly 1/30/2012	4544.64705882353	1649.76470588235	5508.47058823529
With Std dev.

(identifier)	Onload	Throbber	Paint
G2 Multi Native Nightly 1/30/2012	4730.82352941176	1779.11764705882	6094.47058823529
std dev	161.055206099538	180.431040273334	238.715080600037
G2 Single-En Native Nightly 1/30/2012	4637.64705882353	1725.52941176471	6056.17647058824
std dev	113.430012109048	61.0871484510642	288.165151279201
G2 Single-ES Native Nightly 1/30/2012	4544.64705882353	1649.76470588235	5508.47058823529
std dev	85.7772851462368	44.5119778988823	93.4024341539467
(In reply to Brad Lassey [:blassey] from comment #14)
> (In reply to Naoki Hirata :nhirata from comment #12)
> > Based on today's build using the instrumentation:
> > https://docs.google.com/spreadsheet/
> > ccc?key=0Arku3jleCA0UdE5OTGZLdXJCR3gwWjdqV3RTcmNvMGc&hl=en_US#gid=4
> > 
> > I had the builds reversed in my video footage originally.
> > Note: the throbber time is calculated from throbber start to throbber stop,
> > so it may very but should be roughly the same.
> 
> can you just report the numbers in the bug please?

Brad, you or anyone else could have done that.   the numbers are publically accessible in the spreadsheet by anyone.   It even includes extra analysis and observations that Naoki included that was not part of the original request.  

If there's any missing information, please ask.  otherwise, please dont require folks to add extra menial work.
(In reply to Tony Chung [:tchung] from comment #17)
> (In reply to Brad Lassey [:blassey] from comment #14)
> > (In reply to Naoki Hirata :nhirata from comment #12)
> > > Based on today's build using the instrumentation:
> > > https://docs.google.com/spreadsheet/
> > > ccc?key=0Arku3jleCA0UdE5OTGZLdXJCR3gwWjdqV3RTcmNvMGc&hl=en_US#gid=4
> > > 
> > > I had the builds reversed in my video footage originally.
> > > Note: the throbber time is calculated from throbber start to throbber stop,
> > > so it may very but should be roughly the same.
> > 
> > can you just report the numbers in the bug please?
> 
> Brad, you or anyone else could have done that.   the numbers are publically
> accessible in the spreadsheet by anyone.   It even includes extra analysis
> and observations that Naoki included that was not part of the original
> request.  
> 
> If there's any missing information, please ask.  otherwise, please dont
> require folks to add extra menial work.

Tony, I've generally pushed back on putting data in google docs, which is why I asked for this to be reported in the bug and in the past have asked for things to be put in wikis and what not.
(In reply to Tony Chung [:tchung] from comment #17)
> Brad, you or anyone else could have done that.   the numbers are publically
> accessible in the spreadsheet by anyone.   It even includes extra analysis
> and observations that Naoki included that was not part of the original
> request.  

Also, no. There are bunch of numbers in the "Overview Data" sheet of that spread sheet. Which ones am I supposed to be looking at??? Summarizing in the bug would be very useful.

So, looking at this section of the data (reformatted to be readable):
Device	Lang Pack  Build      Onload   Throbber Paint
G2	Multi	   1/25/2012  4673.78  4325     5075
G2	Single-En  1/25/2012  4488.71  4211.11	4655.56

There is ~400ms hit in final paint and ~200ms hit in onload measurement, which looks to be statistically significant.

Also I should note that Naoki asked me on irc if onCreate is a good place to be measuring the start of start up time. Not knowing why he was asking I said it was a good approximation. Seeing now that he was asking for the purposes of comparing single vs. multilocale, it is a very poor approximation. Most of the hit we take from multilocale happens before onCreate gets called. Which of these numbers are tainted by that?
Looking at the information in the log, I see what you mean.  Would starting from the start intent of fennec for the device work instead of onCreate?
(In reply to Naoki Hirata :nhirata from comment #20)
> Looking at the information in the log, I see what you mean.  Would starting
> from the start intent of fennec for the device work instead of onCreate?

Yup
After stopping for a moment, I came to realize that the onload value should always include the hit from the multilocale regardless, because the value that is passed is taken from the start intent to the time of the onload value being pushed out to the log.  

Also, looking at Comment 19, that was from the 1/25/2012 build which was done on video.  The "start time" was calculated from the visual transition from the settings to fennec.  The value from the time of the throbber ending and the paint ending was subtracted from the "start time".  Having said that, I am not sure if I can say with confidence that the multilocale hit had happened at the visual transition or if it had happened prior to the transition.

I believe that the 1/27/2012 was the first build that includes the instrumentation in which values are past to the console, so they are nonvideo automations and have different calculations in terms of the throbber time and paint time.  I did make a mistake in using onstart (which is actually a worse than oncreate to use) as the "start time".  I had saved the builds to my computer and had tested out the droid pro, but the numbers for the paint does not look right so I am withholding the announcement of the values until I figure out why the numbers are looking odd.  I may have to retest again.
blocking-fennec1.0: --- → beta+
It looks like I just forgot to update this bug.  The following shows the Single locale versus Multi locale numbers for 1/30/2012 build

https://docs.google.com/spreadsheet/ccc?key=0Arku3jleCA0UdE5OTGZLdXJCR3gwWjdqV3RTcmNvMGc#gid=4
For those allergic to Google Docs/Spreadsheet:

Droid Pro - Build : 1/30/2012	
	onLoad 			Throbber			Page Paint	
multi	4110.76923076923	1597.30769230793	4968.84615384596
	62.7404094213528	44.1708512776252	76.2253743340281
	
es	4096.5625		1577.18749999981	4936.4666666667
	50.5581760615102	31.0874438749648	78.345267581271		

G2 - Native Nightly 1/30/2012

	Onload			Throbber		Paint
Multi 	4730.82352941176	1779.11764705882	6094.47058823529
	161.055206099538	180.431040273334	238.715080600037

En	4637.64705882353	1725.52941176471	6056.17647058824
	113.430012109048	61.0871484510642	288.165151279201

ES 	4544.64705882353	1649.76470588235	5508.47058823529
	85.7772851462368	44.5119778988823	93.4024341539467
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.