Investigate speedline
Categories
(Testing :: Raptor, enhancement, P3)
Tracking
(Not tracked)
People
(Reporter: tarek, Unassigned)
Details
Per https://github.com/sitespeedio/browsertime/issues/1142
let's see if we can add support to speedline in mach browsertime + mach raptor, and compare it with visualmetrics.py
it's a potential replacement for visualmetrics.py that would greatly simplify deployments because it does not require ffmpeg
Updated•5 years ago
|
Comment 1•5 years ago
|
||
So I took a quick look at the speedline code, and it seems to do the calculations quite differently. First, it doesn't use an orange frame for synchronization, it uses a trace log instead which seems specific to Chrome - so that script is not compatible with Firefox data right now: https://github.com/paulirish/speedline/blob/master/core/lib/frame.js#L112
The second thing I noticed is that it doesn't do any viewport cropping. This difference isn't too problematic, but the perf team has seem better results with cropping than without. Assuming that the two scripts calculate SI the same (which they seem to do), then we'll have this problem here as well. We could always do the cropping before passing the frames to speedline.
Comment 2•5 years ago
|
||
Peter, what are your thoughts on speedline? I think there is at least one large difference between the speedline and visual-metrics scripts (mentioned in the comment above).
Also, I'm not sure which e-mail you want me to ni? so I added both.
Comment 3•5 years ago
|
||
I'm in favour for using reworked version of Speedline: Removing the need for Imagemagick and ffmpeg would be great. I've been focusing on a slim container only with Firefox for end users + using Firefox standalone would just work. That would be perfect for "my" end users and I think it would help one of your use cases too?
However, we need to cherry pick what we need from Speedline. I can reach out to Paul that owns it now to get a blessing if you want, I don't think that would be a problem.
The difference with the first orange frame: we can rebuild it to work the same as VisualMetrics, to search for the "first change" instead of using a timestamp, that should be doable. Cropping: the current version takes the screenshot directly from the Chrome tracelog so there's no borders that needs to be cropped. Or maybe how do you mean?
I did some try runs yesterday for Chrome (its integrated in Browsertime) and compared Speedline metrics with Visual Metrics (using different input though: screenshots from the tracelog vs video). First visual changed matched almost perfectly for the sites I tried but for SpeedIndex and PerceptualSpeedIndex I could see a big difference on some sites. Could be multiple reasons though: limitations in the screenshots in Chrome trace, something wrong in the calculations in Speedline or something "trimmed" in how we calculate things in the Visual Metrics scripts.
I try to use my soulgalore address for browsertime and wikimedia for work more Wikipedia related things.
Comment 4•5 years ago
|
||
Peter, regarding reaching out to Paul, that sounds good to me!
I'm definitely curious to know about why there are differences though - there shouldn't be any. I noticed the speedline script removes white pixels from the frames and I haven't found that in the visualmetrics.py script yet. That could be one thing to look at.
Comment 5•5 years ago
|
||
I think it adds a extra white frame in the beginning right to get perceptual speed index to start from a blank page? I haven't had a chance to break out functionality but I started to try it out for Chrome on Android but got such a difference in first paint compared to the paint api (https://github.com/paulirish/speedline/issues/78) but I don't think it has something to do with Speed Lines implementation, rather a Chrome specific thing.
Updated•2 years ago
|
Comment 6•2 years ago
|
||
(In reply to Peter Hedenskog from comment #3)
I did some try runs yesterday for Chrome (its integrated in Browsertime) and compared Speedline metrics with Visual Metrics (using different input though: screenshots from the tracelog vs video). First visual changed matched almost perfectly for the sites I tried but for SpeedIndex and PerceptualSpeedIndex I could see a big difference on some sites. Could be multiple reasons though: limitations in the screenshots in Chrome trace, something wrong in the calculations in Speedline or something "trimmed" in how we calculate things in the Visual Metrics scripts.
Hi Peter! Speedline came up in some discussions yesterday, and as I'm reading this comment you left, I wonder if the differences we were seeing were from the differences in frame progress calculations that we recently fixed. Do you think you could test out speedline once again to see if there are still big differences?
Updated•2 years ago
|
Comment 7•2 years ago
|
||
Hi Greg, I actually removed SpeedLine a year ago to slim down Browsertime (https://github.com/sitespeedio/browsertime/pull/1727). The reason was that since its Chrome only and like to be frank, the numbers you get out of Lighthouse where its used its kind of broken, and I didn't see any reason to keep it. I could add it back for a test but it will require some work, depends on how important you think it is.
Comment 8•2 years ago
|
||
Ah that's good to know, thanks! I agree with not pursuing this further given those issues. Another issue is that speedline is missing the CSI metric.
Comment 9•1 year ago
|
||
Closing as WONTFIX based on latest comments.
Description
•