Closed Bug 1621879 Opened 5 years ago Closed 1 year ago

Investigate speedline

Categories

(Testing :: Raptor, enhancement, P3)

Version 3
enhancement

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: tarek, Unassigned)

Details

Per https://github.com/sitespeedio/browsertime/issues/1142

let's see if we can add support to speedline in mach browsertime + mach raptor, and compare it with visualmetrics.py

it's a potential replacement for visualmetrics.py that would greatly simplify deployments because it does not require ffmpeg

Priority: P4 → P3

So I took a quick look at the speedline code, and it seems to do the calculations quite differently. First, it doesn't use an orange frame for synchronization, it uses a trace log instead which seems specific to Chrome - so that script is not compatible with Firefox data right now: https://github.com/paulirish/speedline/blob/master/core/lib/frame.js#L112

The second thing I noticed is that it doesn't do any viewport cropping. This difference isn't too problematic, but the perf team has seem better results with cropping than without. Assuming that the two scripts calculate SI the same (which they seem to do), then we'll have this problem here as well. We could always do the cropping before passing the frames to speedline.

Peter, what are your thoughts on speedline? I think there is at least one large difference between the speedline and visual-metrics scripts (mentioned in the comment above).

Also, I'm not sure which e-mail you want me to ni? so I added both.

Flags: needinfo?(peter)
Flags: needinfo?(peter)

I'm in favour for using reworked version of Speedline: Removing the need for Imagemagick and ffmpeg would be great. I've been focusing on a slim container only with Firefox for end users + using Firefox standalone would just work. That would be perfect for "my" end users and I think it would help one of your use cases too?

However, we need to cherry pick what we need from Speedline. I can reach out to Paul that owns it now to get a blessing if you want, I don't think that would be a problem.

The difference with the first orange frame: we can rebuild it to work the same as VisualMetrics, to search for the "first change" instead of using a timestamp, that should be doable. Cropping: the current version takes the screenshot directly from the Chrome tracelog so there's no borders that needs to be cropped. Or maybe how do you mean?

I did some try runs yesterday for Chrome (its integrated in Browsertime) and compared Speedline metrics with Visual Metrics (using different input though: screenshots from the tracelog vs video). First visual changed matched almost perfectly for the sites I tried but for SpeedIndex and PerceptualSpeedIndex I could see a big difference on some sites. Could be multiple reasons though: limitations in the screenshots in Chrome trace, something wrong in the calculations in Speedline or something "trimmed" in how we calculate things in the Visual Metrics scripts.

I try to use my soulgalore address for browsertime and wikimedia for work more Wikipedia related things.

Peter, regarding reaching out to Paul, that sounds good to me!

I'm definitely curious to know about why there are differences though - there shouldn't be any. I noticed the speedline script removes white pixels from the frames and I haven't found that in the visualmetrics.py script yet. That could be one thing to look at.

Flags: needinfo?(peter)

I think it adds a extra white frame in the beginning right to get perceptual speed index to start from a blank page? I haven't had a chance to break out functionality but I started to try it out for Chrome on Android but got such a difference in first paint compared to the paint api (https://github.com/paulirish/speedline/issues/78) but I don't think it has something to do with Speed Lines implementation, rather a Chrome specific thing.

Flags: needinfo?(peter)
Severity: normal → S3

(In reply to Peter Hedenskog from comment #3)

I did some try runs yesterday for Chrome (its integrated in Browsertime) and compared Speedline metrics with Visual Metrics (using different input though: screenshots from the tracelog vs video). First visual changed matched almost perfectly for the sites I tried but for SpeedIndex and PerceptualSpeedIndex I could see a big difference on some sites. Could be multiple reasons though: limitations in the screenshots in Chrome trace, something wrong in the calculations in Speedline or something "trimmed" in how we calculate things in the Visual Metrics scripts.

Hi Peter! Speedline came up in some discussions yesterday, and as I'm reading this comment you left, I wonder if the differences we were seeing were from the differences in frame progress calculations that we recently fixed. Do you think you could test out speedline once again to see if there are still big differences?

Flags: needinfo?(peter)
Component: Performance → Raptor

Hi Greg, I actually removed SpeedLine a year ago to slim down Browsertime (https://github.com/sitespeedio/browsertime/pull/1727). The reason was that since its Chrome only and like to be frank, the numbers you get out of Lighthouse where its used its kind of broken, and I didn't see any reason to keep it. I could add it back for a test but it will require some work, depends on how important you think it is.

Flags: needinfo?(peter)

Ah that's good to know, thanks! I agree with not pursuing this further given those issues. Another issue is that speedline is missing the CSI metric.

Closing as WONTFIX based on latest comments.

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.