463323 - create flatline effect in graphs by repeating last recorded value

Status: RESOLVED FIXED

(Webtools Graveyard :: Graph Server, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

For a given maintenance branch, and o.s., once we stop producing new data on that branch, can we keep repeating the last recorded value? This would give a "flatline" effect in the graphs, and would easily allow people to compare results between different branches even if one branch is not generating new test runs today.

This came up during discussions about mothballing the talos machines on maintenance branches. The concern was about how developers on FF3.next would easily do perf comparison in graph server between FF3.next and the most recent FF3.0 or FF2.0 values. Is this doable?

(Not sure if this the right component, please reassign as needed.)

Comment 1

[Samuel Sidler (old account; do not CC)]

Pretty sure this belongs in the Webtools::Graph Server component...

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Can we make this flatline be a dotted line, so people dont get confused between "real live data", and "extrapolated historical data" ?

Comment 4

[alice nodelman [:alice] [:anode]]

I think that we can approach this pretty simply by extending all graph lines to the end of the display box, perhaps in a different colour/dotted line to indicate that it doesn't correspond to actual test results.  This would make life easier as we wouldn't have to differentiate between active/inactive boxes, we'd just extend all lines.

We wouldn't want this to appear as individual point, especially as that would break the interlink between discrete/continuous graphs that occurs when you right click on a given dot in a continuous graph.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

If making this "dotted line" change will take too much time, another quick option could be using some SQL:

- read the most recent value for a to-be-killed-machine
- insert new records with the same most-recent value. Repeat twice a day for 365 days into the future.

It doesnt give us a nice dotted line, but it does give us a quick solution in an already busy Q4.

Comment 6

[Ryan Doherty (:rdoherty)]

(In reply to comment #5)
> If making this "dotted line" change will take too much time, another quick
> option could be using some SQL:
> 
> - read the most recent value for a to-be-killed-machine
> - insert new records with the same most-recent value. Repeat twice a day for
> 365 days into the future.
> 
> It doesnt give us a nice dotted line, but it does give us a quick solution in
> an already busy Q4.

+1 for this idea.

Comment 7

[Samuel Sidler (old account; do not CC)]

(In reply to comment #5)
> - read the most recent value for a to-be-killed-machine

- ensure that this most recent result for a given machine isn't bogus as they sometimes are

> - insert new records with the same most-recent value. Repeat twice a day for
> 365 days into the future.

- profit!!!

Comment 8

[Ryan Doherty (:rdoherty)]

(In reply to comment #7)
> (In reply to comment #5)
> > - read the most recent value for a to-be-killed-machine
> 
> - ensure that this most recent result for a given machine isn't bogus as they
> sometimes are
This can be solved by grabbing the most recent continuous value, which is averaged and contains bogus values removed.

Comment 9

[alice nodelman [:alice] [:anode]]

I've got most of a patch working that extends the first/last data points in a given set out to the edges of the graph.  Just fixing that last few issues with snap-to and popups.

Bonus here is not having to mess with the db at all - it ended up being just as simple (if not simpler) to fix the display.

Comment 10

[alice nodelman [:alice] [:anode]]

Attachment: extend lines to edges of displayed graph

This will need to do a little baking on stage but from testing on my home set up it correctly extends the lines of stopped machines - including adding information to the pop-up status when hovering over the graph.

Comment 11

[Chris Cooper [:coop] (he/him)]

Comment on attachment 348919 [details] [diff] [review]
extend lines to edges of displayed graph

The results look good, and I see you've got the popup working since last night too.

How does this code deal with a single data point? My assumption is that for any machines we obsolete, we would only keep the final data point. Will this single point be correctly converted into a line? Lines like this scare me a bit in the single point scenario:

> +        if (this.data[this.data.length-2] < startTime) {

Comment 12

[Chris Cooper [:coop] (he/him)]

(In reply to comment #11)
> How does this code deal with a single data point? My assumption is that for any
> machines we obsolete, we would only keep the final data point.

Also, we'll obviously have to revisit our data archiving/expiration procedures to make sure we're not just deleting all data past a certain date.

Comment 13

[alice nodelman [:alice] [:anode]]

Our archiving procedures removes discrete data older than 60 days, but the continuous data remains intact - so we shouldn't have to worry about these types of graphs disappearing.  The continuous data is compact enough that I can't think of a good argument for throwing it away.

But yes, in case there is some pathological graph of a single point we should handle it correctly. :)

Comment 14

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]extend lines to edges of displayed graph, take 2

Some issues cropped up during graphs-stage testing having to do with the styling of the line and the start/end points of the extended lines - especially when highlighting a subsection of a graph.

I have it up and running on graphs-stage so you can poke at it there.

Comment 15

[alice nodelman [:alice] [:anode]]

Comment on attachment 349095 [details] [diff] [review]
[Checked in]extend lines to edges of displayed graph, take 2

changeset: 137:559ff07b2e90

Comment 16

[Ryan Doherty (:rdoherty)]

So is this ready to be pushed to prod? I can merge the change to the production branch and we can probably get it out tomorrow.

Comment 17

[alice nodelman [:alice] [:anode]]

I had filed bug 466229 to get this pushed to production - but it sounds like that isn't the correct route to pushing code anymore.

If there is a production branch to be merged into than please do so, and if that bug is not necessary feel free to close it out.

Comment 18

[Ryan Doherty (:rdoherty)]

(In reply to comment #17)
> I had filed bug 466229 to get this pushed to production - but it sounds like
> that isn't the correct route to pushing code anymore.
> 
> If there is a production branch to be merged into than please do so, and if
> that bug is not necessary feel free to close it out.

Ok, I can merge the change to the production branch and I'll update the bug with the status.

Comment 19

[Ryan Doherty (:rdoherty)]

Done.

Comment 20

[alice nodelman [:alice] [:anode]]

In production and looks good.