486662 - Performance dashboard needs a way to access graph server data

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Johnathan Nightingale [:johnath]]

As discussed in email and in bug 485928, the graph server was straining under the load of periodic bursts of update traffic caused by my perf dashboard ( http://people.mozilla.org/~johnath/pdb2/ ).

While this worked for some time, the theory is that now that the old and new graph servers are operating on the same box, the load has increased, and the queries are now sending it over the edge.

While bug 485928 discusses the symptoms of the problem (graph server timing out queries), it doesn't directly address ways to make the dashboard access less painful to graph server, so this bug can track that.

Comment 1

[Johnathan Nightingale [:johnath]]

After chatting with Dave in IRC, I added a .5 second sleep to the dashboard's fetch script, and got Dave to drop the firewall block.  The script is currently working again, waiting to see whether the impact on graph server load is reduced.

Comment 2

[Ryan Doherty (:rdoherty)]

(In reply to comment #1)
> After chatting with Dave in IRC, I added a .5 second sleep to the dashboard's
> fetch script, and got Dave to drop the firewall block.

Thanks! FYI we have a new architecture and API: https://wiki.mozilla.org/Perfomatic:API

We will be switching graphs.mozilla.org over to it soon (next week most likely). 

Once we do this the load on the graph server should decrease because we will only be pushing data once.

Comment 3

[Johnathan Nightingale [:johnath]]

It's great to see the new server coming up!

Does that mean that my queries to getdata.cgi and the results returned will change?

Comment 4

[Ryan Doherty (:rdoherty)]

(In reply to comment #3)
> It's great to see the new server coming up!
> 
> Does that mean that my queries to getdata.cgi and the results returned will
> change?

Yeah, it means they won't work anymore. We've been trying to publicize the new API for the past month, but we've probably missed some people. If you know of people who should be contacted, let us know.

Let me know if we can be of help (#perfomatic).

Comment 5

[Jeremy Orem [:oremj]]

Any IT action needed on this bug?

Comment 6

[Johnathan Nightingale [:johnath]]

No, I think we can close it off now that the firewall block has been lifted.  The work I'll do to use the new graph server doesn't really need a bug to track it.