634580 - Store endurance tests metrics on disk instead of in memory

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Currently the endurance tests create a result set of the metrics in memory. Seeing as we're gathering metrics such as memory allocation, we need to make sure than the endurance tests don't adversely affect the metrics we're gathering.

This was discussed on IRC. The transcript can be found here: http://etherpad.mozilla.com:9000/endurance-tests-10-01-19

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Each time we push updates to the results member of the endurance object, which gets transferred via JSBridge to the CLI, we should empty this array from the Python side and store in on disk. This probably has to wait until the rewrite of the automation script happened later this month or early March.

Comment 2

[Matt Evans [:mevans]]

I just saw this in the 1:1 notes for dhunt. We should have captured this yesterday during the endurance test meeting. This is probably more important than getting more metrics recorded.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Matt, I have thought about it yesterday but it's not that much in our hands. I think it should be possible in some way to have a listener attached for persisted object transfers in our subclassed Mozmill class, but I'm not sure how complicated it is. Or if we have to wait for Mozmill 2.0. That's why I haven't mentioned it.

I will CC Clint and Jeff and hope at least one of those smart guys could give us more insight what we could do in a short term or not.

Comment 4

[Jeff Hammel]

This is a bit vaguely defined for me to really give a good pointer to right now.  My suggestion from the pastebin (take as a cartoon, not working code to be copy+pasted) is still what I would recommend for the 1.5 series. I can help figure out what exactly to do if there are more specific questions.

Comment 5

[cmtalbert]

(In reply to comment #3)
> Matt, I have thought about it yesterday but it's not that much in our hands.
> I think it should be possible in some way to have a listener attached for
> persisted object transfers in our subclassed Mozmill class, but I'm not sure
> how complicated it is. Or if we have to wait for Mozmill 2.0. That's why I
> haven't mentioned it.
> 
> I will CC Clint and Jeff and hope at least one of those smart guys could
> give us more insight what we could do in a short term or not.
Why does this need to be in mozmill?  You are in chrome JS, nothing stops you from opening a file and dumping the data to that file.  Do you want it in mozmill so that it's easier to upstream into the reporting?  With 2.0, you can use your own reporting handler that knows to look for this file, that's the beauty of 2.0.  For 1.5 we'd have to hard code some mechanism to know the file name or add yet another command parameter.

Can you spec out what you want?  Do you really need some kind of "writeFile" API into the JS side of mozmill?  This feels like a very one off sort of thing, so I think it'd be better in a shared module.  One requirement I have is that I really don't want to put any parsing data into Mozmill for this file, so I'd like to just try to treat this thing as a blob and we upload it into the results POST.  (which means it should probably either be encoded or JSON).

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Clint, please see my comment 3. I don't propose to get it added to Mozmill. I just want to know if we have the possibility to listen for incoming changes of the persisted data object. If we have that, our automation scripts will handle storing that data. Nothing Mozmill has to do.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dave, time has been passed by. Is this still necessary?

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I have no reason to suspect this is no longer necessary, however I'm not sure it will fit into out priorities any time soon.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We are no longer actively running or maintaining the endurance tests.