657676 - Requesting access to Sync load testing machines to test using Vertica as alternate DB backend

Status: RESOLVED INCOMPLETE

(Cloud Services :: Server: Other, defect)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

While I feel this testing is worthwhile, I understand that it likely doesn't take any priority over existing work so I am filing it as minor.

I would like to have some hardware equivalent to what Sync runs in the scl2 datacenter to test out Vertica performance in backing Sync.  On optimal hardware, my estimates based on my understanding of Sync's architecture is that we would see at least an order of magnitude increase in user storage density and in analytic queries while still keeping the OLTP queries as fast or faster.
The SCL2 hardware is not optimal though so I don't hope to get quite that much of a boost. I believe that it might still be considerable and worth evaluating though.

To do this, I just need 3 to 5 machines set aside so we can deploy Vertica and point the Sync service at it.  The SQL DML changes should be minimal. I could either deploy and configure myself or I can provide documentation on how to do so.

Background info:

Metrics uses Vertica for our data warehouse.  It is a clustered column store database.  While proprietary, Mozilla has a permanent license with a 3 year unlimited storage license.  At the end of the 3 years, we can either renew as unlimited, or we can "certify" at our current usage level for our permanent license.

Comment 1

[Jeff Vier [:jvier]]

As we discussed in the morning meeting, this is going to be fairly low priority until after we get load testing done with our existing configs in the load test env.

Comment 2

[:Atoll]

Load testing is done with existing configs.  :dre, has the sync database architecture been ported to Vertica?

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

No, I have not yet looked at any of the Sync server code to see what changes are needed.  I would be happy to take a crack at that.  Could someone give me a pointer at the right place to start? I haven't yet had the pleasure of working on any Sync code, new or old.

Comment 4

[Mike Connor [:mconnor]]

http://hg.mozilla.org/services/server-storage/ is the relevant code.

That said, at this time this is far from urgent, given that we have tons of capacity and spare hardware.  That said, if it works out well, we may want to look at this for the next datacenter we build...

Comment 5

[:Atoll]

This has been idle for a couple quarters, marking as INCOMPLETE. If anyone wishes to take action, feel free to reopen and claim it.