993912 - Configure directory links servers for logging and connecting of data/flow

Status: RESOLVED FIXED

(Content Services Graveyard :: Tiles: Ops, defect)

Description

[Ed Lee :Mardak]

Servers from bug 993911 need to be configured and connected for the data we want to analyze. oyiptong?

Comment 1

[Olivier Yiptong [:oyiptong]]

The infrastructure architecture will be very similar to what is currently deployed on Services infrastructure.

We will have a few kinds of servers:

* user-facing nodes which serve HTTP
* a central logging node which collects data from the user-facing nodes
* a few utility nodes that read from the central logging server and ship data to various backends

The backends will include Amazon Services, such as S3, RedShift and a few custom ones, including ElasticSearch, and other health monitoring services

Comment 2

[Olivier Yiptong [:oyiptong]]

The architecture has been slightly modified after conversations with Services Ops.

We are

Comment 3

[Olivier Yiptong [:oyiptong]]

The architecture for data flow has been slightly modified after conversations with Services Ops.

It looks as follows for business metrics:

* user-facing nodes which serve HTTP
* user-facing nodes send periodically rolled-over logs to Amazon S3
* back-end server pulls data from S3 and stores them into DDFS (a distributed file system)
* periodic jobs are run via Disco and Inferno (tools implementing mapreduce workflow) to compile aggregate metrics and store them in our data warehouse: Amazon RedShift
* an internal tool is used to view and download metrics