Closed Bug 991832 Opened 12 years ago Closed 10 years ago

CDN logs

Categories

(Cloud Services :: Operations: Miscellaneous, task)

Product:
Cloud Services
Component:
Operations: Miscellaneous
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: catlee, Assigned: oremj)

Details

Having logs from our CDNs would really help to analyze our current usage, and where we can make optimizations.
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/135]
Assignee: server-ops-webops → nmaul
Working on this... biggest problem right now is just the sheer volume of data. The way we do updates right now (Firefox makes many range requests for one update) means we get many log entries for one update. It's not a completely insurmountable amount of data, but the CDNs don't deliver it very nicely. For one, both main vendors (Edgecast and Akamai) need their logs sorted... they're not usable out of the box.
AWstats is a bust... just way too much data to process. Webalizer is barely working... processing 9/2's data for just Edgecast (not Akamai or Highwinds) took around 17 hours. I don't think this is likely to work very well in general. It's also pretty static analysis- can't drill down into individual days, or do any sort of filtering or searching or anything in real-time. It generates a set of static HTML files, and that's all you get. Playing around with doing this in Sumologic instead.
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/135] → [kanban:https://webops.kanbanize.com/ctrl_board/2/52]
I spoke with :catlee and this is still "nice to have", so triaging over to CloudOps as they're primary for CDN these days (and have access to all sorts of logstuff that might be useful).
Component: WebOps: Product Delivery → Operations
Product: Infrastructure & Operations → Mozilla Services
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/52]
Assignee: nmaul → oremj
What types are data are you looking for?
Flags: needinfo?(catlee)
Counts of hits and bytes transferred for different update MARs
Flags: needinfo?(catlee)
We can typically get that information from the reporting interfaces of the CDN providers. If you want to go down the log processing route, you are looking at terabytes of logs, each CDN in a different format with a different method of retrieving them.
Do you want me to set up a log transfer, from akamai and cloudfront, or should I close this out? Popular objects are listed in their reporting interfaces, so I can grab you some data on demand if you don't want the logs.
Flags: needinfo?(catlee)
Close this out for now, thanks. I'll request more specific data if I need in the future.
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(catlee)
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.