980110 - Reduce the granularity of the Zeus load balancer time stamps.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Allison Naaktgeboren :ally]

Please feel free to morph bug if this is not the correct component. 

The justifying bug is locked, please let us know who we should add to the cc list for the discussion & rationale.

Comment 1

[Jake Maul [:jakem]]

I'm adding the right people to the CC list here, so you can just copy it over to bug 682472. Thanks!

Comment 2

[Jason Crowe [:jd]]

Hi,

We still do not have access to the bug this bug refers to. You need to add us to the cc list for that bug before we can see that bug. We cannot make any comment or decision on the issue if we can not see the bug explaining the issue. Please add us to the cc list as requested two weeks ago.

Thank You

Comment 3

[Allison Naaktgeboren :ally]

done.

Comment 4

[Allison Naaktgeboren :ally]

In the future, please use the NEEDINFO flag so we know you need something.

Comment 5

[Chris Turra [:cturra]]

:Ally - i gave bug 682472 a quick read and it wasn't obvious to me what the timestamp granularity currently is and what you'd like it to be as a result of this bug?

generally, we would prefer we send the raw data, which could be parse as it comes in by the receiving application.

Comment 6

[Tom Lowenthal]

The desired timestamp granularity is one where any particular recorded timestamp appears on quite a lot of log entries, and is no use at all as a distinctive partial pseudo-identifier for among entries.

Comment 7

[Allison Naaktgeboren :ally]

:StrangeCharm, you filed the original bug. So, do you think there is no longer a problem?

Comment 8

[Jason Crowe [:jd]]

As I understand from comment 6 this is not an issue due to the large volume of entries in the Zeus logs.

I am going to close this out as wontfix, but please do not hesitate to reopen if I am mistaken.

Regards

Comment 9

[Tom Lowenthal]

Jason: what's the anonymity set of a particular Zeus log entry?

Comment 10

[Jason Crowe [:jd]]

I have no clue what you mean. The words 'anonymity set' have no meaning to me. Can you explain this?

Comment 11

[Tom Lowenthal]

An anonymity set refers to the number of other entries/users/&c. sufficiently similar to a particular record, that it can be relatively indistinguishably lost among them.

Comment 12

[Tom Lowenthal]

The goal of this bug is to reduce the granularity of the logs so much that if I had a copy of the loadbalancer logs and knew the exact time and user-agent of a particular request, those details would match so many entries that I couldn't have any practical hope of guessing which one might be the log entry for the time & UA I'm thinking of.

What info is currently in each log entry? How precise is the time field? If I know a time and UA, how many log entries would be a close match?

Comment 13

[Jason Crowe [:jd]]

Tom,

Can we set up a vidyo call to facilitate moving this forward? Please let me know when is a good time for you.

Thanks

Comment 14

[Jake Maul [:jakem]]

I have reduced the timestamps for logs on incoming crash dumps (crash-reports.mozilla.com, port 443) by removing the "seconds" field.

30/Jun/2014:16:10 PDT

Instead of:

30/Jun/2014:16:10:43 -0700

(for those curious, due to some sort of bug in Zeus, I can't actually get it to say the timezone the same way... best I can do is the timezone abbreviation)


The statement we're trying to fix (from bug 682472) is:

"...there's a problem as long as it's practical to look at a crash report, and look at Zeus logs, and reasonably conclude that a Zeus entry corresponds to the submission of the crash report."

Previously (with seconds in the timestamp), there is usually 15-25 crashdumps reported per second, although sometimes there are as few as 1 or as many as 50. This already makes it fairly difficult to correlate, although in rare cases it is still possible.

With seconds omitted, there are always at least 400 crashdumps per minute... usually around 1000-1500. This is well beyond being practical to correlate without other information.