Checkin and run "top20" ispdb log analyzer

RESOLVED FIXED

Status

Mozilla Messaging
Server Operations
P1
critical
RESOLVED FIXED
8 years ago
8 years ago

People

(Reporter: davida, Assigned: gozer)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

8 years ago
Created attachment 431898 [details]
v1

gozer, can you put somewhere appropriate in hg/svn the following script, and run it nightly against the bunzip'ed logs, and email its output to the conversion mailing list (for now)?

(also, we should add blake to that list if he's not on it already).

On the data file you sent me, it generates:

sleet:ipdblogs davida$ python quickparse.py autoconfig-20100217.log 
HITS: 195 domains, accounting for 37880 successes, or 29.7% success rate
MISSES: 29974 domains, accounting for 89652 failures, or 70.3% fail rate
Top 20 misses:
  mail.ru (1466 hits, aka 4.9%)
  online.de (1191 hits, aka 3.9%)
  yandex.ru (971 hits, aka 3.2%)
  alice.it (739 hits, aka 2.4%)
  wp.pl (718 hits, aka 2.4%)
  libero.it (664 hits, aka 2.2%)
  seznam.cz (475 hits, aka 1.6%)
  neuf.fr (454 hits, aka 1.5%)
  bluewin.ch (432 hits, aka 1.4%)
  tiscali.it (424 hits, aka 1.4%)
  msn.com (368 hits, aka 1.2%)
  sfr.fr (365 hits, aka 1.2%)
  o2.pl (356 hits, aka 1.2%)
  cox.net (334 hits, aka 1.1%)
  ewetel.net (330 hits, aka 1.1%)
  aon.at (314 hits, aka 1.0%)
  sbcglobal.net (311 hits, aka 1.0%)
  rambler.ru (296 hits, aka 1.0%)
  ntlworld.com (296 hits, aka 1.0%)
  charter.net (262 hits, aka 0.9%)
adding all 20 would boost our HIT rate by 35.7%


which is useful, IMO =).

We can then tweak the script to answer more questions we may have.  Let me know where that is, as I'll likely want to push patches.
(Assignee)

Comment 1

8 years ago
Committed revision 64061.

Script is in [svn.mozilla.org]/mozillamessaging.com/sites/ispdb.mozillamessaging.com/trunk/tools/quickparse.py
(Assignee)

Comment 2

8 years ago
Quick run on yesterdays logs:

HITS: 176 domains, accounting for 38816 successes, or 29.4% success rate
MISSES: 32040 domains, accounting for 93295 failures, or 70.6% fail rate
Top 20 misses:
  mail.ru (1760 hits, aka 5.5%)
  yandex.ru (1063 hits, aka 3.3%)
  libero.it (811 hits, aka 2.5%)
  alice.it (805 hits, aka 2.5%)
  wp.pl (729 hits, aka 2.3%)
  seznam.cz (452 hits, aka 1.4%)
  bluewin.ch (440 hits, aka 1.4%)
  aon.at (380 hits, aka 1.2%)
  sfr.fr (365 hits, aka 1.1%)
  o2.pl (351 hits, aka 1.1%)
  neuf.fr (337 hits, aka 1.0%)
  rambler.ru (332 hits, aka 1.0%)
  cox.net (325 hits, aka 1.0%)
  tiscali.it (310 hits, aka 1.0%)
  versanet.de (303 hits, aka 0.9%)
  msn.com (278 hits, aka 0.9%)
  btinternet.com (262 hits, aka 0.8%)
  sbcglobal.net (257 hits, aka 0.8%)
  skynet.be (251 hits, aka 0.8%)
  ntlworld.com (234 hits, aka 0.7%)
adding all 20 would boost our HIT rate by 31.2%

real    38m48.731s
user    37m43.982s
sys     1m28.261s
(Assignee)

Comment 3

8 years ago
I'll set it up for nightly run starting tonight.
(Assignee)

Updated

8 years ago
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.