884177 - add simple instrumentation to process_bug.cgi and show_bug.cgi

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[:glob ✱]

i need to add simple instrumentation to process_bug.cgi to capture what's happening with the recent slowness.

i'm planning on capturing the following and sending it to the syslog so we can monitor the data in real-time:
  bug-id
  user-id
  number of bugmail recipients
  total time
  preamble time (everything up to $bug->update)
  update time
  bugmail time
  template time

Comment 1

[:glob ✱]

Committing to: bzr+ssh://[e-mail]/bmo/4.2/
modified process_bug.cgi
modified Bugzilla/Bug.pm
Committed revision 8847.


this outputs the following data to syslog:
  [timings]
  bug-id
  user-id
  total-time
  validation-time
  update-time
  db-time
  bugmail-time
  template-time
  recipient-count

times are in seconds.

Comment 2

[Corey Shields [:cshields]]

Could these timings be sent to Graphite instead of syslog?  Such timers have been immensely helpful with AMO and other sites.  You could poke #webdev on how they do it.

Comment 3

[:glob ✱]

(In reply to Corey Shields [:cshields] from comment #2)
> Could these timings be sent to Graphite instead of syslog?  Such timers have
> been immensely helpful with AMO and other sites.  You could poke #webdev on
> how they do it.

great idea - filed as bug 884442.

Comment 4

[:glob ✱]

after seeing weird results from production (negative time differences), investigation revealed that i should be using clock_gettime(CLOCK_MONOTONIC) instead of time().

Committing to: bzr+ssh://[e-mail]/bmo/4.2/
modified process_bug.cgi
modified show_bug.cgi
Committed revision 8848.

this revision:
  - changes from time() to CLOCK_MONOTONIC
  - splits the first step in process_bug into load-bug and mid-air detection
  - adds timing to show_bug

the fields are now:

process_bug
  bug-id
  user-id
  total-time
  load-bug-time
  mid-air-time
  update-objects-time
  update-database-time
  bugmail-generation-time
  template-time
  bugmail-recipient-count

show_bug
  bug-id
  user-id
  total-time
  login-time
  bug-load-time
  bug-preload-time
  template-time