1024782 - Something making stage very slow

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Selena Deckelmann :selenamarie :selena]

16:41 < nagios-phx1> | Thu 16:41:55 PDT [1986] socorro1.stage.db.phx1.mozilla.com:Ganglia - PostgreSQL Last Reports Update is CRITICAL: 
          CHECKGANGLIA CRITICAL: last_record_reports is 22121.00 (http://m.mozilla.org/Ganglia+-+PostgreSQL+Last+Reports+Update)

We removed ceph, but we're still slow. Need to get to the bottom of the mystery!

Comment 1

[K Lars Lohn [:lars] [:klohn]]

If I understand correctly, this alert is about the processors not having put anything in PG for an alarming amount of time.  No where in the processor logs to for this date do I see any gap in at all.  I call bogus alert.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

Attachment: a LibreOffice spread sheet of hours collected vs moved crashes for the last two weeks.

it appears that Ceph slows down the crash movers quite a bit, especially starting on June 4.  

the June 12 spike at the end is when I removed Ceph from the crashmover...

Comment 3

[Laura Thomson :laura]

What I see in these numbers is crash movers generally keeping up fine, with a couple of blips in perf that recovered. 

The true degradation starts June 4. A bunch of changes happened then - a number of config changes were made on stage and code landed too. I went through them, and here are things that jumped out at me:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1015187 add "FILE_UPLOAD_PERMISSIONS" setting to config (changes tested on stage)
- https://bugzilla.mozilla.org/show_bug.cgi?id=1010327 Permission to delete from /mnt/socorro/symbols_upload/ on stage
- https://bugzilla.mozilla.org/show_bug.cgi?id=1019795 stage auto deploy has stopped working  

I also noted that was when Lars landed his benchmarking code for crash storage. Are we benchmarking on stage? 
- https://bugzilla.mozilla.org/show_bug.cgi?id=1011013 benchmarking crash store

Comment 4

[K Lars Lohn [:lars] [:klohn]]

the landing of the Benchmarking code actually happened in mid-May - I do not know why lonnen switched the milestones on a closed bug on both 5/28 & 6/3.  

the use of the Benchmarking code spanned 2014-05-15 11:52:13,159 - 2014-05-25 03:11:12,996.  We were not benchmarking during this degradation of crashmover performance.

Benchmarking was an ad hoc change, rather than committed to config. Once we verified the numbers for Ceph storage, the plan was to allow puppet to revert the config.  Apparently puppet did not revert for ten days.  Bug 1019795 about auto deploy on stage happened then and I suspect that the benchmarking was lost in staging config at that point.  I've checked and made sure there was not a code regression at that point, too.

Comment 5

[Lonnen :lonnen]

The week of May 11-16 I put code into production on May 15, shortly before the fix for 1011013 landed. The next time we released code into production was Jun 06. 

I targeted this to 87 initially, which was the next milestone at the time. We discussed tagging every week whether or not something landed to make planning easier, so 87 was tagged empty and I moved this to 88.

Comment 6

[Lonnen :lonnen]

Retargetting happened as part of the usual mass retargetting that accompanies tagging.