822414 - Stage postgres is logging lots of errors

Status: RESOLVED WORKSFORME

(Data & BI Services Team :: DB: MySQL, task)

Description

[Selena Deckelmann :selenamarie :selena]

We're seeing tons of errors being logged on postgres. 

Resolved one issue with index errors. Not sure what else is causing problems.

Comment 1

[Selena Deckelmann :selenamarie :selena]

Other actions taken: 

*

Comment 2

[Selena Deckelmann :selenamarie :selena]

Symptoms were:
2012-12-17 00:00:01.403 GMT,,,32238,,50ce55fa.7dee,4022,,2012-12-16 23:15:06 GMT,,0,ERROR,XX000,"xlog flush request ED8/6AC35210 is not satisfied --- flushed only to ED8/52834618",,,,,"writing block 0 of relation base/16408/49884",,,,""
2012-12-17 00:00:01.403 GMT,,,32238,,50ce55fa.7dee,4023,,2012-12-16 23:15:06 GMT,,0,WARNING,58030,"could not write block 0 of base/16408/49884","Multiple failures --- write error might be permanent.",,,,,,,,""

Side note: We should be monitoring for stuff like this!


Other actions I took to diagnose: 

* REINDEX reports_20121210
* Truncated jobs and reports_20121210 (WITH CASCADE)
At this point, the buffer errors stopped being reported

* Truncated priorityjobs (which we couldn't delete anything from)

After the truncate to priorityjobs, processors were able to start back up again.

Possible issues could include: 

* RAM errors - should run a RAM check on this system
* Disk errors - should run a disk diagnostic to detect bad blocks
* Restore of stage database over the weekend was missing WAL or data of some kind

Needs further work to get to the bottom of it.

Comment 3

[Selena Deckelmann :selenamarie :selena]

We've replaced this database a couple times and not experienced the same errors. 

Closing for lack of actionable errors!