Closed Bug 1238052 Opened 9 years ago Closed 9 years ago

reviewboard2.webapp.scl3.mozilla.com:Swap is CRITICAL: SWAP CRITICAL - 13% free (260 MB out of 2047 MB)

Categories

(MozReview Graveyard :: General, defect)

Product:
MozReview Graveyard
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mlankford, Unassigned)

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2424] )

9:00 AM <@nagios-scl3> Fri 09:00:18 PST [5163] reviewboard2.webapp.scl3.mozilla.com:Swap is CRITICAL: SWAP CRITICAL - 13% free (260 MB out of 2047 MB) (http://m.mozilla.org/Swap)
[root@reviewboard2.webapp.scl3 ~]# free -m total used free shared buffers cached Mem: 3832 3689 143 0 0 19 -/+ buffers/cache: 3668 163 Swap: 2047 1362 685 [root@reviewboard2.webapp.scl3 ~]#
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2424]
Assignee: server-ops-webops → nobody
Component: WebOps: Other → MozReview
Product: Infrastructure & Operations → Developer Services
QA Contact: smani
High IO wait on server as well. [root@reviewboard2.webapp.scl3 ~]# top top - 17:30:09 up 342 days, 23:09, 1 user, load average: 1.61, 1.48, 1.51 Tasks: 158 total, 2 running, 156 sleeping, 0 stopped, 0 zombie Cpu(s): 2.0%us, 4.1%sy, 0.0%ni, 43.6%id, 49.8%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 3924636k total, 3776392k used, 148244k free, 2396k buffers Swap: 2097148k total, 1826380k used, 270768k free, 34800k cached ================= root@reviewboard2.webapp.scl3 ~]# grep ' 500 ' /var/log/httpd/reviewboard.mozilla.org/access_log "124.35.199.249, 124.35.199.249, 124.35.199.249" - - [08/Jan/2016:07:25:45 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=949703 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=949703" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "148.122.175.42, 148.122.175.42, 148.122.175.42" - - [08/Jan/2016:09:13:30 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "118.163.10.190, 118.163.10.190, 118.163.10.190" - - [08/Jan/2016:09:14:59 +0000] "PUT /api/review-requests/22565/draft/ HTTP/1.1" 500 245 "https://reviewboard.mozilla.org/r/22565/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" "118.163.10.190, 118.163.10.190, 118.163.10.190" - - [08/Jan/2016:09:19:23 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" "118.163.10.190, 118.163.10.190, 118.163.10.190" - - [08/Jan/2016:09:37:29 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" "118.163.10.190, 118.163.10.190, 118.163.10.190" - - [08/Jan/2016:09:39:01 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0" "118.163.10.190, 118.163.10.190, 118.163.10.190" - - [08/Jan/2016:09:39:38 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" "118.163.10.190, 118.163.10.190, 118.163.10.190" - - [08/Jan/2016:09:43:52 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" "59.115.37.82, 59.115.37.82, 59.115.37.82" - - [08/Jan/2016:09:46:18 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" "27.147.44.12, 27.147.44.12, 27.147.44.12" - - [08/Jan/2016:09:59:19 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/45.0.2454.85 Chrome/45.0.2454.85 Safari/537.36" "159.245.16.100, 159.245.16.100, 159.245.16.100" - - [08/Jan/2016:10:13:23 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1205843 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1205843" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" "192.200.255.237, 192.200.255.237, 192.200.255.237" - - [08/Jan/2016:12:36:10 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=949703 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=949703" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "111.93.218.4, 111.93.218.4, 111.93.218.4" - - [08/Jan/2016:13:17:18 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1231243 HTTP/1.1" 200 500 "https://bugzilla.mozilla.org/show_bug.cgi?id=1231243" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:46.0) Gecko/20100101 Firefox/46.0" "89.202.203.51, 89.202.203.51, 89.202.203.51" - - [08/Jan/2016:13:39:38 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" "37.165.50.218, 37.165.50.218, 37.165.50.218" - - [08/Jan/2016:13:45:42 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0" "5.51.133.172, 5.51.133.172, 5.51.133.172" - - [08/Jan/2016:13:46:23 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" "89.202.203.51, 89.202.203.51, 89.202.203.51" - - [08/Jan/2016:14:20:45 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36" "87.126.70.142, 87.126.70.142, 87.126.70.142" - - [08/Jan/2016:14:36:57 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1231243 HTTP/1.1" 200 500 "https://bugzilla.mozilla.org/show_bug.cgi?id=1231243" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:46.0) Gecko/20100101 Firefox/46.0" "89.202.203.51, 89.202.203.51, 89.202.203.51" - - [08/Jan/2016:14:51:40 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" "104.58.236.72, 104.58.236.72, 104.58.236.72" - - [08/Jan/2016:16:00:40 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1208371 HTTP/1.1" 500 250 "https://bugzilla.mozilla.org/show_bug.cgi?id=1208371" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0" "189.78.55.86, 189.78.55.86, 189.78.55.86" - - [08/Jan/2016:17:27:33 +0000] "GET /api/extensions/mozreview.extension.MozReviewExtension/summary/?bug=1231243 HTTP/1.1" 200 500 "https://bugzilla.mozilla.org/show_bug.cgi?id=1231243" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" [root@reviewboard2.webapp.scl3 ~]# free -m total used free shared buffers cached Mem: 3832 3716 116 0 1 28 -/+ buffers/cache: 3686 146 Swap: 2047 1741 306
Things are mostly back to normal. I suspect this was related to smacleod performing some expensive queries in the web-based admin interface. He reported they were timing out and the alerts happened around the same time.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Version: other → unspecified
Product: Developer Services → MozReview
You need to log in before you can comment on or make changes to this bug.