1329431 - jobqueue[12] using too much memory

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Ludovic Hirlimann [:Usul]]

<nagios-scl3> Sat 08:40:10 PST [5587] jobqueue2.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 49% free (987 MB out of 2047 MB) (http://m.mozilla.org/Swap)
<nagios-scl3> Sat 08:41:10 PST [5589] jobqueue1.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 49% free (985 MB out of 2047 MB) (http://m.mozilla.org/Swap)

I ran sudo swapoff -a and then sudo swapon -a on both nodes and they recovered.

Comment 1

[Ashlee Nguyen [:ashlee]]

They're still happening:

Fri 19:23:47 PST [5187] jobqueue1.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 41% free (833 MB out of 2047 MB)

Fri 19:23:57 PST [5189] jobqueue2.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 41% free (832 MB out of 2047 MB)

Recovered after sudo swapoff -a then sudo swapon -a

Can this be looked into further?

Comment 2

[Salvador Espinoza [:sal]]

happened today 

@nagios-scl3> Sat 13:20:37 PST [5899] jobqueue1.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 45% free (914 MB out of 2047 MB) (http://m.mozilla.org/Swap)
13:45:46 <@nagios-scl3> Sat 13:45:45 PST [5930] jobqueue1.bugs.scl3.mozilla.com:Swap is OK: SWAP OK - 99% free (2023 MB out of 2047 MB) (http://m.mozilla.org/Swap)

Comment 3

[Peter Radcliffe [:pir]]

:ashlee if you're reporting high swap usage you need to report what is using all the swap...

Comment 4

[Kendall Libby [:fubar] (he/him)]

On the jobqueue nodes, this is almost certainly related to BMO enabling HTML email by default. Moving to correct component and CC'ing devs.

Dave/Dylan, any thoughts? IIRC, the jobqueue nodes already have quite a lot of RAM so adding more seems like the wrong choice. Is there any way we can tweak how we're generating/sending messages?

Comment 5

[Ludovic Hirlimann [:Usul]]

Thu 05:46:27 PST [5395] jobqueue1.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 41% free (836 MB out of 2047 MB) (http://m.mozilla.org/Swap)

jobqueues.pl are eating the swap.

Comment 6

[Ludovic Hirlimann [:Usul]]

I swapoff swapon to clear swap

Comment 7

[Salvador Espinoza [:sal]]

This alerted today, swapoff swapon cleared 

@nagios-scl3> Thu 16:09:48 PST [5507] jobqueue1.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 47% free (949 MB out of 2047 MB) (http://m.mozilla.org/Swap)

Comment 8

[Ludovic Hirlimann [:Usul]]

Fri 04:42:46 PST [5326] jobqueue1.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 41% free (819 MB out of 2047 MB) (http://m.mozilla.org/Swap)

Comment 9

[Ludovic Hirlimann [:Usul]]

same process - same solution.

Comment 10

[Ludovic Hirlimann [:Usul]]

Is this fixed ?

Comment 11

[Kendall Libby [:fubar] (he/him)]

Graphs look good and there haven't been any swap alerts in a while. Closing.