Please clean up /Users/cltbld/Library/Caches/TemporaryItems on bld-lion-r5-* hosts

RESOLVED WONTFIX

Status

task
--
trivial
RESOLVED WONTFIX
6 years ago
6 years ago

People

(Reporter: armenzg, Assigned: dustin)

Tracking

Details

Attachments

(2 attachments)

Reporter

Description

6 years ago
ted will be fixing the root cause of bug 880003.

We would like to place a safeguard in case in the future a new test starts producing something similar.

Thanks!
How's this?
Assignee: relops → dustin
Attachment #792268 - Flags: review?(armenzg)
Reporter

Comment 2

6 years ago
Comment on attachment 792268 [details] [diff] [review]
bug906706.patch

Cool!
It's good to see that there was something similar already in place.

Would it remove the actual "TemporaryItems" directory? or would it leave it alone?
If it does, I think it would be problematic but I don't know.
Attachment #792268 - Flags: review?(armenzg) → review+
It won't - just files under it.
Attachment #792268 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Backed out due to bug 906782
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Re-landed at Tue Aug 20 06:26:07 2013 -0700

Which was about 20 minutes *after* use1 took a nose-dive (bug 907158), so the two are unrelated (that, or the use1 failure caused me to land the patch, which seems unlikely).

Load on the puppetmasters seems normal.
..and backed out again, once I saw ruby processes using a lot of RAM on the scl3 puppet master.  I bet this rule is feeding 10,000 filenames back to the puppetmaster, which then wastes a bunch of RAM iterating over that list.

The increased load from the AWS failure isn't helping.

We should switch to using rm -rf.
(and I should add, since I was watching the puppet masters, this didn't cause any further harm than a few puppet runs failing - a normal occurrence)
Actually I'm going to block this on bug 712206, which sets up a system for running pre-flight tasks on slaves.  That will be the better place to do this cleanup.
Depends on: 712206
Severity: normal → trivial
(In reply to Dustin J. Mitchell [:dustin] from comment #8)
> Actually I'm going to block this on bug 712206, which sets up a system for
> running pre-flight tasks on slaves.  That will be the better place to do
> this cleanup.

We could, in the mean time do this with |tidy {}| though, which might make even more sense than recurse+purge

http://docs.puppetlabs.com/references/latest/type.html#tidy
(In reply to Justin Wood (:Callek) from comment #9)
> (In reply to Dustin J. Mitchell [:dustin] from comment #8)
> > Actually I'm going to block this on bug 712206, which sets up a system for
> > running pre-flight tasks on slaves.  That will be the better place to do
> > this cleanup.
> 
> We could, in the mean time do this with |tidy {}| though, which might make
> even more sense than recurse+purge
> 
> http://docs.puppetlabs.com/references/latest/type.html#tidy

errrrrrr, ignore meeeeee. caffeine y u no work.
Reporter

Comment 11

6 years ago
(In reply to Dustin J. Mitchell [:dustin] from comment #8)
> Actually I'm going to block this on bug 712206, which sets up a system for
> running pre-flight tasks on slaves.  That will be the better place to do
> this cleanup.

If I cleaned all bld-lion machines at once, would you be able to re-land the puppet change?
There should not be too many files on boot to remove if they have been cleaned once.

Fixing bug 712206 would take a while before it will happen.
Flags: needinfo?(dustin)
That makes me nervous, but I'm willing to give it a try.
Flags: needinfo?(dustin)
Have you cleaned them all?
Flags: needinfo?(armenzg)
Reporter

Comment 14

6 years ago
I can do it tomorrow morning.
Would that work for you?
Sounds fine - just let me know.  I don't want to redeploy before that!
Reporter

Comment 16

6 years ago
I've cleaned *all* of the bld-lion-r5-* machines with this:
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-1*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-2*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-3*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-4*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-5*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-6*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-7*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-8*
rm -rf /Users/cltbld/Library/Caches/TemporaryItems/cpp-unit-profd-9*

If you could please go ahead and deploy the puppet change it will be great.
Flags: needinfo?(armenzg)
Landed again.  I'll watch load on the masters for a bit.
Posted image mem.png
..and backed out again.  The spike is only on releng-puppet2.srv.releng.scl3, not on releng-puppet1 - curious, because those nodes should be sharing load.
That's graphite being stupid.  Both scl3 masters pegged their RAM.
I think we need to just do bug 712206.
Reporter

Comment 21

6 years ago
Let's then wontfix this.
Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → WONTFIX
Reporter

Comment 22

6 years ago
Would removing the directory and re-creating do the trick?
I don't know - I'm not sure why this particular patch causes problems.  You suggested that maybe it's necessary on other hosts than those you cleaned.  Another possibility is that recursive cleans just eat memory regardless of the number of files to be removed.
You need to log in before you can comment on or make changes to this bug.