minimize json2 memory usage

RESOLVED FIXED

Status

Webtools Graveyard
Tinderbox
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: rhelmer, Assigned: rhelmer)

Tracking

Details

Attachments

(2 attachments)

(Assignee)

Description

8 years ago
json2 was backed out after being put into production due to excessive memory usage (bug 588315). Let's see where it's taking so much memory and make it better.
(Assignee)

Updated

8 years ago
Blocks: 586123
(Assignee)

Comment 1

8 years ago
Created attachment 467212 [details] [diff] [review]
print each entry instead of building data structure

Print out the data as we go instead of building a (potentially very large) data structure.

Unfortunately this means we're back to building JSON output by hand but only for the begin and end, still using JSON->encode() as much as possible.
Attachment #467212 - Flags: review?(bear)
(Assignee)

Comment 2

8 years ago
To confirm this is fixed, I am thinking we could just sample memory usage on tinderbox-stage while hitting it, something like:

while true; do sleep 0.5; ps -e -o rss,cmd; done

We could get more hardcore if we really want to get peak memory usage for each process, and make sure the process is really the one we want etc

However if this is really what caused a problem on production, it should be quite obvious from sampling before/after any fixes while load is happening on tinderbox-stage, whether the problem is getting better or not.

Updated

8 years ago
Attachment #467212 - Flags: review?(bear) → review+
(Assignee)

Comment 3

8 years ago
Checking in tbglobals.pl;
/cvsroot/mozilla/webtools/tinderbox/tbglobals.pl,v  <--  tbglobals.pl
new revision: 1.80; previous revision: 1.79
done
Checking in showbuilds.pl;
/cvsroot/mozilla/webtools/tinderbox/showbuilds.pl,v  <--  showbuilds.pl
new revision: 1.44; previous revision: 1.43
done
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
(Assignee)

Comment 4

8 years ago
Created attachment 476700 [details] [diff] [review]
only print leading comma if buildname is set

I just noticed this testing production (which has attachment 467212 [details] [diff] [review]); I haven't been able to reproduce this in dev yet but I am pretty sure this is because load_buildlog is putting in these fake entries that we don't want, and adding the comma should be after we check for this and call "next;" not before.
Attachment #476700 - Flags: review?(bear)
(Assignee)

Updated

8 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

8 years ago
Attachment #476700 - Flags: review?(bear) → review+
(Assignee)

Comment 5

8 years ago
Checking in tbglobals.pl;
/cvsroot/mozilla/webtools/tinderbox/tbglobals.pl,v  <--  tbglobals.pl
new revision: 1.81; previous revision: 1.80
done
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.