1042247 - Consolidate compression logic in get_about_memory.py

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Eric Rahm [:erahm]]

|get_about_memory.py| gathers a fair amount of logs. Some of them we compress as xz, some of them we compress as gz, some of them we don't compress at all, some of them we decompress, process, and recompress as gz.

I have 3 proposals:

#1 - Client side: we standardize on having all logs uncompressed

Rationale: We actually want to read these logs. Also if we upload them to bugzilla they end up double compressed (at least if they're gz).

Current state:
  - DMD logs are gz compressed, 1 per process
  - GC/CC logs are xz compressed, 1 per process
  - memory-report is uncompressed
  - various other logs are uncompressed, but we don't generate them

#2 - Device side: All generated logs should be compressed in the same format (so dmd, gc/cc, memory-reports).

Rationale: Reduce disk usage on the device (this is an issue with low end devices). Makes transfer time quicker. Also if we do streaming compression we can reduce the in-process memory overhead as well (as in: write the generated report in chunks to the compressor, rather than all in memory, then compressing).

#3 - add an option to compress the entire directory upon completion for easy upload to bugzilla. Additionally add a flag to keep compressing dmd/gc/cc logs if desired.

Comment 1

[Nicholas Nethercote [inactive]]

The bugzilla file size limit is germane to this bug, though I guess it's not that hard to zip things manually before uploading them.

Comment 2

[Eric Rahm [:erahm]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/377

This makes compression of individual files opt-in and adds an option to compress the whole out_dir when completed. It also properly closes DMD files after use.

Andrew can you take a look at this or punt to a more appropriate reviewer?

Comment 3

[Andrew McCreight [:mccr8]]

(In reply to Nicholas Nethercote [:njn] from comment #1)
> The bugzilla file size limit is germane to this bug, though I guess it's not
> that hard to zip things manually before uploading them.

One thing I worry about is that less technical people may find it harder to zip logs, which may prevent them from uploading them to bugzilla.

I also kind of like having the individually compressed logs because when I get the giant dump of partner logs, I can unzip each thing individually.  I suppose one individual memory report isn't that big.

Comment 4

[Andrew McCreight [:mccr8]]

Comment on attachment 8473381 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/377

I guess I minus this now?

Comment 5

[Eric Rahm [:erahm]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/377

This updates the PR to default to using individual file compression which matches the current implementation. Additionally it switches from compressing GC/CC logs with xz to gzip for consistency's sake. This also preps us for handling if/when GC/CC logs are already compressed.

Comment 6

[Andrew McCreight [:mccr8]]

Comment on attachment 8475418 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/377

r=me with the command line arg changed to --uncompressed-logs as discussed in IRC.

Comment 7

[Eric Rahm [:erahm]]

PR updated.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Master: https://github.com/mozilla-b2g/B2G/commit/c733adefa872b17dfbd3402b19c96b900c261e09