1111419 - dmd.py: add a cache for recordKeyParts

Status: RESOLVED FIXED

(Core :: DMD, defect)

Description

[Nicholas Nethercote [inactive]]

dmd.py can be made to run faster.

Comment 1

[Nicholas Nethercote [inactive]]

Attachment: dmd.py: add a cache for recordKeyParts

This reduces the runtime on my Linux machine for one large DMD output file from
235 seconds to 105 seconds.

Comment 2

[Andrew McCreight [:mccr8]]

Comment on attachment 8536287 [details] [diff] [review]
dmd.py: add a cache for recordKeyParts

Review of attachment 8536287 [details] [diff] [review]:
-----------------------------------------------------------------

::: memory/replace/dmd/dmd.py
@@ +367,5 @@
>          # equivalent because the untrimmed frame descriptions (D1 and D2)
>          # match.
> +        #
> +        # Having said all that, for a single invocation of dmd.py on a single
> +        # DMD file a single frameKey value, the record key will always be the

This looks like it is missing a word.  "with a single frameKey value" or something?  Maybe I'm just misreading it.

Comment 3

[Andrew McCreight [:mccr8]]

Nice improvement!  There's a Python profiler that's pretty easy to use if you want to poke at that some more.

Comment 4

[Nicholas Nethercote [inactive]]

> There's a Python profiler that's pretty easy to use if
> you want to poke at that some more.

That's what I used to make this improvement :)

Much of the remaining time is taken up by (a) stack fixing, (b) gunzipping/gzipping, and (c) JSON reading, all of which are difficult to improve because they're all code outside the script. (Well, you can run with --no-fix-stacks when appropriate to avoid (a), and you can unzip your file manually to avoid (b).)

Comment 5

[Nicholas Nethercote [inactive]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/561eb285cf67

Comment 6

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/561eb285cf67