1479484 - Provide perfecthash.py as general usable module and add codegen

Status: RESOLVED FIXED

(Core :: XPCOM, enhancement)

Description

[Jan-Erik Rediger [:janerik]]

perfecthash.py[1] is a nice small implementation of a perfect hash function, useful for mapping elements to integers.
Its C++ part is just 2 functions[2].

This module should be extracted into a common place in order to reuse it in other modules.
Additionally, it should bring functionality for the code generation as well.

The Telemetry module would adopt it to provide a String->Integer map for its internal use. 

[1]: https://searchfox.org/mozilla-central/source/xpcom/reflect/xptinfo/perfecthash.py
[2]: https://searchfox.org/mozilla-central/source/xpcom/reflect/xptinfo/xptinfo.cpp#16-44

Comment 1

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1479484 - Part 1: Add make_dafsa.py to the virtualenv's python path, r=froydnj

The plan is to also expose perfecthash.py from this module on the python path.
This also allows us to stop using explicit module loading to load make_dafsa.py.

make_dafsa.py was moved into tools/ to avoid any extra python files from
accidentally ending up on the python path.

Comment 2

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1479484 - Part 2: Expose endianness from buildconfig, r=gps

This information is already being detected, and it will allow us to avoid
byteorder swaps in XPT on big-endian systems. This patch allows buildconfig to
detect endianness.

It may make sense in the future to also expose this endianness as a define, so
we can avoid custom detection such as in /mfbt/EndianUtils.h:
https://searchfox.org/mozilla-central/rev/196560b95f191b48ff7cba7c2ba9237bba6b5b6a/mfbt/EndianUtils.h#85-148


Depends On D2614

Comment 3

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1479484 - Part 3: Add perfecthash.py with codegen to xpcom/ds/tools, r=froydnj

This is a streamlined version of perfecthash.py from xpcom/reflect/xptinfo.
There are a few major changes here:

Instead of providing '(key, value)' pairs to the constructor, callers provide a
list of 'entries'. An optional 'key' parameter allows overriding the function
used to get an entry's key. The default behaviour is as before, destructuring a
(key, value) tuple.

Keys can now be anything which supports the 'memoryview' protocol with 1-byte
elements, rather than being forced to be a 'bytearray'. This allows passing in
types such as bytestrings.

A new 'cxx_codegen' method is now exposed which generates a series of
fully-contained C++ declarations to perform entry lookups. It is possible to
override many parts of this codegen to make it the best fit for your component.

PerfectHash remembers the 'key' function, meaning lookup methods automatically
verify a valid key was passed in, and return 'None' otherwise.


Depends On D2615

Comment 4

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1479484 - Part 4: Move xptcodegen over to new perfecthash.py, r=froydnj

This patch ports xptcodegen.py over to the new perfecthash.py system, removing
some special-case code generators, and taking advantage of the easier-to-use
interface.

In addition, the code was changed to take advantage of the endianness
information from Part 2, allowing us to avoid having to perform endianness swaps
at runtime when hashing nsIDs.


Depends On D2616

Comment 5

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1479484 - Part 5: Remove old perfecthash.py, r=froydnj

Depends On D2618

Comment 6

[Phabricator Automation]

Comment on attachment 8996791 [details]
Bug 1479484 - Part 1: Add make_dafsa.py to the virtualenv's python path, r=froydnj

Nathan Froyd [:froydnj] has approved the revision.

https://phabricator.services.mozilla.com/D2614

Comment 7

[Phabricator Automation]

Comment on attachment 8996792 [details]
Bug 1479484 - Part 2: Expose endianness from buildconfig, r=gps

Nathan Froyd [:froydnj] has approved the revision.

https://phabricator.services.mozilla.com/D2615

Comment 8

[Phabricator Automation]

Comment on attachment 8996793 [details]
Bug 1479484 - Part 3: Add perfecthash.py with codegen to xpcom/ds/tools, r=froydnj

Nathan Froyd [:froydnj] has approved the revision.

https://phabricator.services.mozilla.com/D2616

Comment 9

[Phabricator Automation]

Comment on attachment 8996795 [details]
Bug 1479484 - Part 4: Move xptcodegen over to new perfecthash.py, r=froydnj

Nathan Froyd [:froydnj] has approved the revision.

https://phabricator.services.mozilla.com/D2618

Comment 10

[Phabricator Automation]

Comment on attachment 8996796 [details]
Bug 1479484 - Part 5: Remove old perfecthash.py, r=froydnj

Nathan Froyd [:froydnj] has approved the revision.

https://phabricator.services.mozilla.com/D2619

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/20668aaf20eb
Part 1: Add make_dafsa.py to the virtualenv's python path, r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/a509c3520b77
Part 2: Expose endianness from buildconfig, r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/fc662efea520
Part 3: Add perfecthash.py with codegen to xpcom/ds/tools, r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/9863b56882dc
Part 4: Move xptcodegen over to new perfecthash.py, r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/6175bdb411f5
Part 5: Remove old perfecthash.py, r=froydnj

Comment 12

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/20668aaf20eb
https://hg.mozilla.org/mozilla-central/rev/a509c3520b77
https://hg.mozilla.org/mozilla-central/rev/fc662efea520
https://hg.mozilla.org/mozilla-central/rev/9863b56882dc
https://hg.mozilla.org/mozilla-central/rev/6175bdb411f5