1610635 - fix the hgweb bootstrap procedure

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, task)

Description

[Connor Sheehan [:sheehan]]

The vcsreplicator bootstrapping procedure has yet to pass under Python 3. Since the bootstrapper is used infrequently it was not prioritized during the port as the rest of the codebase was. After some investigation it seems the breakage came from upgrading the kafka-python version itself, regardless of the Python interpreter being used.

Comment 1

[Connor Sheehan [:sheehan]]

Attachment: vcsreplicator/bootstrap: use custom function to process extra bootstrap messages (Bug 1610635, Bug 1541166) r?mhentges,smacleod,zeid

Between Python 2 and 3, the map builtin function has some slightly
modified behaviour that breaks the hgweb bootstrap processing of
extra messages, namely the parallelization of mapping in Python 3.

This commit switches from a fancy combination of zip and map to
create the future to instead use a simple sequential function to
apply the list of parameters to the handle_message_main function
in the correct order.

Comment 2

[Connor Sheehan [:sheehan]]

Attachment: vcsreplicator/bootstrap: split host list so argument is passed as a list (Bug 1610635) r?mhentges,smacleod,zeid

The API calls for a single host:port string, or a list of
strings. Instead we pass a single string with multiple
host:port combinations, which is obviously wrong.

Comment 3

[Connor Sheehan [:sheehan]]

Attachment: vcsreplicator: call `.topics()` to get metadata from Kafka before requesting list of partitions (Bug 1610635) r?mhentges,smacleod,zeid

This commit adds a call to consumer.topics() to populate metadata
in the client, and also adds a retry around the call to gather
partitions for the required bootstrap topic. Adding this change, as
well as some slightly modified strings in test output, causes tests
to pass.

We didn't see this error in the main vcsreplicator consumers as they
use the older SimpleConsumer APIs.

Comment 4

[Connor Sheehan [:sheehan]]

Attachment: vcsreplicator: fix string escaping in `test-bootstrap.t` output (Bug 1610635, Bug 1541166) r?mhentges,smacleod,zeid

These string previously needed escaping, and did not have the
b'' prefixing.

Comment 5

[Connor Sheehan [:sheehan]]

Attachment: vcsreplicator/bootstrap: remove warning about added SSH keys from `test-bootstrap.t` output (Bug 1610635) r?mhentges,smacleod,zeid

This output no longer appears in the test. Unsure why at the moment but
since it doesn't affect the test itself it should be safe to remove.

Comment 6

[Connor Sheehan [:sheehan]]

Attachment: testing: remove `test-bootstrap.t` from flaky tests list (Bug 1610635, Bug 1541166) r?mhentges,smacleod,zeid

After applying this stack of changes, test-bootstrap.t no longer
fails. It seems that setting futures.ThreadPoolExecutor(1)
produces more deterministic behaviour on Python 3 than on Python 2,
and the test passes reliably on my machine.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b3b26605f232
vcsreplicator/bootstrap: use custom function to process extra bootstrap messages (Bug 1610635, Bug 1541166) r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7c7bbdd4d5a9
vcsreplicator/bootstrap: split host list so argument is passed as a list r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/773d17e3470c
vcsreplicator: call .topics() to get metadata from Kafka before requesting list of partitions r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8089207534f2
vcsreplicator: fix string escaping in test-bootstrap.t output (Bug 1610635, Bug 1541166) r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d3a51af7abc3
vcsreplicator/bootstrap: remove warning about added SSH keys from test-bootstrap.t output r=mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b7b33d10c519
testing: remove test-bootstrap.t from flaky tests list (Bug 1610635, Bug 1541166) r=mhentges