1459003 - Race condition in test-sns-notifier.t

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Connor Sheehan [:sheehan]]

The vcsreplicator test "test-sns-notifier.t" has a race condition that makes the dumped Kafka log output non-deterministic. When we dump the logs we expect to see heartbeat -> repo init -> heartbeat -> repo init, however we are instead seeing heartbeat -> heartbeat -> repo init -> repo init. At https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/pylib/vcsreplicator/vcsreplicator/hgext.py#l280 we write the heartbeat event, but it doesn't go to the same partition as the hg-repo-init-2 event. This causes the heartbeat messages to be consumed and processed faster than the repo init messages and as a result the log output appears in an incorrect order. From the "_original_created" timestamp value we can confirm the messages are produced in the correct order, but they are being consumed incorrectly.

We could fix this by forcing the heartbeat message into the same partition as the corresponding repo init message, which would ensure the messages are consumed in order. We could also wait for the repo creation messages to be consumed for each created repository before dumping the log output.

Comment 1

[Connor Sheehan [:sheehan]]

This has been fixed in a previous commit.