1548372 - Fix, blacklist or remove long standing tests that continually fail

Status: NEW

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[K Lars Lohn [:lars] [:klohn]]

There are several tests that always fail in recent and current versions of the Version Control Tools codebase. These are noise in the test suite that makes it difficult for a someone unfamiliar with the tests to judge what is significant verses insignificant.

These are long standing tests that fail for a module that is apparently eventually going away:
vct/hgext/bzexport/tests/test-basechange.t
vct/hgext/bzexport/tests/test-patch-create.t
vct/hgext/bzexport/tests/test-review.t

This test failure seems to have started with patch 3b362b7a9144 (Bug 1523461) on 2019-02-04
vct/hgext/mozext/tests/test-backedoutby-mozext.t

This test was renamed from test-bugs.t to test-bugs-mozext.t by c033db814e2c (Bug 1481583) on 2018-08-09. Tracing backwards, the test was still failing as far back as 2017-09-12, whereas I've decided not to try to trace it back further.
vct/hgext/mozext/tests/test-bugs-mozext.t.err

It seems like this test has been failing since its introduction in 4c231b256afe. As far as I can tell, it has never not failed.
vct/hgext/wireprotocache/tests/test-cacher-s3.t.err

Comment 1

[Connor Sheehan [:sheehan]]

(In reply to K Lars Lohn [:lars] [:klohn] from comment #0)

There are several tests that always fail in recent and current versions of the Version Control Tools codebase. These are noise in the test suite that makes it difficult for a someone unfamiliar with the tests to judge what is significant verses insignificant.

These are long standing tests that fail for a module that is apparently eventually going away:
vct/hgext/bzexport/tests/test-basechange.t
vct/hgext/bzexport/tests/test-patch-create.t
vct/hgext/bzexport/tests/test-review.t

These tests have all been failing since the bmo docker images were ripped out of the test suite. bzexport itself is candidate for deletion, since we don't support posting patches to Bugzilla any more.

This test failure seems to have started with patch 3b362b7a9144 (Bug 1523461) on 2019-02-04
vct/hgext/mozext/tests/test-backedoutby-mozext.t

This test was renamed from test-bugs.t to test-bugs-mozext.t by c033db814e2c (Bug 1481583) on 2018-08-09. Tracing backwards, the test was still failing as far back as 2017-09-12, whereas I've decided not to try to trace it back further.
vct/hgext/mozext/tests/test-bugs-mozext.t.err

I'm not seeing these tests fail - can you post the failing output?

It seems like this test has been failing since its introduction in 4c231b256afe. As far as I can tell, it has never not failed.
vct/hgext/wireprotocache/tests/test-cacher-s3.t.err

The wireprotocache extension and it's tests should only work under Mercurial 4.8. The extension was committed in a working state but was still being worked on concurrently with wire protocol version two - with that work on pause, it's safe to silence this test.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

with this command:
./run-tests --with-hg=/home/lars/work/wf/vct/venv/mercurials/4.9.1/bin/hg

for ./hgext/mozext/tests/test-backedoutby-mozext.t:

+++ /home/lars/work/wf/vct/hgext/mozext/tests/test-backedoutby-mozext.t.err
@@ -20,4 +20,6 @@
   removing bar
   changeset 2:47fdd3ef011e backs out changeset 1:907f17e15674
   $ hg log -r -2 --template '{backedoutby}\n'
+  devel-warn: old-style template keyword 'backedoutby'
+  (compatibility will be dropped after Mercurial-4.8, update your code.) at: /home/lars/work/wf/vct/venv/mercurials/4.9.1/lib/python2.7/site-packages/mercurial/templateutil.py:769 (evalrawexp)
   47fdd3ef011e5b86fe4ce90a88d362ac03361992

ERROR: test-backedoutby-mozext.t output changed

for ./hgext/mozext/tests/test-bugs-mozext.t:

+++ /home/lars/work/wf/vct/hgext/mozext/tests/test-bugs-mozext.t.err
@@ -13,8 +13,12 @@
   $ hg commit -A -m 'Bug 123456 - Test foo\nFixes bug 100245'
   adding foo
   $ hg log --template '{bug}\n'
+  devel-warn: old-style template keyword 'bug'
+  (compatibility will be dropped after Mercurial-4.8, update your code.) at: /home/lars/work/wf/vct/venv/mercurials/4.9.1/lib/python2.7/site-packages/mercurial/templateutil.py:769 (evalrawexp)
   123456
   $ hg log --template '{join(bugs, " ")}\n'
+  devel-warn: old-style template keyword 'bugs'
+  (compatibility will be dropped after Mercurial-4.8, update your code.) at: /home/lars/work/wf/vct/venv/mercurials/4.9.1/lib/python2.7/site-packages/mercurial/templateutil.py:769 (evalrawexp)
   123456 100245

 bug() revset works

ERROR: test-bugs-mozext.t output changed