1531003 - test_search_counts_across_sessions.py fails when run with QuantumBar

Status: RESOLVED WORKSFORME

(Toolkit :: Telemetry, defect, P1)

Description

[Mark Banner (:standard8)]

I just noticed this was failing when we run it alongside QuantumBar:

[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO - TEST-UNEXPECTED-FAIL | telemetry/marionette/tests/client/test_search_counts_across_sessions.py TestSearchCounts.test_search_counts | AssertionError: {u'range': [1, 2], u'bucket_count': 3, u'histogram_type': 4, u'values': {u'1': 0 [truncated]... != {u'range': [1, 2], u'histogram_type': 4, u'values': {u'1': 0, u'0': 3}, u'bucket [truncated]...
[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO -   {u'bucket_count': 3,
[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO -    u'histogram_type': 4,
[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO -    u'range': [1, 2],
[task 2019-02-27T09:02:57.095Z] 09:02:57     INFO - -  u'sum': 1,
[task 2019-02-27T09:02:57.096Z] 09:02:57     INFO - ?          ^
[task 2019-02-27T09:02:57.096Z] 09:02:57     INFO - 
[task 2019-02-27T09:02:57.097Z] 09:02:57     INFO - +  u'sum': 3,
[task 2019-02-27T09:02:57.097Z] 09:02:57     INFO - ?          ^
[task 2019-02-27T09:02:57.098Z] 09:02:57     INFO - 
[task 2019-02-27T09:02:57.099Z] 09:02:57     INFO - -  u'values': {u'0': 1, u'1': 0}}
[task 2019-02-27T09:02:57.099Z] 09:02:57     INFO - ?                    ^
[task 2019-02-27T09:02:57.100Z] 09:02:57     INFO - 
[task 2019-02-27T09:02:57.101Z] 09:02:57     INFO - +  u'values': {u'0': 3, u'1': 0}}
[task 2019-02-27T09:02:57.101Z] 09:02:57     INFO - ?                    ^

I think the main issue is that the tests aren't waiting to make sure a load has actually happened, and hence a search has been recorded.

I'm a bit surprised this is working reliably with the existing address bar, since that could likely have the same issue.

Comment 1

[Mark Banner (:standard8)]

Attachment: Bug 1531003 - Fix test_search_counts_across_sessions.py to work with QuantumBar by waiting for searches to complete. r?dexter

Comment 2

[Jan-Erik Rediger [:janerik]]

:raphael is probably the better person to review that.

Comment 3

[Mark Banner (:standard8)]

Ok, I've shifted the review request across.

Comment 4

[Mark Banner (:standard8)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ea07a76e7867b79b7c45baba3e5ab113921d98f

Comment 5

[Jan-Erik Rediger [:janerik]]

Marking as P1 as it is assigned and has a patch (with r?) (this way it moves out of our triage)

Comment 6

[Raphael Aurich [:raphael] UTC+01:00]

Thank you! I'm investigating!

Comment 7

[Raphael Aurich [:raphael] UTC+01:00]

See https://phabricator.services.mozilla.com/D21368#643513

Comment 8

[Mark Banner (:standard8)]

Sorry for the delay in responding. I've just been doing some more tests, as I realised the enabling of quantumbar for these tests may have been done in the wrong way since some recent changes.

However, regardless of how I enable the QuantumBar, and if I include Raphael's 5 second-delay, all the tests now seem to be passing.

There's one or two possible intermittents there but they don't seem to be related to this.

So I think we're resolving this as works for me - as I've got not idea what changed to fix this, and it all seems to be ok.

Comment 9

[Mark Banner (:standard8)]

Raphael, Georg, thank you for helping on this. I would have semi-liked to know what fixed this, but for now I'm not sure there's much we can do.