Closed Bug 1531003 Opened 2 years ago Closed 1 year ago

test_search_counts_across_sessions.py fails when run with QuantumBar

Categories

(Toolkit :: Telemetry, defect, P1)

defect

Tracking

()

RESOLVED WORKSFORME

People

(Reporter: standard8, Assigned: standard8)

References

Details

Attachments

(1 obsolete file)

I just noticed this was failing when we run it alongside QuantumBar:

[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO - TEST-UNEXPECTED-FAIL | telemetry/marionette/tests/client/test_search_counts_across_sessions.py TestSearchCounts.test_search_counts | AssertionError: {u'range': [1, 2], u'bucket_count': 3, u'histogram_type': 4, u'values': {u'1': 0 [truncated]... != {u'range': [1, 2], u'histogram_type': 4, u'values': {u'1': 0, u'0': 3}, u'bucket [truncated]...
[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO -   {u'bucket_count': 3,
[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO -    u'histogram_type': 4,
[task 2019-02-27T09:02:57.094Z] 09:02:57     INFO -    u'range': [1, 2],
[task 2019-02-27T09:02:57.095Z] 09:02:57     INFO - -  u'sum': 1,
[task 2019-02-27T09:02:57.096Z] 09:02:57     INFO - ?          ^
[task 2019-02-27T09:02:57.096Z] 09:02:57     INFO - 
[task 2019-02-27T09:02:57.097Z] 09:02:57     INFO - +  u'sum': 3,
[task 2019-02-27T09:02:57.097Z] 09:02:57     INFO - ?          ^
[task 2019-02-27T09:02:57.098Z] 09:02:57     INFO - 
[task 2019-02-27T09:02:57.099Z] 09:02:57     INFO - -  u'values': {u'0': 1, u'1': 0}}
[task 2019-02-27T09:02:57.099Z] 09:02:57     INFO - ?                    ^
[task 2019-02-27T09:02:57.100Z] 09:02:57     INFO - 
[task 2019-02-27T09:02:57.101Z] 09:02:57     INFO - +  u'values': {u'0': 3, u'1': 0}}
[task 2019-02-27T09:02:57.101Z] 09:02:57     INFO - ?                    ^

I think the main issue is that the tests aren't waiting to make sure a load has actually happened, and hence a search has been recorded.

I'm a bit surprised this is working reliably with the existing address bar, since that could likely have the same issue.

:raphael is probably the better person to review that.

Flags: needinfo?(rpierzina)

Ok, I've shifted the review request across.

Flags: needinfo?(rpierzina)

Marking as P1 as it is assigned and has a patch (with r?) (this way it moves out of our triage)

Priority: -- → P1

Thank you! I'm investigating!

Sorry for the delay in responding. I've just been doing some more tests, as I realised the enabling of quantumbar for these tests may have been done in the wrong way since some recent changes.

However, regardless of how I enable the QuantumBar, and if I include Raphael's 5 second-delay, all the tests now seem to be passing.

There's one or two possible intermittents there but they don't seem to be related to this.

So I think we're resolving this as works for me - as I've got not idea what changed to fix this, and it all seems to be ok.

Attachment #9047056 - Attachment is obsolete: true

Raphael, Georg, thank you for helping on this. I would have semi-liked to know what fixed this, but for now I'm not sure there's much we can do.

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.