Closed Bug 1329424 Opened 7 years ago Closed 5 years ago

Intermittent test_screenshot.py TestScreenCaptureChrome.test_capture_flags | AssertionError: u'iVBORw0KGgoA... (images unexpectedly equal)

Categories

(Testing :: Marionette Client and Harness, defect, P1)

Version 3
defect

Tracking

(firefox-esr60 disabled, firefox67 disabled, firefox68 disabled, firefox69 fixed)

RESOLVED FIXED
mozilla69
Tracking Status
firefox-esr60 --- disabled
firefox67 --- disabled
firefox68 --- disabled
firefox69 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disabled])

Attachments

(1 file)

See Also: → 1330560
Summary: Intermittent test_screenshot.py TestScreenCaptureChrome.test_capture_flags | AssertionError: u'iVBORw0KGgoAAAANSUhEUgAAAT4AAAIqCAYAAAC9lyYgAAARQElEQVR4nO3dTVIi+4LG4dwPMnIfPcXIUS/hzHtWGMG0p26gxOjsUS9DywF7qA/r67iDfw8ATRBQUUngfZ6IN24J8lGc6++S6rlZ3d/fFzOzpFVd → Intermittent test_screenshot.py TestScreenCaptureChrome.test_capture_flags | AssertionError: u'iVBORw0KGgoA... (images unexpectedly equal)
This happens mostly on Windows 10 nowadays. Joel, is there anyone checking for test failures on this new platform for tests?
Flags: needinfo?(jmaher)
we did a lot of work to check for new failures- but this is a "fragile" platform still and some jobs get enabled on accident.  Should we leave this running, or disable it for now?  If we just have 1 failure, we should fix it or skip it for the time being so the other tests are getting coverage.
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher) from comment #17)
> we did a lot of work to check for new failures- but this is a "fragile"
> platform still and some jobs get enabled on accident.  Should we leave this
> running, or disable it for now?  If we just have 1 failure, we should fix it
> or skip it for the time being so the other tests are getting coverage.

It's up to you. Last week we got 30 failures so that we hit the threshold of a warning for priority. I think it might make sense to skip it on this platform to get it fixed.
while this has a lot of failures, it seems to be very few failures after the 18th.
:gbrown, I think we should disable this test given the very high failure rate
Flags: needinfo?(gbrown)
Whiteboard: [stockwell needswork]
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1dde9a9f7745
Skip TestScreenCaptureChrome.test_capture_flags for frequent failures; r=me,test-only
Flags: needinfo?(gbrown)
Keywords: leave-open
Whiteboard: [stockwell needswork] → [stockwell disabled]
Keywords: test-disabled
Priority: -- → P3
Priority: P3 → P5

As it looks like this test has two issues:

  1. When the chrome dialog gets opened it gets sometimes moved into the background. As such no focus/blur events will be raised. My patch on bug 1504756 refactors the Mn unit tests kinda heavily for opening a new window. Lets see if that helps here.

  2. We call blur() on the textbox, but don't actually wait for the blur event to happen. This can cause a race condition when trying to compare the screenshots between the focused and not focused textbox.

Depends on: 1504756

We should try to re-enable this test now that bug 1335085 has been fixed. Maybe that also helped here.

Blocks: 1330560
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Keywords: leave-open
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8eb5ac3760e7
[marionette] Re-enable tests TestScreenCapture[Content/Chrome].test_capture_flags. r=maja_zf
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Keywords: test-disabled
Priority: P5 → P1
Product: Testing → Remote Protocol
Moving bug to Testing::Marionette Client and Harness component per bug 1815831.
Component: Marionette → Marionette Client and Harness
Product: Remote Protocol → Testing
You need to log in before you can comment on or make changes to this bug.