Intermittent test_screenshot.py TestScreenCaptureChrome.test_capture_full_area | AssertionError: [truncated]... != [truncated]...

VERIFIED INCOMPLETE

Status

Testing
Marionette
VERIFIED INCOMPLETE
7 months ago
28 days ago

People

(Reporter: Treeherder Bug Filer, Unassigned)

Tracking

({bulk-close-intermittents, intermittent-failure})

Version 3
bulk-close-intermittents, intermittent-failure
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [stockwell unknown])

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

7 months ago
treeherder
Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=98978384&repo=autoland

https://queue.taskcluster.net/v1/task/dIqQoMuQQPulmoTGSOSJew/runs/0/artifacts/public/test_info//marionette_errorsummary.log

Comment 1

7 months ago
37 failures in 188 pushes (0.197 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 23
* mozilla-inbound: 11
* mozilla-central: 3

Platform breakdown:
* linux32: 25
* linux64: 10
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-17&endday=2017-05-17&tree=all

Comment 2

7 months ago
48 failures in 166 pushes (0.289 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 21
* mozilla-inbound: 17
* graphics: 5
* mozilla-central: 3
* try: 2

Platform breakdown:
* linux32: 33
* linux64: 7
* windows7-32-vm: 2
* osx-10-10: 2
* windows8-64: 1
* windows7-32: 1
* windows10-64-vm: 1
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-18&endday=2017-05-18&tree=all
I wonder if this can be affected by recent UI changes to Firefox, and more recently, by https://bugzilla.mozilla.org/show_bug.cgi?id=1355890.  The robot icon that is shown in the address bar when the browser’s remote protocol is activated is a GIF of a robot that occasionally blinks.
When I let both PNGs graphically compare those are identically. I used http://huddle.github.io/Resemble.js/ for that. But binary wise we have a difference. I wonder if we better should not do it binary wise, or what that actually means. I will attach both files in a moment.
Created attachment 8869547 [details]
screenshot1.png
Created attachment 8869548 [details]
screenshot2.png
Maybe only the header is different (eg. different time) but the data is the same? It could easily happen because of large data to transfer between Firefox and the client, so that a second difference can happen.
So, the robot is animated and because of that we have differences in the image. See the rolling eyes. I think that you want to remove this animation. Maybe its better to file a new bug for this.
Depends on: 1366366

Comment 9

7 months ago
43 failures in 146 pushes (0.295 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 20
* mozilla-inbound: 14
* try: 4
* mozilla-central: 4
* oak: 1

Platform breakdown:
* linux32: 26
* linux64: 13
* windows10-64-vm: 2
* windows8-64: 1
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-19&endday=2017-05-19&tree=all

Comment 10

7 months ago
163 failures in 777 pushes (0.21 failures/push) were associated with this bug in the last 7 days. 

This is the #3 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 74
* mozilla-inbound: 60
* mozilla-central: 11
* graphics: 10
* try: 7
* oak: 1

Platform breakdown:
* linux32: 102
* linux64: 36
* windows7-32-vm: 11
* windows8-64: 5
* windows10-64-vm: 3
* osx-10-10: 3
* android-4-3-armv7-api15: 2
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-15&endday=2017-05-21&tree=all

Comment 11

7 months ago
34 failures in 147 pushes (0.231 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 21
* mozilla-inbound: 9
* mozilla-central: 3
* try: 1

Platform breakdown:
* linux32: 19
* linux64: 11
* windows10-64-vm: 2
* windows8-64: 1
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-22&endday=2017-05-22&tree=all

Comment 12

7 months ago
57 failures in 172 pushes (0.331 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 27
* autoland: 21
* graphics: 4
* mozilla-central: 3
* try: 2

Platform breakdown:
* linux32: 33
* linux64: 16
* windows7-32-vm: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-23&endday=2017-05-23&tree=all

Comment 13

7 months ago
47 failures in 161 pushes (0.292 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 14
* autoland: 14
* try: 11
* mozilla-central: 8

Platform breakdown:
* linux32: 24
* linux64: 11
* windows7-32-vm: 9
* windows8-64: 2
* windows10-64-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-24&endday=2017-05-24&tree=all
Whiteboard: [stockwell needswork]
Created attachment 8871362 [details] [diff] [review]
temporarily disable this test everywhere
Attachment #8871362 - Flags: review?(gbrown)
I think this bug may have been fixed, via bug 1366366.
oh, this would be cool, I see the failure rate way down.
Comment on attachment 8871362 [details] [diff] [review]
temporarily disable this test everywhere

Marking as obsolete given that we don't need it anymore. I can have a look in the next days (when it happens the next time) if I can find out what the reason is for this test failure.
Attachment #8871362 - Attachment is obsolete: true
Attachment #8871362 - Flags: review?(gbrown)

Comment 18

7 months ago
156 failures in 891 pushes (0.175 failures/push) were associated with this bug in the last 7 days. 

This is the #5 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 59
* mozilla-inbound: 54
* try: 19
* mozilla-central: 17
* graphics: 5
* oak: 2

Platform breakdown:
* linux32: 84
* linux64: 42
* windows7-32-vm: 23
* windows10-64-vm: 4
* windows8-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-22&endday=2017-05-28&tree=all

Updated

7 months ago
Whiteboard: [stockwell needswork] → [stockwell unknown]
Whiteboard: [stockwell unknown] → [stockwell needswork]
Whiteboard: [stockwell needswork] → [stockwell unknown]

Comment 19

6 months ago
11 failures in 820 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 6
* mozilla-inbound: 4
* try: 1

Platform breakdown:
* linux32: 5
* windows8-64: 4
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-05-29&endday=2017-06-04&tree=all
The recent issue for those failures seems to be the reload button in the toolbar which in one screenshot is disabled, and in the second enabled.

https://dxr.mozilla.org/mozilla-central/rev/cad53f061da634a16ea75887558301b77f65745d/testing/marionette/harness/marionette_harness/tests/unit/test_screenshot.py#194

In ScreenCaptureTestCase.setUp() we explicitly load `about:blank` but we somehow return from `get` for a different URL:

1496328967429	Marionette	TRACE	270 -> [0,7,"get",{"url":"about:blank"}]
1496328967436	Marionette	DEBUG	Received DOM event "beforeunload" for "http://127.0.0.1:38467/windowHandles.html"
1496328967449	Marionette	DEBUG	Received DOM event "pagehide" for "http://127.0.0.1:38467/windowHandles.html"
1496328967451	Marionette	DEBUG	Received DOM event "unload" for "http://127.0.0.1:38467/windowHandles.html"
++DOMWINDOW == 28 (0xd6db6800) [pid = 1661] [serial = 345] [outer = 0xd977cc00]
1496328967888	Marionette	DEBUG	Received DOM event "DOMContentLoaded" for "http://127.0.0.1:38467/windowHandles.html"
1496328967895	Marionette	DEBUG	Received DOM event "pageshow" for "http://127.0.0.1:38467/windowHandles.html"
1496328967902	Marionette	TRACE	270 <- [1,7,null,{}]

And as such the events for `about:blank` might still occur but after we returned from `get`. 

It's not something I currently understand.
I had a look at the test in question:

>     def test_capture_full_area(self):
>         # A full capture is not the outer dimensions of the window,
>         # but instead the bounding box of the window's root node (documentElement).
>         screenshot_full = self.marionette.screenshot()
>         screenshot_root = self.marionette.screenshot(element=self.document_element)
> 
>         self.assert_png(screenshot_full)
>         self.assert_png(screenshot_root)
>         self.assertEqual(screenshot_root, screenshot_full)
>         self.assertEqual(self.scale(self.get_element_dimensions(self.document_element)),
>                          self.get_image_dimensions(screenshot_full))

On the second last line, it’s not clear to me that we actually need to test that the screenshots are 100% equivalent.  It should be enough just to test that we get the right dimensions, which we do in the last line.

If it is indeed the case that the continued intermittents here are caused by changes to the Firefox UI, then I believe we are setting ourselves up for failure.  It’s not vital to the success of this test that the screenshots are the same; that’s not what we’re testing.
In that case we could have left the rolling eyes of the robot!
(In reply to Henrik Skupin (:whimboo) from comment #22)
> In that case we could have left the rolling eyes of the robot!

I suppose this is true.
(In reply to Andreas Tolfsen ‹:ato› from comment #23)
> > In that case we could have left the rolling eyes of the robot!
> 
> I suppose this is true.

Are you willed to fix that, and get your proposal implemented? I'm happy to review. Maybe reverting the rolling eyes might need further changes to other screenshot tests.
(In reply to Henrik Skupin (:whimboo) from comment #24)
> (In reply to Andreas Tolfsen ‹:ato› from comment #23)
> > > In that case we could have left the rolling eyes of the robot!
> > 
> > I suppose this is true.
> 
> Are you willed to fix that, and get your proposal implemented? I'm happy to
> review. Maybe reverting the rolling eyes might need further changes to other
> screenshot tests.

I would say patches are accepted on that.  I don’t consider reverting that
change a priority.
(In reply to Andreas Tolfsen ‹:ato› from comment #25)
> (In reply to Henrik Skupin (:whimboo) from comment #24)
> > (In reply to Andreas Tolfsen ‹:ato› from comment #23)
> > > > In that case we could have left the rolling eyes of the robot!
> > > 
> > > I suppose this is true.
> > 
> > Are you willed to fix that, and get your proposal implemented? I'm happy to
> > review. Maybe reverting the rolling eyes might need further changes to other
> > screenshot tests.
> 
> I would say patches are accepted on that.  I don’t consider reverting that
> change a priority.

Actually, after more consideration, are we sure we want an animated GIF in the
UI?  It sounds like a static image is qualitatively better.
We might want to leave the image as it is right now, given that we still don't know which other side-effects an animation could have. 

What I meant with a patch is what you proposed in comment 21 just to fix the test.

Comment 28

6 months ago
14 failures in 864 pushes (0.016 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 4
* try: 2
* mozilla-central: 1

Platform breakdown:
* linux32: 8
* windows7-32-vm: 3
* windows8-64: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-06-05&endday=2017-06-11&tree=all

Comment 29

6 months ago
17 failures in 814 pushes (0.021 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 8
* autoland: 7
* mozilla-central: 2

Platform breakdown:
* windows8-64: 6
* windows7-32-vm: 6
* linux64: 2
* linux32: 2
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 30

6 months ago
10 failures in 892 pushes (0.011 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 6
* mozilla-inbound: 3
* mozilla-central: 1

Platform breakdown:
* linux64: 4
* windows8-64: 2
* linux32: 2
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 31

5 months ago
9 failures in 718 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 5
* mozilla-inbound: 3
* mozilla-central: 1

Platform breakdown:
* windows8-64: 2
* osx-10-10: 2
* linux32: 2
* windows7-32-vm: 1
* linux64-ccov: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 32

5 months ago
1 failures in 656 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-07-03&endday=2017-07-09&tree=all

Comment 33

5 months ago
2 failures in 720 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* osx-cross: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364762&startday=2017-07-10&endday=2017-07-16&tree=all
Status: NEW → RESOLVED
Last Resolved: 3 months ago
Keywords: bulk-close-intermittents
Resolution: --- → INCOMPLETE
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.