Closed Bug 1577155 Opened 5 months ago Closed 5 months ago

image/test/mochitest/test_bug415761.html fails when run locally

Categories

(Core :: ImageLib, defect, P3)

defect

Tracking

()

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: standard8, Assigned: tnikkel)

Details

Attachments

(2 files)

I'm on a MacBook Pro 15 inch 2018, with a self-built (full) build of Firefox. Running the image mochitests fails:

$ ./mach mochitest image/test/mochitest
...
 0:08.11 GECKO(19657) REFTEST TEST-UNEXPECTED-FAIL | icon | image comparison (==), max difference: 255, number of differing pixels: 50
 0:08.11 GECKO(19657) REFTEST   IMAGE 1 (TEST): see below
 0:08.11 GECKO(19657) REFTEST   IMAGE 2 (REFERENCE): see below
image/test/mochitest/test_bug415761.html
  FAIL reftest comparison: == icon reference icon
SimpleTest.ok@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:277:18
assertSnapshots@chrome://mochikit/content/tests/SimpleTest/WindowSnapshot.js:63:5
finishTest@chrome://mochitests/content/chrome/image/test/mochitest/test_bug415761.html:77:18
refImage.onload@chrome://mochitests/content/chrome/image/test/mochitest/test_bug415761.html:52:15
EventHandlerNonNull*@chrome://mochitests/content/chrome/image/test/mochitest/test_bug415761.html:49:1

Here are links to the data urls:

The test image has a little curly arrow indicating a shortcut, whereas the reference image doesn't.

It passes for me locally on macOS 10.13. What version macOS are you running?

Flags: needinfo?(standard8)

10.14.6 with the 10.4 SDK.

Flags: needinfo?(standard8)

I've never been under the impression our tests are particularly reliable when run anywhere else than CI. Are the image mochitests different in this regard?

(In reply to Alexis Beingessner [:Gankra] from comment #3)

I've never been under the impression our tests are particularly reliable when run anywhere else than CI. Are the image mochitests different in this regard?

I don't think so.

The test copies a file on the filesystem to a filename with a unicode character in it and then checks that asking the OS for the icon that represents both files draws the same at a canvas. So does macOS put a little arrow on files that are copied in 10.14+?

Looking at the files on my machine (in objdir/_tests/testing/mochitest/chrome/image/test/mochitest/) the ref is a symbolic link shows up with the arrow on it in Finder. If I modify the test to leave the created file after finishing the test then it is a full copy and not a link, the icon shown in Finder is a blank white rect, with no arrow, and it does not match the icon for the ref, which has the arrow but also "ico" and some pictures on a white piece of paper. Weird that it passes.

Attached image Test files in finder

Yes, I think I see the same as you're describing as well. The attachment is the test files as they look in finder.

The original source of the ico file is also a blank white square.

However, as I linked to the data urls above, the test seems to be getting the OS-displayed version of the icons, rather than those in the files itself.

I'm currently just about to rebuild with the 10.11.sdk (the one the builders use), and see if that has any affect.

Thanks. If your 10.11 build gets the results we expect (pass) then perhaps we should fix the test by copying the ref to a new file (with only ascii characters in the name) and then copying that to the unicode named file. That way both will be non-link files and should return the same icon from the os.

Using the 10.11 sdk on 10.14.6 also failed as well. So I wonder if this is about what the OS is returning somewhere, rather than the sdk version.

On a whim, I also just tried modifying the original image and adding a one-pixel dot, to see if the OS was detecting the blank/transparent(?) ico and replacing it, unfortunately that didn't seem to make the test pass either.

(In reply to Mark Banner (:standard8) from comment #6)

However, as I linked to the data urls above, the test seems to be getting the OS-displayed version of the icons, rather than those in the files itself.

Oh, that is what the test wants. It's testing moz-icon://, which gets the icon the OS would use to represent the file, not the file contents. So that is expected.

Ok, there's something confusing here: the assertSnapshots call:

assertSnapshots(refCanvas, testCanvas, true, 0, "icon", "reference icon");

assertSnapshots assumes that the first image is the test image, and the second is the reference. So my icons linked to in comment 0 are actually the other way around (although the parameter names and lack of documentation of the assertSnapshots function doesn't help).

If that's the case could we either, detect the reference image is a symlink and follow the symlink, or just create a copy of the source?

This is assuming that 10.14 is exhibiting expected behaviour...

I think the test should just create a copy of the source file. What the test is testing is that we can get valid icons from a file with unicode symbols in it's name, nothing more.

Priority: -- → P3
Pushed by tnikkel@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5e06c1d419b7
Make image/test/mochitest/test_bug415761.html copy both the test and reference icon so the OS doesn't include an extra badge indicating it is a sym link. r=Standard8
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Assignee: nobody → tnikkel
You need to log in before you can comment on or make changes to this bug.