Package hyphenation-dictionaries separately for reftests

RESOLVED FIXED in Firefox 50

Status

()

defect
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: k.krish, Assigned: k.krish)

Tracking

(Blocks 1 bug)

unspecified
Firefox 50
All
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox50 fixed)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
This is an intermediate step for Bug 1095719.

In Bug 1095719 we'll download hyphenation-dictionaries at runtime. we won’t
be able to do this when running reftests. We need to package them separately and install them along with the build.
(Assignee)

Updated

3 years ago
Assignee: nobody → k.krish
Duplicate of this bug: 1276586
(Assignee)

Comment 2

3 years ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=975fbc3008d3&selectedJob=22394515

Setting MOZ_EXCLUDE_HYPHENATION_DICTIONARIES=1, the following retests failed

REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1.html != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-notref.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1a.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-5.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-5-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2481
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-6.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-6-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-7.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-7-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-1.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-1.xhtml != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-notref.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-1a.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-5.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-5-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2481
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-6.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-6-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-7.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-7-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-11a.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-11b.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-13a.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-13b.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-af-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-af-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4608
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-bg-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-bg-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4013
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ca-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ca-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3464
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-cy-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-cy-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4380
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-da-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-da-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4249
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1901-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1901-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4436
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1996-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1996-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4436
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1901-1.html != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1996-1.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-ch-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-ch-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3944
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-eo-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-eo-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4150
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-es-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-es-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4671
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-et-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-et-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5311
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fi-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fi-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5584
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3847
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-gl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-gl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4712
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4567
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hsb-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hsb-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5094
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hu-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hu-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5151
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ia-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ia-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4480
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-is-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-is-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3726
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-it-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-it-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4223
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-kmr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-kmr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3845
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-la-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-la-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3686
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-lt-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-lt-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3730
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-mn-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-mn-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4457
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nb-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nb-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 6100
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4504
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nn-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nn-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 7727
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pt-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pt-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4886
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ru-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ru-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4807
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sh-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sh-1-ref.html | (LOAD ONLY), max difference: 181, number of differing pixels: 1668
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4827
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sr-1-ref.html | (LOAD ONLY), max difference: 181, number of differing pixels: 1668
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3893
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1.html != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1-notref.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-tr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-tr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4149
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-uk-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-uk-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4536
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-transformed-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-transformed-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2753
(Assignee)

Comment 3

3 years ago
When we exclude the hyphenation dictionaries from the build, many reftests will fail. To stop them from failing we have to package the hyphenation-dictionary separately. We should get all the *.dic files and load in it to the device at runtime.

As a first step we have to package the hyphenations-dictionary separately and push it to the profile directory. References taken from Bug1197716 and Bug877203

To test I created a directory mobile/android/hyphenation and dumped all the dictionary files. 

I code to include the files in the temporary profile directory and push it to -sdcard/tests/reftest/profile 

I am able see the hyphenation dictionaries end up in sdcard/tests/reftest/profile/hyphenation/<files>

By doing this we have successfully packaged the hyphenation separately. Unfortunately This does not solve the problem. The reftests fail even after pushing these files to profile directory.

The problem is “Gecko” does not load the hyphenation dictionaries from the profile directory. The Hyphenation dictionaries are loaded in nsHyphenationManager.cpp

http://mxr.mozilla.org/mozilla-central/source/intl/hyphenation/nsHyphenationManager.cpp

I have included a sample code to make gecko look for hyphenation dictionaries from profile directory too.

nsHyphenationManager.cpp


>
>	nsCOMPtr<nsIFile> profileDir;
>	rv = NS_GetSpecialDirectory(NS_APP_USER_PROFILE_LOCAL_50_DIR,
>                                       getter_AddRefs(profileDir));
>	if (NS_SUCCEEDED(rv)) {
>		profileDir->AppendNative(NS_LITERAL_CSTRING("hyphenation"));
>		LoadPatternListFromDir(profileDir);
>	}
>

Now the hyphenation dictionary is being loaded from the sdcard/tests/reftest/profile/ directory and the reftests are passing.

Sample changeset

https://hg.mozilla.org/try/rev/a701a7b3f4bda097732142a6f8f278e43be261d9

Try results

https://treeherder.mozilla.org/#/jobs?repo=try&revision=956609c4d8f0
(Assignee)

Updated

3 years ago
Depends on: 1280757
(Assignee)

Updated

3 years ago
Attachment #8767465 - Flags: review?(gbrown)
(Assignee)

Comment 5

3 years ago
Try Results 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=374316725536

The reftests will pass after the code for Bug 1280757 is merged.
Comment on attachment 8767465 [details]
Bug1276560 Package hyphenation-dictionaries separately for reftests

https://reviewboard.mozilla.org/r/61936/#review58762

r+ with the changes:

::: layout/tools/reftest/mach_commands.py:234
(Diff revision 1)
> +        for (dirpath, dirnames, filenames) in os.walk(hyphenation_path):
> +            for hyphenation_file in filenames:
> +                if hyphenation_file.endswith('.dic'):

nit: The name 'hyphenation_file' is a bit misleading because it's just a file and only after the next check you know if it's a hyphenation dictionary. Maybe use just filename?

::: mobile/android/confvars.sh:28
(Diff revision 1)
> +MOZ_EXCLUDE_HYPHENATION_DICTIONARIES=1
> +

This should not be part of this patch. This is the last step after landing and testing everything related. :)
Attachment #8767465 - Flags: review?(s.kaspari) → review+
Comment on attachment 8767465 [details]
Bug1276560 Package hyphenation-dictionaries separately for reftests

https://reviewboard.mozilla.org/r/61936/#review58852
Attachment #8767465 - Flags: review?(gbrown) → review+
(Assignee)

Comment 8

3 years ago
Comment on attachment 8767465 [details]
Bug1276560 Package hyphenation-dictionaries separately for reftests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61936/diff/1-2/
Attachment #8767465 - Flags: review+
(Assignee)

Comment 9

3 years ago
https://reviewboard.mozilla.org/r/61936/#review58762

> nit: The name 'hyphenation_file' is a bit misleading because it's just a file and only after the next check you know if it's a hyphenation dictionary. Maybe use just filename?

Yeah. Makes much sense :). My bad. Changed the name accordingly

> This should not be part of this patch. This is the last step after landing and testing everything related. :)

Removed the flag.
(Assignee)

Updated

3 years ago
Keywords: checkin-needed

Comment 10

3 years ago
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/integration/fx-team/rev/d28a755c4014
Package hyphenation-dictionaries separately for reftests. r=sebastian
Keywords: checkin-needed

Comment 11

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/d28a755c4014
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 50
You need to log in before you can comment on or make changes to this bug.