Closed Bug 1276560 Opened 8 years ago Closed 8 years ago

Package hyphenation-dictionaries separately for reftests

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Platform:
All
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox50 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 50
Tracking Flags:
Tracking Status
firefox50 --- fixed

People

(Reporter: k.krish, Assigned: k.krish)

References

Details

Attachments

(1 file)

Bug1276560 Package hyphenation-dictionaries separately for reftests
58 bytes, text/x-review-board-request
sebastian
: review+
Details 
This is an intermediate step for Bug 1095719.

In Bug 1095719 we'll download hyphenation-dictionaries at runtime. we won’t
be able to do this when running reftests. We need to package them separately and install them along with the build.
Assignee: nobody → k.krish
https://treeherder.mozilla.org/#/jobs?repo=try&revision=975fbc3008d3&selectedJob=22394515

Setting MOZ_EXCLUDE_HYPHENATION_DICTIONARIES=1, the following retests failed

REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1.html != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-notref.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1a.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-5.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-5-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2481
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-6.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-6-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-7.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-7-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-1.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-1.xhtml != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-notref.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-1a.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-5.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-5-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2481
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-6.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-6-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-7.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-7-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-11a.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-11b.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-13a.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-xmllang-13b.xhtml == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2194
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-af-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-af-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4608
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-bg-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-bg-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4013
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ca-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ca-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3464
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-cy-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-cy-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4380
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-da-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-da-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4249
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1901-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1901-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4436
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1996-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1996-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4436
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1901-1.html != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-1996-1.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-ch-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-de-ch-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3944
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-eo-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-eo-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4150
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-es-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-es-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4671
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-et-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-et-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5311
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fi-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fi-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5584
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-fr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3847
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-gl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-gl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4712
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4567
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hsb-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hsb-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5094
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hu-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-hu-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5151
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ia-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ia-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4480
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-is-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-is-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3726
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-it-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-it-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4223
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-kmr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-kmr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3845
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-la-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-la-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3686
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-lt-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-lt-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3730
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-mn-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-mn-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4457
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nb-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nb-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 6100
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4504
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nn-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-nn-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 5890
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 7727
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pt-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-pt-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4886
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ru-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-ru-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4807
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sh-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sh-1-ref.html | (LOAD ONLY), max difference: 181, number of differing pixels: 1668
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sl-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sl-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4827
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sr-1-ref.html | (LOAD ONLY), max difference: 181, number of differing pixels: 1668
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 3893
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1.html != http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-sv-1-notref.html | (LOAD ONLY)
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-tr-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-tr-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4149
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-uk-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-uk-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 4536
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-transformed-1.html == http://10.0.2.2:8854/tests/layout/reftests/text/auto-hyphenation-transformed-1-ref.html | (LOAD ONLY), max difference: 255, number of differing pixels: 2753
When we exclude the hyphenation dictionaries from the build, many reftests will fail. To stop them from failing we have to package the hyphenation-dictionary separately. We should get all the *.dic files and load in it to the device at runtime.

As a first step we have to package the hyphenations-dictionary separately and push it to the profile directory. References taken from Bug1197716 and Bug877203

To test I created a directory mobile/android/hyphenation and dumped all the dictionary files. 

I code to include the files in the temporary profile directory and push it to -sdcard/tests/reftest/profile 

I am able see the hyphenation dictionaries end up in sdcard/tests/reftest/profile/hyphenation/<files>

By doing this we have successfully packaged the hyphenation separately. Unfortunately This does not solve the problem. The reftests fail even after pushing these files to profile directory.

The problem is “Gecko” does not load the hyphenation dictionaries from the profile directory. The Hyphenation dictionaries are loaded in nsHyphenationManager.cpp

http://mxr.mozilla.org/mozilla-central/source/intl/hyphenation/nsHyphenationManager.cpp

I have included a sample code to make gecko look for hyphenation dictionaries from profile directory too.

nsHyphenationManager.cpp


>
>	nsCOMPtr<nsIFile> profileDir;
>	rv = NS_GetSpecialDirectory(NS_APP_USER_PROFILE_LOCAL_50_DIR,
>                                       getter_AddRefs(profileDir));
>	if (NS_SUCCEEDED(rv)) {
>		profileDir->AppendNative(NS_LITERAL_CSTRING("hyphenation"));
>		LoadPatternListFromDir(profileDir);
>	}
>

Now the hyphenation dictionary is being loaded from the sdcard/tests/reftest/profile/ directory and the reftests are passing.

Sample changeset

https://hg.mozilla.org/try/rev/a701a7b3f4bda097732142a6f8f278e43be261d9

Try results

https://treeherder.mozilla.org/#/jobs?repo=try&revision=956609c4d8f0
Depends on: 1280757
Attachment #8767465 - Flags: review?(gbrown)
Try Results 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=374316725536

The reftests will pass after the code for Bug 1280757 is merged.
Comment on attachment 8767465 [details]
Bug1276560 Package hyphenation-dictionaries separately for reftests

https://reviewboard.mozilla.org/r/61936/#review58762

r+ with the changes:

::: layout/tools/reftest/mach_commands.py:234
(Diff revision 1)
> +        for (dirpath, dirnames, filenames) in os.walk(hyphenation_path):
> +            for hyphenation_file in filenames:
> +                if hyphenation_file.endswith('.dic'):

nit: The name 'hyphenation_file' is a bit misleading because it's just a file and only after the next check you know if it's a hyphenation dictionary. Maybe use just filename?

::: mobile/android/confvars.sh:28
(Diff revision 1)
> +MOZ_EXCLUDE_HYPHENATION_DICTIONARIES=1
> +

This should not be part of this patch. This is the last step after landing and testing everything related. :)
Attachment #8767465 - Flags: review?(s.kaspari) → review+
Comment on attachment 8767465 [details]
Bug1276560 Package hyphenation-dictionaries separately for reftests

https://reviewboard.mozilla.org/r/61936/#review58852
Attachment #8767465 - Flags: review?(gbrown) → review+
Comment on attachment 8767465 [details]
Bug1276560 Package hyphenation-dictionaries separately for reftests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/61936/diff/1-2/
Attachment #8767465 - Flags: review+
https://reviewboard.mozilla.org/r/61936/#review58762

> nit: The name 'hyphenation_file' is a bit misleading because it's just a file and only after the next check you know if it's a hyphenation dictionary. Maybe use just filename?

Yeah. Makes much sense :). My bad. Changed the name accordingly

> This should not be part of this patch. This is the last step after landing and testing everything related. :)

Removed the flag.
Keywords: checkin-needed
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/integration/fx-team/rev/d28a755c4014
Package hyphenation-dictionaries separately for reftests. r=sebastian
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/d28a755c4014
Status: NEW → RESOLVED
Closed: 8 years ago
status-firefox50: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 50
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: