Closed Bug 1607052 Opened 4 years ago Closed 4 years ago

using wrong font since v.72

Categories

(Core :: Layout: Text and Fonts, defect, P2)

Product:
Core
Component:
Layout: Text and Fonts
Version:
72 Branch
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla75
Tracking Flags:
Tracking Status
firefox-esr68 --- unaffected
firefox73 --- wontfix
firefox74 --- wontfix
firefox75 --- fixed

People

(Reporter: andynew327, Assigned: zbraniecki)

References

(Regression)

Details

(Keywords: regression)

Attachments

(2 files)

Bug 1607052 - Cut out the POSIX encoding part from the locale during sanitization.
47 bytes, text/x-phabricator-request
Details | Review
Bug 1607052 - Unify language identifier canonicalization in LocaleService and OSPreferences to use unic_langid, and extend GTK's RegionalPrefs reading to fallback on SystemLocales.
47 bytes, text/x-phabricator-request
Details | Review

User Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

Open web page

Actual results:

uses incorrect font

Expected results:

Firefox should render page using "Terminal Font" and it is using DejaVu Serif
DejaVu Serif
instead of Terminal
Terminal

This info comes from developer tools. The only font that should be used is Terminal. I have it set in Preferences->Default font->Advanced
Serif
Sans-serif
Monospace
are all set to Terminal font.

This is on FreeBSD.

Allow pages to choose their own fonts is unchecked

Component: Untriaged → Layout: Text and Fonts
Product: Firefox → Core

Is there any chance you can run mozregression to see when this broke?

That seems to have started much recently, so probably not? Unless it's been a system change somehow.

Also, what does fc-match "Terminal Font" return on your system? Those settings seem to work (with different fonts) here.

Flags: needinfo?(andynew327)

fc-match "Terminal Font"

DejaVuSans.ttf: "DejaVu Sans" "Book"

Flags: needinfo?(andynew327)

FWIW, I'm seeing the same bug with 72.0.1 on Debian Linux, both with the packaged version and with the pristine one from mozilla.org.
Basically the font settings from Preferences are ignored, both the font face and the font size.
This changed at some point between 71.0 and 72.0.1.

Also reported in the Debian BTS at https://bugs.debian.org/948542

% bin/mozregression -g 71 -b 73                            
 0:01.83 INFO: Using date 2020-01-06 for release 73
 0:03.14 INFO: Using date 2019-10-21 for release 71
...
12:36.32 INFO: No more inbound revisions, bisection finished.
12:36.32 INFO: Last good revision: 37771c6a4bb37fc1197c15b2720571faa25c5b34
12:36.32 INFO: First bad revision: 5ff314660cab8c1791cf94679026507246c8ec35
12:36.32 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=37771c6a4bb37fc1197c15b2720571faa25c5b34&tochange=5ff314660cab8c1791cf94679026507246c8ec35

Thank you! So this is locale-dependent, which makes sense given the regression range (and explains why I couldn't reproduce this).

Zibi, mind taking a look?

Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(gandalf)
Keywords: regression
Priority: -- → P2
Regressed by: 1571915

In case it helps, these are my usual locale settings:

% locale
LANG=
LANGUAGE=
LC_CTYPE=de_AT.utf8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
Has Regression Range: --- → yes

Hi!

I'm not very familiar with font selection code, but I know the language negotiation code we use. I assume that the regression indicates that somehow we now negotiate down to a different locale set and that in some way impacts font selection.

Could you open about:support and copy/paste the bottom block of Localization & Internationalization info about your Firefox instance?

Thank you!

Flags: needinfo?(andynew327)

Err, or gregor of course :-)

sure; and it looks quite, ehm, interesting, at least to me :)

Internationalization & Localization
Application Settings
Requested Locales 	["en-US-u-va-posix"]
Available Locales 	["en-US"]
App Locales 	["und","en-US"]
Regional Preferences 	["und"]
Default Locale 	"und"
Operating System
System Locales 	["en-US-u-va-posix"]
Regional Preferences 	["und"]

Okay! Is there any chance you could pull the same result from a build that works? (pre-regression)?

Flags: needinfo?(gandalf) → needinfo?(gregor+mozilla)
% bin/mozregression --launch 37771c6a4bb37fc1197c15b2720571faa25c5b34
 0:01.96 INFO: 37771c6a4bb37fc1197c15b2720571faa25c5b34 is not a release, assuming it's a hash...
 0:02.89 ERROR: The url 'https://hg.mozilla.org/integration/mozilla-inbound/json-pushes?changeset=37771c6a4bb37fc1197c15b2720571faa25c5b34' returned a 404 error because the push is not in this repo (e.g., not merged yet).
1 (venv) gregoa@jadzia ~/tmp/mozregression/venv % bin/mozregression --launch 71                                        :(
 0:02.36 INFO: Using date 2019-10-21 for release 71
 0:04.90 INFO: Downloading build from: https://archive.mozilla.org/pub/firefox/nightly/2019/10/2019-10-21-21-51-55-mozilla-central/firefox-72.0a1.en-US.linux-x86_64.tar.bz2
===== Downloaded 100% =====
 0:14.08 INFO: Running mozilla-central build for 2019-10-21
 0:25.35 INFO: Launching /home/gregoa/tmp/tmpd1e7y4cc/firefox/firefox
 0:25.35 INFO: Application command: /home/gregoa/tmp/tmpd1e7y4cc/firefox/firefox --allow-downgrade -profile /home/gregoa/tmp/tmpkqqa6m_e.mozrunner
 0:25.36 INFO: application_buildid: 20191021215155
 0:25.36 INFO: application_changeset: 563f437f24b9e12495b0f2c364538a2b8d8a0ca7
 0:25.36 INFO: application_name: Firefox
 0:25.36 INFO: application_repository: https://hg.mozilla.org/mozilla-central
 0:25.36 INFO: application_version: 72.0a1

has the following in about:support:

Internationalization & Localization
Application Settings
Requested Locales 	["en-US"]
Available Locales 	["en-US"]
App Locales 	["en-US"]
Regional Preferences 	["en-US"]
Default Locale 	"en-US"
Operating System
System Locales 	["en-US-u-va-posix"]
Regional Preferences 	["und"]

(where the latter two also look unexpected to me but at least the application settings seem sane)

Flags: needinfo?(gregor+mozilla)

Hah, okay! So, the difference is in those two:

App Locales 	["und","en-US"]
Regional Preferences 	["und"]

Default Locale 	"en-US"

Two things:

  1. We probably shouldn't take und as the default locale. If it's und, we should replace it with last fallback locale as a fallback locale for negotiation. Otherwise we end up with und in app locales :(
  2. This shouldn't return empty array - Services.locale.negotiateLanguages(["en-US-u-va-posix"], ["en-US"]);
Flags: needinfo?(andynew327)

Internationalization & Localization
Application Settings
Requested Locales ["und"]
Available Locales ["en-US"]
App Locales ["und","en-US"]
Regional Preferences ["und"]
Default Locale "und"
Operating System
System Locales ["en-US-u-va-posix"]
Regional Preferences ["und"]

Ok, I think I know what's going on.

We became a bit more strict in what we accept since we switched to unic-langid, and I think it's time to switch all the way, and add a lenient parsing method to unic-langid to handle en_US.UTF8 and similar inputs.

Assignee: nobody → gandalf
Status: NEW → ASSIGNED

Can someone affected try this patch?

In general yes; I'm just wondering if there is a simpler way than rebuilding firefox myself with the patch applied?

And it looks like there is a ci failure:

[task 2020-01-16T20:13:53.430Z]  /builds/worker/checkouts/gecko/intl/locale/LocaleService.cpp
[task 2020-01-16T20:13:53.430Z] /builds/worker/checkouts/gecko/intl/locale/LocaleService.cpp:75:17: error: redefinition of 'locale' [clang-diagnostic-error]
[task 2020-01-16T20:13:53.430Z]   nsAutoCString locale(aLocale);
[task 2020-01-16T20:13:53.430Z]                 ^
[task 2020-01-16T20:13:53.431Z] /builds/worker/checkouts/gecko/intl/locale/LocaleService.cpp:61:17: note: previous definition is here
[task 2020-01-16T20:13:53.431Z]   nsAutoCString locale(aLocale);
[task 2020-01-16T20:13:53.431Z]                 ^

(which shouldn't be to hard to fix :))

Sorry, fixed the patch.

Here's a try build with opt builds for all platforms that should give you a binary to try - https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb38d775b0c1572e45732cda24ce9a8473073c36

Thank you.
I've tried two things now:

  • run bin/mozregression --repo try --launch fb38d775b0c1
  • download a target.tar.bz2 for "Linux 64 opt" from the treeherder page and run firefox/firefox

In both cases the font settings are still ignored, and about:support shows in both cases

Application Settings
Requested Locales       ["en-US"]
Available Locales       ["en-US"]
App Locales     ["en-US"]
Regional Preferences    ["en-US"]
Default Locale  "en-US"
Operating System
System Locales  ["en-US-u-va-posix"]
Regional Preferences    ["und"]

Interestingly, that's the same output as in the case of a 'good' version …

Thank you!

I'm temporarily unassigning myself until I can get clear STRs. I'm on Linux so I hope it won't be very hard.

It seems that the same locales in LocaleService still lead to different outcome, so I'd like someone more familiar with font selection to point me to where this negotiation happens, what are the inputs and what are the outputs.

The patch attached may still be useful to get, since it improves the main localeservice results, but it doesn't seem to fix the problem in this bug.

Assignee: gandalf → nobody
Status: ASSIGNED → NEW
Assignee: nobody → gandalf
Status: NEW → ASSIGNED

I just landed a small fix that should get rid of the und, but doesn't seem to fix the bug for the reporter. I'm still waiting for someone familiar with the font selection to find where it negotiates different results.

Assignee: gandalf → nobody
Status: ASSIGNED → NEW
Keywords: leave-open
Pushed by zbraniecki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/013c0d48d9b0
Cut out the POSIX encoding part from the locale during sanitization. r=jfkthame
Assignee: nobody → gandalf
Status: NEW → ASSIGNED
Assignee: gandalf → nobody
Status: ASSIGNED → NEW

@Jonathan: Anything more we can do about this P2? Might you be able to reproduce?

Flags: needinfo?(jfkthame)

I've tried to reproduce this on my Linux system, setting locale vars similarly to the reporter here, etc., but so far without success.

Gregor and/or andynew327, could you please attach the complete about:support report from your machine? I'm not sure what might be relevant but it'd be good to take a look.

And one thing I wondered: when you're selecting your preferred font in the Firefox preferences, I guess that's the font for Latin script, as indicated by the "Fonts for:" dropdown at the top of the Advanced panel. If you also try setting the font for "Other Writing Systems" (at the end of the dropdown), does that make any difference?

Flags: needinfo?(jfkthame)
Flags: needinfo?(gregor+mozilla)
Flags: needinfo?(andynew327)

Hi @jfkthame,
thanks for following up.

about:support:
Puh, that's alot; let me try with a new empty profile:

Application Basics
------------------

Name: Firefox
Version: 73.0
Build ID: 20200207195153
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0
OS: Linux 5.4.0-3-amd64
Multiprocess Windows: 1/1 Enabled by default
Remote Processes: 3
Enterprise Policies: Inactive
Google Location Service Key: Found
Google Safebrowsing Key: Found
Mozilla Location Service Key: Found
Safe Mode: false

Crash Reports for the Last 3 Days
---------------------------------

Firefox Features
----------------

Name: DoH Roll-Out
Version: 1.3.0
ID: doh-rollout@mozilla.org

Name: Firefox Screenshots
Version: 39.0.0
ID: screenshots@mozilla.org

Name: Form Autofill
Version: 1.0
ID: formautofill@mozilla.org

Name: Web Compat
Version: 7.0.0
ID: webcompat@mozilla.org

Name: WebCompat Reporter
Version: 1.1.0
ID: webcompat-reporter@mozilla.org

Remote Processes
----------------

Type: Web Content
Count: 2 / 8

Type: Extension
Count: 1

Extensions
----------

Name: Amazon.com
Version: 1.1
Enabled: true
ID: amazondotcom@search.mozilla.org

Name: Bing
Version: 1.1
Enabled: true
ID: bing@search.mozilla.org

Name: DuckDuckGo
Version: 1.0
Enabled: true
ID: ddg@search.mozilla.org

Name: eBay
Version: 1.0
Enabled: true
ID: ebay@search.mozilla.org

Name: Google
Version: 1.0
Enabled: true
ID: google@search.mozilla.org

Name: Twitter
Version: 1.0
Enabled: true
ID: twitter@search.mozilla.org

Name: Wikipedia (en)
Version: 1.0
Enabled: true
ID: wikipedia@search.mozilla.org

Graphics
--------

Features
Compositing: Basic
Asynchronous Pan/Zoom: wheel input enabled; scrollbar drag enabled; keyboard enabled; autoscroll enabled
WebGL 1 Driver WSI Info: GLX 1.4 GLX_VENDOR(client): Mesa Project and SGI GLX_VENDOR(server): SGI Extensions: GLX_ARB_create_context GLX_ARB_create_context_no_error GLX_ARB_create_context_profile GLX_ARB_create_context_robustness GLX_ARB_fbconfig_float GLX_ARB_framebuffer_sRGB GLX_ARB_get_proc_address GLX_ARB_multisample GLX_EXT_create_context_es2_profile GLX_EXT_create_context_es_profile GLX_EXT_fbconfig_packed_float GLX_EXT_framebuffer_sRGB GLX_EXT_import_context GLX_EXT_texture_from_pixmap GLX_EXT_visual_info GLX_EXT_visual_rating GLX_INTEL_swap_event GLX_MESA_copy_sub_buffer GLX_MESA_query_renderer GLX_MESA_swap_control GLX_OML_swap_method GLX_OML_sync_control GLX_SGIS_multisample GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGIX_visual_select_group GLX_SGI_make_current_read GLX_SGI_swap_control GLX_SGI_video_sync
WebGL 1 Driver Renderer: Intel Open Source Technology Center -- Mesa DRI Intel(R) Sandybridge Mobile
WebGL 1 Driver Version: 3.0 Mesa 19.3.3
WebGL 1 Driver Extensions: GL_ARB_multisample GL_EXT_abgr GL_EXT_bgra GL_EXT_blend_color GL_EXT_blend_minmax GL_EXT_blend_subtract GL_EXT_copy_texture GL_EXT_subtexture GL_EXT_texture_object GL_EXT_vertex_array GL_EXT_compiled_vertex_array GL_EXT_texture GL_EXT_texture3D GL_IBM_rasterpos_clip GL_ARB_point_parameters GL_EXT_draw_range_elements GL_EXT_packed_pixels GL_EXT_point_parameters GL_EXT_rescale_normal GL_EXT_separate_specular_color GL_EXT_texture_edge_clamp GL_SGIS_generate_mipmap GL_SGIS_texture_border_clamp GL_SGIS_texture_edge_clamp GL_SGIS_texture_lod GL_ARB_framebuffer_sRGB GL_ARB_multitexture GL_EXT_framebuffer_sRGB GL_IBM_multimode_draw_arrays GL_IBM_texture_mirrored_repeat GL_3DFX_texture_compression_FXT1 GL_ARB_texture_cube_map GL_ARB_texture_env_add GL_ARB_transpose_matrix GL_EXT_blend_func_separate GL_EXT_fog_coord GL_EXT_multi_draw_arrays GL_EXT_secondary_color GL_EXT_texture_env_add GL_EXT_texture_filter_anisotropic GL_EXT_texture_lod_bias GL_INGR_blend_func_separate GL_NV_blend_square GL_NV_light_max_exponent GL_NV_texgen_reflection GL_NV_texture_env_combine4 GL_S3_s3tc GL_SUN_multi_draw_arrays GL_ARB_texture_border_clamp GL_ARB_texture_compression GL_EXT_framebuffer_object GL_EXT_texture_compression_s3tc GL_EXT_texture_env_combine GL_EXT_texture_env_dot3 GL_MESA_window_pos GL_NV_packed_depth_stencil GL_NV_texture_rectangle GL_ARB_depth_texture GL_ARB_occlusion_query GL_ARB_shadow GL_ARB_texture_env_combine GL_ARB_texture_env_crossbar GL_ARB_texture_env_dot3 GL_ARB_texture_mirrored_repeat GL_ARB_window_pos GL_EXT_stencil_two_side GL_EXT_texture_cube_map GL_NV_depth_clamp GL_NV_fog_distance GL_APPLE_packed_pixels GL_ARB_draw_buffers GL_ARB_fragment_program GL_ARB_fragment_shader GL_ARB_shader_objects GL_ARB_vertex_program GL_ARB_vertex_shader GL_ATI_draw_buffers GL_ATI_texture_env_combine3 GL_ATI_texture_float GL_EXT_shadow_funcs GL_EXT_stencil_wrap GL_MESA_pack_invert GL_NV_primitive_restart GL_ARB_depth_clamp GL_ARB_fragment_program_shadow GL_ARB_half_float_pixel GL_ARB_occlusion_query2 GL_ARB_point_sprite GL_ARB_shading_language_100 GL_ARB_sync GL_ARB_texture_non_power_of_two GL_ARB_vertex_buffer_object GL_ATI_blend_equation_separate GL_EXT_blend_equation_separate GL_OES_read_format GL_ARB_color_buffer_float GL_ARB_pixel_buffer_object GL_ARB_texture_compression_rgtc GL_ARB_texture_float GL_ARB_texture_rectangle GL_EXT_packed_float GL_EXT_pixel_buffer_object GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_rgtc GL_EXT_texture_rectangle GL_EXT_texture_sRGB GL_EXT_texture_shared_exponent GL_ARB_framebuffer_object GL_EXT_framebuffer_blit GL_EXT_framebuffer_multisample GL_EXT_packed_depth_stencil GL_APPLE_object_purgeable GL_ARB_vertex_array_object GL_ATI_separate_stencil GL_EXT_draw_buffers2 GL_EXT_draw_instanced GL_EXT_gpu_program_parameters GL_EXT_texture_array GL_EXT_texture_integer GL_EXT_texture_sRGB_decode GL_EXT_timer_query GL_OES_EGL_image GL_AMD_texture_texture4 GL_ARB_copy_buffer GL_ARB_depth_buffer_float GL_ARB_draw_instanced GL_ARB_half_float_vertex GL_ARB_instanced_arrays GL_ARB_map_buffer_range GL_ARB_texture_rg GL_ARB_texture_swizzle GL_ARB_vertex_array_bgra GL_EXT_texture_swizzle GL_EXT_vertex_array_bgra GL_NV_conditional_render GL_AMD_draw_buffers_blend GL_AMD_seamless_cubemap_per_texture GL_ARB_ES2_compatibility GL_ARB_blend_func_extended GL_ARB_debug_output GL_ARB_draw_buffers_blend GL_ARB_draw_elements_base_vertex GL_ARB_explicit_attrib_location GL_ARB_fragment_coord_conventions GL_ARB_provoking_vertex GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_seamless_cube_map GL_ARB_shader_texture_lod GL_ARB_texture_cube_map_array GL_ARB_texture_gather GL_ARB_texture_multisample GL_ARB_texture_query_lod GL_ARB_texture_rgb10_a2ui GL_ARB_uniform_buffer_object GL_ARB_vertex_type_2_10_10_10_rev GL_EXT_provoking_vertex GL_EXT_texture_snorm GL_MESA_texture_signed_rgba GL_NV_texture_barrier GL_ARB_get_program_binary GL_ARB_robustness GL_ARB_separate_shader_objects GL_ARB_shader_bit_encoding GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ARB_compressed_texture_pixel_storage GL_ARB_internalformat_query GL_ARB_map_buffer_alignment GL_ARB_shading_language_420pack GL_ARB_shading_language_packing GL_ARB_texture_storage GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_transform_feedback GL_AMD_shader_trinary_minmax GL_ARB_ES3_compatibility GL_ARB_arrays_of_arrays GL_ARB_clear_buffer_object GL_ARB_copy_image GL_ARB_explicit_uniform_location GL_ARB_fragment_layer_viewport GL_ARB_invalidate_subdata GL_ARB_program_interface_query GL_ARB_texture_query_levels GL_ARB_texture_storage_multisample GL_ARB_vertex_attrib_binding GL_KHR_debug GL_KHR_robustness GL_ARB_buffer_storage GL_ARB_clear_texture GL_ARB_internalformat_query2 GL_ARB_multi_bind GL_ARB_seamless_cubemap_per_texture GL_ARB_shader_draw_parameters GL_ARB_shader_group_vote GL_ARB_texture_mirror_clamp_to_edge GL_ARB_vertex_type_10f_11f_11f_rev GL_EXT_shader_integer_mix GL_INTEL_performance_query GL_ARB_clip_control GL_ARB_conditional_render_inverted GL_ARB_cull_distance GL_ARB_get_texture_sub_image GL_ARB_pipeline_statistics_query GL_ARB_texture_barrier GL_ARB_transform_feedback_overflow_query GL_EXT_polygon_offset_clamp GL_KHR_blend_equation_advanced GL_KHR_context_flush_control GL_ARB_parallel_shader_compile GL_KHR_no_error GL_MESA_shader_integer_functions GL_ARB_polygon_offset_clamp GL_ARB_texture_filter_anisotropic GL_KHR_parallel_shader_compile GL_EXT_shader_framebuffer_fetch_non_coherent GL_EXT_texture_sRGB_R8 GL_EXT_texture_shadow_lod GL_EXT_EGL_sync GL_EXT_demote_to_helper_invocation
WebGL 1 Extensions: ANGLE_instanced_arrays EXT_blend_minmax EXT_color_buffer_half_float EXT_disjoint_timer_query EXT_float_blend EXT_frag_depth EXT_shader_texture_lod EXT_sRGB EXT_texture_compression_rgtc EXT_texture_filter_anisotropic OES_element_index_uint OES_fbo_render_mipmap OES_standard_derivatives OES_texture_float OES_texture_float_linear OES_texture_half_float OES_texture_half_float_linear OES_vertex_array_object WEBGL_color_buffer_float WEBGL_compressed_texture_etc WEBGL_compressed_texture_s3tc WEBGL_compressed_texture_s3tc_srgb WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_depth_texture WEBGL_draw_buffers WEBGL_lose_context
WebGL 2 Driver WSI Info: GLX 1.4 GLX_VENDOR(client): Mesa Project and SGI GLX_VENDOR(server): SGI Extensions: GLX_ARB_create_context GLX_ARB_create_context_no_error GLX_ARB_create_context_profile GLX_ARB_create_context_robustness GLX_ARB_fbconfig_float GLX_ARB_framebuffer_sRGB GLX_ARB_get_proc_address GLX_ARB_multisample GLX_EXT_create_context_es2_profile GLX_EXT_create_context_es_profile GLX_EXT_fbconfig_packed_float GLX_EXT_framebuffer_sRGB GLX_EXT_import_context GLX_EXT_texture_from_pixmap GLX_EXT_visual_info GLX_EXT_visual_rating GLX_INTEL_swap_event GLX_MESA_copy_sub_buffer GLX_MESA_query_renderer GLX_MESA_swap_control GLX_OML_swap_method GLX_OML_sync_control GLX_SGIS_multisample GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGIX_visual_select_group GLX_SGI_make_current_read GLX_SGI_swap_control GLX_SGI_video_sync
WebGL 2 Driver Renderer: Intel Open Source Technology Center -- Mesa DRI Intel(R) Sandybridge Mobile
WebGL 2 Driver Version: 3.3 (Core Profile) Mesa 19.3.3
WebGL 2 Driver Extensions: GL_3DFX_texture_compression_FXT1 GL_AMD_draw_buffers_blend GL_AMD_seamless_cubemap_per_texture GL_AMD_shader_trinary_minmax GL_AMD_texture_texture4 GL_AMD_vertex_shader_layer GL_AMD_vertex_shader_viewport_index GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_APPLE_object_purgeable GL_ARB_ES2_compatibility GL_ARB_ES3_compatibility GL_ARB_arrays_of_arrays GL_ARB_base_instance GL_ARB_blend_func_extended GL_ARB_buffer_storage GL_ARB_clear_buffer_object GL_ARB_clear_texture GL_ARB_clip_control GL_ARB_compressed_texture_pixel_storage GL_ARB_conditional_render_inverted GL_ARB_copy_buffer GL_ARB_copy_image GL_ARB_cull_distance GL_ARB_debug_output GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_direct_state_access GL_ARB_draw_buffers GL_ARB_draw_buffers_blend GL_ARB_draw_elements_base_vertex GL_ARB_draw_instanced GL_ARB_enhanced_layouts GL_ARB_explicit_attrib_location GL_ARB_explicit_uniform_location GL_ARB_fragment_coord_conventions GL_ARB_fragment_layer_viewport GL_ARB_fragment_shader GL_ARB_framebuffer_object GL_ARB_framebuffer_sRGB GL_ARB_get_program_binary GL_ARB_get_texture_sub_image GL_ARB_half_float_pixel GL_ARB_half_float_vertex GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_internalformat_query2 GL_ARB_invalidate_subdata GL_ARB_map_buffer_alignment GL_ARB_map_buffer_range GL_ARB_multi_bind GL_ARB_occlusion_query2 GL_ARB_parallel_shader_compile GL_ARB_pipeline_statistics_query GL_ARB_pixel_buffer_object GL_ARB_point_sprite GL_ARB_polygon_offset_clamp GL_ARB_program_interface_query GL_ARB_provoking_vertex GL_ARB_robustness GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_seamless_cube_map GL_ARB_seamless_cubemap_per_texture GL_ARB_separate_shader_objects GL_ARB_shader_bit_encoding GL_ARB_shader_draw_parameters GL_ARB_shader_group_vote GL_ARB_shader_objects GL_ARB_shader_subroutine GL_ARB_shader_texture_lod GL_ARB_shader_viewport_layer_array GL_ARB_shading_language_420pack GL_ARB_shading_language_packing GL_ARB_sync GL_ARB_texture_barrier GL_ARB_texture_buffer_object GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_buffer_range GL_ARB_texture_compression_rgtc GL_ARB_texture_cube_map_array GL_ARB_texture_filter_anisotropic GL_ARB_texture_float GL_ARB_texture_gather GL_ARB_texture_mirror_clamp_to_edge GL_ARB_texture_multisample GL_ARB_texture_non_power_of_two GL_ARB_texture_query_levels GL_ARB_texture_query_lod GL_ARB_texture_rectangle GL_ARB_texture_rg GL_ARB_texture_rgb10_a2ui GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_texture_swizzle GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback_overflow_query GL_ARB_uniform_buffer_object GL_ARB_vertex_array_bgra GL_ARB_vertex_array_object GL_ARB_vertex_attrib_binding GL_ARB_vertex_buffer_object GL_ARB_vertex_shader GL_ARB_vertex_type_10f_11f_11f_rev GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_ATI_blend_equation_separate GL_ATI_texture_float GL_EXT_EGL_sync GL_EXT_abgr GL_EXT_blend_equation_separate GL_EXT_demote_to_helper_invocation GL_EXT_draw_buffers2 GL_EXT_draw_instanced GL_EXT_framebuffer_blit GL_EXT_framebuffer_multisample GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_framebuffer_object GL_EXT_framebuffer_sRGB GL_EXT_packed_depth_stencil GL_EXT_packed_float GL_EXT_pixel_buffer_object GL_EXT_polygon_offset_clamp GL_EXT_provoking_vertex GL_EXT_shader_framebuffer_fetch_non_coherent GL_EXT_shader_integer_mix GL_EXT_texture_array GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_rgtc GL_EXT_texture_compression_s3tc GL_EXT_texture_filter_anisotropic GL_EXT_texture_integer GL_EXT_texture_sRGB GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_decode GL_EXT_texture_shadow_lod GL_EXT_texture_shared_exponent GL_EXT_texture_snorm GL_EXT_texture_swizzle GL_EXT_timer_query GL_EXT_transform_feedback GL_EXT_vertex_array_bgra GL_IBM_multimode_draw_arrays GL_INTEL_performance_query GL_KHR_blend_equation_advanced GL_KHR_context_flush_control GL_KHR_debug GL_KHR_no_error GL_KHR_parallel_shader_compile GL_KHR_robustness GL_MESA_pack_invert GL_MESA_shader_integer_functions GL_MESA_texture_signed_rgba GL_NV_conditional_render GL_NV_depth_clamp GL_NV_packed_depth_stencil GL_NV_texture_barrier GL_OES_EGL_image GL_S3_s3tc
WebGL 2 Extensions: EXT_color_buffer_float EXT_disjoint_timer_query EXT_float_blend EXT_texture_compression_rgtc EXT_texture_filter_anisotropic OES_texture_float_linear WEBGL_compressed_texture_etc WEBGL_compressed_texture_s3tc WEBGL_compressed_texture_s3tc_srgb WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_lose_context
Window Protocol: x11
Off Main Thread Painting Enabled: true
Off Main Thread Painting Worker Count: 3
Target Frame Rate: 60
GPU #1
Active: Yes
Description: Mesa DRI Intel(R) Sandybridge Mobile
Vendor ID: 0x8086
Device ID: 0x0126
Driver Vendor: mesa/i965
Driver Version: 19.3.3.0
RAM: 1536

Diagnostics
AzureCanvasBackend: skia
AzureContentBackend: skia
AzureFallbackCanvasBackend: none
CairoUseXRender: 0
Display0: 1920x1080 default
DisplayCount: 1
Decision Log
HW_COMPOSITING:
blocked by env: Acceleration blocked by platform
OPENGL_COMPOSITING:
unavailable by default: Hardware compositing is disabled
GPU_PROCESS:
unavailable by env: Hardware compositing is unavailable.
WEBRENDER:
opt-in by default: WebRender is an opt-in feature
WEBRENDER_QUALIFIED:
blocked-device-too-old by env: Device too old
WEBRENDER_COMPOSITOR:
disabled by default: Disabled by default
WEBGPU:
disabled by default: Disabled by default




Media
-----

Audio Backend: alsa
Max Channels: 10000
Preferred Sample Rate: 48000
Output Devices
Name: Group
default: default
Input Devices
Name: Group
default: default

Enumerate database

Important Modified Preferences
------------------------------

browser.cache.disk.filesystem_reported: 1
browser.contentblocking.category: standard
browser.startup.homepage_override.buildID: 20200207195153
browser.startup.homepage_override.mstone: 73.0
browser.urlbar.placeholderName: Google
dom.push.userAgentID: 42f83ef6df544136a413c71d73bec23b
extensions.lastAppVersion: 73.0
media.gmp.storage.version.observed: 1
places.history.expiration.transient_current_max_pages: 112348
plugin.flash.arch: x86_64-gcc3
plugin.flash.blockliststate: 0
plugin.flash.desc: Shockwave Flash 32.0 r0
plugin.flash.lastmod_hi: 368
plugin.flash.lastmod_lo: 888031072
plugin.flash.path: /usr/lib/flashplugin-nonfree/libflashplayer.so
plugin.flash.version: 32.0.0.330
privacy.sanitize.pending: [{"id":"newtab-container","itemsToClear":[],"options":{}}]
security.sandbox.content.tempDirSuffix: 5614375e-e66c-4cdc-9772-8095de6d2cf7
security.sandbox.plugin.tempDirSuffix: 7c0f11c3-a9c7-4730-8bed-2055aeb14461
signon.importedFromSqlite: true

Important Locked Preferences
----------------------------

fission.autostart: false

Places Database
---------------

JavaScript
----------

Incremental GC: true

Accessibility
-------------

Activated: false
Prevent Accessibility: 0

Library Versions
----------------

NSPR
Expected minimum version: 4.24
Version in use: 4.24

NSS
Expected minimum version: 3.50
Version in use: 3.50

NSSSMIME
Expected minimum version: 3.50
Version in use: 3.50

NSSSSL
Expected minimum version: 3.50
Version in use: 3.50

NSSUTIL
Expected minimum version: 3.50
Version in use: 3.50

Sandbox
-------

Seccomp-BPF (System Call Filtering): true
Seccomp Thread Synchronization: true
User Namespaces for privileged processes: true
User Namespaces: false
Content Process Sandboxing: true
Media Plugin Sandboxing: true
Content Process Sandbox Level: 4
Effective Content Process Sandbox Level: 4

Rejected System Calls
---------------------

Internationalization & Localization
-----------------------------------

Application Settings
Requested Locales: ["en-US-u-va-posix"]
Available Locales: ["en-US"]
App Locales: ["und","en-US"]
Regional Preferences: ["und"]
Default Locale: "und"
Operating System
System Locales: ["en-US-u-va-posix"]
Regional Preferences: ["und"]

And about the "Latin" vs. "Other writing systems" -- now it gets complicated and interesting and a bit weird. Good catch in any case :)
Let me try to summarise (this is from my usual profile again):

  • Before changing anything, I noticed that on one of my test sites (which doesn't set any font stuff) [0], the font-family and font-size now are respected; which suprised me because I'm pretty sure this was not the case after the last firefox update, where I checked. So something else apparently has changed, and for the better.
  • But it's to early to say hooray and close the bug:
  • There's another site (and that's the Debian bug tracker) where the font size still seems bigger than what I felt I was used to. And here indeed it makes a difference if I apply the same settings as in "Fonts for: Latin" also to "Fonts for: Other Writing Systems". - I can reproduce this increase/decrease of font size at the top of e.g. https://bugs.debian.org/948542 by changing the "Proportional" size for "Other Writing systems" back and forth.

[0] my webcam, in case you want to check: https://info.comodo.priv.at/webcam/

Flags: needinfo?(gregor+mozilla)

Zibi, one thing I did notice when poking around here is that it's still possible for us to end up with

System Locales ["und"]

on Linux, because uloc_getDefault() (used in OSPreferences::ReadSystemLocales) can return an empty string. I haven't pinned down whether or how this would disrupt things, but I'm concerned it may not be handled well. Should we perhaps have a hardcoded fallback to en-US if we fail to get anything from ICU's getDefault API?

(This doesn't directly relate to Gregor's case, as his system locale as reported above is showing "en-US-u-va-posix", but it seemed like something that might lead to problems - e.g. when we try to guess what font prefs to use, in the absence of explicit lang tags.)

WDYT - are we OK with getting und as system locale, or should we force a fallback?

Gregor, could you confirm what the intl.accept_languages pref is set to in about:config for you? (It seems unfortunate that about:support doesn't report that one.)

Flags: needinfo?(gregor+mozilla)

Sure; it's set and its value is "en,en-us,de,de-at" (which matches what I set Accept-Languages to in the preferences dialog many years ago :))

An interesting observation is that in the graphical preferences, "Choose the language to display menus …" is unset/empty (in practice the UI is English, as expected); but "Choose your preferred language for displaying pages" is (unsurprisingly) the same as intl.accept_languages.

Flags: needinfo?(gregor+mozilla)

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #15)

  1. This shouldn't return empty array - Services.locale.negotiateLanguages(["en-US-u-va-posix"], ["en-US"]);

Prior to the switch to unic_langid in bug 1571915, this used to return ["en-US"] as we would expect. So this appears to be a unic_langid problem. And this is presumably why Gregor doesn't see the expected UI language in the preferences menu (comment 32). I'm concerned it could also be affecting font selection, because we may look at the app locale to determine the preferred order of CJK font prefs (although that doesn't seem to explain the original bug reported here).

Flags: needinfo?(gandalf)

Gregor - thank you for all the feedback!

I have one question - could you try a nightly build instead of stable and report the internationalization portion of about:support?

The patch I landed in this bug should hopefully switch your System Locales: ["en-US-u-va-posix"] to System Locales: ["en-US"] and it may fix the issue for you.

Flags: needinfo?(gregor+mozilla)

Sure; with firefox-75.0a1.en-US.linux-x86_64.tar.bz2 from https://www.mozilla.org/en-US/firefox/channel/desktop/ (version in help: 75.0a1 (2020-02-19) (64-bit)), about:support shows (in a new empty profile):

Application Settings
Requested Locales 	["en-US"]
Available Locales 	["en-US"]
App Locales 	["en-US"]
Regional Preferences 	["en-US"]
Default Locale 	"en-US"
Operating System
System Locales 	["en-US-u-va-posix"]
Regional Preferences 	["und"]

In the preferences, the UI language preferences seem to be gone; the page language preferences are set to en-us,en.

Font preferences are still a bit erratic: font-size seems to work (for Latin fonts), font-family still needs changes for "Other writing systems".

The about:support->Localization output is the same when I run bin/mozregression --launch 2020-02-19 (which gives me application_changeset: 1effb11e8e178a9b1f25ba171f4d52f857ebb713). Also the font settings behave the same.

Flags: needinfo?(gregor+mozilla)

Ugh, that's confusing. Thank you!

Can you tell me what's your: LC_ALL, LANGUAGE, LANG, LC_TIME, LC_MESSAGES env virables set to? You can print them with echo $LANG.

It seems that my patch in this bug didn't fix your problem and I'm not sure why - I'd expect that in Nightly your System Locales would be en-US.
I may want to take a bit more drastic action toward cleaning up our canonicalization to produce language tags, not locales.

Flags: needinfo?(gandalf) → needinfo?(gregor+mozilla)

Ah, in fact, just type locale in your command line and it should print all of the relevant ones!

Oh, have I never added them to the ticket before? Here we go:

% locale
LANG=
LANGUAGE=
LC_CTYPE=de_AT.utf8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
Flags: needinfo?(gregor+mozilla)

That's super-helpful, thank you! I was able to reproduce your bug, and I'm diving into it!

Flags: needinfo?(andynew327)

Hey, that's great indeed! Good luck in your quest :)

Assignee: nobody → gandalf
Status: NEW → ASSIGNED

Ok, here's a patch that I think should fix it.

I used Gregor's locale from comment 38 and got

Internationalization & Localization
-----------------------------------

Application Settings
Requested Locales: ["en-US"]
Available Locales: ["en-US"]
App Locales: ["en-US"]
Regional Preferences: ["en-US-posix"]
Default Locale: "en-US"
Operating System
System Locales: ["en-US-posix"]
Regional Preferences: ["en-US-posix"]

which is a set of perfectly fine and negotiable locales.

I unified canonicalization between OSPreferences and LocaleService, and moved us from using ICU's canonicalization to use unic_langid one. This has the benefit of not handling all obsolete scenarios (see [0]) and also not ending up with en_US_POSIX being turned into en-US-u-va-posix which is a workaround ICU used because many of their clients early code couldn't handle variants properly, so en-US-posix failed.

We handle variants just fine and in fact, we don't want to play with unicode extensions yet (we may switch to unic_locale eventually), so this should work quite well for us, be faster than canonicalizing via ICU, reduce ICU dependency, and put us one step closer to migrating LocaleService to store Locale instances rather than strings.

There is one place in the caller code which I had to fix - in GTK OSPreferences, which is the only place where if we failed to canonicalize input, we were throwing an error on a getter.
Now we fallback to get system locales if we can't retrieve pref locales, which I think is generally safer.

[0] https://searchfox.org/mozilla-central/rev/fca0be7e2cf2f922c9b927423ce28e8a04b3fd90/intl/icu/source/common/uloc_tag.cpp#2634

Gregor - in a couple hours there should be try builds ready for any platform here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=edc384f5bf8f022a853eb642cd58613087d78bbc

Can you download one for your platform and test it against your scenario?

Flags: needinfo?(gregor+mozilla)

I'm looking forward to trying the test build tomorrow.

Flags: needinfo?(gregor+mozilla)
Attachment #9127683 - Attachment description: Bug 1607052 - Unify language identifier canonicalization in LocaleService and OSPreferences to use unic_langid. → Bug 1607052 - Unify language identifier canonicalization in LocaleService and OSPreferences to use unic_langid, and extend GTK's RegionalPrefs reading to fallback on SystemLocales.
Pushed by zbraniecki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c3e43b7b1376
Unify language identifier canonicalization in LocaleService and OSPreferences to use unic_langid, and extend GTK's RegionalPrefs reading to fallback on SystemLocales. r=jfkthame

Ok C++, this is how you want the day to go? You want a fight? Fine. I'm taking my toys and moving all my memory management to Rust.
Don't call me.

Flags: needinfo?(gandalf)

To provide some context in a less joking manner. This leaks because I create *LanguageIdentifier hand it to C++ and then never deallocate. I could just add unic_langid_destroy(loc); to the C++ code or use UniquePtr.

Instead, I'm going to just move the whole operation to Rust to avoid bouncing back and forth between C++ and Rust.

Pushed by zbraniecki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4f406ca4f899
Unify language identifier canonicalization in LocaleService and OSPreferences to use unic_langid, and extend GTK's RegionalPrefs reading to fallback on SystemLocales. r=jfkthame

This one is simpler. We just fixed accidentally a bug where we still treated empty string as a valid locale normalized to und. Now we just reject it together with all other bogus input:

Before:

  Services.prefs.setStringPref(
    PREF_REQUESTED_LOCALES,
    "de,2,#$@#,pl,ąó,!a2,DE-at,,;"
  );

  let locales = localeService.requestedLocales;
  Assert.equal(locales[0], "de");
  Assert.equal(locales[1], "pl");
  Assert.equal(locales[2], "de-AT");
  Assert.equal(locales[3], "und");
  Assert.equal(locales.length, 4);

After:

  Services.prefs.setStringPref(
    PREF_REQUESTED_LOCALES,
    "de,2,#$@#,pl,ąó,!a2,DE-at,,;"
  );

  let locales = localeService.requestedLocales;
  Assert.equal(locales[0], "de");
  Assert.equal(locales[1], "pl");
  Assert.equal(locales[2], "de-AT");
  Assert.equal(locales.length, 3);
Flags: needinfo?(gandalf)
Pushed by zbraniecki@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ec0bb875399b
Unify language identifier canonicalization in LocaleService and OSPreferences to use unic_langid, and extend GTK's RegionalPrefs reading to fallback on SystemLocales. r=jfkthame

Gregor - Can you verify if this patch fixed it for you? you should be able to test against today's (or tomorrows) nightly.

Flags: needinfo?(gregor+mozilla)

Ok, I downloaded firefox-75.0a1.en-US.linux-x86_64.tar.bz2, which says 75.0a1 (2020-02-22) (64-bit) in the help/abozut screen, and started it with a new empty profile.

about:support reports:

Application Settings
Requested Locales 	["en-US"]
Available Locales 	["en-US"]
App Locales 	["en-US"]
Regional Preferences 	["en-US-posix"]
Default Locale 	"en-US"
Operating System
System Locales 	["en-US-posix"]
Regional Preferences 	["en-US-posix"]

Adjusting the font-family and font-size works; i.e. my settings for both are immediately respeced, and it is enough if I make the changes for "Latin" script.

So it sems that all is Good™ with the current state of affairs.

Thanks again for your work on this bug.

Flags: needinfo?(gregor+mozilla)

This bug still has the leave-open keyword but it looks like it's fixed now.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Keywords: leave-open
Resolution: --- → FIXED

Thank you gregor!

status-firefox73: --- → wontfix
status-firefox74: --- → wontfix
status-firefox75: --- → fixed
status-firefox-esr68: --- → unaffected
Flags: in-testsuite+
Target Milestone: --- → mozilla75
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: