1439377 - Building a country/language selector page for better crawlability

Status: RESOLVED FIXED

(www.mozilla.org :: General, enhancement)

Description

[Raphael Raue [:raue] [:rraue]]

As described in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1438302 Google has some problems with toosimilar locales versions. 

One puzzle piece could be, that in our actual setup crawlers can only navigate via hreflang from the us to other versions. If the crawler would find the locale versions directly from every document and not just via external links, it could connect and differentiate the content better. 

I suggest, we change the selector in <div class=“lang-switcher"> to a menu with actual a href links. The links should be in the DOM. We should do this on every page of www, where the select menu is now as well.

Comment 1

[Raphael Raue [:raue] [:rraue]]

I digged deeper last week and I am still sure, this problem (now not Serbian but Indonesian) is a Google problem. But I found some issues which could have lead to this Google problem:

1. All our en-GB pages are nearly complete duplicates to the en-US equivalents 
2. Google doesnt like duplicate content, they recomment canonicals from duplicates to the original
3. Our Search traffic is going down for en-GB/firefox/new/ https://drive.google.com/file/d/1MUtEjYM9hTk56NanpiJ9IM5lH9BaWUyp/view?usp=sharing
4. A lot of pages (29 out of 110) in /en-GB/ are flagged as duplicate content and not even indexed https://docs.google.com/spreadsheets/d/16Q7dqjoExW-zSC44uDxCqbxTt6ZUt_bTmug8RGwdNG8/edit?usp=sharing
5. But if I understand it right, we want a UK version in the index (it could be a strategical decision to change to an en version, without localisation of ca, us, gb, that would make it easier but harder in the future to target these markets with a specific campaign or version)

We should try the following to convince Google its not totally unique but localised content. We can be only 100% sure, that something like the reported bug will not happen again, when the content is really unique, but we can try first, if some changes will not do it as well:

The most important pages are /en-GB/, /en-GB/firefox/ and /en-GBfirefox/new/. Apple.com for example has very similar content as well but indicates in the <title> which version it is with (United Kingdom) or (UK). We should do this as well for this three pages to test.

For all en-GB pages we should change <h2 class="nav-logo"><a href=“/en-GB/">Mozilla</a></h2> to <h2 class="nav-logo"><a href=“/en-GB/“>Mozilla United Kingdom</a></h2> or Mozilla (UK)

/en-GB/firefox/new/ doesn’t have the navigation, here we should change: 

<h2><a class="firefox" href="/en-GB/firefox/"><img class="" src="/media/img/logos/firefox/logo-quantum-wordmark-white.bd1944395fb6.png" srcset="/media/img/logos/firefox/logo-quantum-wordmark-white-high-res.3ccf0070280e.png 1.5x" width="133" alt="Firefox" height="50"></a></h2>
        <h2><a class="mozilla" href="/en-GB/"><img src="/media/img/logos/mozilla/wordmark-dark.731d4dab7347.svg" alt="Mozilla" width="101" height="34"></a></h2>


To

<h2><a class="firefox" href="/en-GB/firefox/" title=“Firefox (UK)”><img class="" src="/media/img/logos/firefox/logo-quantum-wordmark-white.bd1944395fb6.png" srcset="/media/img/logos/firefox/logo-quantum-wordmark-white-high-res.3ccf0070280e.png 1.5x" width="133" alt=“Firefox (UK)“ height="50"></a></h2>
        <h2><a class="mozilla" href="/en-GB/" title=“Mozilla (UK)”><img src="/media/img/logos/mozilla/wordmark-dark.731d4dab7347.svg" alt=“Mozilla (UK)“ width="101" height=“34"></a></h2>

These are small changes, but if could help Google to differentiate the pages better. Additional we should solve the crawling locales  problem in this bug, so crawlers can navigate between the pages as well not just via hreflang: https://bugzilla.mozilla.org/show_bug.cgi?id=1439377

Comment 2

[Alex Gibson [:agibson]]

> I suggest, we change the selector in <div class=“lang-switcher"> to a menu with actual a href links.

While fixing the crawlability issue here is someting we need to work out, I'm not sure the best possible way to do this is by rebuilding all the functionality of a native <select> menu using HTML, CSS and JS. Doing so at a high stndard is often needlessly complex, and a great source of accassibility bugs and browser quirks. Rather than try and reinvent the wheel just to satisfy search engines, I think we need to consider a solution that works great for both users and robots.

Comment 3

[Raphael Raue [:raue] [:rraue]]

For a better understanding, why a link menu is better then just the hreflang: hreflang is not a ranking signal. It would be too easy to use for spam, as users do not at all interact with hreflang. Therefore hreflang doesn't pass any pagerank. Only links do and now we do not pass any internal pagerank to our locales. Internal pagerank is very important for rankings and crawlers understanding a sites architecture.

A good example is /?scene=2 as it has far less external links then /new/ but before we set noindex it ranked for several high traffic keywords instead of /new/. Thats because of the internal linking and Google understanding, that we guide users to scene to download, not to /new/.

Internal links do no count as a ranking signal per se but they do as a crawling signal if they are in the DOM (links in javascript are differently treated), but if users use links, they are a very strong ranking signal. Therefor we don't need to necessarily change the lang-switcher, but we should implement a way for users and crawlers to get via a link to the other (language)versions of the document they are right now. This is what I need for the crawler and indirect for users coming from search engines probably not to the right locale.

Comment 4

[Alex Gibson [:agibson]]

We discussed this a bit on Slack, and one alternative possible solution to fixing the issue that :raue suggested (rather than changing the dropdown menu) was building an locale HTML sitemap page, linked to from the site footer. This is similar to Apple's approach:

https://www.apple.com/choose-country-region/

Comment 5

[Raphael Raue [:raue] [:rraue]]

Yes, as a first approach because of the reasons menationed by Alex, we should build such a page.

Comment 6

[Alex Gibson [:agibson]]

Eric, this one will require some design resources.

Comment 7

[Eric Renaud [:erenaud]]

Tagging JRouse and MWeaver to pick up the design work here.

Comment 8

[Jennifer Rouse]

mark is working on it this week.

Comment 9

[Jennifer Rouse]

The design is ready. Assets can be found on github. Please let me know if/when you have questions.

https://github.com/mozilla/bedrock/issues/5827

Comment 10

[Raphael Raue [:raue] [:rraue]]

Fixed