1413289 - Annotate, add nofollow to links to new documents

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Wiki pages, enhancement, P1)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
Users could see that a link (in-content or sidebar) goes to a non-existing document.

Search engine crawlers would not follow links that will lead to a 404.

Who has this problem?
=====================
All visitors to MDN

How do you know that the users identified above have this problem?
==================================================================
The Google crawler sees thousands of non-existent pages returning 404 as it crawls the site. This is expected on a wiki, since a link to a non-existant document is often used as a placeholder. However, Google does spend time crawling these URLs that it could spend crawling content.

How are the users identified above solving this problem now?
============================================================
Google takes note of the 404 and ignores the page.

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
In-content links are already parsed to see if they are to a new document, and an attribute class="new" is added.  This could be extended to add a rel="nofollow" attribute, suggesting to the search engine to ignore them.

The top 404 links are in the Quick Links and Subnav sidebars, since they appear on many pages. The "new docs" parser can be run on these as well, to add the class="new" and rel="nofollow" attributes. 

Is there anything else we should know?
======================================
Tracked in Taiga:
https://tree.taiga.io/project/viya-mdn-durable-team/task/3589

Comment 1

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/3d3c0b66df8de3f39997de6dc89cb186e0f3a68f
bug 1413289: Refactor annotateLinks tests

Convert annotateLinks test from one big test case to several pytest-style
tests.

https://github.com/mozilla/kuma/commit/e4ff389c73c2b9cad63568756dda3d29b6a81003
bug 1413289: Sort class attributes in links

The normalize_html helper does not normalize attribute order, so pick
sorted order.

https://github.com/mozilla/kuma/commit/8f87943972b398885784eb3b00d4f4588e4960a7
bug 1413289: Remove impossible code paths

Remove two if statements that are never False

https://github.com/mozilla/kuma/commit/294b58041b93d0295ced6c7373005d8b79830e2b
bug 1413289: Refactor document parsing tests

Move Document parsing tests to test_models_document, and use pytest
standard conventions.

https://github.com/mozilla/kuma/commit/0ecad7c52edb41e5ab1e8dc05da7ce320cdf1908
bug 1413289: Refactor extractSection tests

Convert to pytest functions, and reduce content of test cases

https://github.com/mozilla/kuma/commit/8c6062ea45ee3dd8507463d65aa73ac231c9a0bd
bug 1413289: Refactor Extractor tests

https://github.com/mozilla/kuma/commit/ba5389bd16bc6c229971ff7724b7eeec954d7345
bug 1413289: Add rel=nofollow to missing docs

This suggests to search crawlers that the links is to a 404 and
shouldn't be followed.

https://github.com/mozilla/kuma/commit/952dcb855f5a8a374de935674707ccad28a1cb12
bug 1413289: Add test for Extractor.section

https://github.com/mozilla/kuma/commit/91344a8e9e45dfa52b3adf549c24e63221ce1f0d
bug 1413289: Annotate 404 links in the sidebars

Annotate links to non-existing pages in the Quick Links sidebar and the
Zone sub-pages navigation sidebar.

https://github.com/mozilla/kuma/commit/5e4461a2f65695ad6cec95b9cef16ac7ff72380a
bug 1413289: Fix docstrings

https://github.com/mozilla/kuma/commit/959d233302f7ea438510f2d9e6f6875934649bae
Merge pull request #4495 from jwhitlock/annotate-links-1413289

Bug 1413289: Annotate links to missing documents with rel=nofollow

Comment 2

[John Whitlock [:jwhitlock]]

Deployed to production. Need to regenerate pages for the styling to take effect.

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/7b95a28f59ef01ae8563061b9b7363df2ecc7ca1
bug 1413289: Handle collation-matching slugs

Under MySQL's utf8_general_ci collation, some slugs like
Glossaire/Recursion and Glossaire/Récursion match (e=é), but the
lowercase versions do not. Fall back to checking each slug one at a
time, which includes checking missing slugs twice.

https://github.com/mozilla/kuma/commit/6746367dc92711c9d05282dde80dd3a32bc32dd9
Merge pull request #4532 from jwhitlock/collation-matches-1413289

bug 1413289: Handle collation-matching slugs

Comment 4

[John Whitlock [:jwhitlock]]

This caused some additional issues, tracked in bug 1418133. Deployed to production.