1364963 - Allow search engines to access tag directories

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: Tags / flags, enhancement)

Description

[John Whitlock [:jwhitlock]]

What feature should be changed? Please provide the URL of the feature if possible.
==================================================================================
A wiki page can be tagged, and the pages with a tag are one click away:

https://developer.mozilla.org/en-US/docs/tag/HTML

This page is currently blocked in robots.txt, and has a meta tag to reject indexing and link following by search engines.

The proposed change:

- Research SEO practices on tag pages, to ensure this is a good idea
- Allow robots to scrape the page
- Allow robots to follow links to and from tag indexes
- Disallow search engines from adding indexes to their index (not valuable content)
- Improve page performance of tag pages

What problems would this solve?
===============================
The tag link may convey useful information to search engines, to link together similar content.

Who would use this?
===================
Search engines may display better results, relating pages that aren't linked in a slug hierarchy.

What would users see?
=====================
Users may see better search results

What would users do? What would happen as a result?
===================================================
Users may get better search results

Is there anything else we should know?
======================================
This was brought up in the work for bug 1353079, which added robots meta tags.

It may be important to fix other issues with tags, such as localization (bug 671721), before allowing scraping of tags.

I believe the page was first excluded in robots.txt because of performance issues, and work will be needed to reduce the database load when rendering these pages.

Comment 1

[Florian Scholz (Open Web Docs)]

SEO agency to comment on this bug

Comment 2

[Eric Shepherd [:sheppy]]

Nilaye, do you have thoughts on this? I wonder if these tag index pages should be noindex but not nofollow, but I'm not an expert.

Comment 3

[Raphael Raue [:raue] [:rraue]]

My 2 cents on this:

- Tag pages on mdn are quite poor quality right now, so they shouldn't be indexed.
- In general are useful tag/catgeory pages worth being indexed, but they really have highlight the best content and they should compete with pages (/tag/html vs /html)

There are two ways of deindexing pages and there is no "better" way in general. It depends.

- The question asked is: how much external inlinks have this pages
- If a lot and valuable backlings noindex, follow is better
- If the pages have nearly no backlinks and are useless, robots.txt is the better place, as it saves crawling budget, what is a value itself for big websites (for small ones I would always recommend noindex, follow)

I will try to answer the backlink question.

Comment 4

[John Whitlock [:jwhitlock]]

I believe re-implementing the tag system will be easier than making the quality of existing tags better. We can revisit indexing of any new tag pages.