1907914 - Google search on bugzilla doesn't include recent bugs

Status: REOPENED

(bugzilla.mozilla.org :: General, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

See https://www.google.com/search?sitesearch=bugzilla.mozilla.org&q=Disable+cert_verifier.cert_trust_evaluation_time+metric+on+Android+

Comment 1

[David Lawrence [:dkl]]

Bug 1907918 has been fixed and deployed so the Google search results should hopefully catch up soon.

Comment 2

[Jeff Muizelaar [:jrmuizel]]

Can you confirm that they're working in the Google Search Console?

It looks the sitemap files are xml.gz that are served as application/octet-stream so it's not obvious that indexers will be able to read them.

Comment 3

[David Lawrence [:dkl]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #2)

Can you confirm that they're working in the Google Search Console?

It looks the sitemap files are xml.gz that are served as application/octet-stream so it's not obvious that indexers will be able to read them.

Nothing has changed recently with regard to content-type and the indexing used to work not long ago. I can check the search console to see if anything is happening.

Comment 4

[Jeff Muizelaar [:jrmuizel]]

The search in the description still doesn't work so I'm reopening. What did the search console say?

Comment 5

[Jeff Muizelaar [:jrmuizel]]

Bug 1898649 is the bug the search is supposed to find

Comment 6

[David Lawrence [:dkl]]

Still working on figuring out this issue. Had to jump some hurdles to get the bugzilla site re-verified so I could see the Search Console but I am in now. It shows that only ~280k pages are being indexed which is low. I am thinking that my robots.txt file is being rejected since its sitemap index file is being hosted by google cloud storage and not under the bugzilla.mozilla.org namespace.

User-agent: *
Disallow: /
Crawl-delay: 30


Allow: /$
Allow: /index.cgi

Allow: /page.cgi
Disallow: /page.cgi*id=voting*
Disallow: /page.cgi*id=productdashboard*
Disallow: /page.cgi*id=user_activity*

Allow: /show_bug.cgi
Disallow: /show_bug.cgi*ctype=*
Disallow: /show_bug.cgi*format=multiple*

Allow: /describecomponents.cgi
Allow: /describekeywords.cgi
Disallow: /showdependency*
Disallow: /banmyipforever.cgi

Allow: /extensions/OpenGraph/web/

Sitemap: https://storage.googleapis.com/storage/v1/b/bugzilla-prod-prod-sitemap/o/sitemap_index.xml?alt=media

According to the sitemap docs, when you link to an external site for a sitemap index file, the external site must also be verified to be owned by the main sites owner. If this is the case, I might have to rework it to dynamically load the cloud storage files and serve them from a cgi under the bugzilla.mozilla.org domain. Not hard just means more work to get this right.

Comment 7

[Jeff Muizelaar [:jrmuizel]]

It looks like Sitemap url is no longer an external site but the query above still doesn't find the bug.

Comment 8

[David Lawrence [:dkl]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #7)

It looks like Sitemap url is no longer an external site but the query above still doesn't find the bug.

Yeah sorry this is not working quite right yet. I have been trying to nail down the exact cause with mixed results. One issue is that it takes a while in between debugging for Google to try and index the site again and report the results. I tweak the robots.txt and sitemap url scheme and then resubmit the sitemap url to Google search console, and it takes a couple days for the result to show up if it worked or now. I am still considering just serving the sitemap files directly from the backend but I am trying one last change first. Sorry for the inconvenience.

Comment 9

[David Lawrence [:dkl]]

Bug 1917348 has landed and is in production. Now it might take a day or two for Google to try the reindexing of BMO. So I will check back then and see if anything has improved.

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Looks like it still doesn't work.

Comment 11

[David Lawrence [:dkl]]

Attachment: image.png

Sorry it is taking so long. I am not sure why Google is take its time catching up. The graph is slowly growing. It also says there are 100k pages that are crawled but not indexed yet which may contain the missing bugs. Not sure how to speed up this process.

Comment 12

[Jeff Muizelaar [:jrmuizel]]

(In reply to David Lawrence [:dkl] from comment #11)

Sorry it is taking so long. I am not sure why Google is take its time catching up. The graph is slowly growing. It also says there are 100k pages that are crawled but not indexed yet which may contain the missing bugs. Not sure how to speed up this process.

Can you check bugzilla.mozilla.org/show_bug.cgi?id=1898649 using the search console "URL inspector" to see why it hasn't been indexed?

Comment 13

[Jeff Muizelaar [:jrmuizel]]

That should at least say whether the sitemap points at it.

Comment 14

[David Lawrence [:dkl]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #12)

(In reply to David Lawrence [:dkl] from comment #11)

Sorry it is taking so long. I am not sure why Google is take its time catching up. The graph is slowly growing. It also says there are 100k pages that are crawled but not indexed yet which may contain the missing bugs. Not sure how to speed up this process.

Can you check bugzilla.mozilla.org/show_bug.cgi?id=1898649 using the search console "URL inspector" to see why it hasn't been indexed?

It says Discovered - currently not indexed along with 100k other URLs so it should be coming soonish.

Comment 15

[Jeff Muizelaar [:jrmuizel]]

Has any progressed been made on those urls?

Comment 16

[David Lawrence [:dkl]]

Attachment: image.png

It is improving but at a snails pace. As you can see by the graph, Google now knows about the bug pages but according to other data if you drill down:

"Discovered - currently not indexed - 1,501,810 pages"

So eventually once all pages are indexed properly, it will only need to process any new pages from that point on and will be much faster.

Comment 17

[Jeff Muizelaar [:jrmuizel]]

https://www.google.com/search?sitesearch=bugzilla.mozilla.org&q=Disable+cert_verifier.cert_trust_evaluation_time+metric+on+Android+ now works.

https://www.google.com/search?q=%22Large+regression+in+Chromium-as-release+performance%22 doesn't yet though.

Dave, can you give an update on how many pages are left to be indexed?

Comment 18

[David Lawrence [:dkl]]

Attachment: image.png

Again, I am very sorry that this taking so long. From all of the docs I have read about this, it is just a matter of waiting til Google gets around to it. The docs state that Google uses an algorithm to deteremine how many pages it will index each day based on perceived performance of the site. That way the crawler does not negatively impact the site. It ignore whatever frequency settings you put in robots.txt. As you can see from the attachment, there are 1,481,851 pages "Disovered - currently not indexed" which should be most all of the public bugs in BMO. The last index date is October 17th. Maybe the next step would be to find someone in Mozilla with a Google contact to file a support ticket to see if things can be sped up in our case.

Comment 19

[Jeff Muizelaar [:jrmuizel]]

What's the state now?

Comment 20

[David Lawrence [:dkl]]

Attachment: bug 1907914 index status

From a BMO standpoint, we are technically doing nothing wrong according the the specifications of sitemaps. etc. It is just that Google is very slow with completing the indexing of pages that it knows exists and are not blocked by the robots.txt file. The above screen shot says that it has been crawled but it not yet indexed. I requested an index of that particular URL just to see how long it takes but would not hold my breath.

Comment 21

[David Lawrence [:dkl]]

Attachment: Sitemap index status as of July 6th

As of July 6th, the sitemap is still being accessed successfully and Google got 1,834,281 pages from it that it is allowed to index. This lines up as the missing bugs are private so we do not add them to the sitemap.

Comment 22

[David Lawrence [:dkl]]

Attachment: Reasons for pages not being indexed

This screenshot shows that 1,404,756 pages out of 1,834,281 total discovered are just simply waiting to be indexed. I do not know why it is being held up or know who I would need to speak to get it moving faster.

Comment 23

[Jeff Muizelaar [:jrmuizel]]

I asked Google for help and got a reply that Google has internal policies that prevent favouritism or influencing search results which makes it difficult for them to help.