1914061 - Sitemap urls are using GCP storage links which are being rejected by Google search indexer since it is not a property we own

Status: RESOLVED FIXED

(Cloud Services :: Operations: Bugzilla, enhancement)

Description

[David Lawrence [:dkl]]

Currently search indexing from Google is not working according to the search console for BMO. After looking through the sitemap upstream specifications, I did not realize when I implemented this that the URL being pointed too needs to be owned by the same owner as the host site. Most people host the sitemap index file(s) on the local web server which is not a problem. Since we moved from AWS to GCP, we not longer have a shared filesystem across all web heads. So I moves the sitemap files to a GCP storage bucket and then put the link to it in the robots.txt file.

Sitemap: https://storage.googleapis.com/storage/v1/b/bugzilla-prod-prod-sitemap/o/sitemap_index.xml?alt=media

Since we do not own https://storage.googleapis.com I think it is being rejected. My question that I brought up with jbuck at MozWeek was should I download the contents on the backend from GCP and then serve it directly to the client, or can we somehow do a redirect from the bugzlla.mozilla.org site to file still hosted on GCP? Option 1 is simple on my end and would require no infrastructure change but would not be performant. Option 2, if implemented, might look something like this:

Sitemap: https://bugzilla.mozilla.org/sitemap/storage/v1/b/bugzilla-prod-prod-sitemap/o/sitemap_index.xml?alt=media

And then you guys would add the redirect in GCP to look for https://bugzilla.mozilla.org/sitemap/ and then redirect it to https://storage.googleapis.com/...

Please let me know if this is still doable and we can set it up on bugzilla-dev.allizom.org for testing.

Thanks

Comment 1

[Jon Buckley [:jbuck]]

Two options I can think of:

1. We can setup a location within nginx to serve requests from GCS like so:

location /sitemap {
  proxy_redirect off;
  # Adding this as a variable forces occasional re-resolution;
  set $gcs_url "https://storage.googleapis.com";
  proxy_pass $gcs_url/{{ .Values.gcsBucket }}$request_uri;
}

2. We could use the Cloud Storage FUSE CSI driver to mount the bucket within a pod and serve sitemaps "locally" https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver

Comment 2

[David Lawrence [:dkl]]

(In reply to Jon Buckley [:jbuck] from comment #1)

Two options I can think of:

1. We can setup a location within nginx to serve requests from GCS like so:

location /sitemap {
  proxy_redirect off;
  # Adding this as a variable forces occasional re-resolution;
  set $gcs_url "https://storage.googleapis.com";
  proxy_pass $gcs_url/{{ .Values.gcsBucket }}$request_uri;
}

Question would be whether the spider would see the end resulting URL and still fail to load it as it is not owned by us. But we could try it and see what happens.

2. We could use the Cloud Storage FUSE CSI driver to mount the bucket within a pod and serve sitemaps "locally" https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver

This would be the most bullet proof method and would serve the same purpose that we used to have with AWS and its shared file system on each web head. Basically boils down to which one is the least amount of maintenance and less likely to break in the future.

Comment 3

[BMO Github Automation]

Attachment: [mozilla-bteam/bmo] Bug 1914061 - Sitemap urls are using GCP storage links which are being rejected by Google search indexer since it is not a property we own (#2305)

Comment 4

[BMO Github Automation]

Authored by https://github.com/dklawren
https://github.com/mozilla-bteam/bmo/commit/3f7b0b0c7473fcd800a2700bc2422be44af36c75
[master] Bug 1914061 - Sitemap urls are using GCP storage links which are being rejected by Google search indexer since it is not a property we own