1606719 - RemoteSettings in stage returns HTTP 403 when writing to cfr-ml-models

Status: RESOLVED FIXED

(Cloud Services :: Server: Remote Settings, defect)

Description

[Victor Ng [:vng] [:crankycoder]]

We're getting HTTP 403 errors in the stage enviroment when writing to remotesettings:

I can't reproduce this error on the dev instance of kinto - I think this is a configuration error on the RemoteSettings side.

[
 {
   "textPayload": "Response: <Response [403]>\n",
   "insertId": "9kjzk5ud7lm49i4am",
   "resource": {
     "type": "k8s_container",
     "labels": {
       "project_id": "moz-fx-data-cfre-nonprod-e985",
       "cluster_name": "cfrexperiment-nonprod-v1",
       "pod_name": "cfrexperiment-stage-data-cfrexperiment-nmcron-1-15779952005fq4l",
       "container_name": "cfrexperiment",
       "namespace_name": "stage-data-cfrexperiment",
       "location": "us-west1"
     }
   },
   "timestamp": "2020-01-02T20:09:41.222362645Z",
   "severity": "INFO",
   "labels": {
     "k8s-pod/app_kubernetes_io/component": "app",
     "k8s-pod/app_kubernetes_io/version": "1.0.0",
     "k8s-pod/jenkins-build-id": "51",
     "k8s-pod/app_kubernetes_io/name": "cfrexperiment",
     "k8s-pod/app_kubernetes_io/instance": "stage",
     "k8s-pod/app_kubernetes_io/part-of": "cfrexperiment",
     "k8s-pod/app_kubernetes_io/managed-by": "jenkins",
     "k8s-pod/controller-uid": "76d837a9-2d9a-11ea-884b-42010a8a0051",
     "k8s-pod/fullname": "cfrexperiment-stage-data-cfrexperiment-nmcron-1",
     "k8s-pod/job-name": "cfrexperiment-stage-data-cfrexperiment-nmcron-1-1577995200"
   },
   "logName": "projects/moz-fx-data-cfre-nonprod-e985/logs/stdout",
   "receiveTimestamp": "2020-01-02T20:09:46.881473891Z"
 }
]

Comment 1

[Victor Ng [:vng] [:crankycoder]]

:wezhou can you take a look at this?

The docs at: https://docs.kinto-storage.org/en/stable/api/1.x/buckets.html seem to indicate we do not have write permissions with an HTTP 403 error.

Comment 2

[mlopatka]

:k88hudson any insights as to where these permissions are managed?

Comment 3

[:wezhou]

:wezhou can you take a look at this?

2 questions in order to help trobleshoot,

1. which user are you using to write?
2. which collection are you trying to write to?

any insights as to where these permissions are managed?

The permissions are managed by https://github.com/mozilla-services/remote-settings-permissions repo.

And just FYI, per https://bugzilla.mozilla.org/show_bug.cgi?id=1601303, we have granted the cfr user write permission to the cfr-ml-models collection.

Comment 4

[Victor Ng [:vng] [:crankycoder]]

This turned out to be a misunderstanding of how to apply the YAML. We were only pulling in the kinto.stage.yaml. To get all the configuration in stage, we needed both stage and prod YAML files.