1578492 - Statuspage operator failing with 401 Client Error: Unauthorized for url: https://api.statuspage.io/v1/pages

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Anthony Miyaguchi [:amiyaguchi]]

All of the statuspage operators failed this weekend (2019-08-30 to 2019-09-02).

Try 2 out of 3
Exception:
401 Client Error: Unauthorized for url: https://api.statuspage.io/v1/pages
Log: Link
Host: airflow-prod-airflow-app-1-worker-0.airflow-prod-airflow-app-1-worker.prod-airflow.svc.cluster.local
Log file: /app/logs/main_summary/main_summary_register/2019-09-02T01:00:00+00:00.log
Mark success: Link

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

AIRFLOW_CTX_TASK_ID=main_summary_register
AIRFLOW_CTX_DAG_ID=main_summary
AIRFLOW_CTX_EXECUTION_DATE=2019-09-02T01:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2019-09-02T01:00:00+00:00
[2019-09-03 05:02:52,307] {logging_mixin.py:95} INFO - [2019-09-03 05:02:52,306] {statuspage_client.py:39} INFO - status-code 401 for https://api.statuspage.io/v1/pages
[2019-09-03 05:02:52,307] {logging_mixin.py:95} INFO - [2019-09-03 05:02:52,307] {statuspage_client.py:41} ERROR - {"error":"Could not authenticate"}
[2019-09-03 05:02:52,307] {models.py:1788} ERROR - 401 Client Error: Unauthorized for url: https://api.statuspage.io/v1/pages
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1657, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 103, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 108, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/app/pvmount/telemetry-airflow/dags/utils/status.py", line 24, in callable
    conn = DatasetStatusHook().get_conn()
  File "/app/pvmount/telemetry-airflow/plugins/statuspage/hook.py", line 30, in get_conn
    return DatasetStatus(self.api_key)
  File "/app/pvmount/telemetry-airflow/plugins/statuspage/dataset_client.py", line 9, in __init__
    api_key, "Firefox Operations", "Data Engineering Datasets"
  File "/app/pvmount/telemetry-airflow/plugins/statuspage/statuspage_client.py", line 21, in __init__
    self.page_id = self.get_page_id(page_name)
  File "/app/pvmount/telemetry-airflow/plugins/statuspage/statuspage_client.py", line 52, in get_page_id
    resp = self._request("get", "pages")
  File "/app/pvmount/telemetry-airflow/plugins/statuspage/statuspage_client.py", line 42, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 401 Client Error: Unauthorized for url: https://api.statuspage.io/v1/pages
[2019-09-03 05:02:52,409] {models.py:1817} INFO - All retries failed; marking task as FAILED

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

I've rotated the API key via the Statuspage UI and replaced the password field in the statuspage_default connection string. I also tested the API key manually using https://api.statuspage.io/v1/pages route -- it seems to work correctly.

However, retriggering the StatuspageOperator seems to fail in the same way.

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: statuspage.py

Attached is the script used to manually verify the key.

$ export STATUSPAGE_API_KEY=...
$ ./statuspage.py | jq '.[0].page_description'
"This page displays the status for services run by Mozilla's Firefox Engineering Operations teams."

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

I've also run this locally with the following steps:

From telemetry-airflow:

make build
make up

# add a new connection: statuspage_default, with type HTTP and the STATUSPAGE_API_KEY as the password

container_id=$(docker ps | grep airflow_web | cut -d' ' -f1)
docker exec -it $container_id bash
airflow test main_summary main_summary_register 20190901

--------------------------------------------------------------------------------
Starting attempt 1 of 3
--------------------------------------------------------------------------------

[2019-09-03 20:42:16,286] {models.py:1593} INFO - Executing <Task(PythonOperator): main_summary_register> on 2019-09-01T00:00:00+00:00
[2019-09-03 20:42:16,309] {python_operator.py:95} INFO - Exporting the following env vars:
AIRFLOW_CTX_TASK_ID=main_summary_register
AIRFLOW_CTX_DAG_ID=main_summary
AIRFLOW_CTX_EXECUTION_DATE=2019-09-01T00:00:00+00:00
[2019-09-03 20:42:16,906] {statuspage_client.py:39} INFO - status-code 200 for https://api.statuspage.io/v1/pages
[2019-09-03 20:42:17,932] {statuspage_client.py:39} INFO - status-code 200 for https://api.statuspage.io/v1/pages/76k9j8n4y3zt/component-groups
[2019-09-03 20:42:18,426] {statuspage_client.py:39} INFO - status-code 200 for https://api.statuspage.io/v1/pages/76k9j8n4y3zt/components
[2019-09-03 20:42:18,428] {status.py:27} INFO - Statuspage component 0f5tk0y5zynh is registered with Main Summary
[2019-09-03 20:42:18,429] {python_operator.py:104} INFO - Done. Returned value was: None

Comment 5

[Anthony Miyaguchi [:amiyaguchi]]

This issue is now resolved with a recent redeploy of Airflow. https://workflow.telemetry.mozilla.org/log?task_id=main_summary_register&dag_id=main_summary&execution_date=2019-08-30T01%3A00%3A00%2B00%3A00

The StatuspageOperator accepts the API key from three sources -- in code, from an environment variable, or set in the connection database. In the production server, the key was being set by the environment variable. Since the environment takes precedence over the connection string, the manually inserted key wasn't being recognized.

The key is no longer being managed by environment variable, but instead through the Airflow UI for connections.