tecken deploy: 2023.09.13
Categories
(Tecken :: General, task, P2)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
References
Details
We want to do a deploy for tecken tagged 2023.09.13.
It consists of the following:
510d292: bug 1746940: fix FileUpload admin; add fake data generator (willkg)
12a8155: bug 1851909: switch to django-admin-cursor-paginator (willkg)
adf576c: bug 1746940: add debug fields to FileUpload model (willkg)
0bfc681: bug 1746940: implement sym data capture at upload (willkg)
0997218: bug 1746940: implement code_file/code_id lookup for download API (willkg)
e4ed26d: bug 1746940: add sym debug data to File page (willkg)
27e4d6e: bug 1746940: remove errant logging.info line (willkg)
23daea4: bug 1746940: fix upload-sym file for Windows modules (willkg)
b84b0ba: bug 1746940: rework download API to support /code_file/code_id/sym_file (willkg)
7614fbc: bug 1746940: fix minor issues with last commit (willkg)
c803233: Update architecture diagram (willkg)
|
Assignee | |
Comment 1•2 years ago
|
||
Tagged 2023.09.13:
tag 2023.09.13
Tagger: Will Kahn-Greene <willkg@mozilla.com>
Date: Wed Sep 13 08:47:18 2023 -0400
Tag 2023.09.13 (bug #1746940)
`510d292`: bug 1746940: fix FileUpload admin; add fake data generator (willkg)
`12a8155`: bug 1851909: switch to django-admin-cursor-paginator (willkg)
`adf576c`: bug 1746940: add debug fields to FileUpload model (willkg)
`0bfc681`: bug 1746940: implement sym data capture at upload (willkg)
`0997218`: bug 1746940: implement code_file/code_id lookup for download API (willkg)
`e4ed26d`: bug 1746940: add sym debug data to File page (willkg)
`27e4d6e`: bug 1746940: remove errant logging.info line (willkg)
`23daea4`: bug 1746940: fix upload-sym file for Windows modules (willkg)
`b84b0ba`: bug 1746940: rework download API to support /code_file/code_id/sym_file (willkg)
`7614fbc`: bug 1746940: fix minor issues with last commit (willkg)
`c803233`: Update architecture diagram (willkg)
Deploy bug #1746940
-----BEGIN PGP SIGNATURE-----
iQEzBAABCgAdFiEEzY3I6/E2Vh8o9nPoGL3RpvBm7BEFAmUBr1YACgkQGL3RpvBm
7BFxWgf/ZRDrFiBD/hC7FTTGnLxGXNRzM6dTWK4cng5Qg0OS+/EuVcGYyscLdv6m
/jsc7QEDZmTyvpWcDLKtFrTv6+iWt5gI0e+RwwFxMNaQ88/y4e09+gTxb9Xv8hkQ
TMhZOalDBxlABE4oDfgPyQjC+rxtRUIx02kOzdZPODDZqVrde7XnU2kWDVO1hO7B
IiYrcmu5tWAk7NNtoE2UD7HZbz2oyf7crqKsJljLUu53Zx5sICjn0XOZlN6iiij3
WI14aCTsfsCp3YdoLKKtYL6+Ef/TAnBK47k02ph9o8Qconalj6XyITvdAeW18vEw
Xuevui4C567d1/9Yfy/gUAsRawMUMw==
=b13O
-----END PGP SIGNATURE-----
commit c8032336b0978ac3defba83a91955719d4d597bc
Author: Will Kahn-Greene <willkg@mozilla.com>
Date: Tue Sep 12 08:03:04 2023 -0400
Update architecture diagram
Tecken no longer has private symbol buckets--there are just symbol
buckets now.
|
|
Assignee | |
Comment 2•2 years ago
|
||
The deploy errored out with test-elbs and then switched the elb to point to the new stack which was all unhealthy. That caused the site to return HTTP 503 for all requests.
Incident report: https://docs.google.com/document/d/18p9HEjeKdqGekdKzx2HXNXJx00UZgW94OJXvczZmQAM/edit
We rolled back to the previous stack which ended the outage.
Mikael and Harold are looking into what happened.
|
|
Assignee | |
Comment 3•2 years ago
|
||
Harold looked at the processes on one of the bad instances and they look fine. It's running everything I would expect it to be running. Harold did a /__heartbeat__ check and that passed.
The way deploys work for Tecken is that it first does a stage deploy and then does a prod deploy. Stage was running 2023.09.13 and working just fine.
I looked through the logs and I didn't see any obvious errors or indications the app failed to start. There's a lot of logs and it's hard to see signal in the noise, so it's possible I missed it.
It sure seems like there was some ephemeral problem with the prod deploy, but we're not sure what exactly happened.
We're going to abandon this deploy and try again tomorrow with a new prod deploy.
Description
•