validate stage infra and deploy pipeline
Categories
(Eliot :: General, task, P2)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
References
Details
After the stage environment has been built, we need to validate the stage infrastructure and deploy pipeline.
This bug covers building a validation plan and executing on it.
Assignee | ||
Comment 1•2 years ago
•
|
||
Rough validation plan:
- validate deploys
- can we trigger deploys?
- do deploys notify the channel?
- do deploys complete?
- is eliot in stage updated after a deploy? -- check
/__version__
for commit sha
- validate infrastructure
- does logging work?
- does monitoring work? is there a dashboard in grafana for eliot on stage?
- does sentry work?
- does symbolication API work? does it return correct responses?
- do dashboards look correct?
Assignee | ||
Comment 2•2 years ago
|
||
We can re-use the symbolication API bits we did in 2021h2 in bug #1674102.
Assignee | ||
Comment 3•2 years ago
|
||
I updated the validation document and fleshed it out with the things I think we need to look at. We can adjust it as we go along.
https://docs.google.com/document/d/1W61N9Ki0zwy-zitiN2AyXAbuPxgwSo-kc8_62-UnOpQ/edit#
I'm setting up the tools I need to validate and load test the stage environment. I'm reusing as much as I can from the tecken-loadtests repo.
https://github.com/mozilla-services/tecken-loadtests/
I'm currently waiting on Sentry access. Once I have that, I can start working on things.
Assignee | ||
Comment 4•2 years ago
|
||
Current status.
Done:
- Deploys to stage are working.
- Logging is working, there's a link in Confluence, I have access to them.
- There's an app monitoring dashboard and an infra monitoring dashboard.
- We verified the HTTP response headers are correct. Eliot GCP stage gets an A+ in the Mozilla Observatory.
Not done:
- Sentry, yet. https://mozilla-hub.atlassian.net/browse/DSRE-1236
- Validate stage environment. https://mozilla-hub.atlassian.net/browse/DSRE-1242
- Load test stage environment. https://mozilla-hub.atlassian.net/browse/DSRE-1243
- Verify the app and infra dashboards are working.
Getting there.
Assignee | ||
Comment 5•2 years ago
|
||
Current status:
Done:
- I verified that Sentry events make it to the Eliot GCP stage Sentry project.
Not done:
- Validate stage environment. https://mozilla-hub.atlassian.net/browse/DSRE-1242
- Load test stage environment. https://mozilla-hub.atlassian.net/browse/DSRE-1243
- Verify the app and infra dashboards are working.
Assignee | ||
Comment 6•2 years ago
|
||
I finished validating the stage environment. I documented (roughly) the tools I used. I updated the repo with the latest versions of everything.
https://github.com/mozilla-services/tecken-loadtests/commit/714d09a2fd83cd351f2532a1f4e18c6047d46daf
Final validation document: https://docs.google.com/document/d/1W61N9Ki0zwy-zitiN2AyXAbuPxgwSo-kc8_62-UnOpQ/edit#
Everything looks good.
Description
•