1177878 - onyx 1.4.11 deployment to production

Status: VERIFIED FIXED

(Content Services Graveyard :: Tiles: Ops, defect)

Description

[Karl Thiessen [:kthiessen, he/him]]

+++ This bug was initially created as a clone of Bug #1176647 +++

onyx 1.4.8 contains changes that allow serving different sets of tiles to user agents based on their channel preferences.

The v3 API endpoint introduced the requirement for the UA to specify the channel in addition to the locale.

This release allows onyx to serve accordingly.

Quoting the CHANGELOG.md:

> * This will require a change in puppet scripts:
>  * there are additional S3 tile index origins
>  * will require puppet template changes, due to cfg change to allow for multiple indexes
>  * cron job will need to download 3 files instead of just 1

Furthermore, this depends on splice 1.1.22 to be deployed

Comment 1

[Daniel Thorn [:relud]]

Onyx wasn't working in stage, due to config problems. This revealed a bug in the test, which was finishing prematurely. Deployment is being delayed until monday, when we can fix the tests.

Comment 2

[Ed Lee :Mardak]

Ah, I was about to file a bug about onyx not serving the right prerelease tiles, but I see this didn't go to production.

In particular https://tiles.cdn.mozilla.net/desktop-prerelease_tile_index_v3.json -> https://tiles.cdn.mozilla.net/desktop-prerelease/US/es-ES.3dfc3f23eb8b75cd300bb495ca4f4e6b742a6ddc.ag.json

while https://tiles.cdn.mozilla.net/desktop_tile_index_v3.json -> https://tiles.cdn.mozilla.net/desktop/US/es-ES.1087740ebef78907842ca797cf12610425f22bc7.ag.json

$ curl -I https://tiles.services.mozilla.com/v3/links/fetch/es-ES/nightly
Location: https://tiles.cdn.mozilla.net/desktop/US/es-ES.1087740ebef78907842ca797cf12610425f22bc7.ag.json

$ curl -I https://tiles.services.mozilla.com/v3/links/fetch/es-ES/release
Location: https://tiles.cdn.mozilla.net/desktop/US/es-ES.1087740ebef78907842ca797cf12610425f22bc7.ag.json

Right now splice prod has release tiles for Sunday and prerelease tiles for Monday. I'm not sure what's the status of bug 1178180, so I haven't pushed new tiles for release.

Comment 3

[Daniel Thorn [:relud]]

fixed release is 1.4.11

Comment 4

[Daniel Thorn [:relud]]

fixed tests require new monitor server, which is currently deploying in prod.

Comment 5

[Daniel Thorn [:relud]]

monitor server deployed and reporting to datadog, ready for onyx deploy

Comment 6

[Daniel Thorn [:relud]]

deploying onyx canary to prod

Comment 7

[Daniel Thorn [:relud]]

canary deployed, geoip test looks good, waiting on redshift propagation test

Comment 8

[Daniel Thorn [:relud]]

redshift propagation test passed, scaling up new onyx to 11 nodes

Comment 9

[Daniel Thorn [:relud]]

new onyx scaled up, scaling down old onyx

Comment 10

[Daniel Thorn [:relud]]

old onyx stack scaled down, enabling auto-scaling on new onyx stack

Comment 11

[Daniel Thorn [:relud]]

geoip test is fully passing in prod

Comment 12

[Daniel Thorn [:relud]]

redshift propagation test passing in prod