1692117 - Migrate data-shared and beam to modern terraform

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, task, P1)

Description

[Wesley Dawson [:whd]]

This is a long time coming, and blocks pub/sub lite and flex template initiatives. The main blocker in bug #1682749 should be resolved next week, but there is a bunch of modernization that needs to be done to the data-shared resources module, as well as replacing the existing bigquery and beam modules with the ones currently homed in data-pioneer. We should leverage update-modules to automatically apply ACL updates as appropriate, and hopefully consolidate sandbox-permissions into resources as a result.

An ancillary goal should be removing deprecated access-groups logic.

The 0.11 exception should be removed form cloudops-infra CI for these projects as well.

Comment 1

[Wesley Dawson [:whd]]

I've finally started work on this in https://github.com/mozilla-services/cloudops-infra/compare/data_modern?expand=1. The plan is to use a separate terraform state during development and do a GCS state swap after all the pieces have been upgraded.

Comment 2

[Wesley Dawson [:whd]]

An update is perhaps in order. I have finished porting and modernizing the data-shared resources module in the above branch, which is the majority of the tricky work. I plan to work on beam and data-shared bigquery modules next, the latter of which should be relatively simple since most of the testing and debugging for that was done in bug #1682749 and deployments of mozdata already use it.

There is still a fair amount of cleanup work and deprecation that needs to be done following the above, but I am hopeful to complete the modernization work in the next few weeks.

Comment 3

[Wesley Dawson [:whd]]

The new data-shared bigquery module is now fully compatible with the old deployment and stage applies cleanly as a no-op. beam is next.

Comment 4

[Wesley Dawson [:whd]]

All of stage is now configured to use terraform 0.14+. This required some significant performance tuning in the bigquery module that has been cherry-picked into the default branch already: https://github.com/mozilla-services/cloudops-infra/pull/3206

I've filed https://github.com/mozilla-services/cloudops-infra/pull/3207 for the stage changes. I plan to develop a prod PR before commencing with various cleanup PRs to remove the 0.11 stack configuration, and also remove myriad unused resources observed while working on this bug.

Comment 5

[Wesley Dawson [:whd]]

https://github.com/mozilla-services/cloudops-infra/pull/3218 migrated prod BQ earlier in the week and the final PR for prod in https://github.com/mozilla-services/cloudops-infra/pull/3219 will likely be merged tomorrow, after which I'm considering this work complete. There is a bunch of cleanup work that will need to be done but I plan to move tracking of that to JIRA.

Comment 6

[Wesley Dawson [:whd]]

This work is complete.