1632495 - Deploy hg bundles to Azure Blob Storage, and serve to Azure hosted infra

Status: REOPENED

(Developer Services :: Mercurial: hg.mozilla.org, task)

Description

[:glob ✱]

Deploy hg bundles to Azure Blob Storage, and serve to Azure hosted infra.

Comment 1

[Zeid Zabaneh [:zeid]]

Attachment: hgserver: upload generated bundles to Azure Blob Storage (Bug 1632495) r=sheehan

WIP DO NOT LAND

This revision adds support for uploading to Azure Blob Storage using their
Python SDK. Some assumptions are made at the moment regarding the method of
authentication (using SAS tokens), however this may change in the future.

Azure Blob Storage requires an alphanumeric name for containers that does not
contain spaces or dashes. The region is specified on the account level, and
thus all containers within an account belong to the same region. The
convention taken at this time is similar to that with S3 and GCP buckets,
and that is to include the region name in the container name.

Various pending TODOs.

Comment 2

[Zeid Zabaneh [:zeid]]

Attachment: scripts: fetch and store ip ranges from Azure (Bug 1632495) r=sheehan

WIP DO NOT LAND

Fetch an authentication token from the Microsoft Identity Platform. Use this
token to authenticate against the Azure REST API which is used to fetch the
service tags.

For more info see:
- https://docs.microsoft.com/en-us/rest/api/azure/
- https://docs.microsoft.com/en-us/azure/active-directory/develop/

Required environment variables:
- AZURE_SUBSCRIPTION_ID
- AZURE_APP_CLIENT_SECRET
- AZURE_APP_CLIENT_ID
- AZURE_APP_TENANT_ID

Various pending TODOs

Depends on D72750

Comment 3

[Zeid Zabaneh [:zeid]]

Attachment: code quality: rename module for python friendliness & black it (Bug 1632495) r=sheehan

Depends on D73387

Comment 4

[Zeid Zabaneh [:zeid]]

Attachment: hgmo: prioritize Azure bundles when cloning from within Azure (Bug 1632495) r=sheehan

WIP DO NOT LAND

Add support for serving clone bundles from Azure, by checking incoming IP
against the IP prefixes file fetched from Azure. Also added some test data.

TODO: add test, refactor some of the code.

Depends on D73387

Comment 5

[Zeid Zabaneh [:zeid]]

Attachment: terraform: create resources to store HG bundles in Azure (Bug 1632495) r=sheehan

WIP DO NOT LAND

Implement some Terraform code to provision a resource group, storage account,
and container for the bundles.

Depends on D73470

Comment 6

[Zeid Zabaneh [:zeid]]

Attachment: azure: implement better auth for ip scraping & bundles (Bug 1632495) r=sheehan

scripts: instead of manually calling the Azure auth API and fetching an auth
token, do this via the ClientSecretCredential object which is provided by the
azure-identity package.

hgmo: instead of using a SAS token to authenticate against Blob Storage,
use the ClientSecretCredential object directly with the storage account.

Depends on D74132

Comment 7

[Zeid Zabaneh [:zeid]]

Abandoning this until further notice.

Comment 8

[Andrew Halberstadt [:ahal]]

I think we should consider looking into this again. Now we have many level 3 Azure tasks, including some on the release pipeline. This came up because an Azure task in the release pipeline timed out after cloning for 1.5 hours and delayed the release (though it's unclear if it was actually taking that long to clone, or just got stuck somehow).

But either way, some Windows builds are now in Azure too, so this will also increase developer productivity.