1522435 - Maven metadata generation Lambda function should create CloudFront invalidations

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Adrian Utrilla [:autrilla]]

:jonalmeida mentioned on Slack that they were seeing different snapshot versions being served to TaskCluster tasks that were running on different regions, namely US east coast and US west coast. Further debugging with https://www.geoscreenshot.com/capture showed that indeed https://snapshots.maven.mozilla.org/maven2/org/mozilla/components/support-ktx/0.40.0-SNAPSHOT/maven-metadata.xml is different in Los Angeles and New Jersey.

I initially thought updating an object in S3 would automatically invalidate that object in CloudFront, but I was wrong and this is actually not the case. We need the Lambda function to create an invalidation in the case where the metadata is being updated and not newly created. Alternatively we could reduce the TTL in CloudFront, but that largely negates the point of using it in the first place.

:jlorenzo, would you be willing to take on the code changes for this? Essentially, we'd need to call https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudfront.html#CloudFront.Client.create_invalidation. We could pass you the CloudFront distribution ID through an environment variable.

Comment 1

[Johan Lorenzo [:jlorenzo] - On PTO - Back on March 18th]

Attachment: [maven-lambda] Invalidate CloudFront cache when metadata is uploaded

Comment 2

[Johan Lorenzo [:jlorenzo] - On PTO - Back on March 18th]

Thanks all for the thorough investigation, Adrian and Jon! I implemented the fix you suggested - and helped me out with -, Adrian. Please let me know what you think of it. The PR can be tested live thanks to https://queue.taskcluster.net/v1/task/CKAlmstFTgWJiOFcHc-Ygg/runs/0/artifacts/public/build/function.zip

Comment 3

[Johan Lorenzo [:jlorenzo] - On PTO - Back on March 18th]

Comment on attachment 9038812 [details] [review]
[maven-lambda] Invalidate CloudFront cache when metadata is uploaded

r+'d by :autrilla at https://github.com/mozilla-releng/maven-lambda/pull/9#pullrequestreview-196066206
Landed on master at: https://github.com/mozilla-releng/maven-lambda/commit/8ea923b3bae6454b1af748470a743b7cd8ec9502

Comment 4

[Johan Lorenzo [:jlorenzo] - On PTO - Back on March 18th]

Adrian, could you let us when this is deployed to all 4 lambda functions? (If people are wondering, for the context: we have 1 for snapshots, 1 for regular releases and the same ones in staging)

Comment 5

[Adrian Utrilla [:autrilla]]

This has been deployed to all 4 environments. I also submitted a CloudFront invalidation for '/*' to all 4 environments to get rid of the inconsistencies.