1567961 - CloudFront invalidation might get repeated CallerReference

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, defect)

Description

[Peter Bengtsson [:peterbe]]

See https://sentry.prod.mozaws.net/operations/mdn-prod/issues/6074311/?environment=oregon%3Aprod

InvalidArgument: An error occurred (InvalidArgument) when calling the CreateInvalidation operation: Your request contains a caller reference that was used for a previous invalidation batch for the same distribution.
  File "celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "newrelic/hooks/application_celery.py", line 85, in wrapper
    return wrapped(*args, **kwargs)
  File "celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "kuma/api/tasks.py", line 188, in request_cdn_cache_invalidation
    'CallerReference': str(time.time())
  File "botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)

When you're doing a CloudFront invalidation request, you have to set a string called CallerReference. At the moment we're using a timestamp formatted as a string with 2 significant figures. E.g '1563813532.18'
The problem with this is that it might be repeated between invalidations. For example, if two concurrent worker tasks at getting two "simultaneous" requests to send an invalidation, within 10ms, they'll get the same CallerReference.

Probably the easiest thing to do is to just massively increase the entropy of that string but using more significant figures. E.g. '1563813821.164395'. Another thing we could do is to make the string depend on the paths list that is available in that loop.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/kuma/pull/5584

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub master push: https://github.com/mozilla/kuma/commit/2aa50afee492d4290da3ef1c0d526a6fe1372700

Commit merged into master by Ryan Johnson (escattone)
Authored by Peter Bengtsson (peterbe)