1896382 - hg bundle generation fails with `The specified copy source is larger than the maximum allowable size for a copy source`

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect, P1)

Description

[Julien Cristau [:jcristau]]

May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: Traceback (most recent call last):
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/bin/generate-hg-s3-bundles", line 33, in <module>
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: sys.exit(load_entry_point('hgmolib==0.0', 'console_scripts', 'generate-hg-s3-bundles')())
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/lib64/python3.6/site-packages/hgmolib/generate_hg_s3_bundles.py", line 655, in main
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: paths[repo] = generate_bundles(repo, upload=upload, **opts)
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/lib64/python3.6/site-packages/hgmolib/generate_hg_s3_bundles.py", line 470, in generate_bundles
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: f.result()
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: return self.__get_result()
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: raise self._exception
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: result = self.fn(*self.args, **self.kwargs)
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/lib64/python3.6/site-packages/hgmolib/generate_hg_s3_bundles.py", line 200, in upload_to_s3
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: MetadataDirective="REPLACE",
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/lib64/python3.6/site-packages/boto3/resources/factory.py", line 580, in do_action
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: response = action(self, *args, **kwargs)
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/lib64/python3.6/site-packages/boto3/resources/action.py", line 88, in __call__
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: response = getattr(parent.meta.client, operation_name)(*args, **params)
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/lib64/python3.6/site-packages/botocore/client.py", line 508, in _api_call
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: return self._make_api_call(operation_name, kwargs)
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: File "/var/hg/venv_bundles/lib64/python3.6/site-packages/botocore/client.py", line 911, in _make_api_call
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: raise error_class(parsed_response, operation_name)
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com generate-hg-s3-bundles[13900]: botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120
May 13 06:04:05 hgssh1.dmz.mdc1.mozilla.com systemd[1]: hg-bundle-generate.service: main process exited, code=exited, status=1/FAILURE

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html documents a 5GB limit for atomic CopyObject operations.

Comment 1

[Connor Sheehan [:sheehan]]

In our case, we don't actually want the entire file to be re-copied. We are just using the copy object API as a hack to reset the object's expiration time. We'll need to do one of the following:

• find a new way to update an object's expiration time when the uploaded bundle would be the same. This is more efficient and involves less data transfer to AWS.
• Remove the check for if a file already exists and always re-upload bundles even if their content hasn't changed. This will make the bundle generation process slower but involves less engineering work.

Comment 2

[Connor Sheehan [:sheehan]]

Attachment: bundles: always upload new bundles instead of resetting bucket lifecycle (Bug 1896382) r?jcristau

The CopyObject operation we were using to reset the lifetime of an
object in S3 has a 5GB limit for atomic operations. Since our bundles
seem to be above 5GB now, we can no longer use this method. Remove the
code that checks for the presence of a key and runs CopyObject and
instead always upload a new bundle file to reset the object lifetime,
while we look for an alternative way to solve this problem.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/70760d908a12
bundles: always upload new bundles instead of resetting bucket lifecycle r=jcristau