The Python version's error is different:
```
{"datetime": "2020-03-20 04:22:57,559", "message": "Starting purge_ttl.py"}
{"datetime": "2020-03-20 04:22:57,559", "message": "For sync:syncdb4"}
{"datetime": "2020-03-20 04:24:02,090", "message": "batches: removed 587 rows, batches_duration: 0:01:04.530411"}
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 79, in next
return six.next(self._wrapped)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 416, in __next__
return self._next()
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 706, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1584681841.118465835","description":"Error received from peer ipv4:74.125.20.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>The above exception was the direct cause of the following exception:Traceback (most recent call last):
File "/app/tools/spanner/purge_ttl.py", line 76, in <module>
spanner_read_data()
File "/app/tools/spanner/purge_ttl.py", line 65, in spanner_read_data
result = database.execute_partitioned_dml(query)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/database.py", line 395, in execute_partitioned_dml
list(result_set) # consume all partials
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/streamed.py", line 143, in __iter__
self._consume_next()
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/streamed.py", line 116, in _consume_next
response = six.next(self._response_iterator)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/snapshot.py", line 45, in _restart_on_unavailable
for item in iterator:
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 81, in next
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
```
According to https://github.com/grpc/grpc/blob/master/doc/statuscodes.md it's possible the call has completed successfully despite an DEADLINE_EXCEEDED error. There's a notion between timeouts and deadlines: https://grpc.io/blog/deadlines/ -- I know the Python client configures a timeout but I wonder if it has an inappropriate deadline for this.
As for the Rust version, grpc clients tend to have retry logic -- and ABORTED signals this failed "typically due to a concurrency issue", indicating a client should retry a certain number of times. I'm not sure our low level rust library (basically the c++ grpcio internals) do any retrying for us.
Bug 1626707 Comment 1 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
The Python version's error is different:
```
{"datetime": "2020-03-20 04:22:57,559", "message": "Starting purge_ttl.py"}
{"datetime": "2020-03-20 04:22:57,559", "message": "For sync:syncdb4"}
{"datetime": "2020-03-20 04:24:02,090", "message": "batches: removed 587 rows, batches_duration: 0:01:04.530411"}
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 79, in next
return six.next(self._wrapped)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 416, in __next__
return self._next()
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 706, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1584681841.118465835","description":"Error received from peer ipv4:74.125.20.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>The above exception was the direct cause of the following exception:Traceback (most recent call last):
File "/app/tools/spanner/purge_ttl.py", line 76, in <module>
spanner_read_data()
File "/app/tools/spanner/purge_ttl.py", line 65, in spanner_read_data
result = database.execute_partitioned_dml(query)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/database.py", line 395, in execute_partitioned_dml
list(result_set) # consume all partials
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/streamed.py", line 143, in __iter__
self._consume_next()
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/streamed.py", line 116, in _consume_next
response = six.next(self._response_iterator)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/snapshot.py", line 45, in _restart_on_unavailable
for item in iterator:
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 81, in next
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
```
According to https://github.com/grpc/grpc/blob/master/doc/statuscodes.md it's possible the call has completed successfully despite a DEADLINE_EXCEEDED error. There's a notion between timeouts and deadlines: https://grpc.io/blog/deadlines/ -- I know the Python client configures a timeout but I wonder if it has an inappropriate deadline for this.
As for the Rust version, grpc clients tend to have retry logic -- and ABORTED signals this failed "typically due to a concurrency issue", indicating a client should retry a certain number of times. I'm not sure our low level rust library (basically the c++ grpcio internals) do any retrying for us.