1624403 - ci keeps failing on "socorro-cmd sqs delete-all"

Status: RESOLVED FIXED

(Socorro :: General, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

CI keeps failing on:

socorro-cmd sqs delete-all

I have to re-run it.

I think we should fix that command to retry over a period of time.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P1 and taking it. It's really irritating and wasting time.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5290: bug 1624403: reduce localstack containers

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5290: "bug 1624403: reduce localstack containers" in 38099b3.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm going to leave this open for a bit. The issue is intermittent, but it happens every other day or so as long as we're making changes and we go a week without seeing it, we can call that a win-but-for-halting-problem-problems.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5292: bug 1624403: fix bucket and queue names

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5292: "bug 1624403: fix bucket and queue names" in 74fd52d.

This fixes issues I was having in the local dev environment that I think stem from better checking by localstack on S3 bucket names.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

It failed again:

https://app.circleci.com/pipelines/github/mozilla-services/socorro/10/workflows/fa99af32-f293-459c-bcc3-96b2de92fe9b/jobs/5503

Same issue:

echo ">>> build sqs things and db things"
+ echo '>>> build sqs things and db things'
>>> build sqs things and db things
# Clear SQS for tests
./socorro-cmd sqs delete-all
+ ./socorro-cmd sqs delete-all
Traceback (most recent call last):
  File "./socorro-cmd", line 205, in <module>
    cmd_main()
  File "./socorro-cmd", line 201, in cmd_main
    import_and_run(runner)
  File "./socorro-cmd", line 123, in import_and_run
    sys.exit(app(sys.argv[1:]))
  File "/app/socorro/scripts/sqs_cli.py", line 196, in main
    sqs_group(argv)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/app/socorro/scripts/sqs_cli.py", line 189, in delete_all
    ctx.invoke(delete, queue=os.environ["resource.boto.standard_queue"])
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/app/socorro/scripts/sqs_cli.py", line 175, in delete
    resp = conn.get_queue_url(QueueName=queue)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (502) when calling the GetQueueUrl operation (reached max retries: 4): Bad Gateway
Makefile:116: recipe for target 'test' failed
make: *** [test] Error 1

I didn't wrap sqs delete-all in a retry since it shouldn't have to operate on a SQS of unknown state--that should get handled by the urlwait. The urlwait is passing for reasons I don't understand. Maybe we should do an improved urlwait that checks for bad gateway?

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5296: bug 1624403: switch SQS check to use waitfor script

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5296: "bug 1624403: switch SQS check to use waitfor script" in cd6ce7f.

Let's see if this works.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

That failed with:

Traceback (most recent call last):
  File "./scripts/waitfor.py", line 67, in <module>
    sys.exit(main(sys.argv[1:]))
  File "./scripts/waitfor.py", line 52, in main
    if error.code in ok_codes:
AttributeError: 'URLError' object has no attribute 'code'
Makefile:116: recipe for target 'test' failed
make: *** [test] Error 1

I'll fix that next.

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5297: bug 1624403: fix case where URLError has no code

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5297: "bug 1624403: fix case where URLError has no code" in 9fbc767.

Let's see if this works.

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

We haven't had another failure for this in a week which included a plethora of PRs from dependabot, so I think we're fine. Marking as FIXED.

Comment 14

[Will Kahn-Greene [:willkg] ET needinfo? me]

Happened twice today. Feels like sqs isn't starting right and when it kicks up a 502, it's the case that it's dead and not "not quite up, yet".

Reopening to investigate further.

Comment 15

[Will Kahn-Greene [:willkg] ET needinfo? me]

This hasn't happened in the last month. I'm going to close it out again as FIXED.