927825 - [ReMo] Cron job cannot talk to Celery

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[Giorgos Logiotatidis [:giorgos]]

We have a cron job on ReMo that among other things tries to contact celery broker to add some jobs.

This fails with "Connection refused" error so I guess there is no network flow between the machine that runs the cron job and the broker.

Can we please fix that?

This is blocking a release so the sooner the better. Thanks!

Comment 1

[Giorgos Logiotatidis [:giorgos]]

This is reproduced on both stage (reps.a.o), dev (reps-dev.a.o)

Please make sure that the flow is open on prod too.

Comment 2

[Giorgos Logiotatidis [:giorgos]]

Ping? 

This is blocking a release a few days now :|

Comment 3

[Chris Turra [:cturra]]

i just had a look at the environments for you and can see each of the web nodes have access to the celery broker, but the admin node does not. i will open a bug with netops to get this flow in place for you.

Comment 4

[Chris Turra [:cturra]]

netflows are now open.

 [[e-mail]1 settings]$ nc -zv engagement-celery1.seamicro.phx1.mozilla.com 5672
 Connection to engagement-celery1.seamicro.phx1.mozilla.com 5672 port [tcp/amqp] succeeded!

Comment 5

[Giorgos Logiotatidis [:giorgos]]

We're still waiting for dev and stage.

Comment 6

[Chris Turra [:cturra]]

(In reply to Giorgos Logiotatidis [:giorgos] from comment #5)
> We're still waiting for dev and stage.

dev is working as expected, we're just waiting on a netflow for stage. 

 [[e-mail]1 remo]$ nc -zv engagement-celery1.dev.seamicro.phx1.mozilla.com 5672
 Connection to engagement-celery1.dev.seamicro.phx1.mozilla.com 5672 port [tcp/amqp] succeeded!

Comment 7

[Chris Turra [:cturra]]

the cron now runs successfully in all environments. for more details see bug 929443.