Setup supervisord to run the bin/run_read_pulse_resultsets command on SCL3 stage/prod

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: camd, Assigned: fubar)

Tracking

Details

(Reporter)

Description

2 years ago
the command is basically just like the one for bin/run_read_pulse_jobs, but for resultsets.

We will need some config information set, which I can give you in a PM when you're ready to do this.

As usual, let's do this on stage then let it soak and then we can do prod.
Comment hidden (obsolete)
(Reporter)

Updated

2 years ago
See Also: → bug 1275717, bug 1266584
(Reporter)

Comment 2

2 years ago
The ETL Nodes will need these values:

GITHUB_CLIENT_ID
GITHUB_CLIENT_SECRET

The node running run_read_pulse_resultsets will need:

PULSE_RESULTSET_SOURCES
(Reporter)

Updated

2 years ago
Depends on: 1264074
Flags: needinfo?(klibby)
(Reporter)

Updated

2 years ago
Blocks: 1297228
(Reporter)

Comment 3

2 years ago
Hey Fubar:  do you think you'll have time to address this soon?  I'd love to get it up and running on stage at least.

Thanks!!
(Assignee)

Comment 4

2 years ago
PM me the info when you wake up and I'll go throw it at stage; sorry for the delay!
Flags: needinfo?(klibby)
(Assignee)

Comment 5

2 years ago
run_read_pulse_resultsets        FATAL     command at '/data/www/treeherder.allizom.org/treeherder-service/bin/run_read_pulse_resultsets' is not executable

you'll need to chmod 755 the file in the repo. I've done it manually to get it running for the moment.

it also throws this error:
> System check identified some issues:
> 
> ERRORS:
> ?: (treeherder.E001) mysqlclient has been compiled against an insecure version of > libmysqlclient (5.1.73).
> HINT: If using Vagrant, run `vagrant provision` and re-login.

... which is from bug 1289156. we don't have mysql-5.7.* anywhere in-house. bug 1289156#c18 *seems* to skip the check, but it isn't happening for this job?

regardless, the job is running, but doesn't seem to be happy:

treeherder-rabbitmq1.stage.private.scl3# strace -f -s 256 -p 15149
Process 15149 attached
restart_syscall(<... resuming interrupted call ...>) = 0
fcntl(4, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(4, F_SETFL, O_RDWR)               = 0
fcntl(4, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
read(4, 0x296fe33, 5)                   = -1 EAGAIN (Resource temporarily unavailable)
Flags: needinfo?(cdawson)

Comment 6

2 years ago
The mysql message is just a warning, django still outputs the message for "skipped" checks, if their error level was error or above. Ie it is expected and can be ignored (it also can be seen in the deploy logs for example).
(Reporter)

Comment 7

2 years ago
I fixed that file to be executable now. I think things are working fine.  But we haven't, as far as I can tell, received a resultset this way.  The traffic is very low.  There was a bmo-development one yesterday, but I think it came in before we deployed this change.

I'll keep an eye on it.
Flags: needinfo?(cdawson)
(Reporter)

Updated

2 years ago
Summary: Setup supervisord to run the bin/run_read_pulse_jobs command on SCL3 stage/prod → Setup supervisord to run the bin/run_read_pulse_resultsets command on SCL3 stage/prod
(Reporter)

Comment 8

2 years ago
Fubar: this appears to be working well on stage.  We are ready to enable this on SCL3 Prod as well.

But please don't bother with this if you're actively working on our move to Heroku.  Once we're there, this won't be necessary anyway.  However, if your Heroku work will take a while and this won't slow you down, enabling it would be great.

You will also need to add this env var:

export PULSE_RESULTSET_SOURCES='[{"exchange": "exchange/taskcluster-github/v1/push", "routing_keys": ["#"]}, {"exchange": "exchange/taskcluster-github/v1/pull-request", "routing_keys": ["#"]}]'

Thanks!
Flags: needinfo?(klibby)
(Assignee)

Comment 9

2 years ago
currently on a plane to MTV for training this week, and PTO next week. this fairly simple so will deploy it when I get a chance (tomorrow, probably).
(Assignee)

Comment 10

2 years ago
done!
Assignee: nobody → klibby
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Flags: needinfo?(klibby)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.