1297208 - Setup supervisord to run the bin/run_read_pulse_resultsets command on SCL3 stage/prod

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Cameron Dawson [:camd]]

the command is basically just like the one for bin/run_read_pulse_jobs, but for resultsets.

We will need some config information set, which I can give you in a PM when you're ready to do this.

As usual, let's do this on stage then let it soak and then we can do prod.

Comment 1

[Cameron Dawson [:camd]]

We will need to set the following env vars:

GITHUB_CLIENT_ID
GITHUB_CLIENT_SECRET
PULSE_RESULTSET_SOURCES

Comment 2

[Cameron Dawson [:camd]]

The ETL Nodes will need these values:

GITHUB_CLIENT_ID
GITHUB_CLIENT_SECRET

The node running run_read_pulse_resultsets will need:

PULSE_RESULTSET_SOURCES

Comment 3

[Cameron Dawson [:camd]]

Hey Fubar:  do you think you'll have time to address this soon?  I'd love to get it up and running on stage at least.

Thanks!!

Comment 4

[Kendall Libby [:fubar] (he/him)]

PM me the info when you wake up and I'll go throw it at stage; sorry for the delay!

Comment 5

[Kendall Libby [:fubar] (he/him)]

run_read_pulse_resultsets        FATAL     command at '/data/www/treeherder.allizom.org/treeherder-service/bin/run_read_pulse_resultsets' is not executable

you'll need to chmod 755 the file in the repo. I've done it manually to get it running for the moment.

it also throws this error:
> System check identified some issues:
> 
> ERRORS:
> ?: (treeherder.E001) mysqlclient has been compiled against an insecure version of > libmysqlclient (5.1.73).
> HINT: If using Vagrant, run `vagrant provision` and re-login.

... which is from bug 1289156. we don't have mysql-5.7.* anywhere in-house. bug 1289156#c18 *seems* to skip the check, but it isn't happening for this job?

regardless, the job is running, but doesn't seem to be happy:

treeherder-rabbitmq1.stage.private.scl3# strace -f -s 256 -p 15149
Process 15149 attached
restart_syscall(<... resuming interrupted call ...>) = 0
fcntl(4, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(4, F_SETFL, O_RDWR)               = 0
fcntl(4, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
read(4, 0x296fe33, 5)                   = -1 EAGAIN (Resource temporarily unavailable)

Comment 6

[Ed Morley [:emorley]]

The mysql message is just a warning, django still outputs the message for "skipped" checks, if their error level was error or above. Ie it is expected and can be ignored (it also can be seen in the deploy logs for example).

Comment 7

[Cameron Dawson [:camd]]

I fixed that file to be executable now. I think things are working fine.  But we haven't, as far as I can tell, received a resultset this way.  The traffic is very low.  There was a bmo-development one yesterday, but I think it came in before we deployed this change.

I'll keep an eye on it.

Comment 8

[Cameron Dawson [:camd]]

Fubar: this appears to be working well on stage.  We are ready to enable this on SCL3 Prod as well.

But please don't bother with this if you're actively working on our move to Heroku.  Once we're there, this won't be necessary anyway.  However, if your Heroku work will take a while and this won't slow you down, enabling it would be great.

You will also need to add this env var:

export PULSE_RESULTSET_SOURCES='[{"exchange": "exchange/taskcluster-github/v1/push", "routing_keys": ["#"]}, {"exchange": "exchange/taskcluster-github/v1/pull-request", "routing_keys": ["#"]}]'

Thanks!

Comment 9

[Kendall Libby [:fubar] (he/him)]

currently on a plane to MTV for training this week, and PTO next week. this fairly simple so will deploy it when I get a chance (tomorrow, probably).

Comment 10

[Kendall Libby [:fubar] (he/him)]

done!