1467242 - spark-streaming - Jenkins job that runs databricks jobs api cmds

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, defect, P1)

Description

[Harold Woo]

- Needs to query only jobs started by the jenkins user

Jenkins job will query databricks jobs api to see if ENV & VERSION already exist.  If it does, then issue a start/stop on existing cluster. 
- Reset(config)
- Stop job
- Poll until Stopped
- Start job

If ENV and VERSION do not exist in db rest api, or do not exist then
- Create jobs api => get ID back 
- Run job with different output path, different checkpoint path, same kafka stream, new cluster, new checkpoint? (dbfs unique checkpoint path)

Comment 1

[Harold Woo]

1. assert creator_user_name == some jenkins user
2. add VERSION to the custom_tags
3. if ENV&&VERSION are the same invoke reset/stop/poll/start else invoke create/get id/run on new cluster with new parameters

curl 'https://dbc-caf9527b-e073.cloud.databricks.com/api/2.0/jobs/list' -X GET -H "Authorization: Bearer xxxx" | jq .


some refs

https://github.com/mozilla/telemetry-streaming/compare/master...akkomar:db_deploy_temp
https://github.com/mozilla/telemetry-streaming/compare/job_definitions
https://docs.databricks.com/api/latest/jobs.html#create