1508787 - Please setup product-details worker pipeline

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Rail Aliiev [:rail]]

As we discussed today, we need to setup a pipeline to deploy a worker (no web head) using the same approach we use for shipit (dockerflow).

Rok is finalizing the last bits and we'll probably have the docker images by the end of this week.

Requirements:

* use the same environments as for shipit (production, staging, testing)
* use the same environment variables we use for the shipit dockerflow instances
* releng to add a `healthcheck` command that would report instances health status

Anything else?

Comment 1

[Jeremy Orem [:oremj]]

Let me know when the image is ready to go.

Comment 2

[Rok Garbas [:garbas]]

docker images for testing and staging are ready (this is dev and dev2 in your case):
- shipit-api-shipit-api.worker_dockerflow-testing
- shipit-api-shipit-api.worker_dockerflow-staging

production image will released under "shipit-api-shipit-api.worker_dockerflow-production" once we test it on staging.

Comment 3

[Rok Garbas [:garbas]]

As we discussed on todays meeting, we should use the same image as we do for shipit backend, except that we need to run ``flask worker`` command.

Comment 4

[Jeremy Orem [:oremj]]

Getting the following traceback when running flask worker:

Traceback (most recent call last):
  File "/nix/store/qybcwdsjmpyqh7md45whck1m2si1203d-python3.7-Flask-1.0.2/bin/.flask-wrapped", line 12, in <module>
    sys.exit(main())
  File "/nix/store/qybcwdsjmpyqh7md45whck1m2si1203d-python3.7-Flask-1.0.2/lib/python3.7/site-packages/flask/cli.py", line 894, in main
    cli.main(args=args, prog_name=name)
  File "/nix/store/qybcwdsjmpyqh7md45whck1m2si1203d-python3.7-Flask-1.0.2/lib/python3.7/site-packages/flask/cli.py", line 557, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/nix/store/yzxb85gg93iw1ws0d64jzmz2mkqw0rrp-python3.7-Click-7.0/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/nix/store/yzxb85gg93iw1ws0d64jzmz2mkqw0rrp-python3.7-Click-7.0/lib/python3.7/site-packages/click/core.py", line 1135, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
AttributeError: 'function' object has no attribute 'make_context'

Comment 5

[Rail Aliiev [:rail]]

I think Rok was going to fix this today/tomorrow.

Comment 6

[Rok Garbas [:garbas]]

:oremj I fixed flask worker command[1] and pushed to staging[2] which means that staging image is now updated[3] (shipit-api-shipit-api.dockerflow-staging tag).

Once you confirm that staging is working I can deploy to production[3] (shipit-api-shipit-api.dockerflow-production tag).


[1] https://github.com/mozilla/release-services/pull/1777
[2] https://tools.taskcluster.net/groups/R_cOBjcWQxyMJO_PaNebZw
[3] https://hub.docker.com/r/mozilla/shipitbackend/tags/

Comment 7

[Jeremy Orem [:oremj]]

The worker claims it is consuming messages now: https://console.cloud.google.com/logs/viewer?interval=P1D&project=moz-fx-shipitapi-nonprod-2690&organizationId=442341870013&folder&minLogLevel=0&expandAll=false&timestamp=2019-01-04T17%3A18%3A27.462000000Z&customFacets&limitCustomFacetWidth=true&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22moz-fx-shipitapi-nonprod-2690%22%0Aresource.labels.location%3D%22us-west1%22%0Aresource.labels.cluster_name%3D%22shipitapi-nonprod-v1%22%0Aresource.labels.namespace_name%3D%22dev-shipitapi%22%0Ametadata.systemLabels.top_level_controller_type%3D%22Deployment%22%0Ametadata.systemLabels.top_level_controller_name%3D%22shipitapi-dev-shipitapi-app-1-worker%22&dateRangeStart=2019-01-03T17%3A18%3A22.574Z&scrollTimestamp=2019-01-04T11%3A56%3A56.000000000Z&dateRangeUnbound=forwardInTime


This has been deployed to api.shipit.staging.mozilla-releng.net and api.shipit.testing.mozilla-releng.net instances.

Comment 8

[Jeremy Orem [:oremj]]

Testing actually has the same issue staging had:

Traceback (most recent call last): File "/nix/store/qybcwdsjmpyqh7md45whck1m2si1203d-python3.7-Flask-1.0.2/bin/.flask-wrapped", line 12, in <module> sys.exit(main()) File "/nix/store/qybcwdsjmpyqh7md45whck1m2si1203d-python3.7-Flask-1.0.2/lib/python3.7/site-packages/flask/cli.py", line 894, in main cli.main(args=args, prog_name=name) File "/nix/store/qybcwdsjmpyqh7md45whck1m2si1203d-python3.7-Flask-1.0.2/lib/python3.7/site-packages/flask/cli.py", line 557, in main return super(FlaskGroup, self).main(*args, **kwargs) File "/nix/store/yzxb85gg93iw1ws0d64jzmz2mkqw0rrp-python3.7-Click-7.0/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/nix/store/yzxb85gg93iw1ws0d64jzmz2mkqw0rrp-python3.7-Click-7.0/lib/python3.7/site-packages/click/core.py", line 1135, in invoke sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) AttributeError: 'function' object has no attribute 'make_context'

Comment 9

[Rok Garbas [:garbas]]

:oremj I pushed changes also to testing[1] once all tasks are green docker image is going to be pushed to docker hub

[1] https://tools.taskcluster.net/groups/IPlx3o2GTWK_3s08tphSUw

Comment 10

[Jeremy Orem [:oremj]]

Looks like the worker in testing is now working. Let me know when I should deploy the worker to pre production and production.

Comment 11

[Rok Garbas [:garbas]]

Production docker image was not published.

Comment 12

[Jeremy Orem [:oremj]]

Given comment 11, should I deploy the worker to production and pre-production?

Comment 13

[Rok Garbas [:garbas]]

Yes, please (replying since :rail is out today)

Comment 14

[Jeremy Orem [:oremj]]

Workers have been deployed to pre-prod and prod.

Comment 15

[Rail Aliiev [:rail]]

Thank you!