1456244 - Expose deployed git sha for Buildhub

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Peter Bengtsson [:peterbe]]

For example, see https://symbols.mozilla.org/__version__

It would be nice to have something like this exposed for https://buildhub.prod.mozaws.net/ and https://buildhub.stage.mozaws.net/

We already generate a version.json file but it's not exposed. 

Once it's exposed we can make a whatsdeployed.io short-URL buildhub. Obviously going to be hard to do something like this for exposing which version of the AWS Lambda job we are running in Stage and Prod.

Comment 1

[Miles Crabill [:miles]]

We wired up a means for displaying the currently deployed version of a lambda function (that has no http api) for socorro. Details here: https://bugzilla.mozilla.org/show_bug.cgi?id=1452681.

The gist is: as part of the deploy process, update a publically viewable version object in S3.

Comment 2

[Philip Cristiano]

For buildhub there appears to be a `__version__` endpoint for them already. https://buildhub.prod.mozaws.net/__version__ Is there another git sha that needs to be added?

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to Philip Cristiano from comment #2)
> For buildhub there appears to be a `__version__` endpoint for them already.
> https://buildhub.prod.mozaws.net/__version__ Is there another git sha that
> needs to be added?

I think that's the git sha for the kinto server. 

Thing is, Buildhub is a cron job and a AWS lambda function. Both of these do stuff with data and write to a Kinto server. 

There's no web server thing for Buildhub itself. There is the web UI thing at https://mozilla-services.github.io/buildhub/ which is just a piece of React that gets its data from that Kinto server.

Comment 4

[Peter Bengtsson [:peterbe]]

(In reply to Miles Crabill [:miles] [also [e-mail]] from comment #1)
> We wired up a means for displaying the currently deployed version of a
> lambda function (that has no http api) for socorro. Details here:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1452681.
> 
> The gist is: as part of the deploy process, update a publically viewable
> version object in S3.

So basically you had to create two S3 buckets just for storing this JSON file. Seems like a hefty change. Especially since you need AWS S3 write access at some point. 

We have a lambda function (a Python file with a lambda_handler() function), can that not be exposed on the web somehow. Then I can write a Python function that responds with a HTTP response somehow. Like this: https://stackoverflow.com/a/39936210/205832

I'd be happy if I can figure out what version of a lambda function we have deployed by doing a curl GET to its public URL. Is that something we can do?

Comment 5

[Miles Crabill [:miles]]

The overhead of having an S3 bucket around is virtually nothing, and it gets you a nice DNS name, SSL, and http access for free. The version object is written at deploy time of the lambda function on the worker that deploys the lambda function - which has all of the necessary AWS access to do both things.

You're describing overloading your lambda function, which at present doesn't respond to HTTP requests, to handle another event type (HTTP requests proxied via API Gateway). This is fine - but it adds complexity to your lambda function beyond its normal operational scope, and will require API Gateway to be configured to proxy those requests, as well as a separate dns entry to be created if you want a pretty url for your version endpoint (as opposed to https://r275xc9bmd.execute-api.us-west-2.amazonaws.com/foobar).

Operationally, it's easier to upload a file to an S3 bucket.

Ansible example of updating the version object (assumes bucket exists):

> - name: create and upload submitter version artifact
>   shell: >
>     ./bin/build_artifact.sh &&
>     aws s3 cp --acl 'public-read'
>     ./version.json s3://{{submitter_version_bucket}}/__version__
>   environment:
>     SUBMITTER_BUILD_ID: '{{stack_id}}'
>     SUBMITTER_SOURCE: '{{submitter_repo}}'
>     SUBMITTER_TAG: '{{submitter_tag}}'

(from https://github.com/mozilla-services/cloudops-deployment/blob/master/projects/socorro/ansible/playbooks/submitter/deploy-lambda.yml#L82-L92)

Comment 6

[Peter Bengtsson [:peterbe]]

Let's take a pause on this. We've started, just today, discussing very different strategies towards the Lambda and the scraping.
As an example, we're pondering to use AWS SQS to transfer the information from S3 to Python that a file was added to S3. 

The Lambda job might be replaced. The scraping might become less important. And, God forbid, peterbe needs to check what version was installed in production or something I can just ping Wei who can ssh in and check.

Comment 7

[Philip Cristiano]

:+1: 

The way Miles suggests would be the way I'd lean as well. Unfortunately Lambdas don't really allow for the same __version__ model that HTTP services use. Exposing the Lambda through an API gateway just to see the sha seems a bit overkill and requires a different mode of operation. 

Uploading to S3 is a decent solution but only says what was deployed, not what is currently running. It's possible for those things to desync.

:peterbe can we close this for now and reopen if this is still the planned course of action?

Comment 8

[Peter Bengtsson [:peterbe]]

I think it's still important to be able to quickly see what version of software we have running in places. Buildhub is a little extra tricky since it has no HTTP server component. 

1. For Lambda, it seems the best course is to do what Miles did with pigeon.

2. We *might* replace Lambda with something else. 

3. Our cron job runs in an EC2 somewhere and talks to a Kinto server via HTTP and that Kinto server shouldn't be upgraded/changed just because cron job changes.

4. One option is to include a metadata field with every event that *either* the Lambda or the cron job executes when it finds a build and sends it to Kinto. E.g. _metadata={"version": <content of version.json>, "environment": "lambda|cron", "timestamp": 1525095123.31859} Then I can look through https://buildhub.prod.mozaws.net/v1/buckets/build-hub/collections/releases/records?_limit=3 and see how the last couple of records got in. 

Considering that, at the time of writing, I'm quite excited about option 4 and that 2 might change a lot in terms of an Cloud Services bug, would you mind if we leave this open a for a little while till we make up our minds?

Comment 9

[Philip Cristiano]

Leaving it open is fine while you the decision is open. Please let us know when you have made a decision!

Comment 10

[:wezhou]

With the new buildhub2 coming up online soon, I think this is probably no longer needed.

Please feel free to re-open it if needed.