1287494 - please deploy balrog 2.2 to stage/prod

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[[e-mail] (:bhearsum)]

We've got quite a few things queued up, including the requested Cache-Control header. master-2016-07-18-16-15 is ready to go to stage for additional verification.

Comment 1

[Benson Wong [:mostlygeek]]

Deployed to stage:

$ curl -i https://aus5.stage.mozaws.net/__version__
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Mon, 18 Jul 2016 17:42:40 GMT
X-Proxy-Cache-Status: MISS
Content-Length: 219
Connection: keep-alive

{
    "commit": "330fb73921b728d58bba22bfc154a280147af0ca",
    "version": "2.2",
    "source": "https://github.com/mozilla/balrog",
    "build": "https://tools.taskcluster.net/task-inspector/#dHbU11vpQiKLCn5gDAjhOg"
}

Note that nginx adds a header: 

   X-Proxy-Cache-Status: MISS

This will help test if caching was working. I tried an endpoint, but it doesn't seem to return any cache-control headers: 

$ curl -i https://aus5.stage.mozaws.net/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: Mon, 18 Jul 2016 17:43:58 GMT
X-Proxy-Cache-Status: MISS
Content-Length: 42
Connection: keep-alive

<?xml version="1.0"?>
<updates>

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Benson Wong [:mostlygeek] from comment #1)
> Deployed to stage:
> 
> $ curl -i https://aus5.stage.mozaws.net/__version__
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Content-Type: application/json
> Date: Mon, 18 Jul 2016 17:42:40 GMT
> X-Proxy-Cache-Status: MISS
> Content-Length: 219
> Connection: keep-alive
> 
> {
>     "commit": "330fb73921b728d58bba22bfc154a280147af0ca",
>     "version": "2.2",
>     "source": "https://github.com/mozilla/balrog",
>     "build":
> "https://tools.taskcluster.net/task-inspector/#dHbU11vpQiKLCn5gDAjhOg"
> }
> 
> Note that nginx adds a header: 
> 
>    X-Proxy-Cache-Status: MISS
> 
> This will help test if caching was working. I tried an endpoint, but it
> doesn't seem to return any cache-control headers: 
> 
> $ curl -i
> https://aus5.stage.mozaws.net/update/3/Firefox/33.0/20141202185629/
> Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/
> update.xml
> HTTP/1.1 200 OK
> Content-Type: text/xml; charset=utf-8
> Date: Mon, 18 Jul 2016 17:43:58 GMT
> X-Proxy-Cache-Status: MISS
> Content-Length: 42
> Connection: keep-alive
> 
> <?xml version="1.0"?>
> <updates>

We debugged this a bit on IRC, turns out the db was completely busted. Benson fixed that up, and we're now getting "Cache-Control: public,max-age=60", but nginx doesn't appear to be respecting it. Benson suggested that we may need a space after the comma, so we'll try that.

That should be happening in the Docker image tagged with master-2016-07-18-21-06, so we can give that a try when you're ready.

Comment 3

[Benson Wong [:mostlygeek]]

I deployed master-2016-07-18-21-06 and it doesn't serve cache-control headers.

$ curl -i https://aus5.stage.mozaws.net/update/3/Firefox/33.0/20141202185629
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: Tue, 19 Jul 2016 18:06:56 GMT
X-Proxy-Cache-Status: MISS
Content-Length: 42
Connection: keep-alive

<?xml version="1.0"?>
<updates>

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Benson Wong [:mostlygeek] from comment #3)
> I deployed master-2016-07-18-21-06 and it doesn't serve cache-control
> headers.
> 
> $ curl -i https://aus5.stage.mozaws.net/update/3/Firefox/33.0/20141202185629
> HTTP/1.1 200 OK
> Content-Type: text/xml; charset=utf-8
> Date: Tue, 19 Jul 2016 18:06:56 GMT
> X-Proxy-Cache-Status: MISS
> Content-Length: 42
> Connection: keep-alive
> 
> <?xml version="1.0"?>
> <updates>

Yeah, that's not a valid endpoint. Works fine with:
➜  ~ curl -IL https://aus5.stage.mozaws.net/update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/nightly/default/default/default/update.xml
HTTP/1.1 200 OK
Cache-Control: public, max-age=60
Content-Length: 529
Content-Type: text/xml; charset=utf-8
Date: Tue, 19 Jul 2016 18:58:21 GMT
X-Proxy-Cache-Status: MISS
Connection: keep-alive

Unfortunately, repeated requests still end up with X-Proxy-Cache-Status: MISS :(.

(Side note: fixing https://bugzilla.mozilla.org/show_bug.cgi?id=1281459 will make URLs like the one you tried returning a 4xx or 5xx response instead of a confusing 200-with-no-caching.)

Comment 5

[Benson Wong [:mostlygeek]]

It seems to be fixed now. Redeployed stage with the latest docker image. Please test.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Benson Wong [:mostlygeek] from comment #5)
> It seems to be fixed now. Redeployed stage with the latest docker image.
> Please test.

Yup, looks good to me now. I did some functional verification of the included changes as well, which all look good. I'm about to leave for the day but we can do the prod push anytime tomorrow onwards - at your convenience (no pressure).

Comment 7

[Benson Wong [:mostlygeek]]

This has been rolled out to production for the web side of things. Looks healthy.

Comment 8

[[e-mail] (:bhearsum)]

Thanks Ben & Chris!