1263038 - universal-search-recommendation deployment at 2016.04.14

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Chuck Harmston [:chuck]]

Tag: 2014.04.07

Build finished successfully. Please deploy to both staging and production and flush memcached for each.

With this, we should be logging request summaries to stdout and sending the appropriate cache control headers on responses.

Thanks!

Comment 1

[Benson Wong [:mostlygeek]]

What are the reasons for flushing memcached on deployments?

Comment 2

[Chuck Harmston [:chuck]]

With each deploy we've updated the format of the responses, which are generated by workers and read directly out of memcached [1]. This blunt force approach won't be needed once we have users, but for now, it's the simplest way to keep them fresh. 

This deploy adds a 7 day/configurable TTL on memcached entries, which will be one way to expire stale entries. Eventually we'll have beat processes that update popular entries in the background.

Is that a burdensome approach? Is there a way we can make it easier on you?

[1] https://github.com/mozilla/universal-search-recommendation/blob/master/recommendation/views/main.py#L33-L43

Comment 3

[Benson Wong [:mostlygeek]]

I think it'd be better for the service if the schema was versioned in some way. During deployment we roll out new servers beside old in the loadbalancer. So for a time both versions of the schema will be in redis and memcache. It's essential the code knows how to handle old and new messages so we don't get weird crashes and behavior.

Comment 4

[Benson Wong [:mostlygeek]]

Also this deployment is blocked on this issue which just paged me: https://github.com/mozilla/universal-search-recommendation/issues/92

Comment 5

[Benson Wong [:mostlygeek]]

Note: I disabled the pingdom checks until the issue #92 is resolved so it doesn't blow up redis in the middle of the night again.

Comment 6

[Chuck Harmston [:chuck]]

The add-on won't crash and burn if it receives an old schema; it will just show partial data, which I think is acceptable. I think that's fine.

As I noted in the bug, I think the Redis pooling issues are configuration-related, not app-related. I did offer a few suggestions; let me know if I can help debug further. Sorry it gave you a late-night ping!

Comment 7

[Benson Wong [:mostlygeek]]

WONTFIX in favor of bug #1263682

Comment 8

[Benson Wong [:mostlygeek]]

Read bug #1263682, reopening this one for deployment

Comment 9

[Benson Wong [:mostlygeek]]

Tested 2016.04.07 in stage with: 

> while true; do curl localhost/__heartbeat__ ; netstat -aln | grep 6379 | grep ESTAB |  wc -l; done

The number of open established connections continue to grow.

Comment 10

[Benson Wong [:mostlygeek]]

The socket exhaustion bug to redis was fixed in 2016.04.14. This has been deployed to stage and tested. I deemed it ready for production and rolled it out. Verified that the socket use bug is gone.

Comment 11

[Chuck Harmston [:chuck]]

\o/ thanks for the help, my friend.