1268165 - Deployment of universal-search-recommendation at 2014.04.27

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task, P1)

Description

[Chuck Harmston [:chuck]]

https://github.com/mozilla/universal-search-recommendation/tree/2014.04.27

This is for an internal release tomorrow.

Comment 1

[Benson Wong [:mostlygeek]]

:chuck

- are you ok with this skipping stage and going straight to prod? 
- do the workers need to be updated to the same version as well?

Comment 2

[Chuck Harmston [:chuck]]

Can we please deploy to both stage and prod, web and workers? I don't think there should ever be a situation where prod is ahead of stage, or web and worker servers in the same environment are out of sync.

For most deployments (unless otherwise noted, I'd say) we will still want to flush_all in memcached, and run a new script at /app/prepopulate.py (which creates ~5000 jobs to prefill the cache) on the web server . Would you like that noted on each new deployment, or should we keep a process doc somewhere?

Comment 3

[Benson Wong [:mostlygeek]]

> For most deployments (unless otherwise noted, I'd say) we will still want to flush_all in memcached, and run a new script at /app/prepopulate.py (which creates ~5000 jobs to prefill the cache) on the web server . Would you like that noted on each new deployment, or should we keep a process doc somewhere?

Actually, I don't think flushing/filling of the cache should be a requirement for deployment. I believe you said the cache flush was because of schema incompatibilities between versions. Is this still the case?

Comment 4

[Chuck Harmston [:chuck]]

That's correct. Looking at the roadmap, I'd wager that in the near future, there will be schema changes for every deployment.

Comment 5

[Benson Wong [:mostlygeek]]

I suggest the schema be versioned and any backwards compatibility logic be managed within the app. In prod, there will always be a mix of new/old servers serving live traffic. Same applies with the workers. We do this so the application can be deployed under load.

Comment 6

[Chuck Harmston [:chuck]]

Generally speaking, I'd agree that that's the right approach. But for what is currently a UI test that may not see much further development, I don't think it's worth the scope it would add. The consumers of the API correctly handle partial schema matches and errored responses, which is good enough for what we're trying to test.

Comment 7

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

> I suggest the schema be versioned and any backwards compatibility logic be managed within the app. In 
> prod, there will always be a mix of new/old servers serving live traffic. Same applies with the workers. We 
> do this so the application can be deployed under load.

That's a good idea, but I think it's a bit more work than we need to do.

The universal search add-on inserts a recommendation if it has one pre-cached. In many cases, no recommendation is shown at all as the user types--and that's ok.

If we ever change the schema of the data packets, causing a few minutes of breakage, users will still see the regular contents of the awesomebar dropdown menu, they just won't see a recommendation for a little bit. So, there's really no significant user-facing consequences to breaking backwards versioning.

On the other hand, as Chuck said, adding backwards compatibility, carefully defining and testing the schema, creating docs for the different versions, etc., would add a lot of overhead to what is ultimately a throwaway prototype. Does this make sense?

Comment 8

[Benson Wong [:mostlygeek]]

Heh. Let's work together to make sure universal search gets to that point then :) 

OK it's out in stage. I also ran the script and it said: 

app@ip-172-31-27-177:/app$ ./prepopulate.py
5845 items queued.

Can the prepopulate.py script do both the flush and prepopulate?

Comment 9

[Chuck Harmston [:chuck]]

Those are orthogonal concerns, I'm not sure I'd want to put them in the same script, but we can definitely write a short one to flush memcached and clear any queued jobs (which might produce old schema results).

https://github.com/mozilla/universal-search-recommendation/issues/111

Comment 10

[Benson Wong [:mostlygeek]]

As requested on IRC, rolling out another stage deploy with updated embedly keys.

Comment 11

[Benson Wong [:mostlygeek]]

OK stage rolled out. Memcache flushed, and prepopulate.py run. Should be ready for another test.

Comment 12

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Updated staging server looks ok to me: I see embedly images coming back in some responses.

Waiting on input from chuck before proceeding.

Comment 13

[Chuck Harmston [:chuck]]

Looks good to me: https://universalsearch.stage.mozaws.net/?q=f

Comment 14

[Benson Wong [:mostlygeek]]

rolled out to production.