Deployment of universal-search-recommendation at 2014.04.27

RESOLVED FIXED

Status

Cloud Services
Operations: Deployment Requests
P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: chuck, Assigned: mostlygeek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
https://github.com/mozilla/universal-search-recommendation/tree/2014.04.27

This is for an internal release tomorrow.
(Assignee)

Comment 1

2 years ago
:chuck

- are you ok with this skipping stage and going straight to prod? 
- do the workers need to be updated to the same version as well?
Flags: needinfo?(charmston)
(Reporter)

Comment 2

2 years ago
Can we please deploy to both stage and prod, web and workers? I don't think there should ever be a situation where prod is ahead of stage, or web and worker servers in the same environment are out of sync.

For most deployments (unless otherwise noted, I'd say) we will still want to flush_all in memcached, and run a new script at /app/prepopulate.py (which creates ~5000 jobs to prefill the cache) on the web server . Would you like that noted on each new deployment, or should we keep a process doc somewhere?
Flags: needinfo?(charmston)
(Assignee)

Comment 3

2 years ago
> For most deployments (unless otherwise noted, I'd say) we will still want to flush_all in memcached, and run a new script at /app/prepopulate.py (which creates ~5000 jobs to prefill the cache) on the web server . Would you like that noted on each new deployment, or should we keep a process doc somewhere?

Actually, I don't think flushing/filling of the cache should be a requirement for deployment. I believe you said the cache flush was because of schema incompatibilities between versions. Is this still the case?
Flags: needinfo?(charmston)
(Reporter)

Comment 4

2 years ago
That's correct. Looking at the roadmap, I'd wager that in the near future, there will be schema changes for every deployment.
Flags: needinfo?(charmston)
(Assignee)

Comment 5

2 years ago
I suggest the schema be versioned and any backwards compatibility logic be managed within the app. In prod, there will always be a mix of new/old servers serving live traffic. Same applies with the workers. We do this so the application can be deployed under load.
(Reporter)

Comment 6

2 years ago
Generally speaking, I'd agree that that's the right approach. But for what is currently a UI test that may not see much further development, I don't think it's worth the scope it would add. The consumers of the API correctly handle partial schema matches and errored responses, which is good enough for what we're trying to test.
> I suggest the schema be versioned and any backwards compatibility logic be managed within the app. In 
> prod, there will always be a mix of new/old servers serving live traffic. Same applies with the workers. We 
> do this so the application can be deployed under load.

That's a good idea, but I think it's a bit more work than we need to do.

The universal search add-on inserts a recommendation if it has one pre-cached. In many cases, no recommendation is shown at all as the user types--and that's ok.

If we ever change the schema of the data packets, causing a few minutes of breakage, users will still see the regular contents of the awesomebar dropdown menu, they just won't see a recommendation for a little bit. So, there's really no significant user-facing consequences to breaking backwards versioning.

On the other hand, as Chuck said, adding backwards compatibility, carefully defining and testing the schema, creating docs for the different versions, etc., would add a lot of overhead to what is ultimately a throwaway prototype. Does this make sense?
(Assignee)

Comment 8

2 years ago
Heh. Let's work together to make sure universal search gets to that point then :) 

OK it's out in stage. I also ran the script and it said: 

app@ip-172-31-27-177:/app$ ./prepopulate.py
5845 items queued.

Can the prepopulate.py script do both the flush and prepopulate?
(Reporter)

Comment 9

2 years ago
Those are orthogonal concerns, I'm not sure I'd want to put them in the same script, but we can definitely write a short one to flush memcached and clear any queued jobs (which might produce old schema results).

https://github.com/mozilla/universal-search-recommendation/issues/111
(Assignee)

Comment 10

2 years ago
As requested on IRC, rolling out another stage deploy with updated embedly keys.
(Assignee)

Comment 11

2 years ago
OK stage rolled out. Memcache flushed, and prepopulate.py run. Should be ready for another test.
Updated staging server looks ok to me: I see embedly images coming back in some responses.

Waiting on input from chuck before proceeding.
(Reporter)

Comment 13

2 years ago
Looks good to me: https://universalsearch.stage.mozaws.net/?q=f
(Assignee)

Comment 14

2 years ago
rolled out to production.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.