583122 - Hash mismatches soon after add-on updates

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Zandr Milewski [:zandr]]

Every time we release a new version of Weave/Sync, we get a flurry of hash mismatches when people try to download the new plugin. Bug 583036 is the latest instance of this. It will go away after a few hours, but it's a problem when we're trying to drive interest around a new release and it looks broken.

We updated Sync at about 9:30 this morning, and were still seeing hash mismatches around noon.

A brief conversation with jbalogh on #webdev seems to indicate that it's a problem with zamboni and remora moving at different speeds, and he has an idea how to fix it.

I suspect there may be issues with caches/load balancers in front of AMO, but one thing at a time.

Comment 1

[Jeff Balogh (:jbalogh)]

We're fighting against three layers of caching here:

1. zamboni gives the addons table a current_version field which is a link to the latest public/approved/experimental version.  This saves us from doing complicated queries with multiple joins every time we want an add-on.
2. We store queries and Addon objects in memcached.
3. Zeus caches full HTML pages.

The problem is with #1, since remora doesn't know about current_version.  It's running the query we wanted to avoid and finding the real latest version.  That's where we get the hash mismatch, which then gets exacerbated by the two levels of caching above.

In bug 582453 we moved the cron that updates current versions from every three hours to every 20 minutes, and made sure the query it runs is not cached.

http://github.com/jbalogh/zamboni/commit/17698c6 does it even better, using a lighter query that we can run every minute.

Bug 583079 will let us turn on redis and enable invalidation of memcached objects.

Bug 583121 will flush zeus caches when we update an add-on.

Comment 2

[Jeff Balogh (:jbalogh)]

Calling this fixed, the dependent bugs are good enough for this all to work.

Comment 4

[Giorgio Maone [:ma1]]

I still periodically (on each public update) receive complaints like this: http://forums.informaction.com/viewtopic.php?p=22425#p22425

They usually fade out in 24 hours or so, but I wouldn't call it "fixed" anyway.

Comment 5

[Jeff Balogh (:jbalogh)]

Bug 591028 is switching the downloads backend to python, so it will match whatever the frontend is thinking.

Comment 6

[Giorgio Maone [:ma1]]

When does it go live?

Comment 7

[Wil Clouser [:clouserw]]

Nov 9th.  See https://mail.mozilla.com/home/[e-mail]/AMO%20Schedule.html