617080 - Collections show modified as 0.01

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Richard Newman [:rnewman]]

(Times below are for 2010-12-03, though this report is still accurate.)

My info/collections right now:

{"clients":0.01,"crypto":0.01,"forms":0.01,"history":0.01,"keys":0.01,"meta":0.01,"bookmarks":0.01,"prefs":0.01,"passwords":0.01,"tabs":1291436692.25}

and yet:

20:41:59 <@zandr> MariaDB [weave3]> select max(modified) as timestamp from wbo36 where username='397236';
20:41:59 <@zandr> +--------------+
20:41:59 <@zandr> | timestamp    |
20:42:00 <@zandr> +--------------+
20:42:00 <@zandr> | 129143634114 |
20:42:00 <@zandr> +--------------+

There's a ton of data in my sync account, with 2 or 3 clients at a time.

(Username: holygoat, maps to 397236 according to atoll.)

Server is phx-sync223.

I see a 200'ed POST to history at 19:55:37, and yet a GET of info/collections at 20:01:57 the above.

The DB seems OK, which points the finger at memcache.

Concerned that memcache is serving stale data, which would cause a whole bevy of problems -- my data just isn't syncing downstream, because every client thinks it's got the latest, but if this is happening to info/collections, it would explain Bug 617047...

Comment 1

[Toby Elliott [:telliott]]

Attachment: Fixes the bad memcache data structure

Comment 2

[:Atoll]

Comment on attachment 495605 [details] [diff] [review]
Fixes the bad memcache data structure

passes syntax and "makes sense" review from ops

Comment 3

[Mike Connor [:mconnor]]

Comment on attachment 495605 [details] [diff] [review]
Fixes the bad memcache data structure

We really need some tests for info/collections in the server test suite.

Comment 4

[Toby Elliott [:telliott]]

We actually have a bunch. One of the things I need to do once we've put the fire out is figure out why they thought things were OK.

Comment 5

[Toby Elliott [:telliott]]

Attachment: Wil self-heal the memcache

I believe this version will allow us to not flush memcache, and will instead invalidate the ones that have an array in them.

Comment 6

[Mike Connor [:mconnor]]

Comment on attachment 495630 [details] [diff] [review]
Wil self-heal the memcache

>diff -r a2a3c9bf6174 1.1/weave_storage/memcache_layer.php

>+			$bad = 0;
>+			foreach ($collections as $k => $v)
>+			{
>+			    if (is_array($v))
>+			        $bad = 1;

should break here, really

with that, r=me

Comment 7

[Toby Elliott [:telliott]]

True, added a break.

We should probably run this code for a week or so, then remove it. Anyone who hasn't accessed it by then should have their memcache expired.

Comment 8

[:Atoll]

Comment on attachment 495630 [details] [diff] [review]
Wil self-heal the memcache

add a break, which mconnor said too

Comment 9

[Toby Elliott [:telliott]]

pushed to production in emergency push, 12/06 around 2:50PM

Comment 10

[Richard Newman [:rnewman]]

My data look good:

{"clients":1291436341.14,"crypto":1291409096.94,"forms":1291617174.67,"history":1291617176.53,"keys":1291338601.74,"meta":1291420911.9,"bookmarks":1291432381.18,"prefs":1291420911.29,"passwords":1291617177.69,"tabs":1291617177.96}

Two thumbs up.

Comment 11

[Doug Turner (:dougt)]

what is the bug number for the "info/collections in the server test suite"?

Comment 12

[Toby Elliott [:telliott]]

What do you mean by bug number? There's a set of functional tests included with the server, that, among other things, check that info/collections is returning the correct data in a series of tests. They failed in this case because the primary bug derived from the fact the the old memcache and the new memcache didn't play well together (and possibly didn't get flushed), and the test suite cleans up after itself.

Comment 13

[Ragavan S [:rags]]

(In reply to comment #12)
> What do you mean by bug number? There's a set of functional tests included with
> the server, that, among other things, check that info/collections is returning
> the correct data in a series of tests. They failed in this case because the
> primary bug derived from the fact the the old memcache and the new memcache
> didn't play well together (and possibly didn't get flushed), and the test suite
> cleans up after itself.

I'm guessing Doug's asking about mconnor's comment#3 above.