Collections show modified as 0.01

RESOLVED FIXED

Status

RESOLVED FIXED
8 years ago
8 years ago

People

(Reporter: rnewman, Assigned: telliott)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

8 years ago
(Times below are for 2010-12-03, though this report is still accurate.)

My info/collections right now:

{"clients":0.01,"crypto":0.01,"forms":0.01,"history":0.01,"keys":0.01,"meta":0.01,"bookmarks":0.01,"prefs":0.01,"passwords":0.01,"tabs":1291436692.25}

and yet:

20:41:59 <@zandr> MariaDB [weave3]> select max(modified) as timestamp from wbo36 where username='397236';
20:41:59 <@zandr> +--------------+
20:41:59 <@zandr> | timestamp    |
20:42:00 <@zandr> +--------------+
20:42:00 <@zandr> | 129143634114 |
20:42:00 <@zandr> +--------------+

There's a ton of data in my sync account, with 2 or 3 clients at a time.

(Username: holygoat, maps to 397236 according to atoll.)

Server is phx-sync223.

I see a 200'ed POST to history at 19:55:37, and yet a GET of info/collections at 20:01:57 the above.

The DB seems OK, which points the finger at memcache.

Concerned that memcache is serving stale data, which would cause a whole bevy of problems -- my data just isn't syncing downstream, because every client thinks it's got the latest, but if this is happening to info/collections, it would explain Bug 617047...
(Reporter)

Updated

8 years ago
Summary: Collections show modified as 0.1 → Collections show modified as 0.01
(Assignee)

Comment 1

8 years ago
Created attachment 495605 [details] [diff] [review]
Fixes the bad memcache data structure
Attachment #495605 - Flags: review?(mconnor)
Assignee: nobody → telliott

Comment 2

8 years ago
Comment on attachment 495605 [details] [diff] [review]
Fixes the bad memcache data structure

passes syntax and "makes sense" review from ops
Attachment #495605 - Flags: review+
Comment on attachment 495605 [details] [diff] [review]
Fixes the bad memcache data structure

We really need some tests for info/collections in the server test suite.
Attachment #495605 - Flags: review?(mconnor) → review+
(Assignee)

Comment 4

8 years ago
We actually have a bunch. One of the things I need to do once we've put the fire out is figure out why they thought things were OK.
(Assignee)

Comment 5

8 years ago
Created attachment 495630 [details] [diff] [review]
Wil self-heal the memcache

I believe this version will allow us to not flush memcache, and will instead invalidate the ones that have an array in them.
Attachment #495630 - Flags: review?(mconnor)
Comment on attachment 495630 [details] [diff] [review]
Wil self-heal the memcache

>diff -r a2a3c9bf6174 1.1/weave_storage/memcache_layer.php

>+			$bad = 0;
>+			foreach ($collections as $k => $v)
>+			{
>+			    if (is_array($v))
>+			        $bad = 1;

should break here, really

with that, r=me
Attachment #495630 - Flags: review?(mconnor) → review+
(Assignee)

Comment 7

8 years ago
True, added a break.

We should probably run this code for a week or so, then remove it. Anyone who hasn't accessed it by then should have their memcache expired.

Comment 8

8 years ago
Comment on attachment 495630 [details] [diff] [review]
Wil self-heal the memcache

add a break, which mconnor said too
Attachment #495630 - Flags: review+
(Assignee)

Comment 9

8 years ago
pushed to production in emergency push, 12/06 around 2:50PM
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
(Reporter)

Comment 10

8 years ago
My data look good:

{"clients":1291436341.14,"crypto":1291409096.94,"forms":1291617174.67,"history":1291617176.53,"keys":1291338601.74,"meta":1291420911.9,"bookmarks":1291432381.18,"prefs":1291420911.29,"passwords":1291617177.69,"tabs":1291617177.96}

Two thumbs up.
what is the bug number for the "info/collections in the server test suite"?
(Assignee)

Comment 12

8 years ago
What do you mean by bug number? There's a set of functional tests included with the server, that, among other things, check that info/collections is returning the correct data in a series of tests. They failed in this case because the primary bug derived from the fact the the old memcache and the new memcache didn't play well together (and possibly didn't get flushed), and the test suite cleans up after itself.
(In reply to comment #12)
> What do you mean by bug number? There's a set of functional tests included with
> the server, that, among other things, check that info/collections is returning
> the correct data in a series of tests. They failed in this case because the
> primary bug derived from the fact the the old memcache and the new memcache
> didn't play well together (and possibly didn't get flushed), and the test suite
> cleans up after itself.

I'm guessing Doug's asking about mconnor's comment#3 above.
You need to log in before you can comment on or make changes to this bug.