Comments on: Managing Versioning

By: cs satellite services

cs satellite services — Wed, 15 May 2013 04:50:03 +0000

What’s up, always i used to check web site posts here in the early hours in the daylight, as i love to learn more and more.

By: Rjw

Rjw — Wed, 16 Nov 2011 19:51:16 +0000

Yep, that looks pretty good…

By: Alexey Ragozin

Alexey Ragozin — Wed, 16 Nov 2011 19:08:37 +0000

You do not need to do it that complex way

All you need just say

// get latest version for key k cache.entrySet(tsHelper.floor(k, null));


See example on http://code.google.com/p/gridkit/wiki/TimeSeriesIndex
Underhood, helper will create TimeSeriesFilter wrapped into KeyAssociationFilter.
KeyAssociationFilter will ensure requiest to be send to key owner only.

This is almost as efficient and cache.get().

By: Rjw

Rjw — Wed, 16 Nov 2011 00:54:24 +0000

I wonder what the cost of
A) getting the key owner of the non versiony part of the key
B) doing an invocation to the owner
C) once on the owner, poking into your versioned index stashed somewhere accessible by your indexawareextractor, to find the latest key. ( everything written in with immutable version numbers)
D) get (local unless partition redistribution happens before this point) – return that as the result of your invocable.

All depends on the cost of a single invocation, and making sure no pointless deser/ser goes on.

Advantages would be: no second copy, no need to do a trigger that causes double the backup traffic/ events, immutable versions (ie near cacheable with no invalidation), should easily scale to get all, and pretty much zero overhead for as-of versioning ( the invocable can know what the highest version acceptable is) if you do clocked/non-local versions in the keys.

But would need to try it out…

By: ben

ben — Tue, 01 Nov 2011 08:58:56 +0000

Thanks Alexey. That is a very useful post. I have a post about snapshotting and timestamping in the pipeline and your method will be very relevant to that so I will incorporate it there.

I see your solution as slightly different to the two mentioned here though. The ones here are simply designed to allow direct key based access to the latest version – this being the version most users are interested in most of the time. Key based access has some important benefits over regular filters:

- It is faster and more scalable (as it is directed rather than hitting each node). Although this problem can be partially addressed with PartitionFilters.
- It allows you to use near caching – something not available to filters.

As I said though – for time based queries which I’ll be posting on and discussing at the SIG your approach looks ace.

By: Alexey Ragozin

Alexey Ragozin — Tue, 01 Nov 2011 07:40:29 +0000

Hi,

Accidently, I’m working on exactly same problem right now.
And I can suggest 3rd aproach – using special kind of indexing structure (via Coherence index API) to find right version of data.
You can read more details here Monday, October 31, 2011
Data Grid Pattern – Time series index for managing versioned data
(source code is also available)

Reagards,
Alexey

By: ben

ben — Thu, 20 Oct 2011 08:06:11 +0000

Hi Ashwin. It’s actually just a Map. Versions can be expired based on time, size or some custom policy.

By: Ashwin Jayaprakash

Ashwin Jayaprakash — Wed, 19 Oct 2011 21:47:21 +0000

Your version map seems like a per-key queue of fixed size. How do you purge old versions beyond, say 3 revisions/versions?

Do you do a findMaxVersion(key) and findMinVersion(key), then if max > min + 3, then remove(key + min)? That looks like a lot of queries.

Redis has readymade constructs for such things – http://redis.io/commands

Ashwin.