April, 2014
Sizing Coherence Indexes
Monday, April 28th, 2014
This post suggests three ways to measure index sizes in Coherence: (a)Using the Coherence MBean (not recommended) (b) Use JMX to GC the cluster (ideally programatically) as you add/remove indexes – this is intrusive (c) Use the SizeOf invocable (recommended) – this requires teh use of a -javaagent option on your command line.
(a) The Coherence MBean
Calculating the data size is pretty easy, you just add a unit calculator and sum over the cluster (there is code to do that here: test, util). Indexes however are more tricky.
Coherence provides an IndexInfo MBean that tries to calculate the size. This is such an important factor in maintaining you cluster it’s worth investigating.
Alas the IndexInfo Footprint is not very accurate. There is a test,IsCoherenceFootprintMBeanAccurate.java,which demonstrates there are huge differences in some cases (5 orders of magnitude). In summary:
– The Footprint is broadly accurate for fairly large fields (~1k) where the index is unique.
– As the cardinality of the index drops the Footprint Mbean starts to underestimate the footprint.
– As the size of the field being indexed gets smaller the MBean starts to underestimate the index.
Probably most importantly for the most likely case, for example the indexed fields is fairly small say 8B, and the cardinality is around half the count, the MBean estimate is out by three orders of magnitude.
Here are the results for the cardinality of half the count and field sizes 8B, 16B, 32B
Ran: 32,768 x 32B fields [1,024KB indexable data], Cardinality of 512 [512 entries in index, each containing 64 values], Coherence MBean measured: 49,152B. JVM increase: 3,162,544B. Difference: 6334%
Ran: 65,536 x 16B fields [1,024KB indexable data], Cardinality of 512 [512 entries in index, each containing 128 values], Coherence MBean measured: 40,960B. JVM increase: 5,095,888B. Difference: 12341%
Ran: 131,072 x 8B fields [1,024KB indexable data], Cardinality of 512 [512 entries in index, each containing 256 values], Coherence MBean measured: 40,960B. JVM increase: 10,196,616B. Difference: 24794%
In short, it’s too inaccurate to be useful.
(b) Using JMX to GC before and after adding indexes
So the we’re left with a more intrusive process to work out our index sizes:
- Load your cluster up with indexes.
- GC a node and take it’s memory footprint via JMX/JConsole/VisualVm
- Drop all indexes
- GC the node again and work out how much the heap dropped by.
I have a script which does this programatically via JMX. It cycles through all the indexes we have doing:
ForEach(Index) { GC->MeasureMemory->DropIndex->GC->MeasureMemory->AddIndexBack }This method works pretty well although it takes a fair while to run if you have a large number of indexes, and is intrusive so you couldn’t run in production. It also relies on our indexes all being statically declared in a single place. This is generally a good idea for any project. I don’t know of a way to extract the ValueExtractor programatically from Coherence so just use the static instance in our code.
(c) Use Java’s Inbuilt Instrumentation
This is the best way (in my opinion). It’s simple and accurate. The only issue is that you need to start your coherence processes with a javaagent as it’s using the Instrumentation API to size indexes.
The instrumentation agent itself is very simple, and uses the library found here. All we need to wrap that is an invocable which executes it on each node in the cluster.
The invocable just loops over each cache service, and each cache within that service, calculating the size of the IndexMap using the instrumentation SizeOf.jar
To implement this yourself:
1) Grab these two classes: SizeOfIndexSizer.java, IndexCountingInvocable.java and add them to your classpath. The first sets the invocable off, handling the results. The second is the invocable that runs on each node and calculates the size of the index map.
2) Take a copy of SizeOf.jar from here and add -javaagent:<pathtojar>/SizeOf.jar to your command line.
3) Call the relevant method on SizeOfIndexSizer.
When is POF a Good Idea?
Saturday, April 12th, 2014
POF is pretty cool. Like Protocol Buffers, which they are broadly similar to, POF provides an space-efficient, byte-packed wire / storage format which is navigable in its binary form. This makes it a better than Java serialisation for most applications (although if you’re not using Coherence then PB are a better bet).
Being a bit-packed format it’s important to understand the performance implications of extracting different parts of the POF stream. This being different to the performance characteristics of other storage formats, in particular fixed width formats such as those used in most databases which provide very fast traversal.
To get an understanding of the POF format see the primer here. In summary:
1. Smaller than standard Java Serialisation: The serialised format is much smaller than java serialisation as only integers are encoded in the stream rather than the fully class/type information.
2. Smaller than fixed-width formats: The bit-packed format provides a small memory footprint when compared to fixed length fields and doesn’t suffer from requiring overflow mechanisms for large values. This makes it versatile.
3. Navigable: The stream can be navigated to read single values without deserialising the whole stream (object graph).
Things to Watch Out For:
1. Access to fields further down the stream is O(n) and this can become dominant for large objects:
Because the stream is ‘packed’, rather than using fixed length fields, traversing the stream is O(n), particularly the further down the stream you go. That’s to say extracting the last element will be slower than extracting the first. Fixed width fields have access times O(1) as they can navigate to a field number directly.
We can measure this using something along the lines of:
Binary pof = ExternalizableHelper.toBinary(object, context);
SimplePofPath path = new SimplePofPath(fieldPos);//vary the position in the stream
PofExtractor pofExtractor = new PofExtractor(ComplexPofObject.class, path);
while (count --> 0) {
PofValue value = PofValueParser.parse(pof, context);
pofExtractor.getNavigator().navigate(value).getValue();
}
If you want to run this yourself it’s available here: howMuchSlowerIsPullingDataFromTheEndOfTheStreamRatherThanTheStart(). This code produces the following output:
> Extraction time for SimplePofPath(indices=1) is 200 ns > Extraction time for SimplePofPath(indices=2) is 212 ns > Extraction time for SimplePofPath(indices=4) is 258 ns > Extraction time for SimplePofPath(indices=8) is 353 ns > Extraction time for SimplePofPath(indices=16) is 564 ns > Extraction time for SimplePofPath(indices=32) is 946 ns > Extraction time for SimplePofPath(indices=64) is 1,708 ns > Extraction time for SimplePofPath(indices=128) is 3,459 ns > Extraction time for SimplePofPath(indices=256) is 6,829 ns > Extraction time for SimplePofPath(indices=512) is 13,595 ns > Extraction time for SimplePofPath(indices=1024) is 27,155 ns
It’s pretty clear (and not really surprising) that the navigation goes O(n). The bigger problem is that this can have an affect on your queries as your datasize grows.
Having 100 fields in a pof object is not unusual, but if you do, the core part of your query is going to run 20 slower when retrieving the last field than it is when you retrieve the first.
For a 100 field object, querying on the 100th field will be 20 times slower than querying the first
This is just a factor of the variable length encoding. The code has no context of the position of a particular field in the stream when it starts traversing it. It has no option but to traverse each value, find it’s length and skip to the next one. Thus the 10th field is found by skipping the first 9 fields. This is in comparison to fixed length formats where extracting the nth field is always O(1).
2) It can be more efficient to deserialise the whole object, and makes your code simpler too
If you’re just using a simple filter (without an index) POF makes a lot of sense, use it, but if you’re doing more complex work that uses multiple fields from the stream then it can be faster to deserialise the whole object. It also makes your code a lot simpler as dealing with POF directly gets pretty ugly as the complexity grows.
We can reason about whether it’s worth deserialising the whole object by comparing serialisation times with the time taken to extract multiple fields.
The test whenDoesPofExtractionStopsBeingMoreEfficient() measures the break even point beyond which we may as well deserialise the whole object. Very broadly speaking it’s 4 extractions, but lets look at the details.
Running the test yields the following output:
On average full deserialisation of a 50 field object took 3225.0ns On average POF extraction of first 5 fields of 50 took 1545.0ns On average POF extraction of last 5 fields of 50 took 4802.0ns On average POF extraction of random 5 fields of 50 took 2934.0ns
Running this test and varying the number of fields in the object leads to the following conclusions.
– for objects of 5 fields the break even point is deserialising 2 pof fields
– for objects of 20 fields the break even point is deserialising 4 pof fields
– for objects of 50 fields the break even point is deserialising 5 pof fields
– for objects of 100 fields the break even point is deserialising 7 pof fields
– for objects of 200 fields the break even point is deserialising 9 pof fields
Or to put it another way, if you have 20 fields in your object and you extracted them one at a time it would be five times slower than deserialising.
In theory the use of the PofValue object should optimise this. The PofValueParser, which is used to create PofValue objects, effectively creates an index over the pof stream meaning that, in theory, reading multiple fields from the pof value should be O(1) each. However in these test I have been unable to see this gain.
Conclusions/Recommendations
Pof is about more than performance. It negates the need to put your classes (which can change) on the server. This in itself is a pretty darn good reason to use it. However it’s worth considering performance. It’s definitely faster than Java serialisation. That’s a given. But you do need to be wary about using pof extractors to get individual fields, particularly if you have large objects.
The degradation is O(n), where n is the number of fields in the object, as the stream must be traversed one field at a time. This is a classic space/time tradeoff. The alternative, faster O(1), fixed-width approach would require more storage which can be costly for in memory technologies.
Fortunately there is a workaround of sorts. If you have large objects, and are using POF extraction for your queries (i.e. you are not using indexes which ensure a deserialised field will be on the heap), then prefer composites of objects to large (long) flat ones. This will reduce the number of skipPofValue() calls that the extractor will have to do.
If you have large objects and are extracting many fields to do their work (more than 5-10 extractions per object) then it may be best to deserialise the whole thing. In cases like this pof-extraction will be counter productive, at least from a performance perspective. Probably more importantly, if you’re doing 5-10 extractions per object, you are doing something fairly complex (but this certainly happens in Coherence projects) so deserialising the object and writing your logic against PoJos is going to make your code look a whole lot better too. If in doubt, measure it!
Ref: JK posted on this too when we first became aware of the problem.
POF Primer
Saturday, April 12th, 2014
This is a brief primer on POF (Portable Object Format) used in Coherence to serialise data. POF is much like Google’s Protocol Buffers so if you’re familiar with those you probably don’t need to read this.
POF a variable length, bit-packed serialisation format used to represent object graphs as byte arrays in as few bytes as possible, without the use of compression. Pof’s key property is that it is navigable. That is to say you can pull a value (object or primitive) out of the stream without having to deserilalise the whole thing. A feature that is very useful if you want to query a field in an object which is not indexed.
The Format
Conceptually simple, each class writes out its fields to a binary stream using a single bit-packed (variable length encoded) integer as an index followed by a value. Various other pieces of metadata are also encoded into the stream using bit-packed ints. It’s simplest to show in pictorially:
Variable Length Encoding using Bit-Packed Values
Variable length encoding uses as few bytes as possible to represent a field. It’s worth focusing on this for a second. Consider the job of representing an Integer in as few bytes as possible. Integers are typically four bytes but you don’t really need four bytes to represent the number 4. You can do that in a few bits.
PackedInts in Coherence take advantage of this to represents an integer in one to five bytes. The first bit of every byte indicates whether subsequent bytes are needed to represent this number. The second bit of the first byte represents the sign of the number. This means there are six ‘useful’ bits in the first byte and 7 ‘useful’ bits in all subsequent ones, where ‘useful’ means ‘can be used to represent our number’.
Taking an example let’s look at representing the number 25 (11001) as a bit-packed stream:
25 //Decimal
11001 //Binary
[0 0 0011001] //POF (leading bits denote: whether more bytes are needed, the sign)
Line 1 shows our decimal, line 2 its binary form. Line 3 shows how it is represented as POF. Note that we have added four zeros to the front of the number denoting that no following bytes are required to represent the number and that the number is positive.
If the number to be encoded is greater than 63 then a second byte is needed. The first bit again signifies whether further bits will be needed to encode the number. There is no sign-bit in the second byte as it’s implied from the first. Also, just to confuse us a little, the encoding of the numeric value is different to the single-byte encoding used above: the binary number is reversed so the least significant byte is first (the whole number appears reversed across the two bytes). So the number 128 (10000000) would be encoded:
128 //Decimal
10000000 //Binary
00000001 //Binary (reversed)
000000 00010 //Aligned
[1 0 000000] [0 00010] //POF
The pattern continues with numbers greater than or equal to 2^13 which need a third byte to represent them. For example 123456 (11110001001000000) would be represented
123456 //Decimal
11110001001000000 //Binary
00000010010001111 //Reversed
000000 1001000 0001111 //Aligned
[1 0 000000] [1 0001001] [0 0001111] //POF
Note again that the binary number is reversed and then laid with the least significant bit first (unlike the single btye encoding above).
In this way the average storage is as small as it can be without actually using compression.
Exploring a POF Stream (see Gist)
We can explore a little further by looking at the Coherence API. Lets start with a simple POF object:
public class PofObject implements PortableObject {
private Object data;
PofObject(Object data) {
this.data = data;
}
public void readExternal(PofReader pofReader) throws IOException {
data = pofReader.readObject(1);
}
public void writeExternal(PofWriter pofWriter) throws IOException {
pofWriter.writeObject(1, data);
}
}
We can explore each element in the stream using the readPackedInt() method to read POF integers and we’ll need a readSafeUTF() for the String value:
SimplePofContext context = new SimplePofContext();
context.registerUserType(1042, PofObject.class, new PortableObjectSerializer(1042));
PofObject object = new PofObject("TheData");
//get the binary & stream
Binary binary = ExternalizableHelper.toBinary(object, context);
ReadBuffer.BufferInput stream = binary.getBufferInput();
System.out.printf("Header btye: %s\n" +
"ClassType is: %s\n" +
"ClassVersion is: %s\n" +
"FieldPofId is: %s\n" +
"Field data type is: %s\n" +
"Field length is: %s\n",
stream.readPackedInt(),
stream.readPackedInt(),
stream.readPackedInt(),
stream.readPackedInt(),
stream.readPackedInt(),
stream.readPackedInt()
);
System.out.printf("Field Value is: %s\n",
binary.toBinary(6, "TheData".length() + 1).getBufferInput().readSafeUTF()
);
Running this code yields:
> Header btye: 21 > ClassType is: 1042 > ClassVersion is: 0 > FieldPofId is: 1 > Field data type is: -15 > Field length is: 7 > Field Value is: TheData
Notice line 25, which reads the UTF String, requires the length as well as the value (it reads bytes 6-15 where 6 is the length and 7-15 are the value).
Finally POF Objects are nested into the stream. So if field 3 is a user’s object, rather than a primitive value, an equivalent POF-stream for the user’s object is nested in the ‘value’ section of the stream, forming a tree that represents the whole object graph.
The code for this is available on Gist and there is more about POF internals in the coherence-bootstrap project on github: PofInternals.java.