Let’s mention also, here, the denormalization method that is one method, used in SN architecture, for enabling all related data to hold on the same node.

One of the interesting aspects of data warehousing is that in many cases, data is only ever added to the database and rows should not be updated. This is often a guiding principle and applies to both fact based and dimensional data. This is often implemented through the use of effective dates. In that context, the requirement to ‘lock’ rows for update is substantially reduced. In this situation the main limiting factor that you have identified is substantially reduced in its effect for SD. How about the following as a suggested amendment :

Shared Disk Architectures are write limited should they require locks that must be coordinated across the cluster. Shared Nothing Architectures are limited should they require a distributed two phase commit.

In fact there is then a close relationship between the two limiting factors – essentially both systems are limited by the requirement to co-ordinate data change across the cluster. In the case of SN, the co-ordination is required to ensure two or more changes are co-ordinated, in the case of SD the co-ordination is required to co-ordinate competing write requests.

This observation is also relevant when considering partitioning of data in a SD system, If data is only ever being added, then data-loading becomes straightforward to parallelise up to the level of a load process per data partition and the SN system becomes the equivalent of a SN system for the load process.

When considering the case of complex queries, then the second limitation of SN you mention (data shipping) becomes the main limiting factor. Again in a DW use case, complex queries that include many tens of joins are the norm – queries with 20-30 are not unusual. Thus the output of each join action will probably not be partitioned on the same key as the next join action and so cross-node shipping of data becomes the norm rather then the exception. The bottleneck for query performance then becomes the network between the nodes. Interestingly, this provides a performance profile for such queries which can provide a reduction in performance as the number of nodes in a SN cluster increases – for 2 nodes, 50% of the data has to be shipped, for 10 nodes, that becomes 90%. I have seen situations where scaling the cluster on SN is near linear until the network is saturated, after that, performance will be flat on extending the cluster. However, as congestion on the network increases, performance will then degrade and so the addition of an node to an SN cluster can result in slower query times.

A final point I would add is that a major factor effecting the performance of many implementations of SN in data warehousing is data skew. Where the partitioning (or sharding) is determined by the value of a data item then if that data item has anything other than a symmetric distribution then it’s likely that the partitions will have unequal sizes and thus unequal scan times. This is not mitigated by hashing (as hashing changes data location but not distribution) but is mitigated by a round-robin partitioning algorithm.

Besides performance/scalability, another difference between shared-disk and shared-nothing has to do with how failures are handling. In a shared-disk model, failure of a node requires a complex recovery/failover procedure in which locks are broken, ownership of storage transferred, etc. This is fairly well understood technology today, but – having been around since we were all inventing that technology together – I continue to see it misapplied. In a shared-nothing model, storage and node failures can be handled the same way. Precisely because any given piece of raw storage is “stranded” on one node, systems built around this paradigm tend to involve replication “further out” from the disks so the requesting node simply uses another replica regardless of which type of failure it was. It’s one of those cases IMO where a worse problem leads to a better solution, because the problem (half solved by the multi-controller RAID hardware that shared-disk types use and shared-nothing types eschew) can’t be ignored.

Another interesting angle here, since you work on Coherence, is how some of these same principles apply to more memory-centric stores. Even with SCI and IB available, memory is essentially shared-nothing so it makes sense if it and disk can be handled in consistent ways instead of having two different systems with sharply different performance and failure-handling characteristics.