From Scale out NAS to Scale out Database for Big Data

This post is the first review of NewSQL DBs, as promised in my last post and we will start with Clustrix. Clustrix is interesting for two reasons: 1. the architect and some of the principle designers came from NAS space (as opposed to database technology) and 2. they appear to be doing the Isilon equivalent in Database space.
Scale Out
 Scale out as a concept became popular with massive growth of Data centre. Essentially scale out means to scale horizontally i.e. adding more compute resources to the infrastructure pool in order to take care of growing demand. Essentially it means if you have one server for n users, to serve m*n users you simply need to add m similar servers. In contrast Scale-up means scaling vertically i.e. changing your server with a more powerful server when numbers of users increase.
Scale Out NAS
So scaling out is simpler, assuming one has enough rack-space in the data centre. During last decade, Scale-out NAS became a strong market driver and many innovation engines took off that promised faster, simpler and cheaper scale out. Huge growth of unstructured data in entreprise data centre pushed the scale-up model to the edge with each hardware replacement cost getting more and more expensive. Addditonally traditional NAS also comes with certain scaling limitations such as,
1. performance is limited by single head [multicore CPUs alleviated the issue just a little]
2. limited Namespace [ typically this is File system limitation brought in to ensure performance does not degrade unacceptably with large number of files]
3. migration of data between two NAS namespace is costly.
3PAR, Spinnaker and Islion were few of the innovation leaders in the space of Scale-out NAS. After NetApp bought Spinnaker, Isilon almost occupied this niche space entirely till EMC bought it.
What did Isilon bring technologically?
Isilon's OneFS clustered File System is the core of its technology. The OneFS software is designed with file-striping functionality across each node in a cluster, a fully distributed lock manager, caching, fully distributed meta-data, and a remote block manager to maintain global coherency and synchronization across the cluster. That enables Isilon to support a massive capacity of 15PB of file storage in a single file system, making EMC position Isilon NAS as the solution for Big Data for File-storage.
Scale-out Database Appliance

Experience of building the distributed lock manager has some relevance to Database and that gave confidence to the Isilon engineers when they started designing Clustrix. Concurrency management is one of the crucial challenge in designing a clustered database.  Clustrix claims to use MVCC [Multi-version concurrency Control] with a share-nothing architecture but the most important part of their innovation is building a distributed Query execution model which they call 'Sierra Distributed Engine'.  True to their philosophy of  "Bring Query to Data and not the Data to the Query",
Sierra’s most basic primitive is a compiled program called a query fragment. They can read, insert, update a container, execute functions, modify control flow, format rows, perform synchronization, and send rows to query fragments on other nodes. These query fragments are run on the nodes that contain the data. The communication between the nodes consists of just the intermediate and result rows needed for the queries. Many, many query fragments can operate simultaneously across the cluster. Those query fragments may be different components of the same query or parts of different queries.
Queries enter the system through the front-end network and are translated to an internal representation used by Sierra. Sierra then executes the queries in an efficient parallel fashion. Sierra uses SSDs to store the data, NVRAM to journal changes, and Infiniband to communicate with other nodes in the cluster. [Clustrix whitepaper]
Performance
Clustrix claims that transaction throughput /million of records in the database scales linearly with addition of nodes.
They support replication with MySQL, both Key-Value based query and SQL query as well as on-the-fly Schema change. The user does not need to worry about sharding / partitioning since basic performance bottleneck observed with RDBMS is removed here. Overall, Clustrix seem to have brought a strong NewSQL product.
For more information take a look at Clustrix resource page.

NewSQL way, neither traditional RDBMS, nor NoSQL

This December, NuoDB [erstwhile NimbusDB] released the Beta 5 of its database software. This database, it claims, "is a NewSQL database. It looks and behaves like a traditional SQL database from the outside but under the covers it's a revolutionary database solution. It is a new class of database for a new class of datacenter."
NewSQL database?
It appears that although the NoSQL variant got popularity with likes of Google, Yahoo, Facebook, it did not make much dent to RDBMS clientale base and main reason, the NewSQL advocates cite, is that people like SQL and irrespective of scalability and other issues, people decided to stay with SQL. The fact that SQL has been in the game for last 20-25 years makes it so entrenched in Business Application space that it is almost impossible to take it away from the Business Application Space. Even if one ignores the large SQL investment, one cannot ignore the value SQL brought to business community. SQL essentially separated data from the code, that protects the data, i.e. Database Engine, enabling the DBMS packages to be commoditized.
SQL also provided a well-undersood and simple Data manipulation interface which Business Applications could use without needing to assimilate full-complexities of  computer programming. Business Applications would have become far more complex if they had to deal with ACID requirement of their data in addition to implement their business logic.
NoSQL, however scalable and flexible they are, comes with a huge cost, one has to design his own data manipulation engine and larger the scale of the data and larger the distribution of computing resources, more intensive is the effort. The uniqueness of the applications demands specific design of the engine that manipulates the applications data and thereby makes it more tied to business logic of the enterprise. While it has its strength, it is obvious that most of the business whose core operation is not about the data itself, has stayed away from NoSQL movement, however large their data are.

Promises of NewSQL : SQL for Big Data
NewSQL tries to bridge this gap by keeping the SQL interface intact but trying to reengineer the basic database engine. Evidently this is far more daunting that coming up with NoSQL alternative. It requires a change in the design that has held its ground for almost 30 years. Only those who understands the intricacies of the original design can venture to take this task of rearchitecting the database engine keeping its original promises intact. What does it mean? It means that the engine must fully support SQL, engine must guarantee ACID [my previous posts elaborated on how NoSQL addressed this requirement]. Additionally engine must 1.  provide support for loosely connected set of computing resources, such as computers connected over Internet and 2. scale the performance with the number of computers.  The last requirement came from NoSQL land, where huge number of computing resources are connected over Internet and are designed to sift through the massive distributed data to find out answer for a single query. Essentially this engine must be capable of building a distributed database spread over huge number of affordable [i.e. cheap] computers connected on Internet and provide a SQL interface for the entire data. That makes the NewSQL truly the database engine for Big Data.
Contenders
 Most of the challengers in this space are started by someone who has participated in the Database software development in early 70's . Let's take the example of VoltDB, started by Michael StoneBraker, a luminary in Database Research who architected Ingres [one of the first Relational DBs], Postgres and many more.
Similarly  NuoDB boasts of Jim Starkey, the person behind DEC's relational DB suites during 75-85.
The other prominent NewSQL venture, ScaleDB was started by another Database legend Vern Watts, the architect of famed DB2 from IBM.
Then there is JustoneDB that proclaim itself as the Relational DB of 21st century, boasts of its CTO, Duncan Pauly.
The list is along one but I must mention of Clustrix that boasts of its CTO, Aaron Passey, with Isilon fame [Isilon brought new definition in mainstream storage clustering]. Clustrix appear to have brought appliance model in the NewSQL Database world.
I am sure there are many more to come in this space and I am sure I missed few in listing down here but given the emeregence of this new technology space, we will have to revisit this topic.
I will try to provide more detailed review of these products in next posts starting with Clustrix [see the post here]
Summary
Here is a quick comparison between three different DB technologies:

SQL-DB	No-SQL DB	NewSQL DB
Basic architecture from 70’s relational Database Model	New [2000] Architecture from likes of Google, Yahoo; designed for single large distributed database	Newer [post-2000] Architecture promises to scale for both standalone and large installation
Centralized Transaction Processing	Distributed processing	Distributed Processing
Fully ACID compliant	Breaks ACID, brings  eventual consistency model	ACID-compliant
Integrates SQL engine	No support for SQL	Full support for SQL
Limited scalability	High Scalability	High scalability, tries to break dependency on any single engine
Mature Technology; has been in the core of all popular OLTP suites	Relatively mature; suits better for SaaS model	Still Evolving; has the potential to scale for both the use-cases