3
1 INTRODUCTION
In view of the fast development of innovative IT-technologies, NoSQL technology is increasingly utilized in
big data and real-time web application in recent years. Because NoSQL stores allow for a more agile
development process and execution, they can replace traditional relational database management
systems (RDBMS) in a large number of industrial application fields. NoSQL technology significantly improves
both database scalability and usability by softening RDBMS features, such as consistency and relational
model.
In this report, bankmark reports on a large series of benchmark experiments to compare publicly available
NoSQL store products with SequoiaDB in in different workload scenarios. For this purpose, the bankmark
team used the Yahoo Cloud Serving Benchmark (YCSB) suite as testing platform. The bankmark team used
preset settings for all systems wherever possible and only adapted settings that caused major performance
bottlenecks. For all databases official documentation as well as information from other publicly available
sources was utilized. All major adaptations are documented in this report, a full report is available on
request that contains all configuration settings.
In the present report, bankmark focused on the performance of each database for different use cases and
ensured a maximum of comparability between different results. One aim of the experiments was to get
out-of-the-box performance. On the other hand, the distributed environment required some amount of
optimization to get the systems running in a clustered environment. All systems were configured for
clustered use and some optimization regarding partitioning / sharding took place to get competitive results
for all systems.
All tests were implemented by the bankmark team. All important details concerning the physical
environment and the testing settings are specified in this testing report, a full report ensuring repeatability
of all experiments is available on request.
2 RESULTS SUMMARY
In our experiments, three systems were compared, SequoiaDB
, Cassandra
, and MongoDB
. All systems
were tested on a 10 node cluster in an in-memory (raw data size ¼ of total RAM) or mostly-in-memory (raw
data size ½ of total RAM) setup. We used the widely accepted YCSB suite as benchmarking platform. In all
experiments, all data was replicated 3 times for fault tolerance. The workloads tested were all using skewed
workloads (with Zipfian or latest distribution). The detailed configuration can be seen below and in an
extended report that is available on request.
The results do not show a clear winner across all experiments. Our tests with the mostly-in-memory setup
show that Cassandra uses most memory and thus has to perform much more disk I/O in read heavy
workloads, leading to a highly decreased performance. In this case, SequoiaDB outperforms the other
systems in most cases, except for write heavy workloads, which are dominated by Cassandra. In a pure in
memory setup (raw data size is ¼ of total RAM), the performance of Cassandra and SequoiaDB with
SequoiaDB being faster for read requests and Cassandra being faster for write request. MongoDB is slowest
in almost all test cases.
http://www.sequoiadb.com/en/
http://cassandra.apache.org/
http://www.mongodb.org/
