Improved benchmark, fixed bugs and SQLite parameters.
- Based on suggestions on the sqlite-users mailing list,
we removed the superfluous index on the primary key
for SQLite's benchmarks, and turned write-ahead logging
("WAL") on. This led to performance improvements for SQLite.
- Based on a suggestion by Florian Weimer on the leveldb
mailing list, we disabled hard drive write-caching via
hdparm when testing synchronous writes. This led to
performance losses for LevelDB and Kyoto TreeDB.
- Fixed a mistake in 2.A.->Random where the bar sizes
were switched for Kyoto TreeDB and SQLite.
git-svn-id: https://leveldb.googlecode.com/svn/trunk@45 62dab493-f737-651d-591e-8d6aee1b9529
<p>In order to test LevelDB's performance, we benchmark it against other well-established database implementations. We compare LevelDB (revision 39) against <ahref="http://www.sqlite.org/">SQLite3</a> (version 3.7.6.3) and <ahref="http://fallabs.com/kyotocabinet/spex.html">Kyoto Cabinet's</a> (version 1.2.67) TreeDB (a B+Tree based key-value store). We would like to acknowledge Scott Hess and Mikio Hirabayashi for their suggestions and contributions to the SQLite3 and Kyoto Cabinet benchmarks, respectively.</p>
<p>Benchmarks were all performed on a six-core Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, with 12288 KB of total L3 cache and 12 GB of DDR3 RAM at 1333 MHz. (Note that LevelDB uses at most two CPUs since the benchmarks are single threaded: one to run the benchmark, and one for background compactions.) We ran the benchmarks on two machines (with identical processors), one with an Ext3 file system and one with an Ext4 file system. The machine with the Ext3 file system has a SATA Hitachi HDS721050CLA362 hard drive. The machine with the Ext4 file system has a SATA Samsung HD502HJ hard drive. Both hard drives spin at 7200 RPM. The numbers reported below are the median of three measurements.</p>
<p>Benchmarks were all performed on a six-core Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, with 12288 KB of total L3 cache and 12 GB of DDR3 RAM at 1333 MHz. (Note that LevelDB uses at most two CPUs since the benchmarks are single threaded: one to run the benchmark, and one for background compactions.) We ran the benchmarks on two machines (with identical processors), one with an Ext3 file system and one with an Ext4 file system. The machine with the Ext3 file system has a SATA Hitachi HDS721050CLA362 hard drive. The machine with the Ext4 file system has a SATA Samsung HD502HJ hard drive. Both hard drives spin at 7200 RPM and have hard drive write-caching enabled (using `hdparm -W 1 [device]`). The numbers reported below are the median of three measurements.</p>
<h4>Benchmark Source Code</h4>
<p>We wrote benchmark tools for SQLite and Kyoto TreeDB based on LevelDB's <spanclass="code">db_bench</span>. The code for each of the benchmarks resides here:</p>
@ -97,9 +97,9 @@ div.bsql {
<h4>Custom Build Specifications</h4>
<ul>
<li>LevelDB: LevelDB was compiled with the <ahref="http://code.google.com/p/google-perftools">tcmalloc</a> library and the <ahref="http://code.google.com/p/snappy/">Snappy</a> compression library. Assertions were disabled.</li>
<li>TreeDB: TreeDB was compiled using the <ahref="http://www.oberhumer.com/opensource/lzo/">LZO</a> compression library. Furthermore, we enabled the TSMALL and TLINEAR options when opening the database in order to reduce the footprint of each record.</li>
<li>SQLite: We tuned SQLite's performance, by setting its locking mode to exclusive. We left SQLite's <ahref="http://www.sqlite.org/draft/wal.html">write-ahead logging</a> disabled since that is the default configuration. (Enabling write-ahead-logging improves SQLite's write performance by roughly 30%, but the character of the comparisons below does not change significantly.)</li>
<li>LevelDB: LevelDB was compiled with the <ahref="http://code.google.com/p/google-perftools">tcmalloc</a> library and the <ahref="http://code.google.com/p/snappy/">Snappy</a> compression library (revision 33). Assertions were disabled.</li>
<li>TreeDB: TreeDB was compiled using the <ahref="http://www.oberhumer.com/opensource/lzo/">LZO</a> compression library (version 2.03). Furthermore, we enabled the TSMALL and TLINEAR options when opening the database in order to reduce the footprint of each record.</li>
<li>SQLite: We tuned SQLite's performance, by setting its locking mode to exclusive. We also enabled SQLite's <ahref="http://www.sqlite.org/draft/wal.html">write-ahead logging</a>.</li>
</ul>
<h2>1. Baseline Performance</h2>
@ -130,8 +130,8 @@ parameters are varied. For the baseline:
<p>LevelDB outperforms both SQLite3 and TreeDB in sequential and random write operations and sequential read operations. Kyoto Cabinet has the fastest random read operations.</p>
@ -178,26 +178,26 @@ parameters are varied. For the baseline:
<p>LevelDB doesn't perform as well with large values of 100,000 bytes each. This is because LevelDB writes keys and values at least twice: first time to the transaction log, and second time (during a compaction) to a sorted file.
With larger values, LevelDB's per-operation efficiency is swamped by the
@ -211,9 +211,9 @@ cost of extra copies of large values.
@ -453,7 +451,7 @@ performance may very well be better with compression if it allows more
of the working set to fit in memory.</p>
<h2>Note about Ext4 Filesystems</h2>
<p>The preceding numbers are for an ext3 file system. Synchronous writes are much slower under <ahref="http://en.wikipedia.org/wiki/Ext4">ext4</a> (LevelDB drops to ~34 writes / second, TreeDB drops to ~5 writes / second; SQLite3 drops to ~24 writes / second) due to ext4's different handling of <spanclass="code">fsync</span> / <spanclass="code">msync</span> calls. Even LevelDB's asynchronous write performance drops somewhat since it spreads its storage across multiple files and issues <spanclass="code">fsync</span> calls when switching to a new file.</p>
<p>The preceding numbers are for an ext3 file system. Synchronous writes are much slower under <ahref="http://en.wikipedia.org/wiki/Ext4">ext4</a> (LevelDB drops to ~31 writes / second and TreeDB drops to ~5 writes / second; SQLite3's synchronous writes do not noticeably drop) due to ext4's different handling of <spanclass="code">fsync</span> / <spanclass="code">msync</span> calls. Even LevelDB's asynchronous write performance drops somewhat since it spreads its storage across multiple files and issues <spanclass="code">fsync</span> calls when switching to a new file.</p>
<h2>Acknowledgements</h2>
<p>Jeff Dean and Sanjay Ghemawat wrote LevelDB. Kevin Tseng wrote and compiled these benchmarks. Mikio Hirabayashi, Scott Hess, and Gabor Cselle provided help and advice.</p>
gabor@google.com
13 years ago