Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)baseline
@ -0,0 +1,36 @@ | |||||
# Contributing | |||||
We'd love to accept your code patches! However, before we can take them, we | |||||
have to jump a couple of legal hurdles. | |||||
## Contributor License Agreements | |||||
Please fill out either the individual or corporate Contributor License | |||||
Agreement as appropriate. | |||||
* If you are an individual writing original source code and you're sure you | |||||
own the intellectual property, then sign an [individual CLA](https://developers.google.com/open-source/cla/individual). | |||||
* If you work for a company that wants to allow you to contribute your work, | |||||
then sign a [corporate CLA](https://developers.google.com/open-source/cla/corporate). | |||||
Follow either of the two links above to access the appropriate CLA and | |||||
instructions for how to sign and return it. | |||||
## Submitting a Patch | |||||
1. Sign the contributors license agreement above. | |||||
2. Decide which code you want to submit. A submission should be a set of changes | |||||
that addresses one issue in the [issue tracker](https://github.com/google/leveldb/issues). | |||||
Please don't mix more than one logical change per submission, because it makes | |||||
the history hard to follow. If you want to make a change | |||||
(e.g. add a sample or feature) that doesn't have a corresponding issue in the | |||||
issue tracker, please create one. | |||||
3. **Submitting**: When you are ready to submit, send us a Pull Request. Be | |||||
sure to include the issue number you fixed and the name you used to sign | |||||
the CLA. | |||||
## Writing Code ## | |||||
If your contribution contains code, please make sure that it follows | |||||
[the style guide](http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml). | |||||
Otherwise we will have to ask you to make changes, and that's no fun for anyone. |
@ -0,0 +1,138 @@ | |||||
**LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.** | |||||
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com) | |||||
# Features | |||||
* Keys and values are arbitrary byte arrays. | |||||
* Data is stored sorted by key. | |||||
* Callers can provide a custom comparison function to override the sort order. | |||||
* The basic operations are `Put(key,value)`, `Get(key)`, `Delete(key)`. | |||||
* Multiple changes can be made in one atomic batch. | |||||
* Users can create a transient snapshot to get a consistent view of data. | |||||
* Forward and backward iteration is supported over the data. | |||||
* Data is automatically compressed using the [Snappy compression library](http://code.google.com/p/snappy). | |||||
* External activity (file system operations etc.) is relayed through a virtual interface so users can customize the operating system interactions. | |||||
* [Detailed documentation](http://htmlpreview.github.io/?https://github.com/google/leveldb/blob/master/doc/index.html) about how to use the library is included with the source code. | |||||
# Limitations | |||||
* This is not a SQL database. It does not have a relational data model, it does not support SQL queries, and it has no support for indexes. | |||||
* Only a single process (possibly multi-threaded) can access a particular database at a time. | |||||
* There is no client-server support builtin to the library. An application that needs such support will have to wrap their own server around the library. | |||||
# Performance | |||||
Here is a performance report (with explanations) from the run of the | |||||
included db_bench program. The results are somewhat noisy, but should | |||||
be enough to get a ballpark performance estimate. | |||||
## Setup | |||||
We use a database with a million entries. Each entry has a 16 byte | |||||
key, and a 100 byte value. Values used by the benchmark compress to | |||||
about half their original size. | |||||
LevelDB: version 1.1 | |||||
Date: Sun May 1 12:11:26 2011 | |||||
CPU: 4 x Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz | |||||
CPUCache: 4096 KB | |||||
Keys: 16 bytes each | |||||
Values: 100 bytes each (50 bytes after compression) | |||||
Entries: 1000000 | |||||
Raw Size: 110.6 MB (estimated) | |||||
File Size: 62.9 MB (estimated) | |||||
## Write performance | |||||
The "fill" benchmarks create a brand new database, in either | |||||
sequential, or random order. The "fillsync" benchmark flushes data | |||||
from the operating system to the disk after every operation; the other | |||||
write operations leave the data sitting in the operating system buffer | |||||
cache for a while. The "overwrite" benchmark does random writes that | |||||
update existing keys in the database. | |||||
fillseq : 1.765 micros/op; 62.7 MB/s | |||||
fillsync : 268.409 micros/op; 0.4 MB/s (10000 ops) | |||||
fillrandom : 2.460 micros/op; 45.0 MB/s | |||||
overwrite : 2.380 micros/op; 46.5 MB/s | |||||
Each "op" above corresponds to a write of a single key/value pair. | |||||
I.e., a random write benchmark goes at approximately 400,000 writes per second. | |||||
Each "fillsync" operation costs much less (0.3 millisecond) | |||||
than a disk seek (typically 10 milliseconds). We suspect that this is | |||||
because the hard disk itself is buffering the update in its memory and | |||||
responding before the data has been written to the platter. This may | |||||
or may not be safe based on whether or not the hard disk has enough | |||||
power to save its memory in the event of a power failure. | |||||
## Read performance | |||||
We list the performance of reading sequentially in both the forward | |||||
and reverse direction, and also the performance of a random lookup. | |||||
Note that the database created by the benchmark is quite small. | |||||
Therefore the report characterizes the performance of leveldb when the | |||||
working set fits in memory. The cost of reading a piece of data that | |||||
is not present in the operating system buffer cache will be dominated | |||||
by the one or two disk seeks needed to fetch the data from disk. | |||||
Write performance will be mostly unaffected by whether or not the | |||||
working set fits in memory. | |||||
readrandom : 16.677 micros/op; (approximately 60,000 reads per second) | |||||
readseq : 0.476 micros/op; 232.3 MB/s | |||||
readreverse : 0.724 micros/op; 152.9 MB/s | |||||
LevelDB compacts its underlying storage data in the background to | |||||
improve read performance. The results listed above were done | |||||
immediately after a lot of random writes. The results after | |||||
compactions (which are usually triggered automatically) are better. | |||||
readrandom : 11.602 micros/op; (approximately 85,000 reads per second) | |||||
readseq : 0.423 micros/op; 261.8 MB/s | |||||
readreverse : 0.663 micros/op; 166.9 MB/s | |||||
Some of the high cost of reads comes from repeated decompression of blocks | |||||
read from disk. If we supply enough cache to the leveldb so it can hold the | |||||
uncompressed blocks in memory, the read performance improves again: | |||||
readrandom : 9.775 micros/op; (approximately 100,000 reads per second before compaction) | |||||
readrandom : 5.215 micros/op; (approximately 190,000 reads per second after compaction) | |||||
## Repository contents | |||||
See doc/index.html for more explanation. See doc/impl.html for a brief overview of the implementation. | |||||
The public interface is in include/*.h. Callers should not include or | |||||
rely on the details of any other header files in this package. Those | |||||
internal APIs may be changed without warning. | |||||
Guide to header files: | |||||
* **include/db.h**: Main interface to the DB: Start here | |||||
* **include/options.h**: Control over the behavior of an entire database, | |||||
and also control over the behavior of individual reads and writes. | |||||
* **include/comparator.h**: Abstraction for user-specified comparison function. | |||||
If you want just bytewise comparison of keys, you can use the default | |||||
comparator, but clients can write their own comparator implementations if they | |||||
want custom ordering (e.g. to handle different character encodings, etc.) | |||||
* **include/iterator.h**: Interface for iterating over data. You can get | |||||
an iterator from a DB object. | |||||
* **include/write_batch.h**: Interface for atomically applying multiple | |||||
updates to a database. | |||||
* **include/slice.h**: A simple module for maintaining a pointer and a | |||||
length into some other byte array. | |||||
* **include/status.h**: Status is returned from many of the public interfaces | |||||
and is used to report success and various kinds of errors. | |||||
* **include/env.h**: | |||||
Abstraction of the OS environment. A posix implementation of this interface is | |||||
in util/env_posix.cc | |||||
* **include/table.h, include/table_builder.h**: Lower-level modules that most | |||||
clients probably won't use directly |
@ -0,0 +1,225 @@ | |||||
// Copyright (c) 2012 The LevelDB Authors. All rights reserved. | |||||
// Use of this source code is governed by a BSD-style license that can be | |||||
// found in the LICENSE file. See the AUTHORS file for names of contributors. | |||||
#include <stdio.h> | |||||
#include "db/dbformat.h" | |||||
#include "db/filename.h" | |||||
#include "db/log_reader.h" | |||||
#include "db/version_edit.h" | |||||
#include "db/write_batch_internal.h" | |||||
#include "leveldb/env.h" | |||||
#include "leveldb/iterator.h" | |||||
#include "leveldb/options.h" | |||||
#include "leveldb/status.h" | |||||
#include "leveldb/table.h" | |||||
#include "leveldb/write_batch.h" | |||||
#include "util/logging.h" | |||||
namespace leveldb { | |||||
namespace { | |||||
bool GuessType(const std::string& fname, FileType* type) { | |||||
size_t pos = fname.rfind('/'); | |||||
std::string basename; | |||||
if (pos == std::string::npos) { | |||||
basename = fname; | |||||
} else { | |||||
basename = std::string(fname.data() + pos + 1, fname.size() - pos - 1); | |||||
} | |||||
uint64_t ignored; | |||||
return ParseFileName(basename, &ignored, type); | |||||
} | |||||
// Notified when log reader encounters corruption. | |||||
class CorruptionReporter : public log::Reader::Reporter { | |||||
public: | |||||
WritableFile* dst_; | |||||
virtual void Corruption(size_t bytes, const Status& status) { | |||||
std::string r = "corruption: "; | |||||
AppendNumberTo(&r, bytes); | |||||
r += " bytes; "; | |||||
r += status.ToString(); | |||||
r.push_back('\n'); | |||||
dst_->Append(r); | |||||
} | |||||
}; | |||||
// Print contents of a log file. (*func)() is called on every record. | |||||
Status PrintLogContents(Env* env, const std::string& fname, | |||||
void (*func)(uint64_t, Slice, WritableFile*), | |||||
WritableFile* dst) { | |||||
SequentialFile* file; | |||||
Status s = env->NewSequentialFile(fname, &file); | |||||
if (!s.ok()) { | |||||
return s; | |||||
} | |||||
CorruptionReporter reporter; | |||||
reporter.dst_ = dst; | |||||
log::Reader reader(file, &reporter, true, 0); | |||||
Slice record; | |||||
std::string scratch; | |||||
while (reader.ReadRecord(&record, &scratch)) { | |||||
(*func)(reader.LastRecordOffset(), record, dst); | |||||
} | |||||
delete file; | |||||
return Status::OK(); | |||||
} | |||||
// Called on every item found in a WriteBatch. | |||||
class WriteBatchItemPrinter : public WriteBatch::Handler { | |||||
public: | |||||
WritableFile* dst_; | |||||
virtual void Put(const Slice& key, const Slice& value) { | |||||
std::string r = " put '"; | |||||
AppendEscapedStringTo(&r, key); | |||||
r += "' '"; | |||||
AppendEscapedStringTo(&r, value); | |||||
r += "'\n"; | |||||
dst_->Append(r); | |||||
} | |||||
virtual void Delete(const Slice& key) { | |||||
std::string r = " del '"; | |||||
AppendEscapedStringTo(&r, key); | |||||
r += "'\n"; | |||||
dst_->Append(r); | |||||
} | |||||
}; | |||||
// Called on every log record (each one of which is a WriteBatch) | |||||
// found in a kLogFile. | |||||
static void WriteBatchPrinter(uint64_t pos, Slice record, WritableFile* dst) { | |||||
std::string r = "--- offset "; | |||||
AppendNumberTo(&r, pos); | |||||
r += "; "; | |||||
if (record.size() < 12) { | |||||
r += "log record length "; | |||||
AppendNumberTo(&r, record.size()); | |||||
r += " is too small\n"; | |||||
dst->Append(r); | |||||
return; | |||||
} | |||||
WriteBatch batch; | |||||
WriteBatchInternal::SetContents(&batch, record); | |||||
r += "sequence "; | |||||
AppendNumberTo(&r, WriteBatchInternal::Sequence(&batch)); | |||||
r.push_back('\n'); | |||||
dst->Append(r); | |||||
WriteBatchItemPrinter batch_item_printer; | |||||
batch_item_printer.dst_ = dst; | |||||
Status s = batch.Iterate(&batch_item_printer); | |||||
if (!s.ok()) { | |||||
dst->Append(" error: " + s.ToString() + "\n"); | |||||
} | |||||
} | |||||
Status DumpLog(Env* env, const std::string& fname, WritableFile* dst) { | |||||
return PrintLogContents(env, fname, WriteBatchPrinter, dst); | |||||
} | |||||
// Called on every log record (each one of which is a WriteBatch) | |||||
// found in a kDescriptorFile. | |||||
static void VersionEditPrinter(uint64_t pos, Slice record, WritableFile* dst) { | |||||
std::string r = "--- offset "; | |||||
AppendNumberTo(&r, pos); | |||||
r += "; "; | |||||
VersionEdit edit; | |||||
Status s = edit.DecodeFrom(record); | |||||
if (!s.ok()) { | |||||
r += s.ToString(); | |||||
r.push_back('\n'); | |||||
} else { | |||||
r += edit.DebugString(); | |||||
} | |||||
dst->Append(r); | |||||
} | |||||
Status DumpDescriptor(Env* env, const std::string& fname, WritableFile* dst) { | |||||
return PrintLogContents(env, fname, VersionEditPrinter, dst); | |||||
} | |||||
Status DumpTable(Env* env, const std::string& fname, WritableFile* dst) { | |||||
uint64_t file_size; | |||||
RandomAccessFile* file = NULL; | |||||
Table* table = NULL; | |||||
Status s = env->GetFileSize(fname, &file_size); | |||||
if (s.ok()) { | |||||
s = env->NewRandomAccessFile(fname, &file); | |||||
} | |||||
if (s.ok()) { | |||||
// We use the default comparator, which may or may not match the | |||||
// comparator used in this database. However this should not cause | |||||
// problems since we only use Table operations that do not require | |||||
// any comparisons. In particular, we do not call Seek or Prev. | |||||
s = Table::Open(Options(), file, file_size, &table); | |||||
} | |||||
if (!s.ok()) { | |||||
delete table; | |||||
delete file; | |||||
return s; | |||||
} | |||||
ReadOptions ro; | |||||
ro.fill_cache = false; | |||||
Iterator* iter = table->NewIterator(ro); | |||||
std::string r; | |||||
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) { | |||||
r.clear(); | |||||
ParsedInternalKey key; | |||||
if (!ParseInternalKey(iter->key(), &key)) { | |||||
r = "badkey '"; | |||||
AppendEscapedStringTo(&r, iter->key()); | |||||
r += "' => '"; | |||||
AppendEscapedStringTo(&r, iter->value()); | |||||
r += "'\n"; | |||||
dst->Append(r); | |||||
} else { | |||||
r = "'"; | |||||
AppendEscapedStringTo(&r, key.user_key); | |||||
r += "' @ "; | |||||
AppendNumberTo(&r, key.sequence); | |||||
r += " : "; | |||||
if (key.type == kTypeDeletion) { | |||||
r += "del"; | |||||
} else if (key.type == kTypeValue) { | |||||
r += "val"; | |||||
} else { | |||||
AppendNumberTo(&r, key.type); | |||||
} | |||||
r += " => '"; | |||||
AppendEscapedStringTo(&r, iter->value()); | |||||
r += "'\n"; | |||||
dst->Append(r); | |||||
} | |||||
} | |||||
s = iter->status(); | |||||
if (!s.ok()) { | |||||
dst->Append("iterator error: " + s.ToString() + "\n"); | |||||
} | |||||
delete iter; | |||||
delete table; | |||||
delete file; | |||||
return Status::OK(); | |||||
} | |||||
} // namespace | |||||
Status DumpFile(Env* env, const std::string& fname, WritableFile* dst) { | |||||
FileType ftype; | |||||
if (!GuessType(fname, &ftype)) { | |||||
return Status::InvalidArgument(fname + ": unknown file type"); | |||||
} | |||||
switch (ftype) { | |||||
case kLogFile: return DumpLog(env, fname, dst); | |||||
case kDescriptorFile: return DumpDescriptor(env, fname, dst); | |||||
case kTableFile: return DumpTable(env, fname, dst); | |||||
default: | |||||
break; | |||||
} | |||||
return Status::InvalidArgument(fname + ": not a dump-able file type"); | |||||
} | |||||
} // namespace leveldb |
@ -0,0 +1,25 @@ | |||||
// Copyright (c) 2014 The LevelDB Authors. All rights reserved. | |||||
// Use of this source code is governed by a BSD-style license that can be | |||||
// found in the LICENSE file. See the AUTHORS file for names of contributors. | |||||
#ifndef STORAGE_LEVELDB_INCLUDE_DUMPFILE_H_ | |||||
#define STORAGE_LEVELDB_INCLUDE_DUMPFILE_H_ | |||||
#include <string> | |||||
#include "leveldb/env.h" | |||||
#include "leveldb/status.h" | |||||
namespace leveldb { | |||||
// Dump the contents of the file named by fname in text format to | |||||
// *dst. Makes a sequence of dst->Append() calls; each call is passed | |||||
// the newline-terminated text corresponding to a single item found | |||||
// in the file. | |||||
// | |||||
// Returns a non-OK result if fname does not name a leveldb storage | |||||
// file, or if the file cannot be read. | |||||
Status DumpFile(Env* env, const std::string& fname, WritableFile* dst); | |||||
} // namespace leveldb | |||||
#endif // STORAGE_LEVELDB_INCLUDE_DUMPFILE_H_ |
@ -0,0 +1,54 @@ | |||||
// Copyright (c) 2011 The LevelDB Authors. All rights reserved. | |||||
// Use of this source code is governed by a BSD-style license that can be | |||||
// found in the LICENSE file. See the AUTHORS file for names of contributors. | |||||
#include "util/hash.h" | |||||
#include "util/testharness.h" | |||||
namespace leveldb { | |||||
class HASH { }; | |||||
TEST(HASH, SignedUnsignedIssue) { | |||||
const unsigned char data1[1] = {0x62}; | |||||
const unsigned char data2[2] = {0xc3, 0x97}; | |||||
const unsigned char data3[3] = {0xe2, 0x99, 0xa5}; | |||||
const unsigned char data4[4] = {0xe1, 0x80, 0xb9, 0x32}; | |||||
const unsigned char data5[48] = { | |||||
0x01, 0xc0, 0x00, 0x00, | |||||
0x00, 0x00, 0x00, 0x00, | |||||
0x00, 0x00, 0x00, 0x00, | |||||
0x00, 0x00, 0x00, 0x00, | |||||
0x14, 0x00, 0x00, 0x00, | |||||
0x00, 0x00, 0x04, 0x00, | |||||
0x00, 0x00, 0x00, 0x14, | |||||
0x00, 0x00, 0x00, 0x18, | |||||
0x28, 0x00, 0x00, 0x00, | |||||
0x00, 0x00, 0x00, 0x00, | |||||
0x02, 0x00, 0x00, 0x00, | |||||
0x00, 0x00, 0x00, 0x00, | |||||
}; | |||||
ASSERT_EQ(Hash(0, 0, 0xbc9f1d34), 0xbc9f1d34); | |||||
ASSERT_EQ( | |||||
Hash(reinterpret_cast<const char*>(data1), sizeof(data1), 0xbc9f1d34), | |||||
0xef1345c4); | |||||
ASSERT_EQ( | |||||
Hash(reinterpret_cast<const char*>(data2), sizeof(data2), 0xbc9f1d34), | |||||
0x5b663814); | |||||
ASSERT_EQ( | |||||
Hash(reinterpret_cast<const char*>(data3), sizeof(data3), 0xbc9f1d34), | |||||
0x323c078f); | |||||
ASSERT_EQ( | |||||
Hash(reinterpret_cast<const char*>(data4), sizeof(data4), 0xbc9f1d34), | |||||
0xed21633a); | |||||
ASSERT_EQ( | |||||
Hash(reinterpret_cast<const char*>(data5), sizeof(data5), 0x12345678), | |||||
0xf333dabb); | |||||
} | |||||
} // namespace leveldb | |||||
int main(int argc, char** argv) { | |||||
return leveldb::test::RunAllTests(); | |||||
} |