作者: 谢瑞阳 10225101483 徐翔宇 10225101535
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

549 lines
21 KiB

  1. <!DOCTYPE html>
  2. <html>
  3. <head>
  4. <link rel="stylesheet" type="text/css" href="doc.css" />
  5. <title>Leveldb</title>
  6. </head>
  7. <body>
  8. <h1>Leveldb</h1>
  9. <address>Jeff Dean, Sanjay Ghemawat</address>
  10. <p>
  11. The <code>leveldb</code> library provides a persistent key value store. Keys and
  12. values are arbitrary byte arrays. The keys are ordered within the key
  13. value store according to a user-specified comparator function.
  14. <p>
  15. <h1>Opening A Database</h1>
  16. <p>
  17. A <code>leveldb</code> database has a name which corresponds to a file system
  18. directory. All of the contents of database are stored in this
  19. directory. The following example shows how to open a database,
  20. creating it if necessary:
  21. <p>
  22. <pre>
  23. #include &lt;cassert&gt;
  24. #include "leveldb/db.h"
  25. leveldb::DB* db;
  26. leveldb::Options options;
  27. options.create_if_missing = true;
  28. leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &amp;db);
  29. assert(status.ok());
  30. ...
  31. </pre>
  32. If you want to raise an error if the database already exists, add
  33. the following line before the <code>leveldb::DB::Open</code> call:
  34. <pre>
  35. options.error_if_exists = true;
  36. </pre>
  37. <h1>Status</h1>
  38. <p>
  39. You may have noticed the <code>leveldb::Status</code> type above. Values of this
  40. type are returned by most functions in <code>leveldb</code> that may encounter an
  41. error. You can check if such a result is ok, and also print an
  42. associated error message:
  43. <p>
  44. <pre>
  45. leveldb::Status s = ...;
  46. if (!s.ok()) cerr &lt;&lt; s.ToString() &lt;&lt; endl;
  47. </pre>
  48. <h1>Closing A Database</h1>
  49. <p>
  50. When you are done with a database, just delete the database object.
  51. Example:
  52. <p>
  53. <pre>
  54. ... open the db as described above ...
  55. ... do something with db ...
  56. delete db;
  57. </pre>
  58. <h1>Reads And Writes</h1>
  59. <p>
  60. The database provides <code>Put</code>, <code>Delete</code>, and <code>Get</code> methods to
  61. modify/query the database. For example, the following code
  62. moves the value stored under key1 to key2.
  63. <pre>
  64. std::string value;
  65. leveldb::Status s = db-&gt;Get(leveldb::ReadOptions(), key1, &amp;value);
  66. if (s.ok()) s = db-&gt;Put(leveldb::WriteOptions(), key2, value);
  67. if (s.ok()) s = db-&gt;Delete(leveldb::WriteOptions(), key1);
  68. </pre>
  69. <h1>Atomic Updates</h1>
  70. <p>
  71. Note that if the process dies after the Put of key2 but before the
  72. delete of key1, the same value may be left stored under multiple keys.
  73. Such problems can be avoided by using the <code>WriteBatch</code> class to
  74. atomically apply a set of updates:
  75. <p>
  76. <pre>
  77. #include "leveldb/write_batch.h"
  78. ...
  79. std::string value;
  80. leveldb::Status s = db-&gt;Get(leveldb::ReadOptions(), key1, &amp;value);
  81. if (s.ok()) {
  82. leveldb::WriteBatch batch;
  83. batch.Delete(key1);
  84. batch.Put(key2, value);
  85. s = db-&gt;Write(leveldb::WriteOptions(), &amp;batch);
  86. }
  87. </pre>
  88. The <code>WriteBatch</code> holds a sequence of edits to be made to the database,
  89. and these edits within the batch are applied in order. Note that we
  90. called <code>Delete</code> before <code>Put</code> so that if <code>key1</code> is identical to <code>key2</code>,
  91. we do not end up erroneously dropping the value entirely.
  92. <p>
  93. Apart from its atomicity benefits, <code>WriteBatch</code> may also be used to
  94. speed up bulk updates by placing lots of individual mutations into the
  95. same batch.
  96. <h1>Synchronous Writes</h1>
  97. By default, each write to <code>leveldb</code> is asynchronous: it
  98. returns after pushing the write from the process into the operating
  99. system. The transfer from operating system memory to the underlying
  100. persistent storage happens asynchronously. The <code>sync</code> flag
  101. can be turned on for a particular write to make the write operation
  102. not return until the data being written has been pushed all the way to
  103. persistent storage. (On Posix systems, this is implemented by calling
  104. either <code>fsync(...)</code> or <code>fdatasync(...)</code> or
  105. <code>msync(..., MS_SYNC)</code> before the write operation returns.)
  106. <pre>
  107. leveldb::WriteOptions write_options;
  108. write_options.sync = true;
  109. db-&gt;Put(write_options, ...);
  110. </pre>
  111. Asynchronous writes are often more than a thousand times as fast as
  112. synchronous writes. The downside of asynchronous writes is that a
  113. crash of the machine may cause the last few updates to be lost. Note
  114. that a crash of just the writing process (i.e., not a reboot) will not
  115. cause any loss since even when <code>sync</code> is false, an update
  116. is pushed from the process memory into the operating system before it
  117. is considered done.
  118. <p>
  119. Asynchronous writes can often be used safely. For example, when
  120. loading a large amount of data into the database you can handle lost
  121. updates by restarting the bulk load after a crash. A hybrid scheme is
  122. also possible where every Nth write is synchronous, and in the event
  123. of a crash, the bulk load is restarted just after the last synchronous
  124. write finished by the previous run. (The synchronous write can update
  125. a marker that describes where to restart on a crash.)
  126. <p>
  127. <code>WriteBatch</code> provides an alternative to asynchronous writes.
  128. Multiple updates may be placed in the same <code>WriteBatch</code> and
  129. applied together using a synchronous write (i.e.,
  130. <code>write_options.sync</code> is set to true). The extra cost of
  131. the synchronous write will be amortized across all of the writes in
  132. the batch.
  133. <p>
  134. <h1>Concurrency</h1>
  135. <p>
  136. A database may only be opened by one process at a time.
  137. The <code>leveldb</code> implementation acquires a lock from the
  138. operating system to prevent misuse. Within a single process, the
  139. same <code>leveldb::DB</code> object may be safely shared by multiple
  140. concurrent threads. I.e., different threads may write into or fetch
  141. iterators or call <code>Get</code> on the same database without any
  142. external synchronization (the leveldb implementation will
  143. automatically do the required synchronization). However other objects
  144. (like Iterator and WriteBatch) may require external synchronization.
  145. If two threads share such an object, they must protect access to it
  146. using their own locking protocol. More details are available in
  147. the public header files.
  148. <p>
  149. <h1>Iteration</h1>
  150. <p>
  151. The following example demonstrates how to print all key,value pairs
  152. in a database.
  153. <p>
  154. <pre>
  155. leveldb::Iterator* it = db-&gt;NewIterator(leveldb::ReadOptions());
  156. for (it-&gt;SeekToFirst(); it-&gt;Valid(); it-&gt;Next()) {
  157. cout &lt;&lt; it-&gt;key().ToString() &lt;&lt; ": " &lt;&lt; it-&gt;value().ToString() &lt;&lt; endl;
  158. }
  159. assert(it-&gt;status().ok()); // Check for any errors found during the scan
  160. delete it;
  161. </pre>
  162. The following variation shows how to process just the keys in the
  163. range <code>[start,limit)</code>:
  164. <p>
  165. <pre>
  166. for (it-&gt;Seek(start);
  167. it-&gt;Valid() &amp;&amp; it-&gt;key().ToString() &lt; limit;
  168. it-&gt;Next()) {
  169. ...
  170. }
  171. </pre>
  172. You can also process entries in reverse order. (Caveat: reverse
  173. iteration may be somewhat slower than forward iteration.)
  174. <p>
  175. <pre>
  176. for (it-&gt;SeekToLast(); it-&gt;Valid(); it-&gt;Prev()) {
  177. ...
  178. }
  179. </pre>
  180. <h1>Snapshots</h1>
  181. <p>
  182. Snapshots provide consistent read-only views over the entire state of
  183. the key-value store. <code>ReadOptions::snapshot</code> may be non-NULL to indicate
  184. that a read should operate on a particular version of the DB state.
  185. If <code>ReadOptions::snapshot</code> is NULL, the read will operate on an
  186. implicit snapshot of the current state.
  187. <p>
  188. Snapshots are created by the DB::GetSnapshot() method:
  189. <p>
  190. <pre>
  191. leveldb::ReadOptions options;
  192. options.snapshot = db-&gt;GetSnapshot();
  193. ... apply some updates to db ...
  194. leveldb::Iterator* iter = db-&gt;NewIterator(options);
  195. ... read using iter to view the state when the snapshot was created ...
  196. delete iter;
  197. db-&gt;ReleaseSnapshot(options.snapshot);
  198. </pre>
  199. Note that when a snapshot is no longer needed, it should be released
  200. using the DB::ReleaseSnapshot interface. This allows the
  201. implementation to get rid of state that was being maintained just to
  202. support reading as of that snapshot.
  203. <h1>Slice</h1>
  204. <p>
  205. The return value of the <code>it->key()</code> and <code>it->value()</code> calls above
  206. are instances of the <code>leveldb::Slice</code> type. <code>Slice</code> is a simple
  207. structure that contains a length and a pointer to an external byte
  208. array. Returning a <code>Slice</code> is a cheaper alternative to returning a
  209. <code>std::string</code> since we do not need to copy potentially large keys and
  210. values. In addition, <code>leveldb</code> methods do not return null-terminated
  211. C-style strings since <code>leveldb</code> keys and values are allowed to
  212. contain '\0' bytes.
  213. <p>
  214. C++ strings and null-terminated C-style strings can be easily converted
  215. to a Slice:
  216. <p>
  217. <pre>
  218. leveldb::Slice s1 = "hello";
  219. std::string str("world");
  220. leveldb::Slice s2 = str;
  221. </pre>
  222. A Slice can be easily converted back to a C++ string:
  223. <pre>
  224. std::string str = s1.ToString();
  225. assert(str == std::string("hello"));
  226. </pre>
  227. Be careful when using Slices since it is up to the caller to ensure that
  228. the external byte array into which the Slice points remains live while
  229. the Slice is in use. For example, the following is buggy:
  230. <p>
  231. <pre>
  232. leveldb::Slice slice;
  233. if (...) {
  234. std::string str = ...;
  235. slice = str;
  236. }
  237. Use(slice);
  238. </pre>
  239. When the <code>if</code> statement goes out of scope, <code>str</code> will be destroyed and the
  240. backing storage for <code>slice</code> will disappear.
  241. <p>
  242. <h1>Comparators</h1>
  243. <p>
  244. The preceding examples used the default ordering function for key,
  245. which orders bytes lexicographically. You can however supply a custom
  246. comparator when opening a database. For example, suppose each
  247. database key consists of two numbers and we should sort by the first
  248. number, breaking ties by the second number. First, define a proper
  249. subclass of <code>leveldb::Comparator</code> that expresses these rules:
  250. <p>
  251. <pre>
  252. class TwoPartComparator : public leveldb::Comparator {
  253. public:
  254. // Three-way comparison function:
  255. // if a &lt; b: negative result
  256. // if a &gt; b: positive result
  257. // else: zero result
  258. int Compare(const leveldb::Slice&amp; a, const leveldb::Slice&amp; b) const {
  259. int a1, a2, b1, b2;
  260. ParseKey(a, &amp;a1, &amp;a2);
  261. ParseKey(b, &amp;b1, &amp;b2);
  262. if (a1 &lt; b1) return -1;
  263. if (a1 &gt; b1) return +1;
  264. if (a2 &lt; b2) return -1;
  265. if (a2 &gt; b2) return +1;
  266. return 0;
  267. }
  268. // Ignore the following methods for now:
  269. const char* Name() const { return "TwoPartComparator"; }
  270. void FindShortestSeparator(std::string*, const leveldb::Slice&amp;) const { }
  271. void FindShortSuccessor(std::string*) const { }
  272. };
  273. </pre>
  274. Now create a database using this custom comparator:
  275. <p>
  276. <pre>
  277. TwoPartComparator cmp;
  278. leveldb::DB* db;
  279. leveldb::Options options;
  280. options.create_if_missing = true;
  281. options.comparator = &amp;cmp;
  282. leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &amp;db);
  283. ...
  284. </pre>
  285. <h2>Backwards compatibility</h2>
  286. <p>
  287. The result of the comparator's <code>Name</code> method is attached to the
  288. database when it is created, and is checked on every subsequent
  289. database open. If the name changes, the <code>leveldb::DB::Open</code> call will
  290. fail. Therefore, change the name if and only if the new key format
  291. and comparison function are incompatible with existing databases, and
  292. it is ok to discard the contents of all existing databases.
  293. <p>
  294. You can however still gradually evolve your key format over time with
  295. a little bit of pre-planning. For example, you could store a version
  296. number at the end of each key (one byte should suffice for most uses).
  297. When you wish to switch to a new key format (e.g., adding an optional
  298. third part to the keys processed by <code>TwoPartComparator</code>),
  299. (a) keep the same comparator name (b) increment the version number
  300. for new keys (c) change the comparator function so it uses the
  301. version numbers found in the keys to decide how to interpret them.
  302. <p>
  303. <h1>Performance</h1>
  304. <p>
  305. Performance can be tuned by changing the default values of the
  306. types defined in <code>include/leveldb/options.h</code>.
  307. <p>
  308. <h2>Block size</h2>
  309. <p>
  310. <code>leveldb</code> groups adjacent keys together into the same block and such a
  311. block is the unit of transfer to and from persistent storage. The
  312. default block size is approximately 4096 uncompressed bytes.
  313. Applications that mostly do bulk scans over the contents of the
  314. database may wish to increase this size. Applications that do a lot
  315. of point reads of small values may wish to switch to a smaller block
  316. size if performance measurements indicate an improvement. There isn't
  317. much benefit in using blocks smaller than one kilobyte, or larger than
  318. a few megabytes. Also note that compression will be more effective
  319. with larger block sizes.
  320. <p>
  321. <h2>Compression</h2>
  322. <p>
  323. Each block is individually compressed before being written to
  324. persistent storage. Compression is on by default since the default
  325. compression method is very fast, and is automatically disabled for
  326. uncompressible data. In rare cases, applications may want to disable
  327. compression entirely, but should only do so if benchmarks show a
  328. performance improvement:
  329. <p>
  330. <pre>
  331. leveldb::Options options;
  332. options.compression = leveldb::kNoCompression;
  333. ... leveldb::DB::Open(options, name, ...) ....
  334. </pre>
  335. <h2>Cache</h2>
  336. <p>
  337. The contents of the database are stored in a set of files in the
  338. filesystem and each file stores a sequence of compressed blocks. If
  339. <code>options.cache</code> is non-NULL, it is used to cache frequently used
  340. uncompressed block contents.
  341. <p>
  342. <pre>
  343. #include "leveldb/cache.h"
  344. leveldb::Options options;
  345. options.cache = leveldb::NewLRUCache(100 * 1048576); // 100MB cache
  346. leveldb::DB* db;
  347. leveldb::DB::Open(options, name, &db);
  348. ... use the db ...
  349. delete db
  350. delete options.cache;
  351. </pre>
  352. Note that the cache holds uncompressed data, and therefore it should
  353. be sized according to application level data sizes, without any
  354. reduction from compression. (Caching of compressed blocks is left to
  355. the operating system buffer cache, or any custom <code>Env</code>
  356. implementation provided by the client.)
  357. <p>
  358. When performing a bulk read, the application may wish to disable
  359. caching so that the data processed by the bulk read does not end up
  360. displacing most of the cached contents. A per-iterator option can be
  361. used to achieve this:
  362. <p>
  363. <pre>
  364. leveldb::ReadOptions options;
  365. options.fill_cache = false;
  366. leveldb::Iterator* it = db-&gt;NewIterator(options);
  367. for (it-&gt;SeekToFirst(); it-&gt;Valid(); it-&gt;Next()) {
  368. ...
  369. }
  370. </pre>
  371. <h2>Key Layout</h2>
  372. <p>
  373. Note that the unit of disk transfer and caching is a block. Adjacent
  374. keys (according to the database sort order) will usually be placed in
  375. the same block. Therefore the application can improve its performance
  376. by placing keys that are accessed together near each other and placing
  377. infrequently used keys in a separate region of the key space.
  378. <p>
  379. For example, suppose we are implementing a simple file system on top
  380. of <code>leveldb</code>. The types of entries we might wish to store are:
  381. <p>
  382. <pre>
  383. filename -&gt; permission-bits, length, list of file_block_ids
  384. file_block_id -&gt; data
  385. </pre>
  386. We might want to prefix <code>filename</code> keys with one letter (say '/') and the
  387. <code>file_block_id</code> keys with a different letter (say '0') so that scans
  388. over just the metadata do not force us to fetch and cache bulky file
  389. contents.
  390. <p>
  391. <h2>Filters</h2>
  392. <p>
  393. Because of the way <code>leveldb</code> data is organized on disk,
  394. a single <code>Get()</code> call may involve multiple reads from disk.
  395. The optional <code>FilterPolicy</code> mechanism can be used to reduce
  396. the number of disk reads substantially.
  397. <pre>
  398. leveldb::Options options;
  399. options.filter_policy = NewBloomFilterPolicy(10);
  400. leveldb::DB* db;
  401. leveldb::DB::Open(options, "/tmp/testdb", &amp;db);
  402. ... use the database ...
  403. delete db;
  404. delete options.filter_policy;
  405. </pre>
  406. The preceding code associates a
  407. <a href="http://en.wikipedia.org/wiki/Bloom_filter">Bloom filter</a>
  408. based filtering policy with the database. Bloom filter based
  409. filtering relies on keeping some number of bits of data in memory per
  410. key (in this case 10 bits per key since that is the argument we passed
  411. to NewBloomFilterPolicy). This filter will reduce the number of unnecessary
  412. disk reads needed for <code>Get()</code> calls by a factor of
  413. approximately a 100. Increasing the bits per key will lead to a
  414. larger reduction at the cost of more memory usage. We recommend that
  415. applications whose working set does not fit in memory and that do a
  416. lot of random reads set a filter policy.
  417. <p>
  418. If you are using a custom comparator, you should ensure that the filter
  419. policy you are using is compatible with your comparator. For example,
  420. consider a comparator that ignores trailing spaces when comparing keys.
  421. <code>NewBloomFilterPolicy</code> must not be used with such a comparator.
  422. Instead, the application should provide a custom filter policy that
  423. also ignores trailing spaces. For example:
  424. <pre>
  425. class CustomFilterPolicy : public leveldb::FilterPolicy {
  426. private:
  427. FilterPolicy* builtin_policy_;
  428. public:
  429. CustomFilterPolicy() : builtin_policy_(NewBloomFilterPolicy(10)) { }
  430. ~CustomFilterPolicy() { delete builtin_policy_; }
  431. const char* Name() const { return "IgnoreTrailingSpacesFilter"; }
  432. void CreateFilter(const Slice* keys, int n, std::string* dst) const {
  433. // Use builtin bloom filter code after removing trailing spaces
  434. std::vector&lt;Slice&gt; trimmed(n);
  435. for (int i = 0; i &lt; n; i++) {
  436. trimmed[i] = RemoveTrailingSpaces(keys[i]);
  437. }
  438. return builtin_policy_-&gt;CreateFilter(&amp;trimmed[i], n, dst);
  439. }
  440. bool KeyMayMatch(const Slice& key, const Slice& filter) const {
  441. // Use builtin bloom filter code after removing trailing spaces
  442. return builtin_policy_-&gt;KeyMayMatch(RemoveTrailingSpaces(key), filter);
  443. }
  444. };
  445. </pre>
  446. <p>
  447. Advanced applications may provide a filter policy that does not use
  448. a bloom filter but uses some other mechanism for summarizing a set
  449. of keys. See <code>leveldb/filter_policy.h</code> for detail.
  450. <p>
  451. <h1>Checksums</h1>
  452. <p>
  453. <code>leveldb</code> associates checksums with all data it stores in the file system.
  454. There are two separate controls provided over how aggressively these
  455. checksums are verified:
  456. <p>
  457. <ul>
  458. <li> <code>ReadOptions::verify_checksums</code> may be set to true to force
  459. checksum verification of all data that is read from the file system on
  460. behalf of a particular read. By default, no such verification is
  461. done.
  462. <p>
  463. <li> <code>Options::paranoid_checks</code> may be set to true before opening a
  464. database to make the database implementation raise an error as soon as
  465. it detects an internal corruption. Depending on which portion of the
  466. database has been corrupted, the error may be raised when the database
  467. is opened, or later by another database operation. By default,
  468. paranoid checking is off so that the database can be used even if
  469. parts of its persistent storage have been corrupted.
  470. <p>
  471. If a database is corrupted (perhaps it cannot be opened when
  472. paranoid checking is turned on), the <code>leveldb::RepairDB</code> function
  473. may be used to recover as much of the data as possible
  474. <p>
  475. </ul>
  476. <h1>Approximate Sizes</h1>
  477. <p>
  478. The <code>GetApproximateSizes</code> method can used to get the approximate
  479. number of bytes of file system space used by one or more key ranges.
  480. <p>
  481. <pre>
  482. leveldb::Range ranges[2];
  483. ranges[0] = leveldb::Range("a", "c");
  484. ranges[1] = leveldb::Range("x", "z");
  485. uint64_t sizes[2];
  486. leveldb::Status s = db-&gt;GetApproximateSizes(ranges, 2, sizes);
  487. </pre>
  488. The preceding call will set <code>sizes[0]</code> to the approximate number of
  489. bytes of file system space used by the key range <code>[a..c)</code> and
  490. <code>sizes[1]</code> to the approximate number of bytes used by the key range
  491. <code>[x..z)</code>.
  492. <p>
  493. <h1>Environment</h1>
  494. <p>
  495. All file operations (and other operating system calls) issued by the
  496. <code>leveldb</code> implementation are routed through a <code>leveldb::Env</code> object.
  497. Sophisticated clients may wish to provide their own <code>Env</code>
  498. implementation to get better control. For example, an application may
  499. introduce artificial delays in the file IO paths to limit the impact
  500. of <code>leveldb</code> on other activities in the system.
  501. <p>
  502. <pre>
  503. class SlowEnv : public leveldb::Env {
  504. .. implementation of the Env interface ...
  505. };
  506. SlowEnv env;
  507. leveldb::Options options;
  508. options.env = &amp;env;
  509. Status s = leveldb::DB::Open(options, ...);
  510. </pre>
  511. <h1>Porting</h1>
  512. <p>
  513. <code>leveldb</code> may be ported to a new platform by providing platform
  514. specific implementations of the types/methods/functions exported by
  515. <code>leveldb/port/port.h</code>. See <code>leveldb/port/port_example.h</code> for more
  516. details.
  517. <p>
  518. In addition, the new platform may need a new default <code>leveldb::Env</code>
  519. implementation. See <code>leveldb/util/env_posix.h</code> for an example.
  520. <h1>Other Information</h1>
  521. <p>
  522. Details about the <code>leveldb</code> implementation may be found in
  523. the following documents:
  524. <ul>
  525. <li> <a href="impl.html">Implementation notes</a>
  526. <li> <a href="table_format.txt">Format of an immutable Table file</a>
  527. <li> <a href="log_format.txt">Format of a log file</a>
  528. </ul>
  529. </body>
  530. </html>