10225501448 李度 10225101546 陈胤遒 10215501422 高宇菲

213 rindas
8.1 KiB

Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
pirms 10 gadiem
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
pirms 10 gadiem
  1. <!DOCTYPE html>
  2. <html>
  3. <head>
  4. <link rel="stylesheet" type="text/css" href="doc.css" />
  5. <title>Leveldb file layout and compactions</title>
  6. </head>
  7. <body>
  8. <h1>Files</h1>
  9. The implementation of leveldb is similar in spirit to the
  10. representation of a single
  11. <a href="http://research.google.com/archive/bigtable.html">
  12. Bigtable tablet (section 5.3)</a>.
  13. However the organization of the files that make up the representation
  14. is somewhat different and is explained below.
  15. <p>
  16. Each database is represented by a set of files stored in a directory.
  17. There are several different types of files as documented below:
  18. <p>
  19. <h2>Log files</h2>
  20. <p>
  21. A log file (*.log) stores a sequence of recent updates. Each update
  22. is appended to the current log file. When the log file reaches a
  23. pre-determined size (approximately 4MB by default), it is converted
  24. to a sorted table (see below) and a new log file is created for future
  25. updates.
  26. <p>
  27. A copy of the current log file is kept in an in-memory structure (the
  28. <code>memtable</code>). This copy is consulted on every read so that read
  29. operations reflect all logged updates.
  30. <p>
  31. <h2>Sorted tables</h2>
  32. <p>
  33. A sorted table (*.sst) stores a sequence of entries sorted by key.
  34. Each entry is either a value for the key, or a deletion marker for the
  35. key. (Deletion markers are kept around to hide obsolete values
  36. present in older sorted tables).
  37. <p>
  38. The set of sorted tables are organized into a sequence of levels. The
  39. sorted table generated from a log file is placed in a special <code>young</code>
  40. level (also called level-0). When the number of young files exceeds a
  41. certain threshold (currently four), all of the young files are merged
  42. together with all of the overlapping level-1 files to produce a
  43. sequence of new level-1 files (we create a new level-1 file for every
  44. 2MB of data.)
  45. <p>
  46. Files in the young level may contain overlapping keys. However files
  47. in other levels have distinct non-overlapping key ranges. Consider
  48. level number L where L >= 1. When the combined size of files in
  49. level-L exceeds (10^L) MB (i.e., 10MB for level-1, 100MB for level-2,
  50. ...), one file in level-L, and all of the overlapping files in
  51. level-(L+1) are merged to form a set of new files for level-(L+1).
  52. These merges have the effect of gradually migrating new updates from
  53. the young level to the largest level using only bulk reads and writes
  54. (i.e., minimizing expensive seeks).
  55. <h2>Manifest</h2>
  56. <p>
  57. A MANIFEST file lists the set of sorted tables that make up each
  58. level, the corresponding key ranges, and other important metadata.
  59. A new MANIFEST file (with a new number embedded in the file name)
  60. is created whenever the database is reopened. The MANIFEST file is
  61. formatted as a log, and changes made to the serving state (as files
  62. are added or removed) are appended to this log.
  63. <p>
  64. <h2>Current</h2>
  65. <p>
  66. CURRENT is a simple text file that contains the name of the latest
  67. MANIFEST file.
  68. <p>
  69. <h2>Info logs</h2>
  70. <p>
  71. Informational messages are printed to files named LOG and LOG.old.
  72. <p>
  73. <h2>Others</h2>
  74. <p>
  75. Other files used for miscellaneous purposes may also be present
  76. (LOCK, *.dbtmp).
  77. <h1>Level 0</h1>
  78. When the log file grows above a certain size (1MB by default):
  79. <ul>
  80. <li>Create a brand new memtable and log file and direct future updates here
  81. <li>In the background:
  82. <ul>
  83. <li>Write the contents of the previous memtable to an sstable
  84. <li>Discard the memtable
  85. <li>Delete the old log file and the old memtable
  86. <li>Add the new sstable to the young (level-0) level.
  87. </ul>
  88. </ul>
  89. <h1>Compactions</h1>
  90. <p>
  91. When the size of level L exceeds its limit, we compact it in a
  92. background thread. The compaction picks a file from level L and all
  93. overlapping files from the next level L+1. Note that if a level-L
  94. file overlaps only part of a level-(L+1) file, the entire file at
  95. level-(L+1) is used as an input to the compaction and will be
  96. discarded after the compaction. Aside: because level-0 is special
  97. (files in it may overlap each other), we treat compactions from
  98. level-0 to level-1 specially: a level-0 compaction may pick more than
  99. one level-0 file in case some of these files overlap each other.
  100. <p>
  101. A compaction merges the contents of the picked files to produce a
  102. sequence of level-(L+1) files. We switch to producing a new
  103. level-(L+1) file after the current output file has reached the target
  104. file size (2MB). We also switch to a new output file when the key
  105. range of the current output file has grown enough to overlap more than
  106. ten level-(L+2) files. This last rule ensures that a later compaction
  107. of a level-(L+1) file will not pick up too much data from level-(L+2).
  108. <p>
  109. The old files are discarded and the new files are added to the serving
  110. state.
  111. <p>
  112. Compactions for a particular level rotate through the key space. In
  113. more detail, for each level L, we remember the ending key of the last
  114. compaction at level L. The next compaction for level L will pick the
  115. first file that starts after this key (wrapping around to the
  116. beginning of the key space if there is no such file).
  117. <p>
  118. Compactions drop overwritten values. They also drop deletion markers
  119. if there are no higher numbered levels that contain a file whose range
  120. overlaps the current key.
  121. <h2>Timing</h2>
  122. Level-0 compactions will read up to four 1MB files from level-0, and
  123. at worst all the level-1 files (10MB). I.e., we will read 14MB and
  124. write 14MB.
  125. <p>
  126. Other than the special level-0 compactions, we will pick one 2MB file
  127. from level L. In the worst case, this will overlap ~ 12 files from
  128. level L+1 (10 because level-(L+1) is ten times the size of level-L,
  129. and another two at the boundaries since the file ranges at level-L
  130. will usually not be aligned with the file ranges at level-L+1). The
  131. compaction will therefore read 26MB and write 26MB. Assuming a disk
  132. IO rate of 100MB/s (ballpark range for modern drives), the worst
  133. compaction cost will be approximately 0.5 second.
  134. <p>
  135. If we throttle the background writing to something small, say 10% of
  136. the full 100MB/s speed, a compaction may take up to 5 seconds. If the
  137. user is writing at 10MB/s, we might build up lots of level-0 files
  138. (~50 to hold the 5*10MB). This may significantly increase the cost of
  139. reads due to the overhead of merging more files together on every
  140. read.
  141. <p>
  142. Solution 1: To reduce this problem, we might want to increase the log
  143. switching threshold when the number of level-0 files is large. Though
  144. the downside is that the larger this threshold, the more memory we will
  145. need to hold the corresponding memtable.
  146. <p>
  147. Solution 2: We might want to decrease write rate artificially when the
  148. number of level-0 files goes up.
  149. <p>
  150. Solution 3: We work on reducing the cost of very wide merges.
  151. Perhaps most of the level-0 files will have their blocks sitting
  152. uncompressed in the cache and we will only need to worry about the
  153. O(N) complexity in the merging iterator.
  154. <h2>Number of files</h2>
  155. Instead of always making 2MB files, we could make larger files for
  156. larger levels to reduce the total file count, though at the expense of
  157. more bursty compactions. Alternatively, we could shard the set of
  158. files into multiple directories.
  159. <p>
  160. An experiment on an <code>ext3</code> filesystem on Feb 04, 2011 shows
  161. the following timings to do 100K file opens in directories with
  162. varying number of files:
  163. <table class="datatable">
  164. <tr><th>Files in directory</th><th>Microseconds to open a file</th></tr>
  165. <tr><td>1000</td><td>9</td>
  166. <tr><td>10000</td><td>10</td>
  167. <tr><td>100000</td><td>16</td>
  168. </table>
  169. So maybe even the sharding is not necessary on modern filesystems?
  170. <h1>Recovery</h1>
  171. <ul>
  172. <li> Read CURRENT to find name of the latest committed MANIFEST
  173. <li> Read the named MANIFEST file
  174. <li> Clean up stale files
  175. <li> We could open all sstables here, but it is probably better to be lazy...
  176. <li> Convert log chunk to a new level-0 sstable
  177. <li> Start directing new writes to a new log file with recovered sequence#
  178. </ul>
  179. <h1>Garbage collection of files</h1>
  180. <code>DeleteObsoleteFiles()</code> is called at the end of every
  181. compaction and at the end of recovery. It finds the names of all
  182. files in the database. It deletes all log files that are not the
  183. current log file. It deletes all table files that are not referenced
  184. from some level and are not the output of an active compaction.
  185. </body>
  186. </html>