作者: 谢瑞阳 10225101483 徐翔宇 10225101535
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1573 regels
48 KiB

Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 jaren geleden
  1. // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
  2. // Use of this source code is governed by a BSD-style license that can be
  3. // found in the LICENSE file. See the AUTHORS file for names of contributors.
  4. #include "db/db_impl.h"
  5. #include <algorithm>
  6. #include <atomic>
  7. #include <cstdint>
  8. #include <cstdio>
  9. #include <set>
  10. #include <string>
  11. #include <vector>
  12. #include "db/builder.h"
  13. #include "db/db_iter.h"
  14. #include "db/dbformat.h"
  15. #include "db/filename.h"
  16. #include "db/log_reader.h"
  17. #include "db/log_writer.h"
  18. #include "db/memtable.h"
  19. #include "db/table_cache.h"
  20. #include "db/version_set.h"
  21. #include "db/write_batch_internal.h"
  22. #include "leveldb/db.h"
  23. #include "leveldb/env.h"
  24. #include "leveldb/status.h"
  25. #include "leveldb/table.h"
  26. #include "leveldb/table_builder.h"
  27. #include "port/port.h"
  28. #include "table/block.h"
  29. #include "table/merger.h"
  30. #include "table/two_level_iterator.h"
  31. #include "util/coding.h"
  32. #include "util/logging.h"
  33. #include "util/mutexlock.h"
  34. namespace leveldb {
  35. const int kNumNonTableCacheFiles = 10;
  36. // Information kept for every waiting writer
  37. struct DBImpl::Writer {
  38. explicit Writer(port::Mutex* mu)
  39. : batch(nullptr), sync(false), done(false), cv(mu) {}
  40. Status status;
  41. WriteBatch* batch;
  42. bool sync;
  43. bool done;
  44. port::CondVar cv;
  45. };
  46. struct DBImpl::CompactionState {
  47. // Files produced by compaction
  48. struct Output {
  49. uint64_t number;
  50. uint64_t file_size;
  51. InternalKey smallest, largest;
  52. };
  53. Output* current_output() { return &outputs[outputs.size() - 1]; }
  54. explicit CompactionState(Compaction* c)
  55. : compaction(c),
  56. smallest_snapshot(0),
  57. outfile(nullptr),
  58. builder(nullptr),
  59. total_bytes(0) {}
  60. Compaction* const compaction;
  61. // Sequence numbers < smallest_snapshot are not significant since we
  62. // will never have to service a snapshot below smallest_snapshot.
  63. // Therefore if we have seen a sequence number S <= smallest_snapshot,
  64. // we can drop all entries for the same key with sequence numbers < S.
  65. SequenceNumber smallest_snapshot;
  66. std::vector<Output> outputs;
  67. // State kept for output being generated
  68. WritableFile* outfile;
  69. TableBuilder* builder;
  70. uint64_t total_bytes;
  71. };
  72. // Fix user-supplied options to be reasonable
  73. template <class T, class V>
  74. static void ClipToRange(T* ptr, V minvalue, V maxvalue) {
  75. if (static_cast<V>(*ptr) > maxvalue) *ptr = maxvalue;
  76. if (static_cast<V>(*ptr) < minvalue) *ptr = minvalue;
  77. }
  78. Options SanitizeOptions(const std::string& dbname,
  79. const InternalKeyComparator* icmp,
  80. const InternalFilterPolicy* ipolicy,
  81. const Options& src) {
  82. Options result = src;
  83. result.comparator = icmp;
  84. result.filter_policy = (src.filter_policy != nullptr) ? ipolicy : nullptr;
  85. ClipToRange(&result.max_open_files, 64 + kNumNonTableCacheFiles, 50000);
  86. ClipToRange(&result.write_buffer_size, 64 << 10, 1 << 30);
  87. ClipToRange(&result.max_file_size, 1 << 20, 1 << 30);
  88. ClipToRange(&result.block_size, 1 << 10, 4 << 20);
  89. if (result.info_log == nullptr) {
  90. // Open a log file in the same directory as the db
  91. src.env->CreateDir(dbname); // In case it does not exist
  92. src.env->RenameFile(InfoLogFileName(dbname), OldInfoLogFileName(dbname));
  93. Status s = src.env->NewLogger(InfoLogFileName(dbname), &result.info_log);
  94. if (!s.ok()) {
  95. // No place suitable for logging
  96. result.info_log = nullptr;
  97. }
  98. }
  99. if (result.block_cache == nullptr) {
  100. result.block_cache = NewLRUCache(8 << 20);
  101. }
  102. return result;
  103. }
  104. static int TableCacheSize(const Options& sanitized_options) {
  105. // Reserve ten files or so for other uses and give the rest to TableCache.
  106. return sanitized_options.max_open_files - kNumNonTableCacheFiles;
  107. }
  108. DBImpl::DBImpl(const Options& raw_options, const std::string& dbname)
  109. : env_(raw_options.env),
  110. internal_comparator_(raw_options.comparator),
  111. internal_filter_policy_(raw_options.filter_policy),
  112. options_(SanitizeOptions(dbname, &internal_comparator_,
  113. &internal_filter_policy_, raw_options)),
  114. owns_info_log_(options_.info_log != raw_options.info_log),
  115. owns_cache_(options_.block_cache != raw_options.block_cache),
  116. dbname_(dbname),
  117. table_cache_(new TableCache(dbname_, options_, TableCacheSize(options_))),
  118. db_lock_(nullptr),
  119. shutting_down_(false),
  120. background_work_finished_signal_(&mutex_),
  121. mem_(nullptr),
  122. imm_(nullptr),
  123. has_imm_(false),
  124. logfile_(nullptr),
  125. logfile_number_(0),
  126. log_(nullptr),
  127. seed_(0),
  128. tmp_batch_(new WriteBatch),
  129. background_compaction_scheduled_(false),
  130. manual_compaction_(nullptr),
  131. versions_(new VersionSet(dbname_, &options_, table_cache_,
  132. &internal_comparator_)) {}
  133. DBImpl::~DBImpl() {
  134. // Wait for background work to finish.
  135. mutex_.Lock();
  136. shutting_down_.store(true, std::memory_order_release);
  137. while (background_compaction_scheduled_) {
  138. background_work_finished_signal_.Wait();
  139. }
  140. mutex_.Unlock();
  141. if (db_lock_ != nullptr) {
  142. env_->UnlockFile(db_lock_);
  143. }
  144. delete versions_;
  145. if (mem_ != nullptr) mem_->Unref();
  146. if (imm_ != nullptr) imm_->Unref();
  147. delete tmp_batch_;
  148. delete log_;
  149. delete logfile_;
  150. delete table_cache_;
  151. if (owns_info_log_) {
  152. delete options_.info_log;
  153. }
  154. if (owns_cache_) {
  155. delete options_.block_cache;
  156. }
  157. }
  158. Status DBImpl::NewDB() {
  159. VersionEdit new_db;
  160. new_db.SetComparatorName(user_comparator()->Name());
  161. new_db.SetLogNumber(0);
  162. new_db.SetNextFile(2);
  163. new_db.SetLastSequence(0);
  164. const std::string manifest = DescriptorFileName(dbname_, 1);
  165. WritableFile* file;
  166. Status s = env_->NewWritableFile(manifest, &file);
  167. if (!s.ok()) {
  168. return s;
  169. }
  170. {
  171. log::Writer log(file);
  172. std::string record;
  173. new_db.EncodeTo(&record);
  174. s = log.AddRecord(record);
  175. if (s.ok()) {
  176. s = file->Sync();
  177. }
  178. if (s.ok()) {
  179. s = file->Close();
  180. }
  181. }
  182. delete file;
  183. if (s.ok()) {
  184. // Make "CURRENT" file that points to the new manifest file.
  185. s = SetCurrentFile(env_, dbname_, 1);
  186. } else {
  187. env_->RemoveFile(manifest);
  188. }
  189. return s;
  190. }
  191. void DBImpl::MaybeIgnoreError(Status* s) const {
  192. if (s->ok() || options_.paranoid_checks) {
  193. // No change needed
  194. } else {
  195. Log(options_.info_log, "Ignoring error %s", s->ToString().c_str());
  196. *s = Status::OK();
  197. }
  198. }
  199. void DBImpl::RemoveObsoleteFiles() {
  200. mutex_.AssertHeld();
  201. if (!bg_error_.ok()) {
  202. // After a background error, we don't know whether a new version may
  203. // or may not have been committed, so we cannot safely garbage collect.
  204. return;
  205. }
  206. // Make a set of all of the live files
  207. std::set<uint64_t> live = pending_outputs_;
  208. versions_->AddLiveFiles(&live);
  209. std::vector<std::string> filenames;
  210. env_->GetChildren(dbname_, &filenames); // Ignoring errors on purpose
  211. uint64_t number;
  212. FileType type;
  213. std::vector<std::string> files_to_delete;
  214. for (std::string& filename : filenames) {
  215. if (ParseFileName(filename, &number, &type)) {
  216. bool keep = true;
  217. switch (type) {
  218. case kLogFile:
  219. keep = ((number >= versions_->LogNumber()) ||
  220. (number == versions_->PrevLogNumber()));
  221. break;
  222. case kDescriptorFile:
  223. // Keep my manifest file, and any newer incarnations'
  224. // (in case there is a race that allows other incarnations)
  225. keep = (number >= versions_->ManifestFileNumber());
  226. break;
  227. case kTableFile:
  228. keep = (live.find(number) != live.end());
  229. break;
  230. case kTempFile:
  231. // Any temp files that are currently being written to must
  232. // be recorded in pending_outputs_, which is inserted into "live"
  233. keep = (live.find(number) != live.end());
  234. break;
  235. case kCurrentFile:
  236. case kDBLockFile:
  237. case kInfoLogFile:
  238. keep = true;
  239. break;
  240. }
  241. if (!keep) {
  242. files_to_delete.push_back(std::move(filename));
  243. if (type == kTableFile) {
  244. table_cache_->Evict(number);
  245. }
  246. Log(options_.info_log, "Delete type=%d #%lld\n", static_cast<int>(type),
  247. static_cast<unsigned long long>(number));
  248. }
  249. }
  250. }
  251. // While deleting all files unblock other threads. All files being deleted
  252. // have unique names which will not collide with newly created files and
  253. // are therefore safe to delete while allowing other threads to proceed.
  254. mutex_.Unlock();
  255. for (const std::string& filename : files_to_delete) {
  256. env_->RemoveFile(dbname_ + "/" + filename);
  257. }
  258. mutex_.Lock();
  259. }
  260. Status DBImpl::Recover(VersionEdit* edit, bool* save_manifest) {
  261. mutex_.AssertHeld();
  262. // Ignore error from CreateDir since the creation of the DB is
  263. // committed only when the descriptor is created, and this directory
  264. // may already exist from a previous failed creation attempt.
  265. env_->CreateDir(dbname_);
  266. assert(db_lock_ == nullptr);
  267. Status s = env_->LockFile(LockFileName(dbname_), &db_lock_);
  268. if (!s.ok()) {
  269. return s;
  270. }
  271. if (!env_->FileExists(CurrentFileName(dbname_))) {
  272. if (options_.create_if_missing) {
  273. Log(options_.info_log, "Creating DB %s since it was missing.",
  274. dbname_.c_str());
  275. s = NewDB();
  276. if (!s.ok()) {
  277. return s;
  278. }
  279. } else {
  280. return Status::InvalidArgument(
  281. dbname_, "does not exist (create_if_missing is false)");
  282. }
  283. } else {
  284. if (options_.error_if_exists) {
  285. return Status::InvalidArgument(dbname_,
  286. "exists (error_if_exists is true)");
  287. }
  288. }
  289. s = versions_->Recover(save_manifest);
  290. if (!s.ok()) {
  291. return s;
  292. }
  293. SequenceNumber max_sequence(0);
  294. // Recover from all newer log files than the ones named in the
  295. // descriptor (new log files may have been added by the previous
  296. // incarnation without registering them in the descriptor).
  297. //
  298. // Note that PrevLogNumber() is no longer used, but we pay
  299. // attention to it in case we are recovering a database
  300. // produced by an older version of leveldb.
  301. const uint64_t min_log = versions_->LogNumber();
  302. const uint64_t prev_log = versions_->PrevLogNumber();
  303. std::vector<std::string> filenames;
  304. s = env_->GetChildren(dbname_, &filenames);
  305. if (!s.ok()) {
  306. return s;
  307. }
  308. std::set<uint64_t> expected;
  309. versions_->AddLiveFiles(&expected);
  310. uint64_t number;
  311. FileType type;
  312. std::vector<uint64_t> logs;
  313. for (size_t i = 0; i < filenames.size(); i++) {
  314. if (ParseFileName(filenames[i], &number, &type)) {
  315. expected.erase(number);
  316. if (type == kLogFile && ((number >= min_log) || (number == prev_log)))
  317. logs.push_back(number);
  318. }
  319. }
  320. if (!expected.empty()) {
  321. char buf[50];
  322. std::snprintf(buf, sizeof(buf), "%d missing files; e.g.",
  323. static_cast<int>(expected.size()));
  324. return Status::Corruption(buf, TableFileName(dbname_, *(expected.begin())));
  325. }
  326. // Recover in the order in which the logs were generated
  327. std::sort(logs.begin(), logs.end());
  328. for (size_t i = 0; i < logs.size(); i++) {
  329. s = RecoverLogFile(logs[i], (i == logs.size() - 1), save_manifest, edit,
  330. &max_sequence);
  331. if (!s.ok()) {
  332. return s;
  333. }
  334. // The previous incarnation may not have written any MANIFEST
  335. // records after allocating this log number. So we manually
  336. // update the file number allocation counter in VersionSet.
  337. versions_->MarkFileNumberUsed(logs[i]);
  338. }
  339. if (versions_->LastSequence() < max_sequence) {
  340. versions_->SetLastSequence(max_sequence);
  341. }
  342. return Status::OK();
  343. }
  344. Status DBImpl::RecoverLogFile(uint64_t log_number, bool last_log,
  345. bool* save_manifest, VersionEdit* edit,
  346. SequenceNumber* max_sequence) {
  347. struct LogReporter : public log::Reader::Reporter {
  348. Env* env;
  349. Logger* info_log;
  350. const char* fname;
  351. Status* status; // null if options_.paranoid_checks==false
  352. void Corruption(size_t bytes, const Status& s) override {
  353. Log(info_log, "%s%s: dropping %d bytes; %s",
  354. (this->status == nullptr ? "(ignoring error) " : ""), fname,
  355. static_cast<int>(bytes), s.ToString().c_str());
  356. if (this->status != nullptr && this->status->ok()) *this->status = s;
  357. }
  358. };
  359. mutex_.AssertHeld();
  360. // Open the log file
  361. std::string fname = LogFileName(dbname_, log_number);
  362. SequentialFile* file;
  363. Status status = env_->NewSequentialFile(fname, &file);
  364. if (!status.ok()) {
  365. MaybeIgnoreError(&status);
  366. return status;
  367. }
  368. // Create the log reader.
  369. LogReporter reporter;
  370. reporter.env = env_;
  371. reporter.info_log = options_.info_log;
  372. reporter.fname = fname.c_str();
  373. reporter.status = (options_.paranoid_checks ? &status : nullptr);
  374. // We intentionally make log::Reader do checksumming even if
  375. // paranoid_checks==false so that corruptions cause entire commits
  376. // to be skipped instead of propagating bad information (like overly
  377. // large sequence numbers).
  378. log::Reader reader(file, &reporter, true /*checksum*/, 0 /*initial_offset*/);
  379. Log(options_.info_log, "Recovering log #%llu",
  380. (unsigned long long)log_number);
  381. // Read all the records and add to a memtable
  382. std::string scratch;
  383. Slice record;
  384. WriteBatch batch;
  385. int compactions = 0;
  386. MemTable* mem = nullptr;
  387. while (reader.ReadRecord(&record, &scratch) && status.ok()) {
  388. if (record.size() < 12) {
  389. reporter.Corruption(record.size(),
  390. Status::Corruption("log record too small"));
  391. continue;
  392. }
  393. WriteBatchInternal::SetContents(&batch, record);
  394. if (mem == nullptr) {
  395. mem = new MemTable(internal_comparator_);
  396. mem->Ref();
  397. }
  398. status = WriteBatchInternal::InsertInto(&batch, mem);
  399. MaybeIgnoreError(&status);
  400. if (!status.ok()) {
  401. break;
  402. }
  403. const SequenceNumber last_seq = WriteBatchInternal::Sequence(&batch) +
  404. WriteBatchInternal::Count(&batch) - 1;
  405. if (last_seq > *max_sequence) {
  406. *max_sequence = last_seq;
  407. }
  408. if (mem->ApproximateMemoryUsage() > options_.write_buffer_size) {
  409. compactions++;
  410. *save_manifest = true;
  411. status = WriteLevel0Table(mem, edit, nullptr);
  412. mem->Unref();
  413. mem = nullptr;
  414. if (!status.ok()) {
  415. // Reflect errors immediately so that conditions like full
  416. // file-systems cause the DB::Open() to fail.
  417. break;
  418. }
  419. }
  420. }
  421. delete file;
  422. // See if we should keep reusing the last log file.
  423. if (status.ok() && options_.reuse_logs && last_log && compactions == 0) {
  424. assert(logfile_ == nullptr);
  425. assert(log_ == nullptr);
  426. assert(mem_ == nullptr);
  427. uint64_t lfile_size;
  428. if (env_->GetFileSize(fname, &lfile_size).ok() &&
  429. env_->NewAppendableFile(fname, &logfile_).ok()) {
  430. Log(options_.info_log, "Reusing old log %s \n", fname.c_str());
  431. log_ = new log::Writer(logfile_, lfile_size);
  432. logfile_number_ = log_number;
  433. if (mem != nullptr) {
  434. mem_ = mem;
  435. mem = nullptr;
  436. } else {
  437. // mem can be nullptr if lognum exists but was empty.
  438. mem_ = new MemTable(internal_comparator_);
  439. mem_->Ref();
  440. }
  441. }
  442. }
  443. if (mem != nullptr) {
  444. // mem did not get reused; compact it.
  445. if (status.ok()) {
  446. *save_manifest = true;
  447. status = WriteLevel0Table(mem, edit, nullptr);
  448. }
  449. mem->Unref();
  450. }
  451. return status;
  452. }
  453. Status DBImpl::WriteLevel0Table(MemTable* mem, VersionEdit* edit,
  454. Version* base) {
  455. mutex_.AssertHeld();
  456. const uint64_t start_micros = env_->NowMicros();
  457. FileMetaData meta;
  458. meta.number = versions_->NewFileNumber();
  459. pending_outputs_.insert(meta.number);
  460. Iterator* iter = mem->NewIterator();
  461. Log(options_.info_log, "Level-0 table #%llu: started",
  462. (unsigned long long)meta.number);
  463. Status s;
  464. {
  465. mutex_.Unlock();
  466. s = BuildTable(dbname_, env_, options_, table_cache_, iter, &meta);
  467. mutex_.Lock();
  468. }
  469. Log(options_.info_log, "Level-0 table #%llu: %lld bytes %s",
  470. (unsigned long long)meta.number, (unsigned long long)meta.file_size,
  471. s.ToString().c_str());
  472. delete iter;
  473. pending_outputs_.erase(meta.number);
  474. // Note that if file_size is zero, the file has been deleted and
  475. // should not be added to the manifest.
  476. int level = 0;
  477. if (s.ok() && meta.file_size > 0) {
  478. const Slice min_user_key = meta.smallest.user_key();
  479. const Slice max_user_key = meta.largest.user_key();
  480. if (base != nullptr) {
  481. level = base->PickLevelForMemTableOutput(min_user_key, max_user_key);
  482. }
  483. edit->AddFile(level, meta.number, meta.file_size, meta.smallest,
  484. meta.largest);
  485. }
  486. CompactionStats stats;
  487. stats.micros = env_->NowMicros() - start_micros;
  488. stats.bytes_written = meta.file_size;
  489. stats_[level].Add(stats);
  490. return s;
  491. }
  492. void DBImpl::CompactMemTable() {
  493. mutex_.AssertHeld();
  494. assert(imm_ != nullptr);
  495. // Save the contents of the memtable as a new Table
  496. VersionEdit edit;
  497. Version* base = versions_->current();
  498. base->Ref();
  499. Status s = WriteLevel0Table(imm_, &edit, base);
  500. base->Unref();
  501. if (s.ok() && shutting_down_.load(std::memory_order_acquire)) {
  502. s = Status::IOError("Deleting DB during memtable compaction");
  503. }
  504. // Replace immutable memtable with the generated Table
  505. if (s.ok()) {
  506. edit.SetPrevLogNumber(0);
  507. edit.SetLogNumber(logfile_number_); // Earlier logs no longer needed
  508. s = versions_->LogAndApply(&edit, &mutex_);
  509. }
  510. if (s.ok()) {
  511. // Commit to the new state
  512. imm_->Unref();
  513. imm_ = nullptr;
  514. has_imm_.store(false, std::memory_order_release);
  515. RemoveObsoleteFiles();
  516. } else {
  517. RecordBackgroundError(s);
  518. }
  519. }
  520. void DBImpl::CompactRange(const Slice* begin, const Slice* end) {
  521. int max_level_with_files = 1;
  522. {
  523. MutexLock l(&mutex_);
  524. Version* base = versions_->current();
  525. for (int level = 1; level < config::kNumLevels; level++) {
  526. if (base->OverlapInLevel(level, begin, end)) {
  527. max_level_with_files = level;
  528. }
  529. }
  530. }
  531. TEST_CompactMemTable(); // TODO(sanjay): Skip if memtable does not overlap
  532. for (int level = 0; level < max_level_with_files; level++) {
  533. TEST_CompactRange(level, begin, end);
  534. }
  535. }
  536. void DBImpl::TEST_CompactRange(int level, const Slice* begin,
  537. const Slice* end) {
  538. assert(level >= 0);
  539. assert(level + 1 < config::kNumLevels);
  540. InternalKey begin_storage, end_storage;
  541. ManualCompaction manual;
  542. manual.level = level;
  543. manual.done = false;
  544. if (begin == nullptr) {
  545. manual.begin = nullptr;
  546. } else {
  547. begin_storage = InternalKey(*begin, kMaxSequenceNumber, kValueTypeForSeek);
  548. manual.begin = &begin_storage;
  549. }
  550. if (end == nullptr) {
  551. manual.end = nullptr;
  552. } else {
  553. end_storage = InternalKey(*end, 0, static_cast<ValueType>(0));
  554. manual.end = &end_storage;
  555. }
  556. MutexLock l(&mutex_);
  557. while (!manual.done && !shutting_down_.load(std::memory_order_acquire) &&
  558. bg_error_.ok()) {
  559. if (manual_compaction_ == nullptr) { // Idle
  560. manual_compaction_ = &manual;
  561. MaybeScheduleCompaction();
  562. } else { // Running either my compaction or another compaction.
  563. background_work_finished_signal_.Wait();
  564. }
  565. }
  566. if (manual_compaction_ == &manual) {
  567. // Cancel my manual compaction since we aborted early for some reason.
  568. manual_compaction_ = nullptr;
  569. }
  570. }
  571. Status DBImpl::TEST_CompactMemTable() {
  572. // nullptr batch means just wait for earlier writes to be done
  573. Status s = Write(WriteOptions(), nullptr);
  574. if (s.ok()) {
  575. // Wait until the compaction completes
  576. MutexLock l(&mutex_);
  577. while (imm_ != nullptr && bg_error_.ok()) {
  578. background_work_finished_signal_.Wait();
  579. }
  580. if (imm_ != nullptr) {
  581. s = bg_error_;
  582. }
  583. }
  584. return s;
  585. }
  586. void DBImpl::RecordBackgroundError(const Status& s) {
  587. mutex_.AssertHeld();
  588. if (bg_error_.ok()) {
  589. bg_error_ = s;
  590. background_work_finished_signal_.SignalAll();
  591. }
  592. }
  593. void DBImpl::MaybeScheduleCompaction() {
  594. mutex_.AssertHeld();
  595. if (background_compaction_scheduled_) {
  596. // Already scheduled
  597. } else if (shutting_down_.load(std::memory_order_acquire)) {
  598. // DB is being deleted; no more background compactions
  599. } else if (!bg_error_.ok()) {
  600. // Already got an error; no more changes
  601. } else if (imm_ == nullptr && manual_compaction_ == nullptr &&
  602. !versions_->NeedsCompaction()) {
  603. // No work to be done
  604. } else {
  605. background_compaction_scheduled_ = true;
  606. env_->Schedule(&DBImpl::BGWork, this);
  607. }
  608. }
  609. void DBImpl::BGWork(void* db) {
  610. reinterpret_cast<DBImpl*>(db)->BackgroundCall();
  611. }
  612. void DBImpl::BackgroundCall() {
  613. MutexLock l(&mutex_);
  614. assert(background_compaction_scheduled_);
  615. if (shutting_down_.load(std::memory_order_acquire)) {
  616. // No more background work when shutting down.
  617. } else if (!bg_error_.ok()) {
  618. // No more background work after a background error.
  619. } else {
  620. BackgroundCompaction();
  621. }
  622. background_compaction_scheduled_ = false;
  623. // Previous compaction may have produced too many files in a level,
  624. // so reschedule another compaction if needed.
  625. MaybeScheduleCompaction();
  626. background_work_finished_signal_.SignalAll();
  627. }
  628. void DBImpl::BackgroundCompaction() {
  629. mutex_.AssertHeld();
  630. if (imm_ != nullptr) {
  631. CompactMemTable();
  632. return;
  633. }
  634. Compaction* c;
  635. bool is_manual = (manual_compaction_ != nullptr);
  636. InternalKey manual_end;
  637. if (is_manual) {
  638. ManualCompaction* m = manual_compaction_;
  639. c = versions_->CompactRange(m->level, m->begin, m->end);
  640. m->done = (c == nullptr);
  641. if (c != nullptr) {
  642. manual_end = c->input(0, c->num_input_files(0) - 1)->largest;
  643. }
  644. Log(options_.info_log,
  645. "Manual compaction at level-%d from %s .. %s; will stop at %s\n",
  646. m->level, (m->begin ? m->begin->DebugString().c_str() : "(begin)"),
  647. (m->end ? m->end->DebugString().c_str() : "(end)"),
  648. (m->done ? "(end)" : manual_end.DebugString().c_str()));
  649. } else {
  650. c = versions_->PickCompaction();
  651. }
  652. Status status;
  653. if (c == nullptr) {
  654. // Nothing to do
  655. } else if (!is_manual && c->IsTrivialMove()) {
  656. // Move file to next level
  657. assert(c->num_input_files(0) == 1);
  658. FileMetaData* f = c->input(0, 0);
  659. c->edit()->RemoveFile(c->level(), f->number);
  660. c->edit()->AddFile(c->level() + 1, f->number, f->file_size, f->smallest,
  661. f->largest);
  662. status = versions_->LogAndApply(c->edit(), &mutex_);
  663. if (!status.ok()) {
  664. RecordBackgroundError(status);
  665. }
  666. VersionSet::LevelSummaryStorage tmp;
  667. Log(options_.info_log, "Moved #%lld to level-%d %lld bytes %s: %s\n",
  668. static_cast<unsigned long long>(f->number), c->level() + 1,
  669. static_cast<unsigned long long>(f->file_size),
  670. status.ToString().c_str(), versions_->LevelSummary(&tmp));
  671. } else {
  672. CompactionState* compact = new CompactionState(c);
  673. status = DoCompactionWork(compact);
  674. if (!status.ok()) {
  675. RecordBackgroundError(status);
  676. }
  677. CleanupCompaction(compact);
  678. c->ReleaseInputs();
  679. RemoveObsoleteFiles();
  680. }
  681. delete c;
  682. if (status.ok()) {
  683. // Done
  684. } else if (shutting_down_.load(std::memory_order_acquire)) {
  685. // Ignore compaction errors found during shutting down
  686. } else {
  687. Log(options_.info_log, "Compaction error: %s", status.ToString().c_str());
  688. }
  689. if (is_manual) {
  690. ManualCompaction* m = manual_compaction_;
  691. if (!status.ok()) {
  692. m->done = true;
  693. }
  694. if (!m->done) {
  695. // We only compacted part of the requested range. Update *m
  696. // to the range that is left to be compacted.
  697. m->tmp_storage = manual_end;
  698. m->begin = &m->tmp_storage;
  699. }
  700. manual_compaction_ = nullptr;
  701. }
  702. }
  703. void DBImpl::CleanupCompaction(CompactionState* compact) {
  704. mutex_.AssertHeld();
  705. if (compact->builder != nullptr) {
  706. // May happen if we get a shutdown call in the middle of compaction
  707. compact->builder->Abandon();
  708. delete compact->builder;
  709. } else {
  710. assert(compact->outfile == nullptr);
  711. }
  712. delete compact->outfile;
  713. for (size_t i = 0; i < compact->outputs.size(); i++) {
  714. const CompactionState::Output& out = compact->outputs[i];
  715. pending_outputs_.erase(out.number);
  716. }
  717. delete compact;
  718. }
  719. Status DBImpl::OpenCompactionOutputFile(CompactionState* compact) {
  720. assert(compact != nullptr);
  721. assert(compact->builder == nullptr);
  722. uint64_t file_number;
  723. {
  724. mutex_.Lock();
  725. file_number = versions_->NewFileNumber();
  726. pending_outputs_.insert(file_number);
  727. CompactionState::Output out;
  728. out.number = file_number;
  729. out.smallest.Clear();
  730. out.largest.Clear();
  731. compact->outputs.push_back(out);
  732. mutex_.Unlock();
  733. }
  734. // Make the output file
  735. std::string fname = TableFileName(dbname_, file_number);
  736. Status s = env_->NewWritableFile(fname, &compact->outfile);
  737. if (s.ok()) {
  738. compact->builder = new TableBuilder(options_, compact->outfile);
  739. }
  740. return s;
  741. }
  742. Status DBImpl::FinishCompactionOutputFile(CompactionState* compact,
  743. Iterator* input) {
  744. assert(compact != nullptr);
  745. assert(compact->outfile != nullptr);
  746. assert(compact->builder != nullptr);
  747. const uint64_t output_number = compact->current_output()->number;
  748. assert(output_number != 0);
  749. // Check for iterator errors
  750. Status s = input->status();
  751. const uint64_t current_entries = compact->builder->NumEntries();
  752. if (s.ok()) {
  753. s = compact->builder->Finish();
  754. } else {
  755. compact->builder->Abandon();
  756. }
  757. const uint64_t current_bytes = compact->builder->FileSize();
  758. compact->current_output()->file_size = current_bytes;
  759. compact->total_bytes += current_bytes;
  760. delete compact->builder;
  761. compact->builder = nullptr;
  762. // Finish and check for file errors
  763. if (s.ok()) {
  764. s = compact->outfile->Sync();
  765. }
  766. if (s.ok()) {
  767. s = compact->outfile->Close();
  768. }
  769. delete compact->outfile;
  770. compact->outfile = nullptr;
  771. if (s.ok() && current_entries > 0) {
  772. // Verify that the table is usable
  773. Iterator* iter =
  774. table_cache_->NewIterator(ReadOptions(), output_number, current_bytes);
  775. s = iter->status();
  776. delete iter;
  777. if (s.ok()) {
  778. Log(options_.info_log, "Generated table #%llu@%d: %lld keys, %lld bytes",
  779. (unsigned long long)output_number, compact->compaction->level(),
  780. (unsigned long long)current_entries,
  781. (unsigned long long)current_bytes);
  782. }
  783. }
  784. return s;
  785. }
  786. Status DBImpl::InstallCompactionResults(CompactionState* compact) {
  787. mutex_.AssertHeld();
  788. Log(options_.info_log, "Compacted %d@%d + %d@%d files => %lld bytes",
  789. compact->compaction->num_input_files(0), compact->compaction->level(),
  790. compact->compaction->num_input_files(1), compact->compaction->level() + 1,
  791. static_cast<long long>(compact->total_bytes));
  792. // Add compaction outputs
  793. compact->compaction->AddInputDeletions(compact->compaction->edit());
  794. const int level = compact->compaction->level();
  795. for (size_t i = 0; i < compact->outputs.size(); i++) {
  796. const CompactionState::Output& out = compact->outputs[i];
  797. compact->compaction->edit()->AddFile(level + 1, out.number, out.file_size,
  798. out.smallest, out.largest);
  799. }
  800. return versions_->LogAndApply(compact->compaction->edit(), &mutex_);
  801. }
  802. Status DBImpl::DoCompactionWork(CompactionState* compact) {
  803. const uint64_t start_micros = env_->NowMicros();
  804. int64_t imm_micros = 0; // Micros spent doing imm_ compactions
  805. Log(options_.info_log, "Compacting %d@%d + %d@%d files",
  806. compact->compaction->num_input_files(0), compact->compaction->level(),
  807. compact->compaction->num_input_files(1),
  808. compact->compaction->level() + 1);
  809. assert(versions_->NumLevelFiles(compact->compaction->level()) > 0);
  810. assert(compact->builder == nullptr);
  811. assert(compact->outfile == nullptr);
  812. if (snapshots_.empty()) {
  813. compact->smallest_snapshot = versions_->LastSequence();
  814. } else {
  815. compact->smallest_snapshot = snapshots_.oldest()->sequence_number();
  816. }
  817. Iterator* input = versions_->MakeInputIterator(compact->compaction);
  818. // Release mutex while we're actually doing the compaction work
  819. mutex_.Unlock();
  820. input->SeekToFirst();
  821. Status status;
  822. ParsedInternalKey ikey;
  823. std::string current_user_key;
  824. bool has_current_user_key = false;
  825. SequenceNumber last_sequence_for_key = kMaxSequenceNumber;
  826. while (input->Valid() && !shutting_down_.load(std::memory_order_acquire)) {
  827. // Prioritize immutable compaction work
  828. if (has_imm_.load(std::memory_order_relaxed)) {
  829. const uint64_t imm_start = env_->NowMicros();
  830. mutex_.Lock();
  831. if (imm_ != nullptr) {
  832. CompactMemTable();
  833. // Wake up MakeRoomForWrite() if necessary.
  834. background_work_finished_signal_.SignalAll();
  835. }
  836. mutex_.Unlock();
  837. imm_micros += (env_->NowMicros() - imm_start);
  838. }
  839. Slice key = input->key();
  840. if (compact->compaction->ShouldStopBefore(key) &&
  841. compact->builder != nullptr) {
  842. status = FinishCompactionOutputFile(compact, input);
  843. if (!status.ok()) {
  844. break;
  845. }
  846. }
  847. // Handle key/value, add to state, etc.
  848. bool drop = false;
  849. if (!ParseInternalKey(key, &ikey)) {
  850. // Do not hide error keys
  851. current_user_key.clear();
  852. has_current_user_key = false;
  853. last_sequence_for_key = kMaxSequenceNumber;
  854. } else {
  855. if (!has_current_user_key ||
  856. user_comparator()->Compare(ikey.user_key, Slice(current_user_key)) !=
  857. 0) {
  858. // First occurrence of this user key
  859. current_user_key.assign(ikey.user_key.data(), ikey.user_key.size());
  860. has_current_user_key = true;
  861. last_sequence_for_key = kMaxSequenceNumber;
  862. }
  863. if (last_sequence_for_key <= compact->smallest_snapshot) {
  864. // Hidden by an newer entry for same user key
  865. drop = true; // (A)
  866. } else if (ikey.type == kTypeDeletion &&
  867. ikey.sequence <= compact->smallest_snapshot &&
  868. compact->compaction->IsBaseLevelForKey(ikey.user_key)) {
  869. // For this user key:
  870. // (1) there is no data in higher levels
  871. // (2) data in lower levels will have larger sequence numbers
  872. // (3) data in layers that are being compacted here and have
  873. // smaller sequence numbers will be dropped in the next
  874. // few iterations of this loop (by rule (A) above).
  875. // Therefore this deletion marker is obsolete and can be dropped.
  876. drop = true;
  877. }
  878. last_sequence_for_key = ikey.sequence;
  879. }
  880. #if 0
  881. Log(options_.info_log,
  882. " Compact: %s, seq %d, type: %d %d, drop: %d, is_base: %d, "
  883. "%d smallest_snapshot: %d",
  884. ikey.user_key.ToString().c_str(),
  885. (int)ikey.sequence, ikey.type, kTypeValue, drop,
  886. compact->compaction->IsBaseLevelForKey(ikey.user_key),
  887. (int)last_sequence_for_key, (int)compact->smallest_snapshot);
  888. #endif
  889. if (!drop) {
  890. // Open output file if necessary
  891. if (compact->builder == nullptr) {
  892. status = OpenCompactionOutputFile(compact);
  893. if (!status.ok()) {
  894. break;
  895. }
  896. }
  897. if (compact->builder->NumEntries() == 0) {
  898. compact->current_output()->smallest.DecodeFrom(key);
  899. }
  900. compact->current_output()->largest.DecodeFrom(key);
  901. compact->builder->Add(key, input->value());
  902. // Close output file if it is big enough
  903. if (compact->builder->FileSize() >=
  904. compact->compaction->MaxOutputFileSize()) {
  905. status = FinishCompactionOutputFile(compact, input);
  906. if (!status.ok()) {
  907. break;
  908. }
  909. }
  910. }
  911. input->Next();
  912. }
  913. if (status.ok() && shutting_down_.load(std::memory_order_acquire)) {
  914. status = Status::IOError("Deleting DB during compaction");
  915. }
  916. if (status.ok() && compact->builder != nullptr) {
  917. status = FinishCompactionOutputFile(compact, input);
  918. }
  919. if (status.ok()) {
  920. status = input->status();
  921. }
  922. delete input;
  923. input = nullptr;
  924. CompactionStats stats;
  925. stats.micros = env_->NowMicros() - start_micros - imm_micros;
  926. for (int which = 0; which < 2; which++) {
  927. for (int i = 0; i < compact->compaction->num_input_files(which); i++) {
  928. stats.bytes_read += compact->compaction->input(which, i)->file_size;
  929. }
  930. }
  931. for (size_t i = 0; i < compact->outputs.size(); i++) {
  932. stats.bytes_written += compact->outputs[i].file_size;
  933. }
  934. mutex_.Lock();
  935. stats_[compact->compaction->level() + 1].Add(stats);
  936. if (status.ok()) {
  937. status = InstallCompactionResults(compact);
  938. }
  939. if (!status.ok()) {
  940. RecordBackgroundError(status);
  941. }
  942. VersionSet::LevelSummaryStorage tmp;
  943. Log(options_.info_log, "compacted to: %s", versions_->LevelSummary(&tmp));
  944. return status;
  945. }
  946. namespace {
  947. struct IterState {
  948. port::Mutex* const mu;
  949. Version* const version GUARDED_BY(mu);
  950. MemTable* const mem GUARDED_BY(mu);
  951. MemTable* const imm GUARDED_BY(mu);
  952. IterState(port::Mutex* mutex, MemTable* mem, MemTable* imm, Version* version)
  953. : mu(mutex), version(version), mem(mem), imm(imm) {}
  954. };
  955. static void CleanupIteratorState(void* arg1, void* arg2) {
  956. IterState* state = reinterpret_cast<IterState*>(arg1);
  957. state->mu->Lock();
  958. state->mem->Unref();
  959. if (state->imm != nullptr) state->imm->Unref();
  960. state->version->Unref();
  961. state->mu->Unlock();
  962. delete state;
  963. }
  964. } // anonymous namespace
  965. Iterator* DBImpl::NewInternalIterator(const ReadOptions& options,
  966. SequenceNumber* latest_snapshot,
  967. uint32_t* seed) {
  968. mutex_.Lock();
  969. *latest_snapshot = versions_->LastSequence();
  970. // Collect together all needed child iterators
  971. std::vector<Iterator*> list;
  972. list.push_back(mem_->NewIterator());
  973. mem_->Ref();
  974. if (imm_ != nullptr) {
  975. list.push_back(imm_->NewIterator());
  976. imm_->Ref();
  977. }
  978. versions_->current()->AddIterators(options, &list);
  979. Iterator* internal_iter =
  980. NewMergingIterator(&internal_comparator_, &list[0], list.size());
  981. versions_->current()->Ref();
  982. IterState* cleanup = new IterState(&mutex_, mem_, imm_, versions_->current());
  983. internal_iter->RegisterCleanup(CleanupIteratorState, cleanup, nullptr);
  984. *seed = ++seed_;
  985. mutex_.Unlock();
  986. return internal_iter;
  987. }
  988. Iterator* DBImpl::TEST_NewInternalIterator() {
  989. SequenceNumber ignored;
  990. uint32_t ignored_seed;
  991. return NewInternalIterator(ReadOptions(), &ignored, &ignored_seed);
  992. }
  993. int64_t DBImpl::TEST_MaxNextLevelOverlappingBytes() {
  994. MutexLock l(&mutex_);
  995. return versions_->MaxNextLevelOverlappingBytes();
  996. }
  997. Status DBImpl::Get(const ReadOptions& options, const Slice& key,
  998. std::string* value) {
  999. Status s;
  1000. MutexLock l(&mutex_);
  1001. SequenceNumber snapshot;
  1002. if (options.snapshot != nullptr) {
  1003. snapshot =
  1004. static_cast<const SnapshotImpl*>(options.snapshot)->sequence_number();
  1005. } else {
  1006. snapshot = versions_->LastSequence();
  1007. }
  1008. MemTable* mem = mem_;
  1009. MemTable* imm = imm_;
  1010. Version* current = versions_->current();
  1011. mem->Ref();
  1012. if (imm != nullptr) imm->Ref();
  1013. current->Ref();
  1014. bool have_stat_update = false;
  1015. Version::GetStats stats;
  1016. // Unlock while reading from files and memtables
  1017. {
  1018. mutex_.Unlock();
  1019. // First look in the memtable, then in the immutable memtable (if any).
  1020. LookupKey lkey(key, snapshot);
  1021. if (mem->Get(lkey, value, &s)) {
  1022. // Done
  1023. } else if (imm != nullptr && imm->Get(lkey, value, &s)) {
  1024. // Done
  1025. } else {
  1026. s = current->Get(options, lkey, value, &stats);
  1027. have_stat_update = true;
  1028. }
  1029. mutex_.Lock();
  1030. }
  1031. if (have_stat_update && current->UpdateStats(stats)) {
  1032. MaybeScheduleCompaction();
  1033. }
  1034. mem->Unref();
  1035. if (imm != nullptr) imm->Unref();
  1036. current->Unref();
  1037. return s;
  1038. }
  1039. Iterator* DBImpl::NewIterator(const ReadOptions& options) {
  1040. SequenceNumber latest_snapshot;
  1041. uint32_t seed;
  1042. Iterator* iter = NewInternalIterator(options, &latest_snapshot, &seed);
  1043. return NewDBIterator(this, user_comparator(), iter,
  1044. (options.snapshot != nullptr
  1045. ? static_cast<const SnapshotImpl*>(options.snapshot)
  1046. ->sequence_number()
  1047. : latest_snapshot),
  1048. seed);
  1049. }
  1050. void DBImpl::RecordReadSample(Slice key) {
  1051. MutexLock l(&mutex_);
  1052. if (versions_->current()->RecordReadSample(key)) {
  1053. MaybeScheduleCompaction();
  1054. }
  1055. }
  1056. const Snapshot* DBImpl::GetSnapshot() {
  1057. MutexLock l(&mutex_);
  1058. return snapshots_.New(versions_->LastSequence());
  1059. }
  1060. void DBImpl::ReleaseSnapshot(const Snapshot* snapshot) {
  1061. MutexLock l(&mutex_);
  1062. snapshots_.Delete(static_cast<const SnapshotImpl*>(snapshot));
  1063. }
  1064. // Convenience methods
  1065. Status DBImpl::Put(const WriteOptions& o, const Slice& key, const Slice& val) {
  1066. return DB::Put(o, key, val);
  1067. }
  1068. Status DBImpl::Delete(const WriteOptions& options, const Slice& key) {
  1069. return DB::Delete(options, key);
  1070. }
  1071. Status DBImpl::Write(const WriteOptions& options, WriteBatch* updates) {
  1072. Writer w(&mutex_);
  1073. w.batch = updates;
  1074. w.sync = options.sync;
  1075. w.done = false;
  1076. MutexLock l(&mutex_);
  1077. writers_.push_back(&w);
  1078. while (!w.done && &w != writers_.front()) {
  1079. w.cv.Wait();
  1080. }
  1081. if (w.done) {
  1082. return w.status;
  1083. }
  1084. // May temporarily unlock and wait.
  1085. Status status = MakeRoomForWrite(updates == nullptr);
  1086. uint64_t last_sequence = versions_->LastSequence();
  1087. Writer* last_writer = &w;
  1088. if (status.ok() && updates != nullptr) { // nullptr batch is for compactions
  1089. WriteBatch* write_batch = BuildBatchGroup(&last_writer);
  1090. WriteBatchInternal::SetSequence(write_batch, last_sequence + 1);
  1091. last_sequence += WriteBatchInternal::Count(write_batch);
  1092. // Add to log and apply to memtable. We can release the lock
  1093. // during this phase since &w is currently responsible for logging
  1094. // and protects against concurrent loggers and concurrent writes
  1095. // into mem_.
  1096. {
  1097. mutex_.Unlock();
  1098. status = log_->AddRecord(WriteBatchInternal::Contents(write_batch));
  1099. bool sync_error = false;
  1100. if (status.ok() && options.sync) {
  1101. status = logfile_->Sync();
  1102. if (!status.ok()) {
  1103. sync_error = true;
  1104. }
  1105. }
  1106. if (status.ok()) {
  1107. status = WriteBatchInternal::InsertInto(write_batch, mem_);
  1108. }
  1109. mutex_.Lock();
  1110. if (sync_error) {
  1111. // The state of the log file is indeterminate: the log record we
  1112. // just added may or may not show up when the DB is re-opened.
  1113. // So we force the DB into a mode where all future writes fail.
  1114. RecordBackgroundError(status);
  1115. }
  1116. }
  1117. if (write_batch == tmp_batch_) tmp_batch_->Clear();
  1118. versions_->SetLastSequence(last_sequence);
  1119. }
  1120. while (true) {
  1121. Writer* ready = writers_.front();
  1122. writers_.pop_front();
  1123. if (ready != &w) {
  1124. ready->status = status;
  1125. ready->done = true;
  1126. ready->cv.Signal();
  1127. }
  1128. if (ready == last_writer) break;
  1129. }
  1130. // Notify new head of write queue
  1131. if (!writers_.empty()) {
  1132. writers_.front()->cv.Signal();
  1133. }
  1134. return status;
  1135. }
  1136. // REQUIRES: Writer list must be non-empty
  1137. // REQUIRES: First writer must have a non-null batch
  1138. WriteBatch* DBImpl::BuildBatchGroup(Writer** last_writer) {
  1139. mutex_.AssertHeld();
  1140. assert(!writers_.empty());
  1141. Writer* first = writers_.front();
  1142. WriteBatch* result = first->batch;
  1143. assert(result != nullptr);
  1144. size_t size = WriteBatchInternal::ByteSize(first->batch);
  1145. // Allow the group to grow up to a maximum size, but if the
  1146. // original write is small, limit the growth so we do not slow
  1147. // down the small write too much.
  1148. size_t max_size = 1 << 20;
  1149. if (size <= (128 << 10)) {
  1150. max_size = size + (128 << 10);
  1151. }
  1152. *last_writer = first;
  1153. std::deque<Writer*>::iterator iter = writers_.begin();
  1154. ++iter; // Advance past "first"
  1155. for (; iter != writers_.end(); ++iter) {
  1156. Writer* w = *iter;
  1157. if (w->sync && !first->sync) {
  1158. // Do not include a sync write into a batch handled by a non-sync write.
  1159. break;
  1160. }
  1161. if (w->batch != nullptr) {
  1162. size += WriteBatchInternal::ByteSize(w->batch);
  1163. if (size > max_size) {
  1164. // Do not make batch too big
  1165. break;
  1166. }
  1167. // Append to *result
  1168. if (result == first->batch) {
  1169. // Switch to temporary batch instead of disturbing caller's batch
  1170. result = tmp_batch_;
  1171. assert(WriteBatchInternal::Count(result) == 0);
  1172. WriteBatchInternal::Append(result, first->batch);
  1173. }
  1174. WriteBatchInternal::Append(result, w->batch);
  1175. }
  1176. *last_writer = w;
  1177. }
  1178. return result;
  1179. }
  1180. // REQUIRES: mutex_ is held
  1181. // REQUIRES: this thread is currently at the front of the writer queue
  1182. Status DBImpl::MakeRoomForWrite(bool force) {
  1183. mutex_.AssertHeld();
  1184. assert(!writers_.empty());
  1185. bool allow_delay = !force;
  1186. Status s;
  1187. while (true) {
  1188. if (!bg_error_.ok()) {
  1189. // Yield previous error
  1190. s = bg_error_;
  1191. break;
  1192. } else if (allow_delay && versions_->NumLevelFiles(0) >=
  1193. config::kL0_SlowdownWritesTrigger) {
  1194. // We are getting close to hitting a hard limit on the number of
  1195. // L0 files. Rather than delaying a single write by several
  1196. // seconds when we hit the hard limit, start delaying each
  1197. // individual write by 1ms to reduce latency variance. Also,
  1198. // this delay hands over some CPU to the compaction thread in
  1199. // case it is sharing the same core as the writer.
  1200. mutex_.Unlock();
  1201. env_->SleepForMicroseconds(1000);
  1202. allow_delay = false; // Do not delay a single write more than once
  1203. mutex_.Lock();
  1204. } else if (!force &&
  1205. (mem_->ApproximateMemoryUsage() <= options_.write_buffer_size)) {
  1206. // There is room in current memtable
  1207. break;
  1208. } else if (imm_ != nullptr) {
  1209. // We have filled up the current memtable, but the previous
  1210. // one is still being compacted, so we wait.
  1211. Log(options_.info_log, "Current memtable full; waiting...\n");
  1212. background_work_finished_signal_.Wait();
  1213. } else if (versions_->NumLevelFiles(0) >= config::kL0_StopWritesTrigger) {
  1214. // There are too many level-0 files.
  1215. Log(options_.info_log, "Too many L0 files; waiting...\n");
  1216. background_work_finished_signal_.Wait();
  1217. } else {
  1218. // Attempt to switch to a new memtable and trigger compaction of old
  1219. assert(versions_->PrevLogNumber() == 0);
  1220. uint64_t new_log_number = versions_->NewFileNumber();
  1221. WritableFile* lfile = nullptr;
  1222. s = env_->NewWritableFile(LogFileName(dbname_, new_log_number), &lfile);
  1223. if (!s.ok()) {
  1224. // Avoid chewing through file number space in a tight loop.
  1225. versions_->ReuseFileNumber(new_log_number);
  1226. break;
  1227. }
  1228. delete log_;
  1229. s = logfile_->Close();
  1230. if (!s.ok()) {
  1231. // We may have lost some data written to the previous log file.
  1232. // Switch to the new log file anyway, but record as a background
  1233. // error so we do not attempt any more writes.
  1234. //
  1235. // We could perhaps attempt to save the memtable corresponding
  1236. // to log file and suppress the error if that works, but that
  1237. // would add more complexity in a critical code path.
  1238. RecordBackgroundError(s);
  1239. }
  1240. delete logfile_;
  1241. logfile_ = lfile;
  1242. logfile_number_ = new_log_number;
  1243. log_ = new log::Writer(lfile);
  1244. imm_ = mem_;
  1245. has_imm_.store(true, std::memory_order_release);
  1246. mem_ = new MemTable(internal_comparator_);
  1247. mem_->Ref();
  1248. force = false; // Do not force another compaction if have room
  1249. MaybeScheduleCompaction();
  1250. }
  1251. }
  1252. return s;
  1253. }
  1254. bool DBImpl::GetProperty(const Slice& property, std::string* value) {
  1255. value->clear();
  1256. MutexLock l(&mutex_);
  1257. Slice in = property;
  1258. Slice prefix("leveldb.");
  1259. if (!in.starts_with(prefix)) return false;
  1260. in.remove_prefix(prefix.size());
  1261. if (in.starts_with("num-files-at-level")) {
  1262. in.remove_prefix(strlen("num-files-at-level"));
  1263. uint64_t level;
  1264. bool ok = ConsumeDecimalNumber(&in, &level) && in.empty();
  1265. if (!ok || level >= config::kNumLevels) {
  1266. return false;
  1267. } else {
  1268. char buf[100];
  1269. std::snprintf(buf, sizeof(buf), "%d",
  1270. versions_->NumLevelFiles(static_cast<int>(level)));
  1271. *value = buf;
  1272. return true;
  1273. }
  1274. } else if (in == "stats") {
  1275. char buf[200];
  1276. std::snprintf(buf, sizeof(buf),
  1277. " Compactions\n"
  1278. "Level Files Size(MB) Time(sec) Read(MB) Write(MB)\n"
  1279. "--------------------------------------------------\n");
  1280. value->append(buf);
  1281. for (int level = 0; level < config::kNumLevels; level++) {
  1282. int files = versions_->NumLevelFiles(level);
  1283. if (stats_[level].micros > 0 || files > 0) {
  1284. std::snprintf(buf, sizeof(buf), "%3d %8d %8.0f %9.0f %8.0f %9.0f\n",
  1285. level, files, versions_->NumLevelBytes(level) / 1048576.0,
  1286. stats_[level].micros / 1e6,
  1287. stats_[level].bytes_read / 1048576.0,
  1288. stats_[level].bytes_written / 1048576.0);
  1289. value->append(buf);
  1290. }
  1291. }
  1292. return true;
  1293. } else if (in == "sstables") {
  1294. *value = versions_->current()->DebugString();
  1295. return true;
  1296. } else if (in == "approximate-memory-usage") {
  1297. size_t total_usage = options_.block_cache->TotalCharge();
  1298. if (mem_) {
  1299. total_usage += mem_->ApproximateMemoryUsage();
  1300. }
  1301. if (imm_) {
  1302. total_usage += imm_->ApproximateMemoryUsage();
  1303. }
  1304. char buf[50];
  1305. std::snprintf(buf, sizeof(buf), "%llu",
  1306. static_cast<unsigned long long>(total_usage));
  1307. value->append(buf);
  1308. return true;
  1309. }
  1310. return false;
  1311. }
  1312. void DBImpl::GetApproximateSizes(const Range* range, int n, uint64_t* sizes) {
  1313. // TODO(opt): better implementation
  1314. MutexLock l(&mutex_);
  1315. Version* v = versions_->current();
  1316. v->Ref();
  1317. for (int i = 0; i < n; i++) {
  1318. // Convert user_key into a corresponding internal key.
  1319. InternalKey k1(range[i].start, kMaxSequenceNumber, kValueTypeForSeek);
  1320. InternalKey k2(range[i].limit, kMaxSequenceNumber, kValueTypeForSeek);
  1321. uint64_t start = versions_->ApproximateOffsetOf(v, k1);
  1322. uint64_t limit = versions_->ApproximateOffsetOf(v, k2);
  1323. sizes[i] = (limit >= start ? limit - start : 0);
  1324. }
  1325. v->Unref();
  1326. }
  1327. // Default implementations of convenience methods that subclasses of DB
  1328. // can call if they wish
  1329. Status DB::Put(const WriteOptions& opt, const Slice& key, const Slice& value) {
  1330. WriteBatch batch;
  1331. batch.Put(key, value);
  1332. return Write(opt, &batch);
  1333. }
  1334. Status DB::Delete(const WriteOptions& opt, const Slice& key) {
  1335. WriteBatch batch;
  1336. batch.Delete(key);
  1337. return Write(opt, &batch);
  1338. }
  1339. DB::~DB() = default;
  1340. Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
  1341. *dbptr = nullptr;
  1342. DBImpl* impl = new DBImpl(options, dbname);
  1343. impl->mutex_.Lock();
  1344. VersionEdit edit;
  1345. // Recover handles create_if_missing, error_if_exists
  1346. bool save_manifest = false;
  1347. Status s = impl->Recover(&edit, &save_manifest);
  1348. if (s.ok() && impl->mem_ == nullptr) {
  1349. // Create new log and a corresponding memtable.
  1350. uint64_t new_log_number = impl->versions_->NewFileNumber();
  1351. WritableFile* lfile;
  1352. s = options.env->NewWritableFile(LogFileName(dbname, new_log_number),
  1353. &lfile);
  1354. if (s.ok()) {
  1355. edit.SetLogNumber(new_log_number);
  1356. impl->logfile_ = lfile;
  1357. impl->logfile_number_ = new_log_number;
  1358. impl->log_ = new log::Writer(lfile);
  1359. impl->mem_ = new MemTable(impl->internal_comparator_);
  1360. impl->mem_->Ref();
  1361. }
  1362. }
  1363. if (s.ok() && save_manifest) {
  1364. edit.SetPrevLogNumber(0); // No older logs needed after recovery.
  1365. edit.SetLogNumber(impl->logfile_number_);
  1366. s = impl->versions_->LogAndApply(&edit, &impl->mutex_);
  1367. }
  1368. if (s.ok()) {
  1369. impl->RemoveObsoleteFiles();
  1370. impl->MaybeScheduleCompaction();
  1371. }
  1372. impl->mutex_.Unlock();
  1373. if (s.ok()) {
  1374. assert(impl->mem_ != nullptr);
  1375. *dbptr = impl;
  1376. } else {
  1377. delete impl;
  1378. }
  1379. return s;
  1380. }
  1381. Snapshot::~Snapshot() = default;
  1382. Status DestroyDB(const std::string& dbname, const Options& options) {
  1383. Env* env = options.env;
  1384. std::vector<std::string> filenames;
  1385. Status result = env->GetChildren(dbname, &filenames);
  1386. if (!result.ok()) {
  1387. // Ignore error in case directory does not exist
  1388. return Status::OK();
  1389. }
  1390. FileLock* lock;
  1391. const std::string lockname = LockFileName(dbname);
  1392. result = env->LockFile(lockname, &lock);
  1393. if (result.ok()) {
  1394. uint64_t number;
  1395. FileType type;
  1396. for (size_t i = 0; i < filenames.size(); i++) {
  1397. if (ParseFileName(filenames[i], &number, &type) &&
  1398. type != kDBLockFile) { // Lock file will be deleted at end
  1399. Status del = env->RemoveFile(dbname + "/" + filenames[i]);
  1400. if (result.ok() && !del.ok()) {
  1401. result = del;
  1402. }
  1403. }
  1404. }
  1405. env->UnlockFile(lock); // Ignore error since state is already gone
  1406. env->RemoveFile(lockname);
  1407. env->RemoveDir(dbname); // Ignore error in case dir contains other files
  1408. }
  1409. return result;
  1410. }
  1411. } // namespace leveldb