作者: 韩晨旭 10225101440 李畅 10225102463
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1631 lines
50 KiB

Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
пре 10 година
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
пре 2 недеља
пре 2 недеља
пре 2 недеља
пре 2 недеља
пре 2 недеља
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
пре 2 недеља
пре 2 недеља
пре 2 недеља
пре 2 недеља
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
пре 10 година
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
пре 4 година
  1. // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
  2. // Use of this source code is governed by a BSD-style license that can be
  3. // found in the LICENSE file. See the AUTHORS file for names of contributors.
  4. #include "db/db_impl.h"
  5. #include "db/builder.h"
  6. #include "db/db_iter.h"
  7. #include "db/dbformat.h"
  8. #include "db/filename.h"
  9. #include "db/log_reader.h"
  10. #include "db/log_writer.h"
  11. #include "db/memtable.h"
  12. #include "db/table_cache.h"
  13. #include "db/version_set.h"
  14. #include "db/write_batch_internal.h"
  15. #include <algorithm>
  16. #include <atomic>
  17. #include <cstdint>
  18. #include <cstdio>
  19. #include <iostream>
  20. #include <ostream>
  21. #include <set>
  22. #include <string>
  23. #include <vector>
  24. #include "leveldb/db.h"
  25. #include "leveldb/env.h"
  26. #include "leveldb/status.h"
  27. #include "leveldb/table.h"
  28. #include "leveldb/table_builder.h"
  29. #include "port/port.h"
  30. #include "table/block.h"
  31. #include "table/merger.h"
  32. #include "table/two_level_iterator.h"
  33. #include "util/coding.h"
  34. #include "util/logging.h"
  35. #include "util/mutexlock.h"
  36. namespace leveldb {
  37. const int kNumNonTableCacheFiles = 10;
  38. // Information kept for every waiting writer
  39. struct DBImpl::Writer {
  40. explicit Writer(port::Mutex* mu)
  41. : batch(nullptr), sync(false), done(false), cv(mu) {}
  42. Status status;
  43. WriteBatch* batch;
  44. bool sync;
  45. bool done;
  46. port::CondVar cv;
  47. };
  48. struct DBImpl::CompactionState {
  49. // Files produced by compaction
  50. struct Output {
  51. uint64_t number;
  52. uint64_t file_size;
  53. InternalKey smallest, largest;
  54. };
  55. Output* current_output() { return &outputs[outputs.size() - 1]; }
  56. explicit CompactionState(Compaction* c)
  57. : compaction(c),
  58. smallest_snapshot(0),
  59. outfile(nullptr),
  60. builder(nullptr),
  61. total_bytes(0) {}
  62. Compaction* const compaction;
  63. // Sequence numbers < smallest_snapshot are not significant since we
  64. // will never have to service a snapshot below smallest_snapshot.
  65. // Therefore if we have seen a sequence number S <= smallest_snapshot,
  66. // we can drop all entries for the same key with sequence numbers < S.
  67. SequenceNumber smallest_snapshot;
  68. std::vector<Output> outputs;
  69. // State kept for output being generated
  70. WritableFile* outfile;
  71. TableBuilder* builder;
  72. uint64_t total_bytes;
  73. };
  74. // Fix user-supplied options to be reasonable
  75. template <class T, class V>
  76. static void ClipToRange(T* ptr, V minvalue, V maxvalue) {
  77. if (static_cast<V>(*ptr) > maxvalue) *ptr = maxvalue;
  78. if (static_cast<V>(*ptr) < minvalue) *ptr = minvalue;
  79. }
  80. Options SanitizeOptions(const std::string& dbname,
  81. const InternalKeyComparator* icmp,
  82. const InternalFilterPolicy* ipolicy,
  83. const Options& src) {
  84. Options result = src;
  85. result.comparator = icmp;
  86. result.filter_policy = (src.filter_policy != nullptr) ? ipolicy : nullptr;
  87. ClipToRange(&result.max_open_files, 64 + kNumNonTableCacheFiles, 50000);
  88. ClipToRange(&result.write_buffer_size, 64 << 10, 1 << 30);
  89. ClipToRange(&result.max_file_size, 1 << 20, 1 << 30);
  90. ClipToRange(&result.block_size, 1 << 10, 4 << 20);
  91. if (result.info_log == nullptr) {
  92. // Open a log file in the same directory as the db
  93. src.env->CreateDir(dbname); // In case it does not exist
  94. src.env->RenameFile(InfoLogFileName(dbname), OldInfoLogFileName(dbname));
  95. Status s = src.env->NewLogger(InfoLogFileName(dbname), &result.info_log);
  96. if (!s.ok()) {
  97. // No place suitable for logging
  98. result.info_log = nullptr;
  99. }
  100. }
  101. if (result.block_cache == nullptr) {
  102. result.block_cache = NewLRUCache(8 << 20);
  103. }
  104. return result;
  105. }
  106. static int TableCacheSize(const Options& sanitized_options) {
  107. // Reserve ten files or so for other uses and give the rest to TableCache.
  108. return sanitized_options.max_open_files - kNumNonTableCacheFiles;
  109. }
  110. DBImpl::DBImpl(const Options& raw_options, const std::string& dbname)
  111. : env_(raw_options.env),
  112. internal_comparator_(raw_options.comparator),
  113. internal_filter_policy_(raw_options.filter_policy),
  114. options_(SanitizeOptions(dbname, &internal_comparator_,
  115. &internal_filter_policy_, raw_options)),
  116. owns_info_log_(options_.info_log != raw_options.info_log),
  117. owns_cache_(options_.block_cache != raw_options.block_cache),
  118. dbname_(dbname),
  119. table_cache_(new TableCache(dbname_, options_, TableCacheSize(options_))),
  120. db_lock_(nullptr),
  121. shutting_down_(false),
  122. background_work_finished_signal_(&mutex_),
  123. mem_(nullptr),
  124. imm_(nullptr),
  125. has_imm_(false),
  126. logfile_(nullptr),
  127. logfile_number_(0),
  128. log_(nullptr),
  129. seed_(0),
  130. tmp_batch_(new WriteBatch),
  131. background_compaction_scheduled_(false),
  132. manual_compaction_(nullptr),
  133. versions_(new VersionSet(dbname_, &options_, table_cache_,
  134. &internal_comparator_)) {}
  135. DBImpl::~DBImpl() {
  136. // Wait for background work to finish.
  137. mutex_.Lock();
  138. shutting_down_.store(true, std::memory_order_release);
  139. while (background_compaction_scheduled_) {
  140. background_work_finished_signal_.Wait();
  141. }
  142. mutex_.Unlock();
  143. if (db_lock_ != nullptr) {
  144. env_->UnlockFile(db_lock_);
  145. }
  146. delete versions_;
  147. if (mem_ != nullptr) mem_->Unref();
  148. if (imm_ != nullptr) imm_->Unref();
  149. delete tmp_batch_;
  150. delete log_;
  151. delete logfile_;
  152. delete table_cache_;
  153. if (owns_info_log_) {
  154. delete options_.info_log;
  155. }
  156. if (owns_cache_) {
  157. delete options_.block_cache;
  158. }
  159. }
  160. Status DBImpl::NewDB() {
  161. VersionEdit new_db;
  162. new_db.SetComparatorName(user_comparator()->Name());
  163. new_db.SetLogNumber(0);
  164. new_db.SetNextFile(2);
  165. new_db.SetLastSequence(0);
  166. const std::string manifest = DescriptorFileName(dbname_, 1);
  167. WritableFile* file;
  168. Status s = env_->NewWritableFile(manifest, &file);
  169. if (!s.ok()) {
  170. return s;
  171. }
  172. {
  173. log::Writer log(file);
  174. std::string record;
  175. new_db.EncodeTo(&record);
  176. s = log.AddRecord(record);
  177. if (s.ok()) {
  178. s = file->Sync();
  179. }
  180. if (s.ok()) {
  181. s = file->Close();
  182. }
  183. }
  184. delete file;
  185. if (s.ok()) {
  186. // Make "CURRENT" file that points to the new manifest file.
  187. s = SetCurrentFile(env_, dbname_, 1);
  188. } else {
  189. env_->RemoveFile(manifest);
  190. }
  191. return s;
  192. }
  193. void DBImpl::MaybeIgnoreError(Status* s) const {
  194. if (s->ok() || options_.paranoid_checks) {
  195. // No change needed
  196. } else {
  197. Log(options_.info_log, "Ignoring error %s", s->ToString().c_str());
  198. *s = Status::OK();
  199. }
  200. }
  201. void DBImpl::RemoveObsoleteFiles() {
  202. mutex_.AssertHeld();
  203. if (!bg_error_.ok()) {
  204. // After a background error, we don't know whether a new version may
  205. // or may not have been committed, so we cannot safely garbage collect.
  206. return;
  207. }
  208. // Make a set of all of the live files
  209. std::set<uint64_t> live = pending_outputs_;
  210. versions_->AddLiveFiles(&live);
  211. std::vector<std::string> filenames;
  212. env_->GetChildren(dbname_, &filenames); // Ignoring errors on purpose
  213. uint64_t number;
  214. FileType type;
  215. std::vector<std::string> files_to_delete;
  216. for (std::string& filename : filenames) {
  217. if (ParseFileName(filename, &number, &type)) {
  218. bool keep = true;
  219. switch (type) {
  220. case kLogFile:
  221. keep = ((number >= versions_->LogNumber()) ||
  222. (number == versions_->PrevLogNumber()));
  223. break;
  224. case kDescriptorFile:
  225. // Keep my manifest file, and any newer incarnations'
  226. // (in case there is a race that allows other incarnations)
  227. keep = (number >= versions_->ManifestFileNumber());
  228. break;
  229. case kTableFile:
  230. keep = (live.find(number) != live.end());
  231. break;
  232. case kTempFile:
  233. // Any temp files that are currently being written to must
  234. // be recorded in pending_outputs_, which is inserted into "live"
  235. keep = (live.find(number) != live.end());
  236. break;
  237. case kCurrentFile:
  238. case kDBLockFile:
  239. case kInfoLogFile:
  240. keep = true;
  241. break;
  242. }
  243. if (!keep) {
  244. files_to_delete.push_back(std::move(filename));
  245. if (type == kTableFile) {
  246. table_cache_->Evict(number);
  247. }
  248. Log(options_.info_log, "Delete type=%d #%lld\n", static_cast<int>(type),
  249. static_cast<unsigned long long>(number));
  250. }
  251. }
  252. }
  253. // While deleting all files unblock other threads. All files being deleted
  254. // have unique names which will not collide with newly created files and
  255. // are therefore safe to delete while allowing other threads to proceed.
  256. mutex_.Unlock();
  257. for (const std::string& filename : files_to_delete) {
  258. env_->RemoveFile(dbname_ + "/" + filename);
  259. }
  260. mutex_.Lock();
  261. }
  262. Status DBImpl::Recover(VersionEdit* edit, bool* save_manifest) {
  263. mutex_.AssertHeld();
  264. // Ignore error from CreateDir since the creation of the DB is
  265. // committed only when the descriptor is created, and this directory
  266. // may already exist from a previous failed creation attempt.
  267. env_->CreateDir(dbname_);
  268. assert(db_lock_ == nullptr);
  269. Status s = env_->LockFile(LockFileName(dbname_), &db_lock_);
  270. if (!s.ok()) {
  271. return s;
  272. }
  273. if (!env_->FileExists(CurrentFileName(dbname_))) {
  274. if (options_.create_if_missing) {
  275. Log(options_.info_log, "Creating DB %s since it was missing.",
  276. dbname_.c_str());
  277. s = NewDB();
  278. if (!s.ok()) {
  279. return s;
  280. }
  281. } else {
  282. return Status::InvalidArgument(
  283. dbname_, "does not exist (create_if_missing is false)");
  284. }
  285. } else {
  286. if (options_.error_if_exists) {
  287. return Status::InvalidArgument(dbname_,
  288. "exists (error_if_exists is true)");
  289. }
  290. }
  291. s = versions_->Recover(save_manifest);
  292. if (!s.ok()) {
  293. return s;
  294. }
  295. SequenceNumber max_sequence(0);
  296. // Recover from all newer log files than the ones named in the
  297. // descriptor (new log files may have been added by the previous
  298. // incarnation without registering them in the descriptor).
  299. //
  300. // Note that PrevLogNumber() is no longer used, but we pay
  301. // attention to it in case we are recovering a database
  302. // produced by an older version of leveldb.
  303. const uint64_t min_log = versions_->LogNumber();
  304. const uint64_t prev_log = versions_->PrevLogNumber();
  305. std::vector<std::string> filenames;
  306. s = env_->GetChildren(dbname_, &filenames);
  307. if (!s.ok()) {
  308. return s;
  309. }
  310. std::set<uint64_t> expected;
  311. versions_->AddLiveFiles(&expected);
  312. uint64_t number;
  313. FileType type;
  314. std::vector<uint64_t> logs;
  315. for (size_t i = 0; i < filenames.size(); i++) {
  316. if (ParseFileName(filenames[i], &number, &type)) {
  317. expected.erase(number);
  318. if (type == kLogFile && ((number >= min_log) || (number == prev_log)))
  319. logs.push_back(number);
  320. }
  321. }
  322. if (!expected.empty()) {
  323. char buf[50];
  324. std::snprintf(buf, sizeof(buf), "%d missing files; e.g.",
  325. static_cast<int>(expected.size()));
  326. return Status::Corruption(buf, TableFileName(dbname_, *(expected.begin())));
  327. }
  328. // Recover in the order in which the logs were generated
  329. std::sort(logs.begin(), logs.end());
  330. for (size_t i = 0; i < logs.size(); i++) {
  331. s = RecoverLogFile(logs[i], (i == logs.size() - 1), save_manifest, edit,
  332. &max_sequence);
  333. if (!s.ok()) {
  334. return s;
  335. }
  336. // The previous incarnation may not have written any MANIFEST
  337. // records after allocating this log number. So we manually
  338. // update the file number allocation counter in VersionSet.
  339. versions_->MarkFileNumberUsed(logs[i]);
  340. }
  341. if (versions_->LastSequence() < max_sequence) {
  342. versions_->SetLastSequence(max_sequence);
  343. }
  344. return Status::OK();
  345. }
  346. Status DBImpl::RecoverLogFile(uint64_t log_number, bool last_log,
  347. bool* save_manifest, VersionEdit* edit,
  348. SequenceNumber* max_sequence) {
  349. struct LogReporter : public log::Reader::Reporter {
  350. Env* env;
  351. Logger* info_log;
  352. const char* fname;
  353. Status* status; // null if options_.paranoid_checks==false
  354. void Corruption(size_t bytes, const Status& s) override {
  355. Log(info_log, "%s%s: dropping %d bytes; %s",
  356. (this->status == nullptr ? "(ignoring error) " : ""), fname,
  357. static_cast<int>(bytes), s.ToString().c_str());
  358. if (this->status != nullptr && this->status->ok()) *this->status = s;
  359. }
  360. };
  361. mutex_.AssertHeld();
  362. // Open the log file
  363. std::string fname = LogFileName(dbname_, log_number);
  364. SequentialFile* file;
  365. Status status = env_->NewSequentialFile(fname, &file);
  366. if (!status.ok()) {
  367. MaybeIgnoreError(&status);
  368. return status;
  369. }
  370. // Create the log reader.
  371. LogReporter reporter;
  372. reporter.env = env_;
  373. reporter.info_log = options_.info_log;
  374. reporter.fname = fname.c_str();
  375. reporter.status = (options_.paranoid_checks ? &status : nullptr);
  376. // We intentionally make log::Reader do checksumming even if
  377. // paranoid_checks==false so that corruptions cause entire commits
  378. // to be skipped instead of propagating bad information (like overly
  379. // large sequence numbers).
  380. log::Reader reader(file, &reporter, true /*checksum*/, 0 /*initial_offset*/);
  381. Log(options_.info_log, "Recovering log #%llu",
  382. (unsigned long long)log_number);
  383. // Read all the records and add to a memtable
  384. std::string scratch;
  385. Slice record;
  386. WriteBatch batch;
  387. int compactions = 0;
  388. MemTable* mem = nullptr;
  389. while (reader.ReadRecord(&record, &scratch) && status.ok()) {
  390. if (record.size() < 12) {
  391. reporter.Corruption(record.size(),
  392. Status::Corruption("log record too small"));
  393. continue;
  394. }
  395. WriteBatchInternal::SetContents(&batch, record);
  396. if (mem == nullptr) {
  397. mem = new MemTable(internal_comparator_);
  398. mem->Ref();
  399. }
  400. status = WriteBatchInternal::InsertInto(&batch, mem);
  401. MaybeIgnoreError(&status);
  402. if (!status.ok()) {
  403. break;
  404. }
  405. const SequenceNumber last_seq = WriteBatchInternal::Sequence(&batch) +
  406. WriteBatchInternal::Count(&batch) - 1;
  407. if (last_seq > *max_sequence) {
  408. *max_sequence = last_seq;
  409. }
  410. if (mem->ApproximateMemoryUsage() > options_.write_buffer_size) {
  411. compactions++;
  412. *save_manifest = true;
  413. status = WriteLevel0Table(mem, edit, nullptr);
  414. mem->Unref();
  415. mem = nullptr;
  416. if (!status.ok()) {
  417. // Reflect errors immediately so that conditions like full
  418. // file-systems cause the DB::Open() to fail.
  419. break;
  420. }
  421. }
  422. }
  423. delete file;
  424. // See if we should keep reusing the last log file.
  425. if (status.ok() && options_.reuse_logs && last_log && compactions == 0) {
  426. assert(logfile_ == nullptr);
  427. assert(log_ == nullptr);
  428. assert(mem_ == nullptr);
  429. uint64_t lfile_size;
  430. if (env_->GetFileSize(fname, &lfile_size).ok() &&
  431. env_->NewAppendableFile(fname, &logfile_).ok()) {
  432. Log(options_.info_log, "Reusing old log %s \n", fname.c_str());
  433. log_ = new log::Writer(logfile_, lfile_size);
  434. logfile_number_ = log_number;
  435. if (mem != nullptr) {
  436. mem_ = mem;
  437. mem = nullptr;
  438. } else {
  439. // mem can be nullptr if lognum exists but was empty.
  440. mem_ = new MemTable(internal_comparator_);
  441. mem_->Ref();
  442. }
  443. }
  444. }
  445. if (mem != nullptr) {
  446. // mem did not get reused; compact it.
  447. if (status.ok()) {
  448. *save_manifest = true;
  449. status = WriteLevel0Table(mem, edit, nullptr);
  450. }
  451. mem->Unref();
  452. }
  453. return status;
  454. }
  455. Status DBImpl::WriteLevel0Table(MemTable* mem, VersionEdit* edit,
  456. Version* base) {
  457. mutex_.AssertHeld();
  458. const uint64_t start_micros = env_->NowMicros();
  459. FileMetaData meta;
  460. meta.number = versions_->NewFileNumber();
  461. pending_outputs_.insert(meta.number);
  462. Iterator* iter = mem->NewIterator();
  463. Log(options_.info_log, "Level-0 table #%llu: started",
  464. (unsigned long long)meta.number);
  465. Status s;
  466. {
  467. mutex_.Unlock();
  468. s = BuildTable(dbname_, env_, options_, table_cache_, iter, &meta);
  469. mutex_.Lock();
  470. }
  471. Log(options_.info_log, "Level-0 table #%llu: %lld bytes %s",
  472. (unsigned long long)meta.number, (unsigned long long)meta.file_size,
  473. s.ToString().c_str());
  474. delete iter;
  475. pending_outputs_.erase(meta.number);
  476. // Note that if file_size is zero, the file has been deleted and
  477. // should not be added to the manifest.
  478. int level = 0;
  479. if (s.ok() && meta.file_size > 0) {
  480. const Slice min_user_key = meta.smallest.user_key();
  481. const Slice max_user_key = meta.largest.user_key();
  482. if (base != nullptr) {
  483. level = base->PickLevelForMemTableOutput(min_user_key, max_user_key);
  484. }
  485. edit->AddFile(level, meta.number, meta.file_size, meta.smallest,
  486. meta.largest);
  487. }
  488. CompactionStats stats;
  489. stats.micros = env_->NowMicros() - start_micros;
  490. stats.bytes_written = meta.file_size;
  491. stats_[level].Add(stats);
  492. return s;
  493. }
  494. void DBImpl::CompactMemTable() {
  495. mutex_.AssertHeld();
  496. assert(imm_ != nullptr);
  497. // Save the contents of the memtable as a new Table
  498. VersionEdit edit;
  499. Version* base = versions_->current();
  500. base->Ref();
  501. Status s = WriteLevel0Table(imm_, &edit, base);
  502. base->Unref();
  503. if (s.ok() && shutting_down_.load(std::memory_order_acquire)) {
  504. s = Status::IOError("Deleting DB during memtable compaction");
  505. }
  506. // Replace immutable memtable with the generated Table
  507. if (s.ok()) {
  508. edit.SetPrevLogNumber(0);
  509. edit.SetLogNumber(logfile_number_); // Earlier logs no longer needed
  510. s = versions_->LogAndApply(&edit, &mutex_);
  511. }
  512. if (s.ok()) {
  513. // Commit to the new state
  514. imm_->Unref();
  515. imm_ = nullptr;
  516. has_imm_.store(false, std::memory_order_release);
  517. RemoveObsoleteFiles();
  518. } else {
  519. RecordBackgroundError(s);
  520. }
  521. }
  522. void DBImpl::CompactRange(const Slice* begin, const Slice* end) {
  523. int max_level_with_files = 1;
  524. {
  525. MutexLock l(&mutex_);
  526. Version* base = versions_->current();
  527. for (int level = 1; level < config::kNumLevels; level++) {
  528. if (base->OverlapInLevel(level, begin, end)) {
  529. max_level_with_files = level;
  530. }
  531. }
  532. }
  533. TEST_CompactMemTable(); // TODO(sanjay): Skip if memtable does not overlap
  534. for (int level = 0; level < max_level_with_files + 1; level++) {
  535. TEST_CompactRange(level, begin, end, level == max_level_with_files);
  536. }
  537. }
  538. void DBImpl::TEST_CompactRange(int level, const Slice* begin,
  539. const Slice* end, bool is_last_level) {
  540. assert(level >= 0);
  541. assert(level < config::kNumLevels);
  542. InternalKey begin_storage, end_storage;
  543. ManualCompaction manual;
  544. manual.level = level;
  545. manual.is_last_level = is_last_level;
  546. manual.done = false;
  547. if (begin == nullptr) {
  548. manual.begin = nullptr;
  549. } else {
  550. begin_storage = InternalKey(*begin, kMaxSequenceNumber, kValueTypeForSeek);
  551. manual.begin = &begin_storage;
  552. }
  553. if (end == nullptr) {
  554. manual.end = nullptr;
  555. } else {
  556. end_storage = InternalKey(*end, 0, static_cast<ValueType>(0));
  557. manual.end = &end_storage;
  558. }
  559. MutexLock l(&mutex_);
  560. while (!manual.done && !shutting_down_.load(std::memory_order_acquire) &&
  561. bg_error_.ok()) {
  562. if (manual_compaction_ == nullptr) { // Idle
  563. manual_compaction_ = &manual;
  564. MaybeScheduleCompaction();
  565. } else { // Running either my compaction or another compaction.
  566. background_work_finished_signal_.Wait();
  567. }
  568. }
  569. // Finish current background compaction in the case where
  570. // `background_work_finished_signal_` was signalled due to an error.
  571. while (background_compaction_scheduled_) {
  572. background_work_finished_signal_.Wait();
  573. }
  574. if (manual_compaction_ == &manual) {
  575. // Cancel my manual compaction since we aborted early for some reason.
  576. manual_compaction_ = nullptr;
  577. }
  578. }
  579. Status DBImpl::TEST_CompactMemTable() {
  580. // nullptr batch means just wait for earlier writes to be done
  581. Status s = Write(WriteOptions(), nullptr);
  582. if (s.ok()) {
  583. // Wait until the compaction completes
  584. MutexLock l(&mutex_);
  585. while (imm_ != nullptr && bg_error_.ok()) {
  586. background_work_finished_signal_.Wait();
  587. }
  588. if (imm_ != nullptr) {
  589. s = bg_error_;
  590. }
  591. }
  592. return s;
  593. }
  594. void DBImpl::RecordBackgroundError(const Status& s) {
  595. mutex_.AssertHeld();
  596. if (bg_error_.ok()) {
  597. bg_error_ = s;
  598. background_work_finished_signal_.SignalAll();
  599. }
  600. }
  601. void DBImpl::MaybeScheduleCompaction() {
  602. mutex_.AssertHeld();
  603. if (background_compaction_scheduled_) {
  604. // Already scheduled
  605. } else if (shutting_down_.load(std::memory_order_acquire)) {
  606. // DB is being deleted; no more background compactions
  607. } else if (!bg_error_.ok()) {
  608. // Already got an error; no more changes
  609. } else if (imm_ == nullptr && manual_compaction_ == nullptr &&
  610. !versions_->NeedsCompaction()) {
  611. // No work to be done
  612. } else {
  613. background_compaction_scheduled_ = true;
  614. env_->Schedule(&DBImpl::BGWork, this);
  615. }
  616. }
  617. void DBImpl::BGWork(void* db) {
  618. reinterpret_cast<DBImpl*>(db)->BackgroundCall();
  619. }
  620. void DBImpl::BackgroundCall() {
  621. MutexLock l(&mutex_);
  622. assert(background_compaction_scheduled_);
  623. if (shutting_down_.load(std::memory_order_acquire)) {
  624. // No more background work when shutting down.
  625. } else if (!bg_error_.ok()) {
  626. // No more background work after a background error.
  627. } else {
  628. BackgroundCompaction();
  629. }
  630. background_compaction_scheduled_ = false;
  631. // Previous compaction may have produced too many files in a level,
  632. // so reschedule another compaction if needed.
  633. MaybeScheduleCompaction();
  634. background_work_finished_signal_.SignalAll();
  635. }
  636. void DBImpl::BackgroundCompaction() {
  637. mutex_.AssertHeld();
  638. if (imm_ != nullptr) {
  639. CompactMemTable();
  640. return;
  641. }
  642. Compaction* c;
  643. bool is_manual = (manual_compaction_ != nullptr);
  644. InternalKey manual_end;
  645. if (is_manual) {
  646. ManualCompaction* m = manual_compaction_;
  647. c = versions_->CompactRange(m->level, m->begin, m->end, m->is_last_level);
  648. m->done = (c == nullptr);
  649. if (c != nullptr) {
  650. manual_end = c->input(0, c->num_input_files(0) - 1)->largest;
  651. }
  652. Log(options_.info_log,
  653. "Manual compaction at level-%d from %s .. %s; will stop at %s\n",
  654. m->level, (m->begin ? m->begin->DebugString().c_str() : "(begin)"),
  655. (m->end ? m->end->DebugString().c_str() : "(end)"),
  656. (m->done ? "(end)" : manual_end.DebugString().c_str()));
  657. } else {
  658. c = versions_->PickCompaction();
  659. }
  660. Status status;
  661. if (c == nullptr) {
  662. // Nothing to do
  663. } else if (!is_manual && c->IsTrivialMove()) {
  664. // Move file to next level
  665. assert(c->num_input_files(0) == 1);
  666. FileMetaData* f = c->input(0, 0);
  667. c->edit()->RemoveFile(c->level(), f->number);
  668. c->edit()->AddFile(c->level() + 1, f->number, f->file_size, f->smallest,
  669. f->largest);
  670. status = versions_->LogAndApply(c->edit(), &mutex_);
  671. if (!status.ok()) {
  672. RecordBackgroundError(status);
  673. }
  674. VersionSet::LevelSummaryStorage tmp;
  675. Log(options_.info_log, "Moved #%lld to level-%d %lld bytes %s: %s\n",
  676. static_cast<unsigned long long>(f->number), c->level() + 1,
  677. static_cast<unsigned long long>(f->file_size),
  678. status.ToString().c_str(), versions_->LevelSummary(&tmp));
  679. } else {
  680. CompactionState* compact = new CompactionState(c);
  681. status = DoCompactionWork(compact);
  682. if (!status.ok()) {
  683. RecordBackgroundError(status);
  684. }
  685. CleanupCompaction(compact);
  686. c->ReleaseInputs();
  687. RemoveObsoleteFiles();
  688. }
  689. delete c;
  690. if (status.ok()) {
  691. // Done
  692. } else if (shutting_down_.load(std::memory_order_acquire)) {
  693. // Ignore compaction errors found during shutting down
  694. } else {
  695. Log(options_.info_log, "Compaction error: %s", status.ToString().c_str());
  696. }
  697. if (is_manual) {
  698. ManualCompaction* m = manual_compaction_;
  699. if (!status.ok()) {
  700. m->done = true;
  701. }
  702. if (!m->done) {
  703. // We only compacted part of the requested range. Update *m
  704. // to the range that is left to be compacted.
  705. m->tmp_storage = manual_end;
  706. m->begin = &m->tmp_storage;
  707. }
  708. manual_compaction_ = nullptr;
  709. }
  710. }
  711. void DBImpl::CleanupCompaction(CompactionState* compact) {
  712. mutex_.AssertHeld();
  713. if (compact->builder != nullptr) {
  714. // May happen if we get a shutdown call in the middle of compaction
  715. compact->builder->Abandon();
  716. delete compact->builder;
  717. } else {
  718. assert(compact->outfile == nullptr);
  719. }
  720. delete compact->outfile;
  721. for (size_t i = 0; i < compact->outputs.size(); i++) {
  722. const CompactionState::Output& out = compact->outputs[i];
  723. pending_outputs_.erase(out.number);
  724. }
  725. delete compact;
  726. }
  727. Status DBImpl::OpenCompactionOutputFile(CompactionState* compact) {
  728. assert(compact != nullptr);
  729. assert(compact->builder == nullptr);
  730. uint64_t file_number;
  731. {
  732. mutex_.Lock();
  733. file_number = versions_->NewFileNumber();
  734. pending_outputs_.insert(file_number);
  735. CompactionState::Output out;
  736. out.number = file_number;
  737. out.smallest.Clear();
  738. out.largest.Clear();
  739. compact->outputs.push_back(out);
  740. mutex_.Unlock();
  741. }
  742. // Make the output file
  743. std::string fname = TableFileName(dbname_, file_number);
  744. Status s = env_->NewWritableFile(fname, &compact->outfile);
  745. if (s.ok()) {
  746. compact->builder = new TableBuilder(options_, compact->outfile);
  747. }
  748. return s;
  749. }
  750. Status DBImpl::FinishCompactionOutputFile(CompactionState* compact,
  751. Iterator* input) {
  752. assert(compact != nullptr);
  753. assert(compact->outfile != nullptr);
  754. assert(compact->builder != nullptr);
  755. const uint64_t output_number = compact->current_output()->number;
  756. assert(output_number != 0);
  757. // Check for iterator errors
  758. Status s = input->status();
  759. const uint64_t current_entries = compact->builder->NumEntries();
  760. if (s.ok()) {
  761. s = compact->builder->Finish();
  762. } else {
  763. compact->builder->Abandon();
  764. }
  765. const uint64_t current_bytes = compact->builder->FileSize();
  766. compact->current_output()->file_size = current_bytes;
  767. compact->total_bytes += current_bytes;
  768. delete compact->builder;
  769. compact->builder = nullptr;
  770. // Finish and check for file errors
  771. if (s.ok()) {
  772. s = compact->outfile->Sync();
  773. }
  774. if (s.ok()) {
  775. s = compact->outfile->Close();
  776. }
  777. delete compact->outfile;
  778. compact->outfile = nullptr;
  779. if (s.ok() && current_entries > 0) {
  780. // Verify that the table is usable
  781. Iterator* iter =
  782. table_cache_->NewIterator(ReadOptions(), output_number, current_bytes);
  783. s = iter->status();
  784. delete iter;
  785. if (s.ok()) {
  786. Log(options_.info_log, "Generated table #%llu@%d: %lld keys, %lld bytes",
  787. (unsigned long long)output_number, compact->compaction->level(),
  788. (unsigned long long)current_entries,
  789. (unsigned long long)current_bytes);
  790. }
  791. }
  792. return s;
  793. }
  794. Status DBImpl::InstallCompactionResults(CompactionState* compact) {
  795. mutex_.AssertHeld();
  796. Log(options_.info_log, "Compacted %d@%d + %d@%d files => %lld bytes",
  797. compact->compaction->num_input_files(0), compact->compaction->level(),
  798. compact->compaction->num_input_files(1), compact->compaction->level() + 1,
  799. static_cast<long long>(compact->total_bytes));
  800. // Add compaction outputs
  801. compact->compaction->AddInputDeletions(compact->compaction->edit());
  802. const int level = compact->compaction->level();
  803. for (size_t i = 0; i < compact->outputs.size(); i++) {
  804. const CompactionState::Output& out = compact->outputs[i];
  805. if (!compact->compaction->is_last_level()) {
  806. compact->compaction->edit()->AddFile(level + 1, out.number, out.file_size,
  807. out.smallest, out.largest);
  808. } else {
  809. // TTL: outputs of last level compaction should be writen to last level itself
  810. compact->compaction->edit()->AddFile(level, out.number, out.file_size,
  811. out.smallest, out.largest);
  812. }
  813. }
  814. return versions_->LogAndApply(compact->compaction->edit(), &mutex_);
  815. }
  816. Status DBImpl::DoCompactionWork(CompactionState* compact) {
  817. const uint64_t start_micros = env_->NowMicros();
  818. int64_t imm_micros = 0; // Micros spent doing imm_ compactions
  819. Log(options_.info_log, "Compacting %d@%d + %d@%d files",
  820. compact->compaction->num_input_files(0), compact->compaction->level(),
  821. compact->compaction->num_input_files(1),
  822. compact->compaction->level() + 1);
  823. assert(versions_->NumLevelFiles(compact->compaction->level()) > 0);
  824. assert(compact->builder == nullptr);
  825. assert(compact->outfile == nullptr);
  826. if (snapshots_.empty()) {
  827. compact->smallest_snapshot = versions_->LastSequence();
  828. } else {
  829. compact->smallest_snapshot = snapshots_.oldest()->sequence_number();
  830. }
  831. Iterator* input = versions_->MakeInputIterator(compact->compaction);
  832. // Release mutex while we're actually doing the compaction work
  833. mutex_.Unlock();
  834. input->SeekToFirst();
  835. Status status;
  836. ParsedInternalKey ikey;
  837. std::string current_user_key;
  838. bool has_current_user_key = false;
  839. SequenceNumber last_sequence_for_key = kMaxSequenceNumber;
  840. while (input->Valid() && !shutting_down_.load(std::memory_order_acquire)) {
  841. // Prioritize immutable compaction work
  842. if (has_imm_.load(std::memory_order_relaxed)) {
  843. const uint64_t imm_start = env_->NowMicros();
  844. mutex_.Lock();
  845. if (imm_ != nullptr) {
  846. CompactMemTable();
  847. // Wake up MakeRoomForWrite() if necessary.
  848. background_work_finished_signal_.SignalAll();
  849. }
  850. mutex_.Unlock();
  851. imm_micros += (env_->NowMicros() - imm_start);
  852. }
  853. Slice key = input->key();
  854. if (compact->compaction->ShouldStopBefore(key) &&
  855. compact->builder != nullptr) {
  856. status = FinishCompactionOutputFile(compact, input);
  857. if (!status.ok()) {
  858. break;
  859. }
  860. }
  861. // Handle key/value, add to state, etc.
  862. bool drop = false;
  863. if (!ParseInternalKey(key, &ikey)) {
  864. // Do not hide error keys
  865. current_user_key.clear();
  866. has_current_user_key = false;
  867. last_sequence_for_key = kMaxSequenceNumber;
  868. } else {
  869. if (!has_current_user_key ||
  870. user_comparator()->Compare(ikey.user_key, Slice(current_user_key)) !=
  871. 0) {
  872. // First occurrence of this user key
  873. current_user_key.assign(ikey.user_key.data(), ikey.user_key.size());
  874. has_current_user_key = true;
  875. last_sequence_for_key = kMaxSequenceNumber;
  876. }
  877. std::string value = input->value().ToString();
  878. uint64_t ddl;
  879. DecodeDeadLineValue(&value, ddl);
  880. if (last_sequence_for_key <= compact->smallest_snapshot) {
  881. // Hidden by an newer entry for same user key
  882. drop = true; // (A)
  883. } else if (ikey.type == kTypeDeletion &&
  884. ikey.sequence <= compact->smallest_snapshot &&
  885. compact->compaction->IsBaseLevelForKey(ikey.user_key)) {
  886. // For this user key:
  887. // (1) there is no data in higher levels
  888. // (2) data in lower levels will have larger sequence numbers
  889. // (3) data in layers that are being compacted here and have
  890. // smaller sequence numbers will be dropped in the next
  891. // few iterations of this loop (by rule (A) above).
  892. // Therefore this deletion marker is obsolete and can be dropped.
  893. drop = true;
  894. } else if (ddl <= std::time(nullptr)) {
  895. // TTL: data expired
  896. drop = true;
  897. }
  898. last_sequence_for_key = ikey.sequence;
  899. }
  900. #if 0
  901. Log(options_.info_log,
  902. " Compact: %s, seq %d, type: %d %d, drop: %d, is_base: %d, "
  903. "%d smallest_snapshot: %d",
  904. ikey.user_key.ToString().c_str(),
  905. (int)ikey.sequence, ikey.type, kTypeValue, drop,
  906. compact->compaction->IsBaseLevelForKey(ikey.user_key),
  907. (int)last_sequence_for_key, (int)compact->smallest_snapshot);
  908. #endif
  909. if (!drop) {
  910. // Open output file if necessary
  911. if (compact->builder == nullptr) {
  912. status = OpenCompactionOutputFile(compact);
  913. if (!status.ok()) {
  914. break;
  915. }
  916. }
  917. if (compact->builder->NumEntries() == 0) {
  918. compact->current_output()->smallest.DecodeFrom(key);
  919. }
  920. compact->current_output()->largest.DecodeFrom(key);
  921. compact->builder->Add(key, input->value());
  922. // Close output file if it is big enough
  923. if (compact->builder->FileSize() >=
  924. compact->compaction->MaxOutputFileSize()) {
  925. status = FinishCompactionOutputFile(compact, input);
  926. if (!status.ok()) {
  927. break;
  928. }
  929. }
  930. }
  931. input->Next();
  932. }
  933. if (status.ok() && shutting_down_.load(std::memory_order_acquire)) {
  934. status = Status::IOError("Deleting DB during compaction");
  935. }
  936. if (status.ok() && compact->builder != nullptr) {
  937. status = FinishCompactionOutputFile(compact, input);
  938. }
  939. if (status.ok()) {
  940. status = input->status();
  941. }
  942. delete input;
  943. input = nullptr;
  944. CompactionStats stats;
  945. stats.micros = env_->NowMicros() - start_micros - imm_micros;
  946. for (int which = 0; which < 2; which++) {
  947. for (int i = 0; i < compact->compaction->num_input_files(which); i++) {
  948. stats.bytes_read += compact->compaction->input(which, i)->file_size;
  949. }
  950. }
  951. for (size_t i = 0; i < compact->outputs.size(); i++) {
  952. stats.bytes_written += compact->outputs[i].file_size;
  953. }
  954. mutex_.Lock();
  955. if (!compact->compaction->is_last_level()) {
  956. stats_[compact->compaction->level() + 1].Add(stats);
  957. } else {
  958. // TTL: compaction for last level
  959. stats_[compact->compaction->level()].Add(stats);
  960. }
  961. if (status.ok()) {
  962. status = InstallCompactionResults(compact);
  963. }
  964. if (!status.ok()) {
  965. RecordBackgroundError(status);
  966. }
  967. VersionSet::LevelSummaryStorage tmp;
  968. Log(options_.info_log, "compacted to: %s", versions_->LevelSummary(&tmp));
  969. return status;
  970. }
  971. namespace {
  972. struct IterState {
  973. port::Mutex* const mu;
  974. Version* const version GUARDED_BY(mu);
  975. MemTable* const mem GUARDED_BY(mu);
  976. MemTable* const imm GUARDED_BY(mu);
  977. IterState(port::Mutex* mutex, MemTable* mem, MemTable* imm, Version* version)
  978. : mu(mutex), version(version), mem(mem), imm(imm) {}
  979. };
  980. static void CleanupIteratorState(void* arg1, void* arg2) {
  981. IterState* state = reinterpret_cast<IterState*>(arg1);
  982. state->mu->Lock();
  983. state->mem->Unref();
  984. if (state->imm != nullptr) state->imm->Unref();
  985. state->version->Unref();
  986. state->mu->Unlock();
  987. delete state;
  988. }
  989. } // anonymous namespace
  990. Iterator* DBImpl::NewInternalIterator(const ReadOptions& options,
  991. SequenceNumber* latest_snapshot,
  992. uint32_t* seed) {
  993. mutex_.Lock();
  994. *latest_snapshot = versions_->LastSequence();
  995. // Collect together all needed child iterators
  996. std::vector<Iterator*> list;
  997. list.push_back(mem_->NewIterator());
  998. mem_->Ref();
  999. if (imm_ != nullptr) {
  1000. list.push_back(imm_->NewIterator());
  1001. imm_->Ref();
  1002. }
  1003. versions_->current()->AddIterators(options, &list);
  1004. Iterator* internal_iter =
  1005. NewMergingIterator(&internal_comparator_, &list[0], list.size());
  1006. versions_->current()->Ref();
  1007. IterState* cleanup = new IterState(&mutex_, mem_, imm_, versions_->current());
  1008. internal_iter->RegisterCleanup(CleanupIteratorState, cleanup, nullptr);
  1009. *seed = ++seed_;
  1010. mutex_.Unlock();
  1011. return internal_iter;
  1012. }
  1013. Iterator* DBImpl::TEST_NewInternalIterator() {
  1014. SequenceNumber ignored;
  1015. uint32_t ignored_seed;
  1016. return NewInternalIterator(ReadOptions(), &ignored, &ignored_seed);
  1017. }
  1018. int64_t DBImpl::TEST_MaxNextLevelOverlappingBytes() {
  1019. MutexLock l(&mutex_);
  1020. return versions_->MaxNextLevelOverlappingBytes();
  1021. }
  1022. Status DBImpl::Get(const ReadOptions& options, const Slice& key,
  1023. std::string* value) {
  1024. Status s;
  1025. MutexLock l(&mutex_);
  1026. SequenceNumber snapshot;
  1027. if (options.snapshot != nullptr) {
  1028. snapshot =
  1029. static_cast<const SnapshotImpl*>(options.snapshot)->sequence_number();
  1030. } else {
  1031. snapshot = versions_->LastSequence();
  1032. }
  1033. MemTable* mem = mem_;
  1034. MemTable* imm = imm_;
  1035. Version* current = versions_->current();
  1036. mem->Ref();
  1037. if (imm != nullptr) imm->Ref();
  1038. current->Ref();
  1039. bool have_stat_update = false;
  1040. Version::GetStats stats;
  1041. // Unlock while reading from files and memtables
  1042. {
  1043. mutex_.Unlock();
  1044. // First look in the memtable, then in the immutable memtable (if any).
  1045. LookupKey lkey(key, snapshot);
  1046. if (mem->Get(lkey, value, &s)) {
  1047. // Done
  1048. } else if (imm != nullptr && imm->Get(lkey, value, &s)) {
  1049. // Done
  1050. } else {
  1051. s = current->Get(options, lkey, value, &stats);
  1052. have_stat_update = true;
  1053. }
  1054. // TTL: Get the true value and make sure the data is still living
  1055. if(!value->empty()) {
  1056. uint64_t dead_line;
  1057. DecodeDeadLineValue(value, dead_line);
  1058. if (dead_line != 0) {
  1059. // use TTL
  1060. if (std::time(nullptr) >= dead_line) {
  1061. // data expired
  1062. *value = "";
  1063. s = Status::NotFound("Data expired");
  1064. }
  1065. } else {
  1066. // TTL not set
  1067. }
  1068. }
  1069. mutex_.Lock();
  1070. }
  1071. if (have_stat_update && current->UpdateStats(stats)) {
  1072. MaybeScheduleCompaction();
  1073. }
  1074. mem->Unref();
  1075. if (imm != nullptr) imm->Unref();
  1076. current->Unref();
  1077. return s;
  1078. }
  1079. Iterator* DBImpl::NewIterator(const ReadOptions& options) {
  1080. SequenceNumber latest_snapshot;
  1081. uint32_t seed;
  1082. Iterator* iter = NewInternalIterator(options, &latest_snapshot, &seed);
  1083. return NewDBIterator(this, user_comparator(), iter,
  1084. (options.snapshot != nullptr
  1085. ? static_cast<const SnapshotImpl*>(options.snapshot)
  1086. ->sequence_number()
  1087. : latest_snapshot),
  1088. seed);
  1089. }
  1090. void DBImpl::RecordReadSample(Slice key) {
  1091. MutexLock l(&mutex_);
  1092. if (versions_->current()->RecordReadSample(key)) {
  1093. MaybeScheduleCompaction();
  1094. }
  1095. }
  1096. const Snapshot* DBImpl::GetSnapshot() {
  1097. MutexLock l(&mutex_);
  1098. return snapshots_.New(versions_->LastSequence());
  1099. }
  1100. void DBImpl::ReleaseSnapshot(const Snapshot* snapshot) {
  1101. MutexLock l(&mutex_);
  1102. snapshots_.Delete(static_cast<const SnapshotImpl*>(snapshot));
  1103. }
  1104. // Convenience methods
  1105. Status DBImpl::Put(const WriteOptions& o, const Slice& key, const Slice& val) {
  1106. return DB::Put(o, key, val);
  1107. }
  1108. Status DBImpl::Put(const WriteOptions& options, const Slice& key,
  1109. const Slice& value, uint64_t ttl) {
  1110. return DB::Put(options, key, value, ttl);
  1111. }
  1112. Status DBImpl::Delete(const WriteOptions& options, const Slice& key) {
  1113. return DB::Delete(options, key);
  1114. }
  1115. Status DBImpl::Write(const WriteOptions& options, WriteBatch* updates) {
  1116. Writer w(&mutex_);
  1117. w.batch = updates;
  1118. w.sync = options.sync;
  1119. w.done = false;
  1120. MutexLock l(&mutex_);
  1121. writers_.push_back(&w);
  1122. while (!w.done && &w != writers_.front()) {
  1123. w.cv.Wait();
  1124. }
  1125. if (w.done) {
  1126. return w.status;
  1127. }
  1128. // May temporarily unlock and wait.
  1129. Status status = MakeRoomForWrite(updates == nullptr);
  1130. uint64_t last_sequence = versions_->LastSequence();
  1131. Writer* last_writer = &w;
  1132. if (status.ok() && updates != nullptr) { // nullptr batch is for compactions
  1133. WriteBatch* write_batch = BuildBatchGroup(&last_writer);
  1134. WriteBatchInternal::SetSequence(write_batch, last_sequence + 1);
  1135. last_sequence += WriteBatchInternal::Count(write_batch);
  1136. // Add to log and apply to memtable. We can release the lock
  1137. // during this phase since &w is currently responsible for logging
  1138. // and protects against concurrent loggers and concurrent writes
  1139. // into mem_.
  1140. {
  1141. mutex_.Unlock();
  1142. status = log_->AddRecord(WriteBatchInternal::Contents(write_batch));
  1143. bool sync_error = false;
  1144. if (status.ok() && options.sync) {
  1145. status = logfile_->Sync();
  1146. if (!status.ok()) {
  1147. sync_error = true;
  1148. }
  1149. }
  1150. if (status.ok()) {
  1151. status = WriteBatchInternal::InsertInto(write_batch, mem_);
  1152. }
  1153. mutex_.Lock();
  1154. if (sync_error) {
  1155. // The state of the log file is indeterminate: the log record we
  1156. // just added may or may not show up when the DB is re-opened.
  1157. // So we force the DB into a mode where all future writes fail.
  1158. RecordBackgroundError(status);
  1159. }
  1160. }
  1161. if (write_batch == tmp_batch_) tmp_batch_->Clear();
  1162. versions_->SetLastSequence(last_sequence);
  1163. }
  1164. while (true) {
  1165. Writer* ready = writers_.front();
  1166. writers_.pop_front();
  1167. if (ready != &w) {
  1168. ready->status = status;
  1169. ready->done = true;
  1170. ready->cv.Signal();
  1171. }
  1172. if (ready == last_writer) break;
  1173. }
  1174. // Notify new head of write queue
  1175. if (!writers_.empty()) {
  1176. writers_.front()->cv.Signal();
  1177. }
  1178. return status;
  1179. }
  1180. // REQUIRES: Writer list must be non-empty
  1181. // REQUIRES: First writer must have a non-null batch
  1182. WriteBatch* DBImpl::BuildBatchGroup(Writer** last_writer) {
  1183. mutex_.AssertHeld();
  1184. assert(!writers_.empty());
  1185. Writer* first = writers_.front();
  1186. WriteBatch* result = first->batch;
  1187. assert(result != nullptr);
  1188. size_t size = WriteBatchInternal::ByteSize(first->batch);
  1189. // Allow the group to grow up to a maximum size, but if the
  1190. // original write is small, limit the growth so we do not slow
  1191. // down the small write too much.
  1192. size_t max_size = 1 << 20;
  1193. if (size <= (128 << 10)) {
  1194. max_size = size + (128 << 10);
  1195. }
  1196. *last_writer = first;
  1197. std::deque<Writer*>::iterator iter = writers_.begin();
  1198. ++iter; // Advance past "first"
  1199. for (; iter != writers_.end(); ++iter) {
  1200. Writer* w = *iter;
  1201. if (w->sync && !first->sync) {
  1202. // Do not include a sync write into a batch handled by a non-sync write.
  1203. break;
  1204. }
  1205. if (w->batch != nullptr) {
  1206. size += WriteBatchInternal::ByteSize(w->batch);
  1207. if (size > max_size) {
  1208. // Do not make batch too big
  1209. break;
  1210. }
  1211. // Append to *result
  1212. if (result == first->batch) {
  1213. // Switch to temporary batch instead of disturbing caller's batch
  1214. result = tmp_batch_;
  1215. assert(WriteBatchInternal::Count(result) == 0);
  1216. WriteBatchInternal::Append(result, first->batch);
  1217. }
  1218. WriteBatchInternal::Append(result, w->batch);
  1219. }
  1220. *last_writer = w;
  1221. }
  1222. return result;
  1223. }
  1224. // REQUIRES: mutex_ is held
  1225. // REQUIRES: this thread is currently at the front of the writer queue
  1226. Status DBImpl::MakeRoomForWrite(bool force) {
  1227. mutex_.AssertHeld();
  1228. assert(!writers_.empty());
  1229. bool allow_delay = !force;
  1230. Status s;
  1231. while (true) {
  1232. if (!bg_error_.ok()) {
  1233. // Yield previous error
  1234. s = bg_error_;
  1235. break;
  1236. } else if (allow_delay && versions_->NumLevelFiles(0) >=
  1237. config::kL0_SlowdownWritesTrigger) {
  1238. // We are getting close to hitting a hard limit on the number of
  1239. // L0 files. Rather than delaying a single write by several
  1240. // seconds when we hit the hard limit, start delaying each
  1241. // individual write by 1ms to reduce latency variance. Also,
  1242. // this delay hands over some CPU to the compaction thread in
  1243. // case it is sharing the same core as the writer.
  1244. mutex_.Unlock();
  1245. env_->SleepForMicroseconds(1000);
  1246. allow_delay = false; // Do not delay a single write more than once
  1247. mutex_.Lock();
  1248. } else if (!force &&
  1249. (mem_->ApproximateMemoryUsage() <= options_.write_buffer_size)) {
  1250. // There is room in current memtable
  1251. break;
  1252. } else if (imm_ != nullptr) {
  1253. // We have filled up the current memtable, but the previous
  1254. // one is still being compacted, so we wait.
  1255. Log(options_.info_log, "Current memtable full; waiting...\n");
  1256. background_work_finished_signal_.Wait();
  1257. } else if (versions_->NumLevelFiles(0) >= config::kL0_StopWritesTrigger) {
  1258. // There are too many level-0 files.
  1259. Log(options_.info_log, "Too many L0 files; waiting...\n");
  1260. background_work_finished_signal_.Wait();
  1261. } else {
  1262. // Attempt to switch to a new memtable and trigger compaction of old
  1263. assert(versions_->PrevLogNumber() == 0);
  1264. uint64_t new_log_number = versions_->NewFileNumber();
  1265. WritableFile* lfile = nullptr;
  1266. s = env_->NewWritableFile(LogFileName(dbname_, new_log_number), &lfile);
  1267. if (!s.ok()) {
  1268. // Avoid chewing through file number space in a tight loop.
  1269. versions_->ReuseFileNumber(new_log_number);
  1270. break;
  1271. }
  1272. delete log_;
  1273. s = logfile_->Close();
  1274. if (!s.ok()) {
  1275. // We may have lost some data written to the previous log file.
  1276. // Switch to the new log file anyway, but record as a background
  1277. // error so we do not attempt any more writes.
  1278. //
  1279. // We could perhaps attempt to save the memtable corresponding
  1280. // to log file and suppress the error if that works, but that
  1281. // would add more complexity in a critical code path.
  1282. RecordBackgroundError(s);
  1283. }
  1284. delete logfile_;
  1285. logfile_ = lfile;
  1286. logfile_number_ = new_log_number;
  1287. log_ = new log::Writer(lfile);
  1288. imm_ = mem_;
  1289. has_imm_.store(true, std::memory_order_release);
  1290. mem_ = new MemTable(internal_comparator_);
  1291. mem_->Ref();
  1292. force = false; // Do not force another compaction if have room
  1293. MaybeScheduleCompaction();
  1294. }
  1295. }
  1296. return s;
  1297. }
  1298. bool DBImpl::GetProperty(const Slice& property, std::string* value) {
  1299. value->clear();
  1300. MutexLock l(&mutex_);
  1301. Slice in = property;
  1302. Slice prefix("leveldb.");
  1303. if (!in.starts_with(prefix)) return false;
  1304. in.remove_prefix(prefix.size());
  1305. if (in.starts_with("num-files-at-level")) {
  1306. in.remove_prefix(strlen("num-files-at-level"));
  1307. uint64_t level;
  1308. bool ok = ConsumeDecimalNumber(&in, &level) && in.empty();
  1309. if (!ok || level >= config::kNumLevels) {
  1310. return false;
  1311. } else {
  1312. char buf[100];
  1313. std::snprintf(buf, sizeof(buf), "%d",
  1314. versions_->NumLevelFiles(static_cast<int>(level)));
  1315. *value = buf;
  1316. return true;
  1317. }
  1318. } else if (in == "stats") {
  1319. char buf[200];
  1320. std::snprintf(buf, sizeof(buf),
  1321. " Compactions\n"
  1322. "Level Files Size(MB) Time(sec) Read(MB) Write(MB)\n"
  1323. "--------------------------------------------------\n");
  1324. value->append(buf);
  1325. for (int level = 0; level < config::kNumLevels; level++) {
  1326. int files = versions_->NumLevelFiles(level);
  1327. if (stats_[level].micros > 0 || files > 0) {
  1328. std::snprintf(buf, sizeof(buf), "%3d %8d %8.0f %9.0f %8.0f %9.0f\n",
  1329. level, files, versions_->NumLevelBytes(level) / 1048576.0,
  1330. stats_[level].micros / 1e6,
  1331. stats_[level].bytes_read / 1048576.0,
  1332. stats_[level].bytes_written / 1048576.0);
  1333. value->append(buf);
  1334. }
  1335. }
  1336. return true;
  1337. } else if (in == "sstables") {
  1338. *value = versions_->current()->DebugString();
  1339. return true;
  1340. } else if (in == "approximate-memory-usage") {
  1341. size_t total_usage = options_.block_cache->TotalCharge();
  1342. if (mem_) {
  1343. total_usage += mem_->ApproximateMemoryUsage();
  1344. }
  1345. if (imm_) {
  1346. total_usage += imm_->ApproximateMemoryUsage();
  1347. }
  1348. char buf[50];
  1349. std::snprintf(buf, sizeof(buf), "%llu",
  1350. static_cast<unsigned long long>(total_usage));
  1351. value->append(buf);
  1352. return true;
  1353. }
  1354. return false;
  1355. }
  1356. void DBImpl::GetApproximateSizes(const Range* range, int n, uint64_t* sizes) {
  1357. // TODO(opt): better implementation
  1358. MutexLock l(&mutex_);
  1359. Version* v = versions_->current();
  1360. v->Ref();
  1361. for (int i = 0; i < n; i++) {
  1362. // Convert user_key into a corresponding internal key.
  1363. InternalKey k1(range[i].start, kMaxSequenceNumber, kValueTypeForSeek);
  1364. InternalKey k2(range[i].limit, kMaxSequenceNumber, kValueTypeForSeek);
  1365. uint64_t start = versions_->ApproximateOffsetOf(v, k1);
  1366. uint64_t limit = versions_->ApproximateOffsetOf(v, k2);
  1367. sizes[i] = (limit >= start ? limit - start : 0);
  1368. }
  1369. v->Unref();
  1370. }
  1371. // Default implementations of convenience methods that subclasses of DB
  1372. // can call if they wish
  1373. // TTL: Update TTL Encode
  1374. Status DB::Put(const WriteOptions& opt, const Slice& key, const Slice& value) {
  1375. WriteBatch batch;
  1376. batch.Put(key, EncodeDeadLine(0, value));
  1377. return Write(opt, &batch);
  1378. }
  1379. // TTL: Put methods for ttl
  1380. Status DB::Put(const WriteOptions& options, const Slice& key,
  1381. const Slice& value, uint64_t ttl) {
  1382. WriteBatch batch;
  1383. auto dead_line = std::time(nullptr) + ttl;
  1384. batch.Put(key, EncodeDeadLine(dead_line, value));
  1385. return Write(options, &batch);
  1386. }
  1387. Status DB::Delete(const WriteOptions& opt, const Slice& key) {
  1388. WriteBatch batch;
  1389. batch.Delete(key);
  1390. return Write(opt, &batch);
  1391. }
  1392. DB::~DB() = default;
  1393. Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
  1394. *dbptr = nullptr;
  1395. DBImpl* impl = new DBImpl(options, dbname);
  1396. impl->mutex_.Lock();
  1397. VersionEdit edit;
  1398. // Recover handles create_if_missing, error_if_exists
  1399. bool save_manifest = false;
  1400. Status s = impl->Recover(&edit, &save_manifest);
  1401. if (s.ok() && impl->mem_ == nullptr) {
  1402. // Create new log and a corresponding memtable.
  1403. uint64_t new_log_number = impl->versions_->NewFileNumber();
  1404. WritableFile* lfile;
  1405. s = options.env->NewWritableFile(LogFileName(dbname, new_log_number),
  1406. &lfile);
  1407. if (s.ok()) {
  1408. edit.SetLogNumber(new_log_number);
  1409. impl->logfile_ = lfile;
  1410. impl->logfile_number_ = new_log_number;
  1411. impl->log_ = new log::Writer(lfile);
  1412. impl->mem_ = new MemTable(impl->internal_comparator_);
  1413. impl->mem_->Ref();
  1414. }
  1415. }
  1416. if (s.ok() && save_manifest) {
  1417. edit.SetPrevLogNumber(0); // No older logs needed after recovery.
  1418. edit.SetLogNumber(impl->logfile_number_);
  1419. s = impl->versions_->LogAndApply(&edit, &impl->mutex_);
  1420. }
  1421. if (s.ok()) {
  1422. impl->RemoveObsoleteFiles();
  1423. impl->MaybeScheduleCompaction();
  1424. }
  1425. impl->mutex_.Unlock();
  1426. if (s.ok()) {
  1427. assert(impl->mem_ != nullptr);
  1428. *dbptr = impl;
  1429. } else {
  1430. delete impl;
  1431. }
  1432. return s;
  1433. }
  1434. Snapshot::~Snapshot() = default;
  1435. Status DestroyDB(const std::string& dbname, const Options& options) {
  1436. Env* env = options.env;
  1437. std::vector<std::string> filenames;
  1438. Status result = env->GetChildren(dbname, &filenames);
  1439. if (!result.ok()) {
  1440. // Ignore error in case directory does not exist
  1441. return Status::OK();
  1442. }
  1443. FileLock* lock;
  1444. const std::string lockname = LockFileName(dbname);
  1445. result = env->LockFile(lockname, &lock);
  1446. if (result.ok()) {
  1447. uint64_t number;
  1448. FileType type;
  1449. for (size_t i = 0; i < filenames.size(); i++) {
  1450. if (ParseFileName(filenames[i], &number, &type) &&
  1451. type != kDBLockFile) { // Lock file will be deleted at end
  1452. Status del = env->RemoveFile(dbname + "/" + filenames[i]);
  1453. if (result.ok() && !del.ok()) {
  1454. result = del;
  1455. }
  1456. }
  1457. }
  1458. env->UnlockFile(lock); // Ignore error since state is already gone
  1459. env->RemoveFile(lockname);
  1460. env->RemoveDir(dbname); // Ignore error in case dir contains other files
  1461. }
  1462. return result;
  1463. }
  1464. } // namespace leveldb