作者: 韩晨旭 10225101440 李畅 10225102463
Вы не можете выбрать более 25 тем Темы должны начинаться с буквы или цифры, могут содержать дефисы(-) и должны содержать не более 35 символов.

2366 строки
70 KiB

Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 лет назад
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 лет назад
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 лет назад
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 лет назад
Add Env::Remove{File,Dir} which obsolete Env::Delete{File,Dir}. The "DeleteFile" method name causes pain for Windows developers, because <windows.h> #defines a DeleteFile macro to DeleteFileW or DeleteFileA. Current code uses workarounds, like #undefining DeleteFile everywhere an Env is declared, implemented, or used. This CL removes the need for workarounds by renaming Env::DeleteFile to Env::RemoveFile. For consistency, Env::DeleteDir is also renamed to Env::RemoveDir. A few internal methods are also renamed for consistency. Software that supports Windows is expected to migrate any Env implementations and usage to Remove{File,Dir}, and never use the name Env::Delete{File,Dir} in its code. The renaming is done in a backwards-compatible way, at the risk of making it slightly more difficult to build a new correct Env implementation. The backwards compatibility is achieved using the following hacks: 1) Env::Remove{File,Dir} methods are added, with a default implementation that calls into Env::Delete{File,Dir}. This makes old Env implementations compatible with code that calls into the updated API. 2) The Env::Delete{File,Dir} methods are no longer pure virtuals. Instead, they gain a default implementation that calls into Env::Remove{File,Dir}. This makes updated Env implementations compatible with code that calls into the old API. The cost of this approach is that it's possible to write an Env without overriding either Rename{File,Dir} or Delete{File,Dir}, without getting a compiler warning. However, attempting to run the test suite will immediately fail with an infinite call stack ending in {Remove,Delete}{File,Dir}, making developers aware of the problem. PiperOrigin-RevId: 288710907
4 лет назад
  1. // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
  2. // Use of this source code is governed by a BSD-style license that can be
  3. // found in the LICENSE file. See the AUTHORS file for names of contributors.
  4. #include "leveldb/db.h"
  5. #include <atomic>
  6. #include <cinttypes>
  7. #include <string>
  8. #include "gtest/gtest.h"
  9. #include "benchmark/benchmark.h"
  10. #include "db/db_impl.h"
  11. #include "db/filename.h"
  12. #include "db/version_set.h"
  13. #include "db/write_batch_internal.h"
  14. #include "leveldb/cache.h"
  15. #include "leveldb/env.h"
  16. #include "leveldb/filter_policy.h"
  17. #include "leveldb/table.h"
  18. #include "port/port.h"
  19. #include "port/thread_annotations.h"
  20. #include "util/hash.h"
  21. #include "util/logging.h"
  22. #include "util/mutexlock.h"
  23. #include "util/testutil.h"
  24. namespace leveldb {
  25. static std::string RandomString(Random* rnd, int len) {
  26. std::string r;
  27. test::RandomString(rnd, len, &r);
  28. return r;
  29. }
  30. static std::string RandomKey(Random* rnd) {
  31. int len =
  32. (rnd->OneIn(3) ? 1 // Short sometimes to encourage collisions
  33. : (rnd->OneIn(100) ? rnd->Skewed(10) : rnd->Uniform(10)));
  34. return test::RandomKey(rnd, len);
  35. }
  36. namespace {
  37. class AtomicCounter {
  38. public:
  39. AtomicCounter() : count_(0) {}
  40. void Increment() { IncrementBy(1); }
  41. void IncrementBy(int count) LOCKS_EXCLUDED(mu_) {
  42. MutexLock l(&mu_);
  43. count_ += count;
  44. }
  45. int Read() LOCKS_EXCLUDED(mu_) {
  46. MutexLock l(&mu_);
  47. return count_;
  48. }
  49. void Reset() LOCKS_EXCLUDED(mu_) {
  50. MutexLock l(&mu_);
  51. count_ = 0;
  52. }
  53. private:
  54. port::Mutex mu_;
  55. int count_ GUARDED_BY(mu_);
  56. };
  57. void DelayMilliseconds(int millis) {
  58. Env::Default()->SleepForMicroseconds(millis * 1000);
  59. }
  60. } // namespace
  61. // Test Env to override default Env behavior for testing.
  62. class TestEnv : public EnvWrapper {
  63. public:
  64. explicit TestEnv(Env* base) : EnvWrapper(base), ignore_dot_files_(false) {}
  65. void SetIgnoreDotFiles(bool ignored) { ignore_dot_files_ = ignored; }
  66. Status GetChildren(const std::string& dir,
  67. std::vector<std::string>* result) override {
  68. Status s = target()->GetChildren(dir, result);
  69. if (!s.ok() || !ignore_dot_files_) {
  70. return s;
  71. }
  72. std::vector<std::string>::iterator it = result->begin();
  73. while (it != result->end()) {
  74. if ((*it == ".") || (*it == "..")) {
  75. it = result->erase(it);
  76. } else {
  77. ++it;
  78. }
  79. }
  80. return s;
  81. }
  82. private:
  83. bool ignore_dot_files_;
  84. };
  85. // Special Env used to delay background operations.
  86. class SpecialEnv : public EnvWrapper {
  87. public:
  88. // sstable/log Sync() calls are blocked while this pointer is non-null.
  89. std::atomic<bool> delay_data_sync_;
  90. // sstable/log Sync() calls return an error.
  91. std::atomic<bool> data_sync_error_;
  92. // Simulate no-space errors while this pointer is non-null.
  93. std::atomic<bool> no_space_;
  94. // Simulate non-writable file system while this pointer is non-null.
  95. std::atomic<bool> non_writable_;
  96. // Force sync of manifest files to fail while this pointer is non-null.
  97. std::atomic<bool> manifest_sync_error_;
  98. // Force write to manifest files to fail while this pointer is non-null.
  99. std::atomic<bool> manifest_write_error_;
  100. bool count_random_reads_;
  101. AtomicCounter random_read_counter_;
  102. explicit SpecialEnv(Env* base)
  103. : EnvWrapper(base),
  104. delay_data_sync_(false),
  105. data_sync_error_(false),
  106. no_space_(false),
  107. non_writable_(false),
  108. manifest_sync_error_(false),
  109. manifest_write_error_(false),
  110. count_random_reads_(false) {}
  111. Status NewWritableFile(const std::string& f, WritableFile** r) {
  112. class DataFile : public WritableFile {
  113. private:
  114. SpecialEnv* const env_;
  115. WritableFile* const base_;
  116. public:
  117. DataFile(SpecialEnv* env, WritableFile* base) : env_(env), base_(base) {}
  118. ~DataFile() { delete base_; }
  119. Status Append(const Slice& data) {
  120. if (env_->no_space_.load(std::memory_order_acquire)) {
  121. // Drop writes on the floor
  122. return Status::OK();
  123. } else {
  124. return base_->Append(data);
  125. }
  126. }
  127. Status Close() { return base_->Close(); }
  128. Status Flush() { return base_->Flush(); }
  129. Status Sync() {
  130. if (env_->data_sync_error_.load(std::memory_order_acquire)) {
  131. return Status::IOError("simulated data sync error");
  132. }
  133. while (env_->delay_data_sync_.load(std::memory_order_acquire)) {
  134. DelayMilliseconds(100);
  135. }
  136. return base_->Sync();
  137. }
  138. };
  139. class ManifestFile : public WritableFile {
  140. private:
  141. SpecialEnv* env_;
  142. WritableFile* base_;
  143. public:
  144. ManifestFile(SpecialEnv* env, WritableFile* b) : env_(env), base_(b) {}
  145. ~ManifestFile() { delete base_; }
  146. Status Append(const Slice& data) {
  147. if (env_->manifest_write_error_.load(std::memory_order_acquire)) {
  148. return Status::IOError("simulated writer error");
  149. } else {
  150. return base_->Append(data);
  151. }
  152. }
  153. Status Close() { return base_->Close(); }
  154. Status Flush() { return base_->Flush(); }
  155. Status Sync() {
  156. if (env_->manifest_sync_error_.load(std::memory_order_acquire)) {
  157. return Status::IOError("simulated sync error");
  158. } else {
  159. return base_->Sync();
  160. }
  161. }
  162. };
  163. if (non_writable_.load(std::memory_order_acquire)) {
  164. return Status::IOError("simulated write error");
  165. }
  166. Status s = target()->NewWritableFile(f, r);
  167. if (s.ok()) {
  168. if (strstr(f.c_str(), ".ldb") != nullptr ||
  169. strstr(f.c_str(), ".log") != nullptr) {
  170. *r = new DataFile(this, *r);
  171. } else if (strstr(f.c_str(), "MANIFEST") != nullptr) {
  172. *r = new ManifestFile(this, *r);
  173. }
  174. }
  175. return s;
  176. }
  177. Status NewRandomAccessFile(const std::string& f, RandomAccessFile** r) {
  178. class CountingFile : public RandomAccessFile {
  179. private:
  180. RandomAccessFile* target_;
  181. AtomicCounter* counter_;
  182. public:
  183. CountingFile(RandomAccessFile* target, AtomicCounter* counter)
  184. : target_(target), counter_(counter) {}
  185. ~CountingFile() override { delete target_; }
  186. Status Read(uint64_t offset, size_t n, Slice* result,
  187. char* scratch) const override {
  188. counter_->Increment();
  189. return target_->Read(offset, n, result, scratch);
  190. }
  191. };
  192. Status s = target()->NewRandomAccessFile(f, r);
  193. if (s.ok() && count_random_reads_) {
  194. *r = new CountingFile(*r, &random_read_counter_);
  195. }
  196. return s;
  197. }
  198. };
  199. class DBTest : public testing::Test {
  200. public:
  201. std::string dbname_;
  202. SpecialEnv* env_;
  203. DB* db_;
  204. Options last_options_;
  205. DBTest() : env_(new SpecialEnv(Env::Default())), option_config_(kDefault) {
  206. filter_policy_ = NewBloomFilterPolicy(10);
  207. dbname_ = testing::TempDir() + "db_test";
  208. DestroyDB(dbname_, Options());
  209. db_ = nullptr;
  210. Reopen();
  211. }
  212. ~DBTest() {
  213. delete db_;
  214. DestroyDB(dbname_, Options());
  215. delete env_;
  216. delete filter_policy_;
  217. }
  218. // Switch to a fresh database with the next option configuration to
  219. // test. Return false if there are no more configurations to test.
  220. bool ChangeOptions() {
  221. option_config_++;
  222. if (option_config_ >= kEnd) {
  223. return false;
  224. } else {
  225. DestroyAndReopen();
  226. return true;
  227. }
  228. }
  229. // Return the current option configuration.
  230. Options CurrentOptions() {
  231. Options options;
  232. options.reuse_logs = false;
  233. switch (option_config_) {
  234. case kReuse:
  235. options.reuse_logs = true;
  236. break;
  237. case kFilter:
  238. options.filter_policy = filter_policy_;
  239. break;
  240. case kUncompressed:
  241. options.compression = kNoCompression;
  242. break;
  243. default:
  244. break;
  245. }
  246. return options;
  247. }
  248. DBImpl* dbfull() { return reinterpret_cast<DBImpl*>(db_); }
  249. void Reopen(Options* options = nullptr) {
  250. ASSERT_LEVELDB_OK(TryReopen(options));
  251. }
  252. void Close() {
  253. delete db_;
  254. db_ = nullptr;
  255. }
  256. void DestroyAndReopen(Options* options = nullptr) {
  257. delete db_;
  258. db_ = nullptr;
  259. DestroyDB(dbname_, Options());
  260. ASSERT_LEVELDB_OK(TryReopen(options));
  261. }
  262. Status TryReopen(Options* options) {
  263. delete db_;
  264. db_ = nullptr;
  265. Options opts;
  266. if (options != nullptr) {
  267. opts = *options;
  268. } else {
  269. opts = CurrentOptions();
  270. opts.create_if_missing = true;
  271. }
  272. last_options_ = opts;
  273. return DB::Open(opts, dbname_, &db_);
  274. }
  275. Status Put(const std::string& k, const std::string& v) {
  276. return db_->Put(WriteOptions(), k, v);
  277. }
  278. Status Delete(const std::string& k) { return db_->Delete(WriteOptions(), k); }
  279. std::string Get(const std::string& k, const Snapshot* snapshot = nullptr) {
  280. ReadOptions options;
  281. options.snapshot = snapshot;
  282. std::string result;
  283. Status s = db_->Get(options, k, &result);
  284. if (s.IsNotFound()) {
  285. result = "NOT_FOUND";
  286. } else if (!s.ok()) {
  287. result = s.ToString();
  288. }
  289. return result;
  290. }
  291. // Return a string that contains all key,value pairs in order,
  292. // formatted like "(k1->v1)(k2->v2)".
  293. std::string Contents() {
  294. std::vector<std::string> forward;
  295. std::string result;
  296. Iterator* iter = db_->NewIterator(ReadOptions());
  297. for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
  298. std::string s = IterStatus(iter);
  299. result.push_back('(');
  300. result.append(s);
  301. result.push_back(')');
  302. forward.push_back(s);
  303. }
  304. // Check reverse iteration results are the reverse of forward results
  305. size_t matched = 0;
  306. for (iter->SeekToLast(); iter->Valid(); iter->Prev()) {
  307. EXPECT_LT(matched, forward.size());
  308. EXPECT_EQ(IterStatus(iter), forward[forward.size() - matched - 1]);
  309. matched++;
  310. }
  311. EXPECT_EQ(matched, forward.size());
  312. delete iter;
  313. return result;
  314. }
  315. std::string AllEntriesFor(const Slice& user_key) {
  316. Iterator* iter = dbfull()->TEST_NewInternalIterator();
  317. InternalKey target(user_key, kMaxSequenceNumber, kTypeValue);
  318. iter->Seek(target.Encode());
  319. std::string result;
  320. if (!iter->status().ok()) {
  321. result = iter->status().ToString();
  322. } else {
  323. result = "[ ";
  324. bool first = true;
  325. while (iter->Valid()) {
  326. ParsedInternalKey ikey;
  327. if (!ParseInternalKey(iter->key(), &ikey)) {
  328. result += "CORRUPTED";
  329. } else {
  330. if (last_options_.comparator->Compare(ikey.user_key, user_key) != 0) {
  331. break;
  332. }
  333. if (!first) {
  334. result += ", ";
  335. }
  336. first = false;
  337. switch (ikey.type) {
  338. case kTypeValue:
  339. result += iter->value().ToString();
  340. break;
  341. case kTypeDeletion:
  342. result += "DEL";
  343. break;
  344. }
  345. }
  346. iter->Next();
  347. }
  348. if (!first) {
  349. result += " ";
  350. }
  351. result += "]";
  352. }
  353. delete iter;
  354. return result;
  355. }
  356. int NumTableFilesAtLevel(int level) {
  357. std::string property;
  358. EXPECT_TRUE(db_->GetProperty(
  359. "leveldb.num-files-at-level" + NumberToString(level), &property));
  360. return std::stoi(property);
  361. }
  362. int TotalTableFiles() {
  363. int result = 0;
  364. for (int level = 0; level < config::kNumLevels; level++) {
  365. result += NumTableFilesAtLevel(level);
  366. }
  367. return result;
  368. }
  369. // Return spread of files per level
  370. std::string FilesPerLevel() {
  371. std::string result;
  372. int last_non_zero_offset = 0;
  373. for (int level = 0; level < config::kNumLevels; level++) {
  374. int f = NumTableFilesAtLevel(level);
  375. char buf[100];
  376. std::snprintf(buf, sizeof(buf), "%s%d", (level ? "," : ""), f);
  377. result += buf;
  378. if (f > 0) {
  379. last_non_zero_offset = result.size();
  380. }
  381. }
  382. result.resize(last_non_zero_offset);
  383. return result;
  384. }
  385. int CountFiles() {
  386. std::vector<std::string> files;
  387. env_->GetChildren(dbname_, &files);
  388. return static_cast<int>(files.size());
  389. }
  390. uint64_t Size(const Slice& start, const Slice& limit) {
  391. Range r(start, limit);
  392. uint64_t size;
  393. db_->GetApproximateSizes(&r, 1, &size);
  394. return size;
  395. }
  396. void Compact(const Slice& start, const Slice& limit) {
  397. db_->CompactRange(&start, &limit);
  398. }
  399. // Do n memtable compactions, each of which produces an sstable
  400. // covering the range [small_key,large_key].
  401. void MakeTables(int n, const std::string& small_key,
  402. const std::string& large_key) {
  403. for (int i = 0; i < n; i++) {
  404. Put(small_key, "begin");
  405. Put(large_key, "end");
  406. dbfull()->TEST_CompactMemTable();
  407. }
  408. }
  409. // Prevent pushing of new sstables into deeper levels by adding
  410. // tables that cover a specified range to all levels.
  411. void FillLevels(const std::string& smallest, const std::string& largest) {
  412. MakeTables(config::kNumLevels, smallest, largest);
  413. }
  414. void DumpFileCounts(const char* label) {
  415. std::fprintf(stderr, "---\n%s:\n", label);
  416. std::fprintf(
  417. stderr, "maxoverlap: %lld\n",
  418. static_cast<long long>(dbfull()->TEST_MaxNextLevelOverlappingBytes()));
  419. for (int level = 0; level < config::kNumLevels; level++) {
  420. int num = NumTableFilesAtLevel(level);
  421. if (num > 0) {
  422. std::fprintf(stderr, " level %3d : %d files\n", level, num);
  423. }
  424. }
  425. }
  426. std::string DumpSSTableList() {
  427. std::string property;
  428. db_->GetProperty("leveldb.sstables", &property);
  429. return property;
  430. }
  431. std::string IterStatus(Iterator* iter) {
  432. std::string result;
  433. if (iter->Valid()) {
  434. result = iter->key().ToString() + "->" + iter->value().ToString();
  435. } else {
  436. result = "(invalid)";
  437. }
  438. return result;
  439. }
  440. bool DeleteAnSSTFile() {
  441. std::vector<std::string> filenames;
  442. EXPECT_LEVELDB_OK(env_->GetChildren(dbname_, &filenames));
  443. uint64_t number;
  444. FileType type;
  445. for (size_t i = 0; i < filenames.size(); i++) {
  446. if (ParseFileName(filenames[i], &number, &type) && type == kTableFile) {
  447. EXPECT_LEVELDB_OK(env_->RemoveFile(TableFileName(dbname_, number)));
  448. return true;
  449. }
  450. }
  451. return false;
  452. }
  453. // Returns number of files renamed.
  454. int RenameLDBToSST() {
  455. std::vector<std::string> filenames;
  456. EXPECT_LEVELDB_OK(env_->GetChildren(dbname_, &filenames));
  457. uint64_t number;
  458. FileType type;
  459. int files_renamed = 0;
  460. for (size_t i = 0; i < filenames.size(); i++) {
  461. if (ParseFileName(filenames[i], &number, &type) && type == kTableFile) {
  462. const std::string from = TableFileName(dbname_, number);
  463. const std::string to = SSTTableFileName(dbname_, number);
  464. EXPECT_LEVELDB_OK(env_->RenameFile(from, to));
  465. files_renamed++;
  466. }
  467. }
  468. return files_renamed;
  469. }
  470. private:
  471. // Sequence of option configurations to try
  472. enum OptionConfig { kDefault, kReuse, kFilter, kUncompressed, kEnd };
  473. const FilterPolicy* filter_policy_;
  474. int option_config_;
  475. };
  476. TEST_F(DBTest, Empty) {
  477. do {
  478. ASSERT_TRUE(db_ != nullptr);
  479. ASSERT_EQ("NOT_FOUND", Get("foo"));
  480. } while (ChangeOptions());
  481. }
  482. TEST_F(DBTest, EmptyKey) {
  483. do {
  484. ASSERT_LEVELDB_OK(Put("", "v1"));
  485. ASSERT_EQ("v1", Get(""));
  486. ASSERT_LEVELDB_OK(Put("", "v2"));
  487. ASSERT_EQ("v2", Get(""));
  488. } while (ChangeOptions());
  489. }
  490. TEST_F(DBTest, EmptyValue) {
  491. do {
  492. ASSERT_LEVELDB_OK(Put("key", "v1"));
  493. ASSERT_EQ("v1", Get("key"));
  494. ASSERT_LEVELDB_OK(Put("key", ""));
  495. ASSERT_EQ("", Get("key"));
  496. ASSERT_LEVELDB_OK(Put("key", "v2"));
  497. ASSERT_EQ("v2", Get("key"));
  498. } while (ChangeOptions());
  499. }
  500. TEST_F(DBTest, ReadWrite) {
  501. do {
  502. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  503. ASSERT_EQ("v1", Get("foo"));
  504. ASSERT_LEVELDB_OK(Put("bar", "v2"));
  505. ASSERT_LEVELDB_OK(Put("foo", "v3"));
  506. ASSERT_EQ("v3", Get("foo"));
  507. ASSERT_EQ("v2", Get("bar"));
  508. } while (ChangeOptions());
  509. }
  510. TEST_F(DBTest, PutDeleteGet) {
  511. do {
  512. ASSERT_LEVELDB_OK(db_->Put(WriteOptions(), "foo", "v1"));
  513. ASSERT_EQ("v1", Get("foo"));
  514. ASSERT_LEVELDB_OK(db_->Put(WriteOptions(), "foo", "v2"));
  515. ASSERT_EQ("v2", Get("foo"));
  516. ASSERT_LEVELDB_OK(db_->Delete(WriteOptions(), "foo"));
  517. ASSERT_EQ("NOT_FOUND", Get("foo"));
  518. } while (ChangeOptions());
  519. }
  520. TEST_F(DBTest, GetFromImmutableLayer) {
  521. do {
  522. Options options = CurrentOptions();
  523. options.env = env_;
  524. options.write_buffer_size = 100000; // Small write buffer
  525. Reopen(&options);
  526. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  527. ASSERT_EQ("v1", Get("foo"));
  528. // Block sync calls.
  529. env_->delay_data_sync_.store(true, std::memory_order_release);
  530. Put("k1", std::string(100000, 'x')); // Fill memtable.
  531. Put("k2", std::string(100000, 'y')); // Trigger compaction.
  532. ASSERT_EQ("v1", Get("foo"));
  533. // Release sync calls.
  534. env_->delay_data_sync_.store(false, std::memory_order_release);
  535. } while (ChangeOptions());
  536. }
  537. TEST_F(DBTest, GetFromVersions) {
  538. do {
  539. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  540. dbfull()->TEST_CompactMemTable();
  541. ASSERT_EQ("v1", Get("foo"));
  542. } while (ChangeOptions());
  543. }
  544. TEST_F(DBTest, GetMemUsage) {
  545. do {
  546. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  547. std::string val;
  548. ASSERT_TRUE(db_->GetProperty("leveldb.approximate-memory-usage", &val));
  549. int mem_usage = std::stoi(val);
  550. ASSERT_GT(mem_usage, 0);
  551. ASSERT_LT(mem_usage, 5 * 1024 * 1024);
  552. } while (ChangeOptions());
  553. }
  554. TEST_F(DBTest, GetSnapshot) {
  555. do {
  556. // Try with both a short key and a long key
  557. for (int i = 0; i < 2; i++) {
  558. std::string key = (i == 0) ? std::string("foo") : std::string(200, 'x');
  559. ASSERT_LEVELDB_OK(Put(key, "v1"));
  560. const Snapshot* s1 = db_->GetSnapshot();
  561. ASSERT_LEVELDB_OK(Put(key, "v2"));
  562. ASSERT_EQ("v2", Get(key));
  563. ASSERT_EQ("v1", Get(key, s1));
  564. dbfull()->TEST_CompactMemTable();
  565. ASSERT_EQ("v2", Get(key));
  566. ASSERT_EQ("v1", Get(key, s1));
  567. db_->ReleaseSnapshot(s1);
  568. }
  569. } while (ChangeOptions());
  570. }
  571. TEST_F(DBTest, GetIdenticalSnapshots) {
  572. do {
  573. // Try with both a short key and a long key
  574. for (int i = 0; i < 2; i++) {
  575. std::string key = (i == 0) ? std::string("foo") : std::string(200, 'x');
  576. ASSERT_LEVELDB_OK(Put(key, "v1"));
  577. const Snapshot* s1 = db_->GetSnapshot();
  578. const Snapshot* s2 = db_->GetSnapshot();
  579. const Snapshot* s3 = db_->GetSnapshot();
  580. ASSERT_LEVELDB_OK(Put(key, "v2"));
  581. ASSERT_EQ("v2", Get(key));
  582. ASSERT_EQ("v1", Get(key, s1));
  583. ASSERT_EQ("v1", Get(key, s2));
  584. ASSERT_EQ("v1", Get(key, s3));
  585. db_->ReleaseSnapshot(s1);
  586. dbfull()->TEST_CompactMemTable();
  587. ASSERT_EQ("v2", Get(key));
  588. ASSERT_EQ("v1", Get(key, s2));
  589. db_->ReleaseSnapshot(s2);
  590. ASSERT_EQ("v1", Get(key, s3));
  591. db_->ReleaseSnapshot(s3);
  592. }
  593. } while (ChangeOptions());
  594. }
  595. TEST_F(DBTest, IterateOverEmptySnapshot) {
  596. do {
  597. const Snapshot* snapshot = db_->GetSnapshot();
  598. ReadOptions read_options;
  599. read_options.snapshot = snapshot;
  600. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  601. ASSERT_LEVELDB_OK(Put("foo", "v2"));
  602. Iterator* iterator1 = db_->NewIterator(read_options);
  603. iterator1->SeekToFirst();
  604. ASSERT_TRUE(!iterator1->Valid());
  605. delete iterator1;
  606. dbfull()->TEST_CompactMemTable();
  607. Iterator* iterator2 = db_->NewIterator(read_options);
  608. iterator2->SeekToFirst();
  609. ASSERT_TRUE(!iterator2->Valid());
  610. delete iterator2;
  611. db_->ReleaseSnapshot(snapshot);
  612. } while (ChangeOptions());
  613. }
  614. TEST_F(DBTest, GetLevel0Ordering) {
  615. do {
  616. // Check that we process level-0 files in correct order. The code
  617. // below generates two level-0 files where the earlier one comes
  618. // before the later one in the level-0 file list since the earlier
  619. // one has a smaller "smallest" key.
  620. ASSERT_LEVELDB_OK(Put("bar", "b"));
  621. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  622. dbfull()->TEST_CompactMemTable();
  623. ASSERT_LEVELDB_OK(Put("foo", "v2"));
  624. dbfull()->TEST_CompactMemTable();
  625. ASSERT_EQ("v2", Get("foo"));
  626. } while (ChangeOptions());
  627. }
  628. TEST_F(DBTest, GetOrderedByLevels) {
  629. do {
  630. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  631. Compact("a", "z");
  632. ASSERT_EQ("v1", Get("foo"));
  633. ASSERT_LEVELDB_OK(Put("foo", "v2"));
  634. ASSERT_EQ("v2", Get("foo"));
  635. dbfull()->TEST_CompactMemTable();
  636. ASSERT_EQ("v2", Get("foo"));
  637. } while (ChangeOptions());
  638. }
  639. TEST_F(DBTest, GetPicksCorrectFile) {
  640. do {
  641. // Arrange to have multiple files in a non-level-0 level.
  642. ASSERT_LEVELDB_OK(Put("a", "va"));
  643. Compact("a", "b");
  644. ASSERT_LEVELDB_OK(Put("x", "vx"));
  645. Compact("x", "y");
  646. ASSERT_LEVELDB_OK(Put("f", "vf"));
  647. Compact("f", "g");
  648. ASSERT_EQ("va", Get("a"));
  649. ASSERT_EQ("vf", Get("f"));
  650. ASSERT_EQ("vx", Get("x"));
  651. } while (ChangeOptions());
  652. }
  653. TEST_F(DBTest, GetEncountersEmptyLevel) {
  654. do {
  655. // Arrange for the following to happen:
  656. // * sstable A in level 0
  657. // * nothing in level 1
  658. // * sstable B in level 2
  659. // Then do enough Get() calls to arrange for an automatic compaction
  660. // of sstable A. A bug would cause the compaction to be marked as
  661. // occurring at level 1 (instead of the correct level 0).
  662. // Step 1: First place sstables in levels 0 and 2
  663. int compaction_count = 0;
  664. while (NumTableFilesAtLevel(0) == 0 || NumTableFilesAtLevel(2) == 0) {
  665. ASSERT_LE(compaction_count, 100) << "could not fill levels 0 and 2";
  666. compaction_count++;
  667. Put("a", "begin");
  668. Put("z", "end");
  669. dbfull()->TEST_CompactMemTable();
  670. }
  671. // Step 2: clear level 1 if necessary.
  672. dbfull()->TEST_CompactRange(1, nullptr, nullptr);
  673. ASSERT_EQ(NumTableFilesAtLevel(0), 1);
  674. ASSERT_EQ(NumTableFilesAtLevel(1), 0);
  675. ASSERT_EQ(NumTableFilesAtLevel(2), 1);
  676. // Step 3: read a bunch of times
  677. for (int i = 0; i < 1000; i++) {
  678. ASSERT_EQ("NOT_FOUND", Get("missing"));
  679. }
  680. // Step 4: Wait for compaction to finish
  681. DelayMilliseconds(1000);
  682. ASSERT_EQ(NumTableFilesAtLevel(0), 0);
  683. } while (ChangeOptions());
  684. }
  685. TEST_F(DBTest, IterEmpty) {
  686. Iterator* iter = db_->NewIterator(ReadOptions());
  687. iter->SeekToFirst();
  688. ASSERT_EQ(IterStatus(iter), "(invalid)");
  689. iter->SeekToLast();
  690. ASSERT_EQ(IterStatus(iter), "(invalid)");
  691. iter->Seek("foo");
  692. ASSERT_EQ(IterStatus(iter), "(invalid)");
  693. delete iter;
  694. }
  695. TEST_F(DBTest, IterSingle) {
  696. ASSERT_LEVELDB_OK(Put("a", "va"));
  697. Iterator* iter = db_->NewIterator(ReadOptions());
  698. iter->SeekToFirst();
  699. ASSERT_EQ(IterStatus(iter), "a->va");
  700. iter->Next();
  701. ASSERT_EQ(IterStatus(iter), "(invalid)");
  702. iter->SeekToFirst();
  703. ASSERT_EQ(IterStatus(iter), "a->va");
  704. iter->Prev();
  705. ASSERT_EQ(IterStatus(iter), "(invalid)");
  706. iter->SeekToLast();
  707. ASSERT_EQ(IterStatus(iter), "a->va");
  708. iter->Next();
  709. ASSERT_EQ(IterStatus(iter), "(invalid)");
  710. iter->SeekToLast();
  711. ASSERT_EQ(IterStatus(iter), "a->va");
  712. iter->Prev();
  713. ASSERT_EQ(IterStatus(iter), "(invalid)");
  714. iter->Seek("");
  715. ASSERT_EQ(IterStatus(iter), "a->va");
  716. iter->Next();
  717. ASSERT_EQ(IterStatus(iter), "(invalid)");
  718. iter->Seek("a");
  719. ASSERT_EQ(IterStatus(iter), "a->va");
  720. iter->Next();
  721. ASSERT_EQ(IterStatus(iter), "(invalid)");
  722. iter->Seek("b");
  723. ASSERT_EQ(IterStatus(iter), "(invalid)");
  724. delete iter;
  725. }
  726. TEST_F(DBTest, IterMulti) {
  727. ASSERT_LEVELDB_OK(Put("a", "va"));
  728. ASSERT_LEVELDB_OK(Put("b", "vb"));
  729. ASSERT_LEVELDB_OK(Put("c", "vc"));
  730. Iterator* iter = db_->NewIterator(ReadOptions());
  731. iter->SeekToFirst();
  732. ASSERT_EQ(IterStatus(iter), "a->va");
  733. iter->Next();
  734. ASSERT_EQ(IterStatus(iter), "b->vb");
  735. iter->Next();
  736. ASSERT_EQ(IterStatus(iter), "c->vc");
  737. iter->Next();
  738. ASSERT_EQ(IterStatus(iter), "(invalid)");
  739. iter->SeekToFirst();
  740. ASSERT_EQ(IterStatus(iter), "a->va");
  741. iter->Prev();
  742. ASSERT_EQ(IterStatus(iter), "(invalid)");
  743. iter->SeekToLast();
  744. ASSERT_EQ(IterStatus(iter), "c->vc");
  745. iter->Prev();
  746. ASSERT_EQ(IterStatus(iter), "b->vb");
  747. iter->Prev();
  748. ASSERT_EQ(IterStatus(iter), "a->va");
  749. iter->Prev();
  750. ASSERT_EQ(IterStatus(iter), "(invalid)");
  751. iter->SeekToLast();
  752. ASSERT_EQ(IterStatus(iter), "c->vc");
  753. iter->Next();
  754. ASSERT_EQ(IterStatus(iter), "(invalid)");
  755. iter->Seek("");
  756. ASSERT_EQ(IterStatus(iter), "a->va");
  757. iter->Seek("a");
  758. ASSERT_EQ(IterStatus(iter), "a->va");
  759. iter->Seek("ax");
  760. ASSERT_EQ(IterStatus(iter), "b->vb");
  761. iter->Seek("b");
  762. ASSERT_EQ(IterStatus(iter), "b->vb");
  763. iter->Seek("z");
  764. ASSERT_EQ(IterStatus(iter), "(invalid)");
  765. // Switch from reverse to forward
  766. iter->SeekToLast();
  767. iter->Prev();
  768. iter->Prev();
  769. iter->Next();
  770. ASSERT_EQ(IterStatus(iter), "b->vb");
  771. // Switch from forward to reverse
  772. iter->SeekToFirst();
  773. iter->Next();
  774. iter->Next();
  775. iter->Prev();
  776. ASSERT_EQ(IterStatus(iter), "b->vb");
  777. // Make sure iter stays at snapshot
  778. ASSERT_LEVELDB_OK(Put("a", "va2"));
  779. ASSERT_LEVELDB_OK(Put("a2", "va3"));
  780. ASSERT_LEVELDB_OK(Put("b", "vb2"));
  781. ASSERT_LEVELDB_OK(Put("c", "vc2"));
  782. ASSERT_LEVELDB_OK(Delete("b"));
  783. iter->SeekToFirst();
  784. ASSERT_EQ(IterStatus(iter), "a->va");
  785. iter->Next();
  786. ASSERT_EQ(IterStatus(iter), "b->vb");
  787. iter->Next();
  788. ASSERT_EQ(IterStatus(iter), "c->vc");
  789. iter->Next();
  790. ASSERT_EQ(IterStatus(iter), "(invalid)");
  791. iter->SeekToLast();
  792. ASSERT_EQ(IterStatus(iter), "c->vc");
  793. iter->Prev();
  794. ASSERT_EQ(IterStatus(iter), "b->vb");
  795. iter->Prev();
  796. ASSERT_EQ(IterStatus(iter), "a->va");
  797. iter->Prev();
  798. ASSERT_EQ(IterStatus(iter), "(invalid)");
  799. delete iter;
  800. }
  801. TEST_F(DBTest, IterSmallAndLargeMix) {
  802. ASSERT_LEVELDB_OK(Put("a", "va"));
  803. ASSERT_LEVELDB_OK(Put("b", std::string(100000, 'b')));
  804. ASSERT_LEVELDB_OK(Put("c", "vc"));
  805. ASSERT_LEVELDB_OK(Put("d", std::string(100000, 'd')));
  806. ASSERT_LEVELDB_OK(Put("e", std::string(100000, 'e')));
  807. Iterator* iter = db_->NewIterator(ReadOptions());
  808. iter->SeekToFirst();
  809. ASSERT_EQ(IterStatus(iter), "a->va");
  810. iter->Next();
  811. ASSERT_EQ(IterStatus(iter), "b->" + std::string(100000, 'b'));
  812. iter->Next();
  813. ASSERT_EQ(IterStatus(iter), "c->vc");
  814. iter->Next();
  815. ASSERT_EQ(IterStatus(iter), "d->" + std::string(100000, 'd'));
  816. iter->Next();
  817. ASSERT_EQ(IterStatus(iter), "e->" + std::string(100000, 'e'));
  818. iter->Next();
  819. ASSERT_EQ(IterStatus(iter), "(invalid)");
  820. iter->SeekToLast();
  821. ASSERT_EQ(IterStatus(iter), "e->" + std::string(100000, 'e'));
  822. iter->Prev();
  823. ASSERT_EQ(IterStatus(iter), "d->" + std::string(100000, 'd'));
  824. iter->Prev();
  825. ASSERT_EQ(IterStatus(iter), "c->vc");
  826. iter->Prev();
  827. ASSERT_EQ(IterStatus(iter), "b->" + std::string(100000, 'b'));
  828. iter->Prev();
  829. ASSERT_EQ(IterStatus(iter), "a->va");
  830. iter->Prev();
  831. ASSERT_EQ(IterStatus(iter), "(invalid)");
  832. delete iter;
  833. }
  834. TEST_F(DBTest, IterMultiWithDelete) {
  835. do {
  836. ASSERT_LEVELDB_OK(Put("a", "va"));
  837. ASSERT_LEVELDB_OK(Put("b", "vb"));
  838. ASSERT_LEVELDB_OK(Put("c", "vc"));
  839. ASSERT_LEVELDB_OK(Delete("b"));
  840. ASSERT_EQ("NOT_FOUND", Get("b"));
  841. Iterator* iter = db_->NewIterator(ReadOptions());
  842. iter->Seek("c");
  843. ASSERT_EQ(IterStatus(iter), "c->vc");
  844. iter->Prev();
  845. ASSERT_EQ(IterStatus(iter), "a->va");
  846. delete iter;
  847. } while (ChangeOptions());
  848. }
  849. TEST_F(DBTest, IterMultiWithDeleteAndCompaction) {
  850. do {
  851. ASSERT_LEVELDB_OK(Put("b", "vb"));
  852. ASSERT_LEVELDB_OK(Put("c", "vc"));
  853. ASSERT_LEVELDB_OK(Put("a", "va"));
  854. dbfull()->TEST_CompactMemTable();
  855. ASSERT_LEVELDB_OK(Delete("b"));
  856. ASSERT_EQ("NOT_FOUND", Get("b"));
  857. Iterator* iter = db_->NewIterator(ReadOptions());
  858. iter->Seek("c");
  859. ASSERT_EQ(IterStatus(iter), "c->vc");
  860. iter->Prev();
  861. ASSERT_EQ(IterStatus(iter), "a->va");
  862. iter->Seek("b");
  863. ASSERT_EQ(IterStatus(iter), "c->vc");
  864. delete iter;
  865. } while (ChangeOptions());
  866. }
  867. TEST_F(DBTest, Recover) {
  868. do {
  869. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  870. ASSERT_LEVELDB_OK(Put("baz", "v5"));
  871. Reopen();
  872. ASSERT_EQ("v1", Get("foo"));
  873. ASSERT_EQ("v1", Get("foo"));
  874. ASSERT_EQ("v5", Get("baz"));
  875. ASSERT_LEVELDB_OK(Put("bar", "v2"));
  876. ASSERT_LEVELDB_OK(Put("foo", "v3"));
  877. Reopen();
  878. ASSERT_EQ("v3", Get("foo"));
  879. ASSERT_LEVELDB_OK(Put("foo", "v4"));
  880. ASSERT_EQ("v4", Get("foo"));
  881. ASSERT_EQ("v2", Get("bar"));
  882. ASSERT_EQ("v5", Get("baz"));
  883. } while (ChangeOptions());
  884. }
  885. TEST_F(DBTest, RecoveryWithEmptyLog) {
  886. do {
  887. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  888. ASSERT_LEVELDB_OK(Put("foo", "v2"));
  889. Reopen();
  890. Reopen();
  891. ASSERT_LEVELDB_OK(Put("foo", "v3"));
  892. Reopen();
  893. ASSERT_EQ("v3", Get("foo"));
  894. } while (ChangeOptions());
  895. }
  896. // Check that writes done during a memtable compaction are recovered
  897. // if the database is shutdown during the memtable compaction.
  898. TEST_F(DBTest, RecoverDuringMemtableCompaction) {
  899. do {
  900. Options options = CurrentOptions();
  901. options.env = env_;
  902. options.write_buffer_size = 1000000;
  903. Reopen(&options);
  904. // Trigger a long memtable compaction and reopen the database during it
  905. ASSERT_LEVELDB_OK(Put("foo", "v1")); // Goes to 1st log file
  906. ASSERT_LEVELDB_OK(
  907. Put("big1", std::string(10000000, 'x'))); // Fills memtable
  908. ASSERT_LEVELDB_OK(
  909. Put("big2", std::string(1000, 'y'))); // Triggers compaction
  910. ASSERT_LEVELDB_OK(Put("bar", "v2")); // Goes to new log file
  911. Reopen(&options);
  912. ASSERT_EQ("v1", Get("foo"));
  913. ASSERT_EQ("v2", Get("bar"));
  914. ASSERT_EQ(std::string(10000000, 'x'), Get("big1"));
  915. ASSERT_EQ(std::string(1000, 'y'), Get("big2"));
  916. } while (ChangeOptions());
  917. }
  918. static std::string Key(int i) {
  919. char buf[100];
  920. std::snprintf(buf, sizeof(buf), "key%06d", i);
  921. return std::string(buf);
  922. }
  923. TEST_F(DBTest, MinorCompactionsHappen) {
  924. Options options = CurrentOptions();
  925. options.write_buffer_size = 10000;
  926. Reopen(&options);
  927. const int N = 500;
  928. int starting_num_tables = TotalTableFiles();
  929. for (int i = 0; i < N; i++) {
  930. ASSERT_LEVELDB_OK(Put(Key(i), Key(i) + std::string(1000, 'v')));
  931. }
  932. int ending_num_tables = TotalTableFiles();
  933. ASSERT_GT(ending_num_tables, starting_num_tables);
  934. for (int i = 0; i < N; i++) {
  935. ASSERT_EQ(Key(i) + std::string(1000, 'v'), Get(Key(i)));
  936. }
  937. Reopen();
  938. for (int i = 0; i < N; i++) {
  939. ASSERT_EQ(Key(i) + std::string(1000, 'v'), Get(Key(i)));
  940. }
  941. }
  942. TEST_F(DBTest, RecoverWithLargeLog) {
  943. {
  944. Options options = CurrentOptions();
  945. Reopen(&options);
  946. ASSERT_LEVELDB_OK(Put("big1", std::string(200000, '1')));
  947. ASSERT_LEVELDB_OK(Put("big2", std::string(200000, '2')));
  948. ASSERT_LEVELDB_OK(Put("small3", std::string(10, '3')));
  949. ASSERT_LEVELDB_OK(Put("small4", std::string(10, '4')));
  950. ASSERT_EQ(NumTableFilesAtLevel(0), 0);
  951. }
  952. // Make sure that if we re-open with a small write buffer size that
  953. // we flush table files in the middle of a large log file.
  954. Options options = CurrentOptions();
  955. options.write_buffer_size = 100000;
  956. Reopen(&options);
  957. ASSERT_EQ(NumTableFilesAtLevel(0), 3);
  958. ASSERT_EQ(std::string(200000, '1'), Get("big1"));
  959. ASSERT_EQ(std::string(200000, '2'), Get("big2"));
  960. ASSERT_EQ(std::string(10, '3'), Get("small3"));
  961. ASSERT_EQ(std::string(10, '4'), Get("small4"));
  962. ASSERT_GT(NumTableFilesAtLevel(0), 1);
  963. }
  964. TEST_F(DBTest, CompactionsGenerateMultipleFiles) {
  965. Options options = CurrentOptions();
  966. options.write_buffer_size = 100000000; // Large write buffer
  967. Reopen(&options);
  968. Random rnd(301);
  969. // Write 8MB (80 values, each 100K)
  970. ASSERT_EQ(NumTableFilesAtLevel(0), 0);
  971. std::vector<std::string> values;
  972. for (int i = 0; i < 80; i++) {
  973. values.push_back(RandomString(&rnd, 100000));
  974. ASSERT_LEVELDB_OK(Put(Key(i), values[i]));
  975. }
  976. // Reopening moves updates to level-0
  977. Reopen(&options);
  978. dbfull()->TEST_CompactRange(0, nullptr, nullptr);
  979. ASSERT_EQ(NumTableFilesAtLevel(0), 0);
  980. ASSERT_GT(NumTableFilesAtLevel(1), 1);
  981. for (int i = 0; i < 80; i++) {
  982. ASSERT_EQ(Get(Key(i)), values[i]);
  983. }
  984. }
  985. TEST_F(DBTest, RepeatedWritesToSameKey) {
  986. Options options = CurrentOptions();
  987. options.env = env_;
  988. options.write_buffer_size = 100000; // Small write buffer
  989. Reopen(&options);
  990. // We must have at most one file per level except for level-0,
  991. // which may have up to kL0_StopWritesTrigger files.
  992. const int kMaxFiles = config::kNumLevels + config::kL0_StopWritesTrigger;
  993. Random rnd(301);
  994. std::string value = RandomString(&rnd, 2 * options.write_buffer_size);
  995. for (int i = 0; i < 5 * kMaxFiles; i++) {
  996. Put("key", value);
  997. ASSERT_LE(TotalTableFiles(), kMaxFiles);
  998. std::fprintf(stderr, "after %d: %d files\n", i + 1, TotalTableFiles());
  999. }
  1000. }
  1001. TEST_F(DBTest, SparseMerge) {
  1002. Options options = CurrentOptions();
  1003. options.compression = kNoCompression;
  1004. Reopen(&options);
  1005. FillLevels("A", "Z");
  1006. // Suppose there is:
  1007. // small amount of data with prefix A
  1008. // large amount of data with prefix B
  1009. // small amount of data with prefix C
  1010. // and that recent updates have made small changes to all three prefixes.
  1011. // Check that we do not do a compaction that merges all of B in one shot.
  1012. const std::string value(1000, 'x');
  1013. Put("A", "va");
  1014. // Write approximately 100MB of "B" values
  1015. for (int i = 0; i < 100000; i++) {
  1016. char key[100];
  1017. std::snprintf(key, sizeof(key), "B%010d", i);
  1018. Put(key, value);
  1019. }
  1020. Put("C", "vc");
  1021. dbfull()->TEST_CompactMemTable();
  1022. dbfull()->TEST_CompactRange(0, nullptr, nullptr);
  1023. // Make sparse update
  1024. Put("A", "va2");
  1025. Put("B100", "bvalue2");
  1026. Put("C", "vc2");
  1027. dbfull()->TEST_CompactMemTable();
  1028. // Compactions should not cause us to create a situation where
  1029. // a file overlaps too much data at the next level.
  1030. ASSERT_LE(dbfull()->TEST_MaxNextLevelOverlappingBytes(), 20 * 1048576);
  1031. dbfull()->TEST_CompactRange(0, nullptr, nullptr);
  1032. ASSERT_LE(dbfull()->TEST_MaxNextLevelOverlappingBytes(), 20 * 1048576);
  1033. dbfull()->TEST_CompactRange(1, nullptr, nullptr);
  1034. ASSERT_LE(dbfull()->TEST_MaxNextLevelOverlappingBytes(), 20 * 1048576);
  1035. }
  1036. static bool Between(uint64_t val, uint64_t low, uint64_t high) {
  1037. bool result = (val >= low) && (val <= high);
  1038. if (!result) {
  1039. std::fprintf(stderr, "Value %llu is not in range [%llu, %llu]\n",
  1040. (unsigned long long)(val), (unsigned long long)(low),
  1041. (unsigned long long)(high));
  1042. }
  1043. return result;
  1044. }
  1045. TEST_F(DBTest, ApproximateSizes) {
  1046. do {
  1047. Options options = CurrentOptions();
  1048. options.write_buffer_size = 100000000; // Large write buffer
  1049. options.compression = kNoCompression;
  1050. DestroyAndReopen();
  1051. ASSERT_TRUE(Between(Size("", "xyz"), 0, 0));
  1052. Reopen(&options);
  1053. ASSERT_TRUE(Between(Size("", "xyz"), 0, 0));
  1054. // Write 8MB (80 values, each 100K)
  1055. ASSERT_EQ(NumTableFilesAtLevel(0), 0);
  1056. const int N = 80;
  1057. static const int S1 = 100000;
  1058. static const int S2 = 105000; // Allow some expansion from metadata
  1059. Random rnd(301);
  1060. for (int i = 0; i < N; i++) {
  1061. ASSERT_LEVELDB_OK(Put(Key(i), RandomString(&rnd, S1)));
  1062. }
  1063. // 0 because GetApproximateSizes() does not account for memtable space
  1064. ASSERT_TRUE(Between(Size("", Key(50)), 0, 0));
  1065. if (options.reuse_logs) {
  1066. // Recovery will reuse memtable, and GetApproximateSizes() does not
  1067. // account for memtable usage;
  1068. Reopen(&options);
  1069. ASSERT_TRUE(Between(Size("", Key(50)), 0, 0));
  1070. continue;
  1071. }
  1072. // Check sizes across recovery by reopening a few times
  1073. for (int run = 0; run < 3; run++) {
  1074. Reopen(&options);
  1075. for (int compact_start = 0; compact_start < N; compact_start += 10) {
  1076. for (int i = 0; i < N; i += 10) {
  1077. ASSERT_TRUE(Between(Size("", Key(i)), S1 * i, S2 * i));
  1078. ASSERT_TRUE(Between(Size("", Key(i) + ".suffix"), S1 * (i + 1),
  1079. S2 * (i + 1)));
  1080. ASSERT_TRUE(Between(Size(Key(i), Key(i + 10)), S1 * 10, S2 * 10));
  1081. }
  1082. ASSERT_TRUE(Between(Size("", Key(50)), S1 * 50, S2 * 50));
  1083. ASSERT_TRUE(Between(Size("", Key(50) + ".suffix"), S1 * 50, S2 * 50));
  1084. std::string cstart_str = Key(compact_start);
  1085. std::string cend_str = Key(compact_start + 9);
  1086. Slice cstart = cstart_str;
  1087. Slice cend = cend_str;
  1088. dbfull()->TEST_CompactRange(0, &cstart, &cend);
  1089. }
  1090. ASSERT_EQ(NumTableFilesAtLevel(0), 0);
  1091. ASSERT_GT(NumTableFilesAtLevel(1), 0);
  1092. }
  1093. } while (ChangeOptions());
  1094. }
  1095. TEST_F(DBTest, ApproximateSizes_MixOfSmallAndLarge) {
  1096. do {
  1097. Options options = CurrentOptions();
  1098. options.compression = kNoCompression;
  1099. Reopen();
  1100. Random rnd(301);
  1101. std::string big1 = RandomString(&rnd, 100000);
  1102. ASSERT_LEVELDB_OK(Put(Key(0), RandomString(&rnd, 10000)));
  1103. ASSERT_LEVELDB_OK(Put(Key(1), RandomString(&rnd, 10000)));
  1104. ASSERT_LEVELDB_OK(Put(Key(2), big1));
  1105. ASSERT_LEVELDB_OK(Put(Key(3), RandomString(&rnd, 10000)));
  1106. ASSERT_LEVELDB_OK(Put(Key(4), big1));
  1107. ASSERT_LEVELDB_OK(Put(Key(5), RandomString(&rnd, 10000)));
  1108. ASSERT_LEVELDB_OK(Put(Key(6), RandomString(&rnd, 300000)));
  1109. ASSERT_LEVELDB_OK(Put(Key(7), RandomString(&rnd, 10000)));
  1110. if (options.reuse_logs) {
  1111. // Need to force a memtable compaction since recovery does not do so.
  1112. ASSERT_LEVELDB_OK(dbfull()->TEST_CompactMemTable());
  1113. }
  1114. // Check sizes across recovery by reopening a few times
  1115. for (int run = 0; run < 3; run++) {
  1116. Reopen(&options);
  1117. ASSERT_TRUE(Between(Size("", Key(0)), 0, 0));
  1118. ASSERT_TRUE(Between(Size("", Key(1)), 10000, 11000));
  1119. ASSERT_TRUE(Between(Size("", Key(2)), 20000, 21000));
  1120. ASSERT_TRUE(Between(Size("", Key(3)), 120000, 121000));
  1121. ASSERT_TRUE(Between(Size("", Key(4)), 130000, 131000));
  1122. ASSERT_TRUE(Between(Size("", Key(5)), 230000, 231000));
  1123. ASSERT_TRUE(Between(Size("", Key(6)), 240000, 241000));
  1124. ASSERT_TRUE(Between(Size("", Key(7)), 540000, 541000));
  1125. ASSERT_TRUE(Between(Size("", Key(8)), 550000, 560000));
  1126. ASSERT_TRUE(Between(Size(Key(3), Key(5)), 110000, 111000));
  1127. dbfull()->TEST_CompactRange(0, nullptr, nullptr);
  1128. }
  1129. } while (ChangeOptions());
  1130. }
  1131. TEST_F(DBTest, IteratorPinsRef) {
  1132. Put("foo", "hello");
  1133. // Get iterator that will yield the current contents of the DB.
  1134. Iterator* iter = db_->NewIterator(ReadOptions());
  1135. // Write to force compactions
  1136. Put("foo", "newvalue1");
  1137. for (int i = 0; i < 100; i++) {
  1138. ASSERT_LEVELDB_OK(
  1139. Put(Key(i), Key(i) + std::string(100000, 'v'))); // 100K values
  1140. }
  1141. Put("foo", "newvalue2");
  1142. iter->SeekToFirst();
  1143. ASSERT_TRUE(iter->Valid());
  1144. ASSERT_EQ("foo", iter->key().ToString());
  1145. ASSERT_EQ("hello", iter->value().ToString());
  1146. iter->Next();
  1147. ASSERT_TRUE(!iter->Valid());
  1148. delete iter;
  1149. }
  1150. TEST_F(DBTest, Snapshot) {
  1151. do {
  1152. Put("foo", "v1");
  1153. const Snapshot* s1 = db_->GetSnapshot();
  1154. Put("foo", "v2");
  1155. const Snapshot* s2 = db_->GetSnapshot();
  1156. Put("foo", "v3");
  1157. const Snapshot* s3 = db_->GetSnapshot();
  1158. Put("foo", "v4");
  1159. ASSERT_EQ("v1", Get("foo", s1));
  1160. ASSERT_EQ("v2", Get("foo", s2));
  1161. ASSERT_EQ("v3", Get("foo", s3));
  1162. ASSERT_EQ("v4", Get("foo"));
  1163. db_->ReleaseSnapshot(s3);
  1164. ASSERT_EQ("v1", Get("foo", s1));
  1165. ASSERT_EQ("v2", Get("foo", s2));
  1166. ASSERT_EQ("v4", Get("foo"));
  1167. db_->ReleaseSnapshot(s1);
  1168. ASSERT_EQ("v2", Get("foo", s2));
  1169. ASSERT_EQ("v4", Get("foo"));
  1170. db_->ReleaseSnapshot(s2);
  1171. ASSERT_EQ("v4", Get("foo"));
  1172. } while (ChangeOptions());
  1173. }
  1174. TEST_F(DBTest, HiddenValuesAreRemoved) {
  1175. do {
  1176. Random rnd(301);
  1177. FillLevels("a", "z");
  1178. std::string big = RandomString(&rnd, 50000);
  1179. Put("foo", big);
  1180. Put("pastfoo", "v");
  1181. const Snapshot* snapshot = db_->GetSnapshot();
  1182. Put("foo", "tiny");
  1183. Put("pastfoo2", "v2"); // Advance sequence number one more
  1184. ASSERT_LEVELDB_OK(dbfull()->TEST_CompactMemTable());
  1185. ASSERT_GT(NumTableFilesAtLevel(0), 0);
  1186. ASSERT_EQ(big, Get("foo", snapshot));
  1187. ASSERT_TRUE(Between(Size("", "pastfoo"), 50000, 60000));
  1188. db_->ReleaseSnapshot(snapshot);
  1189. ASSERT_EQ(AllEntriesFor("foo"), "[ tiny, " + big + " ]");
  1190. Slice x("x");
  1191. dbfull()->TEST_CompactRange(0, nullptr, &x);
  1192. ASSERT_EQ(AllEntriesFor("foo"), "[ tiny ]");
  1193. ASSERT_EQ(NumTableFilesAtLevel(0), 0);
  1194. ASSERT_GE(NumTableFilesAtLevel(1), 1);
  1195. dbfull()->TEST_CompactRange(1, nullptr, &x);
  1196. ASSERT_EQ(AllEntriesFor("foo"), "[ tiny ]");
  1197. ASSERT_TRUE(Between(Size("", "pastfoo"), 0, 1000));
  1198. } while (ChangeOptions());
  1199. }
  1200. TEST_F(DBTest, DeletionMarkers1) {
  1201. Put("foo", "v1");
  1202. ASSERT_LEVELDB_OK(dbfull()->TEST_CompactMemTable());
  1203. const int last = config::kMaxMemCompactLevel;
  1204. ASSERT_EQ(NumTableFilesAtLevel(last), 1); // foo => v1 is now in last level
  1205. // Place a table at level last-1 to prevent merging with preceding mutation
  1206. Put("a", "begin");
  1207. Put("z", "end");
  1208. dbfull()->TEST_CompactMemTable();
  1209. ASSERT_EQ(NumTableFilesAtLevel(last), 1);
  1210. ASSERT_EQ(NumTableFilesAtLevel(last - 1), 1);
  1211. Delete("foo");
  1212. Put("foo", "v2");
  1213. ASSERT_EQ(AllEntriesFor("foo"), "[ v2, DEL, v1 ]");
  1214. ASSERT_LEVELDB_OK(dbfull()->TEST_CompactMemTable()); // Moves to level last-2
  1215. ASSERT_EQ(AllEntriesFor("foo"), "[ v2, DEL, v1 ]");
  1216. Slice z("z");
  1217. dbfull()->TEST_CompactRange(last - 2, nullptr, &z);
  1218. // DEL eliminated, but v1 remains because we aren't compacting that level
  1219. // (DEL can be eliminated because v2 hides v1).
  1220. ASSERT_EQ(AllEntriesFor("foo"), "[ v2, v1 ]");
  1221. dbfull()->TEST_CompactRange(last - 1, nullptr, nullptr);
  1222. // Merging last-1 w/ last, so we are the base level for "foo", so
  1223. // DEL is removed. (as is v1).
  1224. ASSERT_EQ(AllEntriesFor("foo"), "[ v2 ]");
  1225. }
  1226. TEST_F(DBTest, DeletionMarkers2) {
  1227. Put("foo", "v1");
  1228. ASSERT_LEVELDB_OK(dbfull()->TEST_CompactMemTable());
  1229. const int last = config::kMaxMemCompactLevel;
  1230. ASSERT_EQ(NumTableFilesAtLevel(last), 1); // foo => v1 is now in last level
  1231. // Place a table at level last-1 to prevent merging with preceding mutation
  1232. Put("a", "begin");
  1233. Put("z", "end");
  1234. dbfull()->TEST_CompactMemTable();
  1235. ASSERT_EQ(NumTableFilesAtLevel(last), 1);
  1236. ASSERT_EQ(NumTableFilesAtLevel(last - 1), 1);
  1237. Delete("foo");
  1238. ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, v1 ]");
  1239. ASSERT_LEVELDB_OK(dbfull()->TEST_CompactMemTable()); // Moves to level last-2
  1240. ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, v1 ]");
  1241. dbfull()->TEST_CompactRange(last - 2, nullptr, nullptr);
  1242. // DEL kept: "last" file overlaps
  1243. ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, v1 ]");
  1244. dbfull()->TEST_CompactRange(last - 1, nullptr, nullptr);
  1245. // Merging last-1 w/ last, so we are the base level for "foo", so
  1246. // DEL is removed. (as is v1).
  1247. ASSERT_EQ(AllEntriesFor("foo"), "[ ]");
  1248. }
  1249. TEST_F(DBTest, OverlapInLevel0) {
  1250. do {
  1251. ASSERT_EQ(config::kMaxMemCompactLevel, 2) << "Fix test to match config";
  1252. // Fill levels 1 and 2 to disable the pushing of new memtables to levels >
  1253. // 0.
  1254. ASSERT_LEVELDB_OK(Put("100", "v100"));
  1255. ASSERT_LEVELDB_OK(Put("999", "v999"));
  1256. dbfull()->TEST_CompactMemTable();
  1257. ASSERT_LEVELDB_OK(Delete("100"));
  1258. ASSERT_LEVELDB_OK(Delete("999"));
  1259. dbfull()->TEST_CompactMemTable();
  1260. ASSERT_EQ("0,1,1", FilesPerLevel());
  1261. // Make files spanning the following ranges in level-0:
  1262. // files[0] 200 .. 900
  1263. // files[1] 300 .. 500
  1264. // Note that files are sorted by smallest key.
  1265. ASSERT_LEVELDB_OK(Put("300", "v300"));
  1266. ASSERT_LEVELDB_OK(Put("500", "v500"));
  1267. dbfull()->TEST_CompactMemTable();
  1268. ASSERT_LEVELDB_OK(Put("200", "v200"));
  1269. ASSERT_LEVELDB_OK(Put("600", "v600"));
  1270. ASSERT_LEVELDB_OK(Put("900", "v900"));
  1271. dbfull()->TEST_CompactMemTable();
  1272. ASSERT_EQ("2,1,1", FilesPerLevel());
  1273. // Compact away the placeholder files we created initially
  1274. dbfull()->TEST_CompactRange(1, nullptr, nullptr);
  1275. dbfull()->TEST_CompactRange(2, nullptr, nullptr);
  1276. ASSERT_EQ("2", FilesPerLevel());
  1277. // Do a memtable compaction. Before bug-fix, the compaction would
  1278. // not detect the overlap with level-0 files and would incorrectly place
  1279. // the deletion in a deeper level.
  1280. ASSERT_LEVELDB_OK(Delete("600"));
  1281. dbfull()->TEST_CompactMemTable();
  1282. ASSERT_EQ("3", FilesPerLevel());
  1283. ASSERT_EQ("NOT_FOUND", Get("600"));
  1284. } while (ChangeOptions());
  1285. }
  1286. TEST_F(DBTest, L0_CompactionBug_Issue44_a) {
  1287. Reopen();
  1288. ASSERT_LEVELDB_OK(Put("b", "v"));
  1289. Reopen();
  1290. ASSERT_LEVELDB_OK(Delete("b"));
  1291. ASSERT_LEVELDB_OK(Delete("a"));
  1292. Reopen();
  1293. ASSERT_LEVELDB_OK(Delete("a"));
  1294. Reopen();
  1295. ASSERT_LEVELDB_OK(Put("a", "v"));
  1296. Reopen();
  1297. Reopen();
  1298. ASSERT_EQ("(a->v)", Contents());
  1299. DelayMilliseconds(1000); // Wait for compaction to finish
  1300. ASSERT_EQ("(a->v)", Contents());
  1301. }
  1302. TEST_F(DBTest, L0_CompactionBug_Issue44_b) {
  1303. Reopen();
  1304. Put("", "");
  1305. Reopen();
  1306. Delete("e");
  1307. Put("", "");
  1308. Reopen();
  1309. Put("c", "cv");
  1310. Reopen();
  1311. Put("", "");
  1312. Reopen();
  1313. Put("", "");
  1314. DelayMilliseconds(1000); // Wait for compaction to finish
  1315. Reopen();
  1316. Put("d", "dv");
  1317. Reopen();
  1318. Put("", "");
  1319. Reopen();
  1320. Delete("d");
  1321. Delete("b");
  1322. Reopen();
  1323. ASSERT_EQ("(->)(c->cv)", Contents());
  1324. DelayMilliseconds(1000); // Wait for compaction to finish
  1325. ASSERT_EQ("(->)(c->cv)", Contents());
  1326. }
  1327. TEST_F(DBTest, Fflush_Issue474) {
  1328. static const int kNum = 100000;
  1329. Random rnd(test::RandomSeed());
  1330. for (int i = 0; i < kNum; i++) {
  1331. std::fflush(nullptr);
  1332. ASSERT_LEVELDB_OK(Put(RandomKey(&rnd), RandomString(&rnd, 100)));
  1333. }
  1334. }
  1335. TEST_F(DBTest, ComparatorCheck) {
  1336. class NewComparator : public Comparator {
  1337. public:
  1338. const char* Name() const override { return "leveldb.NewComparator"; }
  1339. int Compare(const Slice& a, const Slice& b) const override {
  1340. return BytewiseComparator()->Compare(a, b);
  1341. }
  1342. void FindShortestSeparator(std::string* s, const Slice& l) const override {
  1343. BytewiseComparator()->FindShortestSeparator(s, l);
  1344. }
  1345. void FindShortSuccessor(std::string* key) const override {
  1346. BytewiseComparator()->FindShortSuccessor(key);
  1347. }
  1348. };
  1349. NewComparator cmp;
  1350. Options new_options = CurrentOptions();
  1351. new_options.comparator = &cmp;
  1352. Status s = TryReopen(&new_options);
  1353. ASSERT_TRUE(!s.ok());
  1354. ASSERT_TRUE(s.ToString().find("comparator") != std::string::npos)
  1355. << s.ToString();
  1356. }
  1357. TEST_F(DBTest, CustomComparator) {
  1358. class NumberComparator : public Comparator {
  1359. public:
  1360. const char* Name() const override { return "test.NumberComparator"; }
  1361. int Compare(const Slice& a, const Slice& b) const override {
  1362. return ToNumber(a) - ToNumber(b);
  1363. }
  1364. void FindShortestSeparator(std::string* s, const Slice& l) const override {
  1365. ToNumber(*s); // Check format
  1366. ToNumber(l); // Check format
  1367. }
  1368. void FindShortSuccessor(std::string* key) const override {
  1369. ToNumber(*key); // Check format
  1370. }
  1371. private:
  1372. static int ToNumber(const Slice& x) {
  1373. // Check that there are no extra characters.
  1374. EXPECT_TRUE(x.size() >= 2 && x[0] == '[' && x[x.size() - 1] == ']')
  1375. << EscapeString(x);
  1376. int val;
  1377. char ignored;
  1378. EXPECT_TRUE(sscanf(x.ToString().c_str(), "[%i]%c", &val, &ignored) == 1)
  1379. << EscapeString(x);
  1380. return val;
  1381. }
  1382. };
  1383. NumberComparator cmp;
  1384. Options new_options = CurrentOptions();
  1385. new_options.create_if_missing = true;
  1386. new_options.comparator = &cmp;
  1387. new_options.filter_policy = nullptr; // Cannot use bloom filters
  1388. new_options.write_buffer_size = 1000; // Compact more often
  1389. DestroyAndReopen(&new_options);
  1390. ASSERT_LEVELDB_OK(Put("[10]", "ten"));
  1391. ASSERT_LEVELDB_OK(Put("[0x14]", "twenty"));
  1392. for (int i = 0; i < 2; i++) {
  1393. ASSERT_EQ("ten", Get("[10]"));
  1394. ASSERT_EQ("ten", Get("[0xa]"));
  1395. ASSERT_EQ("twenty", Get("[20]"));
  1396. ASSERT_EQ("twenty", Get("[0x14]"));
  1397. ASSERT_EQ("NOT_FOUND", Get("[15]"));
  1398. ASSERT_EQ("NOT_FOUND", Get("[0xf]"));
  1399. Compact("[0]", "[9999]");
  1400. }
  1401. for (int run = 0; run < 2; run++) {
  1402. for (int i = 0; i < 1000; i++) {
  1403. char buf[100];
  1404. std::snprintf(buf, sizeof(buf), "[%d]", i * 10);
  1405. ASSERT_LEVELDB_OK(Put(buf, buf));
  1406. }
  1407. Compact("[0]", "[1000000]");
  1408. }
  1409. }
  1410. TEST_F(DBTest, ManualCompaction) {
  1411. ASSERT_EQ(config::kMaxMemCompactLevel, 2)
  1412. << "Need to update this test to match kMaxMemCompactLevel";
  1413. MakeTables(3, "p", "q");
  1414. ASSERT_EQ("1,1,1", FilesPerLevel());
  1415. // Compaction range falls before files
  1416. Compact("", "c");
  1417. ASSERT_EQ("1,1,1", FilesPerLevel());
  1418. // Compaction range falls after files
  1419. Compact("r", "z");
  1420. ASSERT_EQ("1,1,1", FilesPerLevel());
  1421. // Compaction range overlaps files
  1422. Compact("p1", "p9");
  1423. ASSERT_EQ("0,0,1", FilesPerLevel());
  1424. // Populate a different range
  1425. MakeTables(3, "c", "e");
  1426. ASSERT_EQ("1,1,2", FilesPerLevel());
  1427. // Compact just the new range
  1428. Compact("b", "f");
  1429. ASSERT_EQ("0,0,2", FilesPerLevel());
  1430. // Compact all
  1431. MakeTables(1, "a", "z");
  1432. ASSERT_EQ("0,1,2", FilesPerLevel());
  1433. db_->CompactRange(nullptr, nullptr);
  1434. ASSERT_EQ("0,0,1", FilesPerLevel());
  1435. }
  1436. TEST_F(DBTest, DBOpen_Options) {
  1437. std::string dbname = testing::TempDir() + "db_options_test";
  1438. DestroyDB(dbname, Options());
  1439. // Does not exist, and create_if_missing == false: error
  1440. DB* db = nullptr;
  1441. Options opts;
  1442. opts.create_if_missing = false;
  1443. Status s = DB::Open(opts, dbname, &db);
  1444. ASSERT_TRUE(strstr(s.ToString().c_str(), "does not exist") != nullptr);
  1445. ASSERT_TRUE(db == nullptr);
  1446. // Does not exist, and create_if_missing == true: OK
  1447. opts.create_if_missing = true;
  1448. s = DB::Open(opts, dbname, &db);
  1449. ASSERT_LEVELDB_OK(s);
  1450. ASSERT_TRUE(db != nullptr);
  1451. delete db;
  1452. db = nullptr;
  1453. // Does exist, and error_if_exists == true: error
  1454. opts.create_if_missing = false;
  1455. opts.error_if_exists = true;
  1456. s = DB::Open(opts, dbname, &db);
  1457. ASSERT_TRUE(strstr(s.ToString().c_str(), "exists") != nullptr);
  1458. ASSERT_TRUE(db == nullptr);
  1459. // Does exist, and error_if_exists == false: OK
  1460. opts.create_if_missing = true;
  1461. opts.error_if_exists = false;
  1462. s = DB::Open(opts, dbname, &db);
  1463. ASSERT_LEVELDB_OK(s);
  1464. ASSERT_TRUE(db != nullptr);
  1465. delete db;
  1466. db = nullptr;
  1467. }
  1468. TEST_F(DBTest, DestroyEmptyDir) {
  1469. std::string dbname = testing::TempDir() + "db_empty_dir";
  1470. TestEnv env(Env::Default());
  1471. env.RemoveDir(dbname);
  1472. ASSERT_TRUE(!env.FileExists(dbname));
  1473. Options opts;
  1474. opts.env = &env;
  1475. ASSERT_LEVELDB_OK(env.CreateDir(dbname));
  1476. ASSERT_TRUE(env.FileExists(dbname));
  1477. std::vector<std::string> children;
  1478. ASSERT_LEVELDB_OK(env.GetChildren(dbname, &children));
  1479. // The stock Env's do not filter out '.' and '..' special files.
  1480. ASSERT_EQ(2, children.size());
  1481. ASSERT_LEVELDB_OK(DestroyDB(dbname, opts));
  1482. ASSERT_TRUE(!env.FileExists(dbname));
  1483. // Should also be destroyed if Env is filtering out dot files.
  1484. env.SetIgnoreDotFiles(true);
  1485. ASSERT_LEVELDB_OK(env.CreateDir(dbname));
  1486. ASSERT_TRUE(env.FileExists(dbname));
  1487. ASSERT_LEVELDB_OK(env.GetChildren(dbname, &children));
  1488. ASSERT_EQ(0, children.size());
  1489. ASSERT_LEVELDB_OK(DestroyDB(dbname, opts));
  1490. ASSERT_TRUE(!env.FileExists(dbname));
  1491. }
  1492. TEST_F(DBTest, DestroyOpenDB) {
  1493. std::string dbname = testing::TempDir() + "open_db_dir";
  1494. env_->RemoveDir(dbname);
  1495. ASSERT_TRUE(!env_->FileExists(dbname));
  1496. Options opts;
  1497. opts.create_if_missing = true;
  1498. DB* db = nullptr;
  1499. ASSERT_LEVELDB_OK(DB::Open(opts, dbname, &db));
  1500. ASSERT_TRUE(db != nullptr);
  1501. // Must fail to destroy an open db.
  1502. ASSERT_TRUE(env_->FileExists(dbname));
  1503. ASSERT_TRUE(!DestroyDB(dbname, Options()).ok());
  1504. ASSERT_TRUE(env_->FileExists(dbname));
  1505. delete db;
  1506. db = nullptr;
  1507. // Should succeed destroying a closed db.
  1508. ASSERT_LEVELDB_OK(DestroyDB(dbname, Options()));
  1509. ASSERT_TRUE(!env_->FileExists(dbname));
  1510. }
  1511. TEST_F(DBTest, Locking) {
  1512. DB* db2 = nullptr;
  1513. Status s = DB::Open(CurrentOptions(), dbname_, &db2);
  1514. ASSERT_TRUE(!s.ok()) << "Locking did not prevent re-opening db";
  1515. }
  1516. // Check that number of files does not grow when we are out of space
  1517. TEST_F(DBTest, NoSpace) {
  1518. Options options = CurrentOptions();
  1519. options.env = env_;
  1520. Reopen(&options);
  1521. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  1522. ASSERT_EQ("v1", Get("foo"));
  1523. Compact("a", "z");
  1524. const int num_files = CountFiles();
  1525. // Force out-of-space errors.
  1526. env_->no_space_.store(true, std::memory_order_release);
  1527. for (int i = 0; i < 10; i++) {
  1528. for (int level = 0; level < config::kNumLevels - 1; level++) {
  1529. dbfull()->TEST_CompactRange(level, nullptr, nullptr);
  1530. }
  1531. }
  1532. env_->no_space_.store(false, std::memory_order_release);
  1533. ASSERT_LT(CountFiles(), num_files + 3);
  1534. }
  1535. TEST_F(DBTest, NonWritableFileSystem) {
  1536. Options options = CurrentOptions();
  1537. options.write_buffer_size = 1000;
  1538. options.env = env_;
  1539. Reopen(&options);
  1540. ASSERT_LEVELDB_OK(Put("foo", "v1"));
  1541. // Force errors for new files.
  1542. env_->non_writable_.store(true, std::memory_order_release);
  1543. std::string big(100000, 'x');
  1544. int errors = 0;
  1545. for (int i = 0; i < 20; i++) {
  1546. std::fprintf(stderr, "iter %d; errors %d\n", i, errors);
  1547. if (!Put("foo", big).ok()) {
  1548. errors++;
  1549. DelayMilliseconds(100);
  1550. }
  1551. }
  1552. ASSERT_GT(errors, 0);
  1553. env_->non_writable_.store(false, std::memory_order_release);
  1554. }
  1555. TEST_F(DBTest, WriteSyncError) {
  1556. // Check that log sync errors cause the DB to disallow future writes.
  1557. // (a) Cause log sync calls to fail
  1558. Options options = CurrentOptions();
  1559. options.env = env_;
  1560. Reopen(&options);
  1561. env_->data_sync_error_.store(true, std::memory_order_release);
  1562. // (b) Normal write should succeed
  1563. WriteOptions w;
  1564. ASSERT_LEVELDB_OK(db_->Put(w, "k1", "v1"));
  1565. ASSERT_EQ("v1", Get("k1"));
  1566. // (c) Do a sync write; should fail
  1567. w.sync = true;
  1568. ASSERT_TRUE(!db_->Put(w, "k2", "v2").ok());
  1569. ASSERT_EQ("v1", Get("k1"));
  1570. ASSERT_EQ("NOT_FOUND", Get("k2"));
  1571. // (d) make sync behave normally
  1572. env_->data_sync_error_.store(false, std::memory_order_release);
  1573. // (e) Do a non-sync write; should fail
  1574. w.sync = false;
  1575. ASSERT_TRUE(!db_->Put(w, "k3", "v3").ok());
  1576. ASSERT_EQ("v1", Get("k1"));
  1577. ASSERT_EQ("NOT_FOUND", Get("k2"));
  1578. ASSERT_EQ("NOT_FOUND", Get("k3"));
  1579. }
  1580. TEST_F(DBTest, ManifestWriteError) {
  1581. // Test for the following problem:
  1582. // (a) Compaction produces file F
  1583. // (b) Log record containing F is written to MANIFEST file, but Sync() fails
  1584. // (c) GC deletes F
  1585. // (d) After reopening DB, reads fail since deleted F is named in log record
  1586. // We iterate twice. In the second iteration, everything is the
  1587. // same except the log record never makes it to the MANIFEST file.
  1588. for (int iter = 0; iter < 2; iter++) {
  1589. std::atomic<bool>* error_type = (iter == 0) ? &env_->manifest_sync_error_
  1590. : &env_->manifest_write_error_;
  1591. // Insert foo=>bar mapping
  1592. Options options = CurrentOptions();
  1593. options.env = env_;
  1594. options.create_if_missing = true;
  1595. options.error_if_exists = false;
  1596. DestroyAndReopen(&options);
  1597. ASSERT_LEVELDB_OK(Put("foo", "bar"));
  1598. ASSERT_EQ("bar", Get("foo"));
  1599. // Memtable compaction (will succeed)
  1600. dbfull()->TEST_CompactMemTable();
  1601. ASSERT_EQ("bar", Get("foo"));
  1602. const int last = config::kMaxMemCompactLevel;
  1603. ASSERT_EQ(NumTableFilesAtLevel(last), 1); // foo=>bar is now in last level
  1604. // Merging compaction (will fail)
  1605. error_type->store(true, std::memory_order_release);
  1606. dbfull()->TEST_CompactRange(last, nullptr, nullptr); // Should fail
  1607. ASSERT_EQ("bar", Get("foo"));
  1608. // Recovery: should not lose data
  1609. error_type->store(false, std::memory_order_release);
  1610. Reopen(&options);
  1611. ASSERT_EQ("bar", Get("foo"));
  1612. }
  1613. }
  1614. TEST_F(DBTest, MissingSSTFile) {
  1615. ASSERT_LEVELDB_OK(Put("foo", "bar"));
  1616. ASSERT_EQ("bar", Get("foo"));
  1617. // Dump the memtable to disk.
  1618. dbfull()->TEST_CompactMemTable();
  1619. ASSERT_EQ("bar", Get("foo"));
  1620. Close();
  1621. ASSERT_TRUE(DeleteAnSSTFile());
  1622. Options options = CurrentOptions();
  1623. options.paranoid_checks = true;
  1624. Status s = TryReopen(&options);
  1625. ASSERT_TRUE(!s.ok());
  1626. ASSERT_TRUE(s.ToString().find("issing") != std::string::npos) << s.ToString();
  1627. }
  1628. TEST_F(DBTest, StillReadSST) {
  1629. ASSERT_LEVELDB_OK(Put("foo", "bar"));
  1630. ASSERT_EQ("bar", Get("foo"));
  1631. // Dump the memtable to disk.
  1632. dbfull()->TEST_CompactMemTable();
  1633. ASSERT_EQ("bar", Get("foo"));
  1634. Close();
  1635. ASSERT_GT(RenameLDBToSST(), 0);
  1636. Options options = CurrentOptions();
  1637. options.paranoid_checks = true;
  1638. Status s = TryReopen(&options);
  1639. ASSERT_TRUE(s.ok());
  1640. ASSERT_EQ("bar", Get("foo"));
  1641. }
  1642. TEST_F(DBTest, FilesDeletedAfterCompaction) {
  1643. ASSERT_LEVELDB_OK(Put("foo", "v2"));
  1644. Compact("a", "z");
  1645. const int num_files = CountFiles();
  1646. for (int i = 0; i < 10; i++) {
  1647. ASSERT_LEVELDB_OK(Put("foo", "v2"));
  1648. Compact("a", "z");
  1649. }
  1650. ASSERT_EQ(CountFiles(), num_files);
  1651. }
  1652. TEST_F(DBTest, BloomFilter) {
  1653. env_->count_random_reads_ = true;
  1654. Options options = CurrentOptions();
  1655. options.env = env_;
  1656. options.block_cache = NewLRUCache(0); // Prevent cache hits
  1657. options.filter_policy = NewBloomFilterPolicy(10);
  1658. Reopen(&options);
  1659. // Populate multiple layers
  1660. const int N = 10000;
  1661. for (int i = 0; i < N; i++) {
  1662. ASSERT_LEVELDB_OK(Put(Key(i), Key(i)));
  1663. }
  1664. Compact("a", "z");
  1665. for (int i = 0; i < N; i += 100) {
  1666. ASSERT_LEVELDB_OK(Put(Key(i), Key(i)));
  1667. }
  1668. dbfull()->TEST_CompactMemTable();
  1669. // Prevent auto compactions triggered by seeks
  1670. env_->delay_data_sync_.store(true, std::memory_order_release);
  1671. // Lookup present keys. Should rarely read from small sstable.
  1672. env_->random_read_counter_.Reset();
  1673. for (int i = 0; i < N; i++) {
  1674. ASSERT_EQ(Key(i), Get(Key(i)));
  1675. }
  1676. int reads = env_->random_read_counter_.Read();
  1677. std::fprintf(stderr, "%d present => %d reads\n", N, reads);
  1678. ASSERT_GE(reads, N);
  1679. ASSERT_LE(reads, N + 2 * N / 100);
  1680. // Lookup present keys. Should rarely read from either sstable.
  1681. env_->random_read_counter_.Reset();
  1682. for (int i = 0; i < N; i++) {
  1683. ASSERT_EQ("NOT_FOUND", Get(Key(i) + ".missing"));
  1684. }
  1685. reads = env_->random_read_counter_.Read();
  1686. std::fprintf(stderr, "%d missing => %d reads\n", N, reads);
  1687. ASSERT_LE(reads, 3 * N / 100);
  1688. env_->delay_data_sync_.store(false, std::memory_order_release);
  1689. Close();
  1690. delete options.block_cache;
  1691. delete options.filter_policy;
  1692. }
  1693. // Multi-threaded test:
  1694. namespace {
  1695. static const int kNumThreads = 4;
  1696. static const int kTestSeconds = 10;
  1697. static const int kNumKeys = 1000;
  1698. struct MTState {
  1699. DBTest* test;
  1700. std::atomic<bool> stop;
  1701. std::atomic<int> counter[kNumThreads];
  1702. std::atomic<bool> thread_done[kNumThreads];
  1703. };
  1704. struct MTThread {
  1705. MTState* state;
  1706. int id;
  1707. };
  1708. static void MTThreadBody(void* arg) {
  1709. MTThread* t = reinterpret_cast<MTThread*>(arg);
  1710. int id = t->id;
  1711. DB* db = t->state->test->db_;
  1712. int counter = 0;
  1713. std::fprintf(stderr, "... starting thread %d\n", id);
  1714. Random rnd(1000 + id);
  1715. std::string value;
  1716. char valbuf[1500];
  1717. while (!t->state->stop.load(std::memory_order_acquire)) {
  1718. t->state->counter[id].store(counter, std::memory_order_release);
  1719. int key = rnd.Uniform(kNumKeys);
  1720. char keybuf[20];
  1721. std::snprintf(keybuf, sizeof(keybuf), "%016d", key);
  1722. if (rnd.OneIn(2)) {
  1723. // Write values of the form <key, my id, counter>.
  1724. // We add some padding for force compactions.
  1725. std::snprintf(valbuf, sizeof(valbuf), "%d.%d.%-1000d", key, id,
  1726. static_cast<int>(counter));
  1727. ASSERT_LEVELDB_OK(db->Put(WriteOptions(), Slice(keybuf), Slice(valbuf)));
  1728. } else {
  1729. // Read a value and verify that it matches the pattern written above.
  1730. Status s = db->Get(ReadOptions(), Slice(keybuf), &value);
  1731. if (s.IsNotFound()) {
  1732. // Key has not yet been written
  1733. } else {
  1734. // Check that the writer thread counter is >= the counter in the value
  1735. ASSERT_LEVELDB_OK(s);
  1736. int k, w, c;
  1737. ASSERT_EQ(3, sscanf(value.c_str(), "%d.%d.%d", &k, &w, &c)) << value;
  1738. ASSERT_EQ(k, key);
  1739. ASSERT_GE(w, 0);
  1740. ASSERT_LT(w, kNumThreads);
  1741. ASSERT_LE(c, t->state->counter[w].load(std::memory_order_acquire));
  1742. }
  1743. }
  1744. counter++;
  1745. }
  1746. t->state->thread_done[id].store(true, std::memory_order_release);
  1747. std::fprintf(stderr, "... stopping thread %d after %d ops\n", id, counter);
  1748. }
  1749. } // namespace
  1750. TEST_F(DBTest, MultiThreaded) {
  1751. do {
  1752. // Initialize state
  1753. MTState mt;
  1754. mt.test = this;
  1755. mt.stop.store(false, std::memory_order_release);
  1756. for (int id = 0; id < kNumThreads; id++) {
  1757. mt.counter[id].store(false, std::memory_order_release);
  1758. mt.thread_done[id].store(false, std::memory_order_release);
  1759. }
  1760. // Start threads
  1761. MTThread thread[kNumThreads];
  1762. for (int id = 0; id < kNumThreads; id++) {
  1763. thread[id].state = &mt;
  1764. thread[id].id = id;
  1765. env_->StartThread(MTThreadBody, &thread[id]);
  1766. }
  1767. // Let them run for a while
  1768. DelayMilliseconds(kTestSeconds * 1000);
  1769. // Stop the threads and wait for them to finish
  1770. mt.stop.store(true, std::memory_order_release);
  1771. for (int id = 0; id < kNumThreads; id++) {
  1772. while (!mt.thread_done[id].load(std::memory_order_acquire)) {
  1773. DelayMilliseconds(100);
  1774. }
  1775. }
  1776. } while (ChangeOptions());
  1777. }
  1778. namespace {
  1779. typedef std::map<std::string, std::string> KVMap;
  1780. }
  1781. class ModelDB : public DB {
  1782. public:
  1783. class ModelSnapshot : public Snapshot {
  1784. public:
  1785. KVMap map_;
  1786. };
  1787. explicit ModelDB(const Options& options) : options_(options) {}
  1788. ~ModelDB() override = default;
  1789. Status Put(const WriteOptions& o, const Slice& k, const Slice& v) override {
  1790. return DB::Put(o, k, v);
  1791. }
  1792. Status Delete(const WriteOptions& o, const Slice& key) override {
  1793. return DB::Delete(o, key);
  1794. }
  1795. Status Get(const ReadOptions& options, const Slice& key,
  1796. std::string* value) override {
  1797. assert(false); // Not implemented
  1798. return Status::NotFound(key);
  1799. }
  1800. Iterator* NewIterator(const ReadOptions& options) override {
  1801. if (options.snapshot == nullptr) {
  1802. KVMap* saved = new KVMap;
  1803. *saved = map_;
  1804. return new ModelIter(saved, true);
  1805. } else {
  1806. const KVMap* snapshot_state =
  1807. &(reinterpret_cast<const ModelSnapshot*>(options.snapshot)->map_);
  1808. return new ModelIter(snapshot_state, false);
  1809. }
  1810. }
  1811. const Snapshot* GetSnapshot() override {
  1812. ModelSnapshot* snapshot = new ModelSnapshot;
  1813. snapshot->map_ = map_;
  1814. return snapshot;
  1815. }
  1816. void ReleaseSnapshot(const Snapshot* snapshot) override {
  1817. delete reinterpret_cast<const ModelSnapshot*>(snapshot);
  1818. }
  1819. Status Write(const WriteOptions& options, WriteBatch* batch) override {
  1820. class Handler : public WriteBatch::Handler {
  1821. public:
  1822. KVMap* map_;
  1823. void Put(const Slice& key, const Slice& value) override {
  1824. (*map_)[key.ToString()] = value.ToString();
  1825. }
  1826. void Delete(const Slice& key) override { map_->erase(key.ToString()); }
  1827. };
  1828. Handler handler;
  1829. handler.map_ = &map_;
  1830. return batch->Iterate(&handler);
  1831. }
  1832. bool GetProperty(const Slice& property, std::string* value) override {
  1833. return false;
  1834. }
  1835. void GetApproximateSizes(const Range* r, int n, uint64_t* sizes) override {
  1836. for (int i = 0; i < n; i++) {
  1837. sizes[i] = 0;
  1838. }
  1839. }
  1840. void CompactRange(const Slice* start, const Slice* end) override {}
  1841. private:
  1842. class ModelIter : public Iterator {
  1843. public:
  1844. ModelIter(const KVMap* map, bool owned)
  1845. : map_(map), owned_(owned), iter_(map_->end()) {}
  1846. ~ModelIter() override {
  1847. if (owned_) delete map_;
  1848. }
  1849. bool Valid() const override { return iter_ != map_->end(); }
  1850. void SeekToFirst() override { iter_ = map_->begin(); }
  1851. void SeekToLast() override {
  1852. if (map_->empty()) {
  1853. iter_ = map_->end();
  1854. } else {
  1855. iter_ = map_->find(map_->rbegin()->first);
  1856. }
  1857. }
  1858. void Seek(const Slice& k) override {
  1859. iter_ = map_->lower_bound(k.ToString());
  1860. }
  1861. void Next() override { ++iter_; }
  1862. void Prev() override { --iter_; }
  1863. Slice key() const override { return iter_->first; }
  1864. Slice value() const override { return iter_->second; }
  1865. Status status() const override { return Status::OK(); }
  1866. private:
  1867. const KVMap* const map_;
  1868. const bool owned_; // Do we own map_
  1869. KVMap::const_iterator iter_;
  1870. };
  1871. const Options options_;
  1872. KVMap map_;
  1873. };
  1874. static bool CompareIterators(int step, DB* model, DB* db,
  1875. const Snapshot* model_snap,
  1876. const Snapshot* db_snap) {
  1877. ReadOptions options;
  1878. options.snapshot = model_snap;
  1879. Iterator* miter = model->NewIterator(options);
  1880. options.snapshot = db_snap;
  1881. Iterator* dbiter = db->NewIterator(options);
  1882. bool ok = true;
  1883. int count = 0;
  1884. std::vector<std::string> seek_keys;
  1885. // Compare equality of all elements using Next(). Save some of the keys for
  1886. // comparing Seek equality.
  1887. for (miter->SeekToFirst(), dbiter->SeekToFirst();
  1888. ok && miter->Valid() && dbiter->Valid(); miter->Next(), dbiter->Next()) {
  1889. count++;
  1890. if (miter->key().compare(dbiter->key()) != 0) {
  1891. std::fprintf(stderr, "step %d: Key mismatch: '%s' vs. '%s'\n", step,
  1892. EscapeString(miter->key()).c_str(),
  1893. EscapeString(dbiter->key()).c_str());
  1894. ok = false;
  1895. break;
  1896. }
  1897. if (miter->value().compare(dbiter->value()) != 0) {
  1898. std::fprintf(stderr,
  1899. "step %d: Value mismatch for key '%s': '%s' vs. '%s'\n",
  1900. step, EscapeString(miter->key()).c_str(),
  1901. EscapeString(miter->value()).c_str(),
  1902. EscapeString(miter->value()).c_str());
  1903. ok = false;
  1904. break;
  1905. }
  1906. if (count % 10 == 0) {
  1907. seek_keys.push_back(miter->key().ToString());
  1908. }
  1909. }
  1910. if (ok) {
  1911. if (miter->Valid() != dbiter->Valid()) {
  1912. std::fprintf(stderr, "step %d: Mismatch at end of iterators: %d vs. %d\n",
  1913. step, miter->Valid(), dbiter->Valid());
  1914. ok = false;
  1915. }
  1916. }
  1917. if (ok) {
  1918. // Validate iterator equality when performing seeks.
  1919. for (auto kiter = seek_keys.begin(); ok && kiter != seek_keys.end();
  1920. ++kiter) {
  1921. miter->Seek(*kiter);
  1922. dbiter->Seek(*kiter);
  1923. if (!miter->Valid() || !dbiter->Valid()) {
  1924. std::fprintf(stderr, "step %d: Seek iterators invalid: %d vs. %d\n",
  1925. step, miter->Valid(), dbiter->Valid());
  1926. ok = false;
  1927. }
  1928. if (miter->key().compare(dbiter->key()) != 0) {
  1929. std::fprintf(stderr, "step %d: Seek key mismatch: '%s' vs. '%s'\n",
  1930. step, EscapeString(miter->key()).c_str(),
  1931. EscapeString(dbiter->key()).c_str());
  1932. ok = false;
  1933. break;
  1934. }
  1935. if (miter->value().compare(dbiter->value()) != 0) {
  1936. std::fprintf(
  1937. stderr,
  1938. "step %d: Seek value mismatch for key '%s': '%s' vs. '%s'\n", step,
  1939. EscapeString(miter->key()).c_str(),
  1940. EscapeString(miter->value()).c_str(),
  1941. EscapeString(miter->value()).c_str());
  1942. ok = false;
  1943. break;
  1944. }
  1945. }
  1946. }
  1947. std::fprintf(stderr, "%d entries compared: ok=%d\n", count, ok);
  1948. delete miter;
  1949. delete dbiter;
  1950. return ok;
  1951. }
  1952. TEST_F(DBTest, Randomized) {
  1953. Random rnd(test::RandomSeed());
  1954. do {
  1955. ModelDB model(CurrentOptions());
  1956. const int N = 10000;
  1957. const Snapshot* model_snap = nullptr;
  1958. const Snapshot* db_snap = nullptr;
  1959. std::string k, v;
  1960. for (int step = 0; step < N; step++) {
  1961. if (step % 100 == 0) {
  1962. std::fprintf(stderr, "Step %d of %d\n", step, N);
  1963. }
  1964. // TODO(sanjay): Test Get() works
  1965. int p = rnd.Uniform(100);
  1966. if (p < 45) { // Put
  1967. k = RandomKey(&rnd);
  1968. v = RandomString(
  1969. &rnd, rnd.OneIn(20) ? 100 + rnd.Uniform(100) : rnd.Uniform(8));
  1970. ASSERT_LEVELDB_OK(model.Put(WriteOptions(), k, v));
  1971. ASSERT_LEVELDB_OK(db_->Put(WriteOptions(), k, v));
  1972. } else if (p < 90) { // Delete
  1973. k = RandomKey(&rnd);
  1974. ASSERT_LEVELDB_OK(model.Delete(WriteOptions(), k));
  1975. ASSERT_LEVELDB_OK(db_->Delete(WriteOptions(), k));
  1976. } else { // Multi-element batch
  1977. WriteBatch b;
  1978. const int num = rnd.Uniform(8);
  1979. for (int i = 0; i < num; i++) {
  1980. if (i == 0 || !rnd.OneIn(10)) {
  1981. k = RandomKey(&rnd);
  1982. } else {
  1983. // Periodically re-use the same key from the previous iter, so
  1984. // we have multiple entries in the write batch for the same key
  1985. }
  1986. if (rnd.OneIn(2)) {
  1987. v = RandomString(&rnd, rnd.Uniform(10));
  1988. b.Put(k, v);
  1989. } else {
  1990. b.Delete(k);
  1991. }
  1992. }
  1993. ASSERT_LEVELDB_OK(model.Write(WriteOptions(), &b));
  1994. ASSERT_LEVELDB_OK(db_->Write(WriteOptions(), &b));
  1995. }
  1996. if ((step % 100) == 0) {
  1997. ASSERT_TRUE(CompareIterators(step, &model, db_, nullptr, nullptr));
  1998. ASSERT_TRUE(CompareIterators(step, &model, db_, model_snap, db_snap));
  1999. // Save a snapshot from each DB this time that we'll use next
  2000. // time we compare things, to make sure the current state is
  2001. // preserved with the snapshot
  2002. if (model_snap != nullptr) model.ReleaseSnapshot(model_snap);
  2003. if (db_snap != nullptr) db_->ReleaseSnapshot(db_snap);
  2004. Reopen();
  2005. ASSERT_TRUE(CompareIterators(step, &model, db_, nullptr, nullptr));
  2006. model_snap = model.GetSnapshot();
  2007. db_snap = db_->GetSnapshot();
  2008. }
  2009. }
  2010. if (model_snap != nullptr) model.ReleaseSnapshot(model_snap);
  2011. if (db_snap != nullptr) db_->ReleaseSnapshot(db_snap);
  2012. } while (ChangeOptions());
  2013. }
  2014. std::string MakeKey(unsigned int num) {
  2015. char buf[30];
  2016. std::snprintf(buf, sizeof(buf), "%016u", num);
  2017. return std::string(buf);
  2018. }
  2019. static void BM_LogAndApply(benchmark::State& state) {
  2020. const int num_base_files = state.range(0);
  2021. std::string dbname = testing::TempDir() + "leveldb_test_benchmark";
  2022. DestroyDB(dbname, Options());
  2023. DB* db = nullptr;
  2024. Options opts;
  2025. opts.create_if_missing = true;
  2026. Status s = DB::Open(opts, dbname, &db);
  2027. ASSERT_LEVELDB_OK(s);
  2028. ASSERT_TRUE(db != nullptr);
  2029. delete db;
  2030. db = nullptr;
  2031. Env* env = Env::Default();
  2032. port::Mutex mu;
  2033. MutexLock l(&mu);
  2034. InternalKeyComparator cmp(BytewiseComparator());
  2035. Options options;
  2036. VersionSet vset(dbname, &options, nullptr, &cmp);
  2037. bool save_manifest;
  2038. ASSERT_LEVELDB_OK(vset.Recover(&save_manifest));
  2039. VersionEdit vbase;
  2040. uint64_t fnum = 1;
  2041. for (int i = 0; i < num_base_files; i++) {
  2042. InternalKey start(MakeKey(2 * fnum), 1, kTypeValue);
  2043. InternalKey limit(MakeKey(2 * fnum + 1), 1, kTypeDeletion);
  2044. vbase.AddFile(2, fnum++, 1 /* file size */, start, limit);
  2045. }
  2046. ASSERT_LEVELDB_OK(vset.LogAndApply(&vbase, &mu));
  2047. uint64_t start_micros = env->NowMicros();
  2048. for (auto st : state) {
  2049. VersionEdit vedit;
  2050. vedit.RemoveFile(2, fnum);
  2051. InternalKey start(MakeKey(2 * fnum), 1, kTypeValue);
  2052. InternalKey limit(MakeKey(2 * fnum + 1), 1, kTypeDeletion);
  2053. vedit.AddFile(2, fnum++, 1 /* file size */, start, limit);
  2054. vset.LogAndApply(&vedit, &mu);
  2055. }
  2056. uint64_t stop_micros = env->NowMicros();
  2057. unsigned int us = stop_micros - start_micros;
  2058. char buf[16];
  2059. std::snprintf(buf, sizeof(buf), "%d", num_base_files);
  2060. std::fprintf(stderr,
  2061. "BM_LogAndApply/%-6s %8" PRIu64
  2062. " iters : %9u us (%7.0f us / iter)\n",
  2063. buf, state.iterations(), us, ((float)us) / state.iterations());
  2064. }
  2065. BENCHMARK(BM_LogAndApply)->Arg(1)->Arg(100)->Arg(10000)->Arg(100000);
  2066. } // namespace leveldb
  2067. int main(int argc, char** argv) {
  2068. testing::InitGoogleTest(&argc, argv);
  2069. benchmark::RunSpecifiedBenchmarks();
  2070. return RUN_ALL_TESTS();
  2071. }