749 rivejä
18 KiB

Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 vuotta sitten
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 vuotta sitten
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 vuotta sitten
  1. // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
  2. // Use of this source code is governed by a BSD-style license that can be
  3. // found in the LICENSE file. See the AUTHORS file for names of contributors.
  4. #include <dirent.h>
  5. #include <errno.h>
  6. #include <fcntl.h>
  7. #include <pthread.h>
  8. #include <stdio.h>
  9. #include <stdlib.h>
  10. #include <string.h>
  11. #include <sys/mman.h>
  12. #include <sys/resource.h>
  13. #include <sys/stat.h>
  14. #include <sys/time.h>
  15. #include <sys/types.h>
  16. #include <time.h>
  17. #include <unistd.h>
  18. #include <deque>
  19. #include <limits>
  20. #include <set>
  21. #include "leveldb/env.h"
  22. #include "leveldb/slice.h"
  23. #include "port/port.h"
  24. #include "util/logging.h"
  25. #include "util/mutexlock.h"
  26. #include "util/posix_logger.h"
  27. #include "util/env_posix_test_helper.h"
  28. namespace leveldb {
  29. namespace {
  30. static int open_read_only_file_limit = -1;
  31. static int mmap_limit = -1;
  32. static const size_t kBufSize = 65536;
  33. static Status PosixError(const std::string& context, int err_number) {
  34. if (err_number == ENOENT) {
  35. return Status::NotFound(context, strerror(err_number));
  36. } else {
  37. return Status::IOError(context, strerror(err_number));
  38. }
  39. }
  40. // Helper class to limit resource usage to avoid exhaustion.
  41. // Currently used to limit read-only file descriptors and mmap file usage
  42. // so that we do not end up running out of file descriptors, virtual memory,
  43. // or running into kernel performance problems for very large databases.
  44. class Limiter {
  45. public:
  46. // Limit maximum number of resources to |n|.
  47. Limiter(intptr_t n) {
  48. SetAllowed(n);
  49. }
  50. // If another resource is available, acquire it and return true.
  51. // Else return false.
  52. bool Acquire() {
  53. if (GetAllowed() <= 0) {
  54. return false;
  55. }
  56. MutexLock l(&mu_);
  57. intptr_t x = GetAllowed();
  58. if (x <= 0) {
  59. return false;
  60. } else {
  61. SetAllowed(x - 1);
  62. return true;
  63. }
  64. }
  65. // Release a resource acquired by a previous call to Acquire() that returned
  66. // true.
  67. void Release() {
  68. MutexLock l(&mu_);
  69. SetAllowed(GetAllowed() + 1);
  70. }
  71. private:
  72. port::Mutex mu_;
  73. port::AtomicPointer allowed_;
  74. intptr_t GetAllowed() const {
  75. return reinterpret_cast<intptr_t>(allowed_.Acquire_Load());
  76. }
  77. // REQUIRES: mu_ must be held
  78. void SetAllowed(intptr_t v) {
  79. allowed_.Release_Store(reinterpret_cast<void*>(v));
  80. }
  81. Limiter(const Limiter&);
  82. void operator=(const Limiter&);
  83. };
  84. class PosixSequentialFile: public SequentialFile {
  85. private:
  86. std::string filename_;
  87. int fd_;
  88. public:
  89. PosixSequentialFile(const std::string& fname, int fd)
  90. : filename_(fname), fd_(fd) {}
  91. virtual ~PosixSequentialFile() { close(fd_); }
  92. virtual Status Read(size_t n, Slice* result, char* scratch) {
  93. Status s;
  94. while (true) {
  95. ssize_t r = read(fd_, scratch, n);
  96. if (r < 0) {
  97. if (errno == EINTR) {
  98. continue; // Retry
  99. }
  100. s = PosixError(filename_, errno);
  101. break;
  102. }
  103. *result = Slice(scratch, r);
  104. break;
  105. }
  106. return s;
  107. }
  108. virtual Status Skip(uint64_t n) {
  109. if (lseek(fd_, n, SEEK_CUR) == static_cast<off_t>(-1)) {
  110. return PosixError(filename_, errno);
  111. }
  112. return Status::OK();
  113. }
  114. };
  115. // pread() based random-access
  116. class PosixRandomAccessFile: public RandomAccessFile {
  117. private:
  118. std::string filename_;
  119. bool temporary_fd_; // If true, fd_ is -1 and we open on every read.
  120. int fd_;
  121. Limiter* limiter_;
  122. public:
  123. PosixRandomAccessFile(const std::string& fname, int fd, Limiter* limiter)
  124. : filename_(fname), fd_(fd), limiter_(limiter) {
  125. temporary_fd_ = !limiter->Acquire();
  126. if (temporary_fd_) {
  127. // Open file on every access.
  128. close(fd_);
  129. fd_ = -1;
  130. }
  131. }
  132. virtual ~PosixRandomAccessFile() {
  133. if (!temporary_fd_) {
  134. close(fd_);
  135. limiter_->Release();
  136. }
  137. }
  138. virtual Status Read(uint64_t offset, size_t n, Slice* result,
  139. char* scratch) const {
  140. int fd = fd_;
  141. if (temporary_fd_) {
  142. fd = open(filename_.c_str(), O_RDONLY);
  143. if (fd < 0) {
  144. return PosixError(filename_, errno);
  145. }
  146. }
  147. Status s;
  148. ssize_t r = pread(fd, scratch, n, static_cast<off_t>(offset));
  149. *result = Slice(scratch, (r < 0) ? 0 : r);
  150. if (r < 0) {
  151. // An error: return a non-ok status
  152. s = PosixError(filename_, errno);
  153. }
  154. if (temporary_fd_) {
  155. // Close the temporary file descriptor opened earlier.
  156. close(fd);
  157. }
  158. return s;
  159. }
  160. };
  161. // mmap() based random-access
  162. class PosixMmapReadableFile: public RandomAccessFile {
  163. private:
  164. std::string filename_;
  165. void* mmapped_region_;
  166. size_t length_;
  167. Limiter* limiter_;
  168. public:
  169. // base[0,length-1] contains the mmapped contents of the file.
  170. PosixMmapReadableFile(const std::string& fname, void* base, size_t length,
  171. Limiter* limiter)
  172. : filename_(fname), mmapped_region_(base), length_(length),
  173. limiter_(limiter) {
  174. }
  175. virtual ~PosixMmapReadableFile() {
  176. munmap(mmapped_region_, length_);
  177. limiter_->Release();
  178. }
  179. virtual Status Read(uint64_t offset, size_t n, Slice* result,
  180. char* scratch) const {
  181. Status s;
  182. if (offset + n > length_) {
  183. *result = Slice();
  184. s = PosixError(filename_, EINVAL);
  185. } else {
  186. *result = Slice(reinterpret_cast<char*>(mmapped_region_) + offset, n);
  187. }
  188. return s;
  189. }
  190. };
  191. class PosixWritableFile : public WritableFile {
  192. private:
  193. // buf_[0, pos_-1] contains data to be written to fd_.
  194. std::string filename_;
  195. int fd_;
  196. char buf_[kBufSize];
  197. size_t pos_;
  198. public:
  199. PosixWritableFile(const std::string& fname, int fd)
  200. : filename_(fname), fd_(fd), pos_(0) { }
  201. ~PosixWritableFile() {
  202. if (fd_ >= 0) {
  203. // Ignoring any potential errors
  204. Close();
  205. }
  206. }
  207. virtual Status Append(const Slice& data) {
  208. size_t n = data.size();
  209. const char* p = data.data();
  210. // Fit as much as possible into buffer.
  211. size_t copy = std::min(n, kBufSize - pos_);
  212. memcpy(buf_ + pos_, p, copy);
  213. p += copy;
  214. n -= copy;
  215. pos_ += copy;
  216. if (n == 0) {
  217. return Status::OK();
  218. }
  219. // Can't fit in buffer, so need to do at least one write.
  220. Status s = FlushBuffered();
  221. if (!s.ok()) {
  222. return s;
  223. }
  224. // Small writes go to buffer, large writes are written directly.
  225. if (n < kBufSize) {
  226. memcpy(buf_, p, n);
  227. pos_ = n;
  228. return Status::OK();
  229. }
  230. return WriteRaw(p, n);
  231. }
  232. virtual Status Close() {
  233. Status result = FlushBuffered();
  234. const int r = close(fd_);
  235. if (r < 0 && result.ok()) {
  236. result = PosixError(filename_, errno);
  237. }
  238. fd_ = -1;
  239. return result;
  240. }
  241. virtual Status Flush() {
  242. return FlushBuffered();
  243. }
  244. Status SyncDirIfManifest() {
  245. const char* f = filename_.c_str();
  246. const char* sep = strrchr(f, '/');
  247. Slice basename;
  248. std::string dir;
  249. if (sep == NULL) {
  250. dir = ".";
  251. basename = f;
  252. } else {
  253. dir = std::string(f, sep - f);
  254. basename = sep + 1;
  255. }
  256. Status s;
  257. if (basename.starts_with("MANIFEST")) {
  258. int fd = open(dir.c_str(), O_RDONLY);
  259. if (fd < 0) {
  260. s = PosixError(dir, errno);
  261. } else {
  262. if (fsync(fd) < 0) {
  263. s = PosixError(dir, errno);
  264. }
  265. close(fd);
  266. }
  267. }
  268. return s;
  269. }
  270. virtual Status Sync() {
  271. // Ensure new files referred to by the manifest are in the filesystem.
  272. Status s = SyncDirIfManifest();
  273. if (!s.ok()) {
  274. return s;
  275. }
  276. s = FlushBuffered();
  277. if (s.ok()) {
  278. if (fdatasync(fd_) != 0) {
  279. s = PosixError(filename_, errno);
  280. }
  281. }
  282. return s;
  283. }
  284. private:
  285. Status FlushBuffered() {
  286. Status s = WriteRaw(buf_, pos_);
  287. pos_ = 0;
  288. return s;
  289. }
  290. Status WriteRaw(const char* p, size_t n) {
  291. while (n > 0) {
  292. ssize_t r = write(fd_, p, n);
  293. if (r < 0) {
  294. if (errno == EINTR) {
  295. continue; // Retry
  296. }
  297. return PosixError(filename_, errno);
  298. }
  299. p += r;
  300. n -= r;
  301. }
  302. return Status::OK();
  303. }
  304. };
  305. static int LockOrUnlock(int fd, bool lock) {
  306. errno = 0;
  307. struct flock f;
  308. memset(&f, 0, sizeof(f));
  309. f.l_type = (lock ? F_WRLCK : F_UNLCK);
  310. f.l_whence = SEEK_SET;
  311. f.l_start = 0;
  312. f.l_len = 0; // Lock/unlock entire file
  313. return fcntl(fd, F_SETLK, &f);
  314. }
  315. class PosixFileLock : public FileLock {
  316. public:
  317. int fd_;
  318. std::string name_;
  319. };
  320. // Set of locked files. We keep a separate set instead of just
  321. // relying on fcntrl(F_SETLK) since fcntl(F_SETLK) does not provide
  322. // any protection against multiple uses from the same process.
  323. class PosixLockTable {
  324. private:
  325. port::Mutex mu_;
  326. std::set<std::string> locked_files_;
  327. public:
  328. bool Insert(const std::string& fname) {
  329. MutexLock l(&mu_);
  330. return locked_files_.insert(fname).second;
  331. }
  332. void Remove(const std::string& fname) {
  333. MutexLock l(&mu_);
  334. locked_files_.erase(fname);
  335. }
  336. };
  337. class PosixEnv : public Env {
  338. public:
  339. PosixEnv();
  340. virtual ~PosixEnv() {
  341. char msg[] = "Destroying Env::Default()\n";
  342. fwrite(msg, 1, sizeof(msg), stderr);
  343. abort();
  344. }
  345. virtual Status NewSequentialFile(const std::string& fname,
  346. SequentialFile** result) {
  347. int fd = open(fname.c_str(), O_RDONLY);
  348. if (fd < 0) {
  349. *result = NULL;
  350. return PosixError(fname, errno);
  351. } else {
  352. *result = new PosixSequentialFile(fname, fd);
  353. return Status::OK();
  354. }
  355. }
  356. virtual Status NewRandomAccessFile(const std::string& fname,
  357. RandomAccessFile** result) {
  358. *result = NULL;
  359. Status s;
  360. int fd = open(fname.c_str(), O_RDONLY);
  361. if (fd < 0) {
  362. s = PosixError(fname, errno);
  363. } else if (mmap_limit_.Acquire()) {
  364. uint64_t size;
  365. s = GetFileSize(fname, &size);
  366. if (s.ok()) {
  367. void* base = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
  368. if (base != MAP_FAILED) {
  369. *result = new PosixMmapReadableFile(fname, base, size, &mmap_limit_);
  370. } else {
  371. s = PosixError(fname, errno);
  372. }
  373. }
  374. close(fd);
  375. if (!s.ok()) {
  376. mmap_limit_.Release();
  377. }
  378. } else {
  379. *result = new PosixRandomAccessFile(fname, fd, &fd_limit_);
  380. }
  381. return s;
  382. }
  383. virtual Status NewWritableFile(const std::string& fname,
  384. WritableFile** result) {
  385. Status s;
  386. int fd = open(fname.c_str(), O_TRUNC | O_WRONLY | O_CREAT, 0644);
  387. if (fd < 0) {
  388. *result = NULL;
  389. s = PosixError(fname, errno);
  390. } else {
  391. *result = new PosixWritableFile(fname, fd);
  392. }
  393. return s;
  394. }
  395. virtual Status NewAppendableFile(const std::string& fname,
  396. WritableFile** result) {
  397. Status s;
  398. int fd = open(fname.c_str(), O_APPEND | O_WRONLY | O_CREAT, 0644);
  399. if (fd < 0) {
  400. *result = NULL;
  401. s = PosixError(fname, errno);
  402. } else {
  403. *result = new PosixWritableFile(fname, fd);
  404. }
  405. return s;
  406. }
  407. virtual bool FileExists(const std::string& fname) {
  408. return access(fname.c_str(), F_OK) == 0;
  409. }
  410. virtual Status GetChildren(const std::string& dir,
  411. std::vector<std::string>* result) {
  412. result->clear();
  413. DIR* d = opendir(dir.c_str());
  414. if (d == NULL) {
  415. return PosixError(dir, errno);
  416. }
  417. struct dirent* entry;
  418. while ((entry = readdir(d)) != NULL) {
  419. result->push_back(entry->d_name);
  420. }
  421. closedir(d);
  422. return Status::OK();
  423. }
  424. virtual Status DeleteFile(const std::string& fname) {
  425. Status result;
  426. if (unlink(fname.c_str()) != 0) {
  427. result = PosixError(fname, errno);
  428. }
  429. return result;
  430. }
  431. virtual Status CreateDir(const std::string& name) {
  432. Status result;
  433. if (mkdir(name.c_str(), 0755) != 0) {
  434. result = PosixError(name, errno);
  435. }
  436. return result;
  437. }
  438. virtual Status DeleteDir(const std::string& name) {
  439. Status result;
  440. if (rmdir(name.c_str()) != 0) {
  441. result = PosixError(name, errno);
  442. }
  443. return result;
  444. }
  445. virtual Status GetFileSize(const std::string& fname, uint64_t* size) {
  446. Status s;
  447. struct stat sbuf;
  448. if (stat(fname.c_str(), &sbuf) != 0) {
  449. *size = 0;
  450. s = PosixError(fname, errno);
  451. } else {
  452. *size = sbuf.st_size;
  453. }
  454. return s;
  455. }
  456. virtual Status RenameFile(const std::string& src, const std::string& target) {
  457. Status result;
  458. if (rename(src.c_str(), target.c_str()) != 0) {
  459. result = PosixError(src, errno);
  460. }
  461. return result;
  462. }
  463. virtual Status LockFile(const std::string& fname, FileLock** lock) {
  464. *lock = NULL;
  465. Status result;
  466. int fd = open(fname.c_str(), O_RDWR | O_CREAT, 0644);
  467. if (fd < 0) {
  468. result = PosixError(fname, errno);
  469. } else if (!locks_.Insert(fname)) {
  470. close(fd);
  471. result = Status::IOError("lock " + fname, "already held by process");
  472. } else if (LockOrUnlock(fd, true) == -1) {
  473. result = PosixError("lock " + fname, errno);
  474. close(fd);
  475. locks_.Remove(fname);
  476. } else {
  477. PosixFileLock* my_lock = new PosixFileLock;
  478. my_lock->fd_ = fd;
  479. my_lock->name_ = fname;
  480. *lock = my_lock;
  481. }
  482. return result;
  483. }
  484. virtual Status UnlockFile(FileLock* lock) {
  485. PosixFileLock* my_lock = reinterpret_cast<PosixFileLock*>(lock);
  486. Status result;
  487. if (LockOrUnlock(my_lock->fd_, false) == -1) {
  488. result = PosixError("unlock", errno);
  489. }
  490. locks_.Remove(my_lock->name_);
  491. close(my_lock->fd_);
  492. delete my_lock;
  493. return result;
  494. }
  495. virtual void Schedule(void (*function)(void*), void* arg);
  496. virtual void StartThread(void (*function)(void* arg), void* arg);
  497. virtual Status GetTestDirectory(std::string* result) {
  498. const char* env = getenv("TEST_TMPDIR");
  499. if (env && env[0] != '\0') {
  500. *result = env;
  501. } else {
  502. char buf[100];
  503. snprintf(buf, sizeof(buf), "/tmp/leveldbtest-%d", int(geteuid()));
  504. *result = buf;
  505. }
  506. // Directory may already exist
  507. CreateDir(*result);
  508. return Status::OK();
  509. }
  510. static uint64_t gettid() {
  511. pthread_t tid = pthread_self();
  512. uint64_t thread_id = 0;
  513. memcpy(&thread_id, &tid, std::min(sizeof(thread_id), sizeof(tid)));
  514. return thread_id;
  515. }
  516. virtual Status NewLogger(const std::string& fname, Logger** result) {
  517. FILE* f = fopen(fname.c_str(), "w");
  518. if (f == NULL) {
  519. *result = NULL;
  520. return PosixError(fname, errno);
  521. } else {
  522. *result = new PosixLogger(f, &PosixEnv::gettid);
  523. return Status::OK();
  524. }
  525. }
  526. virtual uint64_t NowMicros() {
  527. struct timeval tv;
  528. gettimeofday(&tv, NULL);
  529. return static_cast<uint64_t>(tv.tv_sec) * 1000000 + tv.tv_usec;
  530. }
  531. virtual void SleepForMicroseconds(int micros) {
  532. usleep(micros);
  533. }
  534. private:
  535. void PthreadCall(const char* label, int result) {
  536. if (result != 0) {
  537. fprintf(stderr, "pthread %s: %s\n", label, strerror(result));
  538. abort();
  539. }
  540. }
  541. // BGThread() is the body of the background thread
  542. void BGThread();
  543. static void* BGThreadWrapper(void* arg) {
  544. reinterpret_cast<PosixEnv*>(arg)->BGThread();
  545. return NULL;
  546. }
  547. pthread_mutex_t mu_;
  548. pthread_cond_t bgsignal_;
  549. pthread_t bgthread_;
  550. bool started_bgthread_;
  551. // Entry per Schedule() call
  552. struct BGItem { void* arg; void (*function)(void*); };
  553. typedef std::deque<BGItem> BGQueue;
  554. BGQueue queue_;
  555. PosixLockTable locks_;
  556. Limiter mmap_limit_;
  557. Limiter fd_limit_;
  558. };
  559. // Return the maximum number of concurrent mmaps.
  560. static int MaxMmaps() {
  561. if (mmap_limit >= 0) {
  562. return mmap_limit;
  563. }
  564. // Up to 1000 mmaps for 64-bit binaries; none for smaller pointer sizes.
  565. mmap_limit = sizeof(void*) >= 8 ? 1000 : 0;
  566. return mmap_limit;
  567. }
  568. // Return the maximum number of read-only files to keep open.
  569. static intptr_t MaxOpenFiles() {
  570. if (open_read_only_file_limit >= 0) {
  571. return open_read_only_file_limit;
  572. }
  573. struct rlimit rlim;
  574. if (getrlimit(RLIMIT_NOFILE, &rlim)) {
  575. // getrlimit failed, fallback to hard-coded default.
  576. open_read_only_file_limit = 50;
  577. } else if (rlim.rlim_cur == RLIM_INFINITY) {
  578. open_read_only_file_limit = std::numeric_limits<int>::max();
  579. } else {
  580. // Allow use of 20% of available file descriptors for read-only files.
  581. open_read_only_file_limit = rlim.rlim_cur / 5;
  582. }
  583. return open_read_only_file_limit;
  584. }
  585. PosixEnv::PosixEnv()
  586. : started_bgthread_(false),
  587. mmap_limit_(MaxMmaps()),
  588. fd_limit_(MaxOpenFiles()) {
  589. PthreadCall("mutex_init", pthread_mutex_init(&mu_, NULL));
  590. PthreadCall("cvar_init", pthread_cond_init(&bgsignal_, NULL));
  591. }
  592. void PosixEnv::Schedule(void (*function)(void*), void* arg) {
  593. PthreadCall("lock", pthread_mutex_lock(&mu_));
  594. // Start background thread if necessary
  595. if (!started_bgthread_) {
  596. started_bgthread_ = true;
  597. PthreadCall(
  598. "create thread",
  599. pthread_create(&bgthread_, NULL, &PosixEnv::BGThreadWrapper, this));
  600. }
  601. // If the queue is currently empty, the background thread may currently be
  602. // waiting.
  603. if (queue_.empty()) {
  604. PthreadCall("signal", pthread_cond_signal(&bgsignal_));
  605. }
  606. // Add to priority queue
  607. queue_.push_back(BGItem());
  608. queue_.back().function = function;
  609. queue_.back().arg = arg;
  610. PthreadCall("unlock", pthread_mutex_unlock(&mu_));
  611. }
  612. void PosixEnv::BGThread() {
  613. while (true) {
  614. // Wait until there is an item that is ready to run
  615. PthreadCall("lock", pthread_mutex_lock(&mu_));
  616. while (queue_.empty()) {
  617. PthreadCall("wait", pthread_cond_wait(&bgsignal_, &mu_));
  618. }
  619. void (*function)(void*) = queue_.front().function;
  620. void* arg = queue_.front().arg;
  621. queue_.pop_front();
  622. PthreadCall("unlock", pthread_mutex_unlock(&mu_));
  623. (*function)(arg);
  624. }
  625. }
  626. namespace {
  627. struct StartThreadState {
  628. void (*user_function)(void*);
  629. void* arg;
  630. };
  631. }
  632. static void* StartThreadWrapper(void* arg) {
  633. StartThreadState* state = reinterpret_cast<StartThreadState*>(arg);
  634. state->user_function(state->arg);
  635. delete state;
  636. return NULL;
  637. }
  638. void PosixEnv::StartThread(void (*function)(void* arg), void* arg) {
  639. pthread_t t;
  640. StartThreadState* state = new StartThreadState;
  641. state->user_function = function;
  642. state->arg = arg;
  643. PthreadCall("start thread",
  644. pthread_create(&t, NULL, &StartThreadWrapper, state));
  645. }
  646. } // namespace
  647. static pthread_once_t once = PTHREAD_ONCE_INIT;
  648. static Env* default_env;
  649. static void InitDefaultEnv() { default_env = new PosixEnv; }
  650. void EnvPosixTestHelper::SetReadOnlyFDLimit(int limit) {
  651. assert(default_env == NULL);
  652. open_read_only_file_limit = limit;
  653. }
  654. void EnvPosixTestHelper::SetReadOnlyMMapLimit(int limit) {
  655. assert(default_env == NULL);
  656. mmap_limit = limit;
  657. }
  658. Env* Env::Default() {
  659. pthread_once(&once, InitDefaultEnv);
  660. return default_env;
  661. }
  662. } // namespace leveldb