FileState::Read (used by InMemoryEnv) creates a new Slice when reading.
If all the bytes for the read are in the first block then the Slice
points to the private block data in FileState and is not copied to the
|scratch| buffer.
A recent change allows files in InMemEnv to be overwritten which deletes
these blocks and in this case can result in a Slice having a dangling
pointer. This change fixes this bug by always copying to the |scratch|
buffer.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=239301930
Env's (like the POSIX Env) which use an actual filesystem behave
differently than InMemoryEnv with regards to writing data to a currently
open file.
InMemoryEnv::NewWritableFile would previously delete that file,
if it was open, before creating a new file so any previously
open file would be unlinked. This change truncates an open file
so that subsequent reads will read that new data.
This should have no impact on leveldb as it never has the same
file open for both read and write access. This change is only
being made for tests (specifically a future change to corruption_test)
to allow them to be decoupled from the underlying platform and
allow them to use an Env.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=237858231
helpers/memenv/memenv.cc used SIZE_MAX without including <stdint.h>.
Since we're fixing this problem, replace SIZE_MAX with
std::numeric_limits<size_t>::max(), which is clearer.
Fixes https://github.com/google/leveldb/issues/562
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189821707
This CL makes it easier to reason about thread safety by:
1) Adding Clang thread safety annotations according to comments.
2) Expanding a couple of variable names, without adding extra lines of code.
3) Adding const in a couple of places.
4) Replacing an always-non-null const pointer with a reference.
5) Fixing style warnings in the modified files.
This CL does not annotate the DBImpl members that claim to be protected
by the instance mutex, but are accessed without the mutex being held.
Those members (and their unprotected accesses) will be addressed in
future CLs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=189354657
(Based on a suggestion by cmumford.)
"open" benchmark on my workstation speeds up significantly since we
can now avoid three fdatasync calls and a compaction per open:
Before: ~80000 microseconds
After: ~130 microseconds
Details:
(1) Added Options::reuse_logs (currently defaults to false) to control
new behavior. The intention is to change the default to true after some
baking.
(2) Added Env::NewAppendableFile() whose default implementation returns
a not-supported error.
(3) VersionSet::Recovery attempts to reuse the MANIFEST from which
it is recovering.
(4) DBImpl recovery attempts to reuse the last log file and memtable.
(5) db_test.cc now tests a new configuration that sets reuse_logs to true.
(6) fault_injection_test also tests a reuse_logs==true config.
(7) Added a new recovery_test.
Changes are:
* Update version number to 1.18
* Replace the basic fprintf call with a call to fwrite in order to
work around the apparent compiler optimization/rewrite failure that we are
seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8.
* Fix ALL the header guards.
* Createed a README.md with the LevelDB project description.
* A new CONTRIBUTING file.
* Don't implicitly convert uint64_t to size_t or int. Either preserve it as
uint64_t, or explicitly cast. This fixes MSVC warnings about possible value
truncation when compiling this code in Chromium.
* Added a DumpFile() library function that encapsulates the guts of the
"leveldbutil dump" command. This will allow clients to dump
data to their log files instead of stdout. It will also allow clients to
supply their own environment.
* leveldb: Remove unused function 'ConsumeChar'.
* leveldbutil: Remove unused member variables from WriteBatchItemPrinter.
* OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define
PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes:
* issue #143
* issue #198
* issue #249
* Switch from <cstdatomic> to <atomic>. The former never made it into the
standard and doesn't exist in modern gcc versions at all. The later contains
everything that leveldb was using from the former.
This problem was noticed when porting to Portable Native Client where no memory
barrier is defined. The fact that <cstdatomic> is missing normally goes
unnoticed since memory barriers are defined for most architectures.
* Make Hash() treat its input as unsigned. Before this change LevelDB files
from platforms with different signedness of char were not compatible. This
change fixes: issue #243
* Verify checksums of index/meta/filter blocks when paranoid_checks set.
* Invoke all tools for iOS with xcrun. (This was causing problems with the new
XCode 5.1.1 image on pulse.)
* include <sys/stat.h> only once, and fix the following linter warning:
"Found C system header after C++ system header"
* When encountering a corrupted table file, return Status::Corruption instead of
Status::InvalidArgument.
* Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188
* Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159
* Fix typos and comments, and address the following two issues:
* issue #166
* issue #241
* Add missing db synchronize after "fillseq" in the benchmark.
* Removed unused variable in SeekRandom: value (issue #201)
Highlights
----------
Mmap at most 1000 files on Posix to improve performance for large databases.
Support for more architectures (thanks to Alexander K.)
Building and porting
--------------------
HP/UX support (issue 126)
AtomicPointer for ia64 (issue 123)
Sparc v9 support (issue 124)
Atomic ops for powerpc
Use -fno-builtin-memcmp only when using g++
Simplify IOS build rules (issue 114)
Use CXXFLAGS instead of CFLAGS when invoking C++ compiler (issue 118)
Fix snappy shared library problem (issue 94)
Fix shared library installation path regression
Endian-ness detection tweak for FreeBSD
Bug fixes
---------
Stop ignoring FLAGS_open_files in db_bench
Make bloom test behavior agnostic to endian-ness
Performance
-----------
Limit number of mmapped files to 1000 to improve perf for large dbs
Do not delay for 1 second on shutdown path (issue 125)
Misc
----
Make InMemoryEnv return a no-op logger
C binding now has a wrapper for free (issue 117)
Add thread-safety annotations
Added an in-process lock table (issue 120)
Make RandomAccessFile and SequentialFile non-copyable
- Replace raw slice comparison with a call to user comparator.
Added test for custom comparators.
- Fix end of namespace comments.
- Fixed bug in picking inputs for a level-0 compaction.
When finding overlapping files, the covered range may expand
as files are added to the input set. We now correctly expand
the range when this happens instead of continuing to use the
old range. For example, suppose L0 contains files with the
following ranges:
F1: a .. d
F2: c .. g
F3: f .. j
and the initial compaction target is F3. We used to search
for range f..j which yielded {F2,F3}. However we now expand
the range as soon as another file is added. In this case,
when F2 is added, we expand the range to c..j and restart the
search. That picks up file F1 as well.
This change fixes a bug related to deleted keys showing up
incorrectly after a compaction as described in Issue 44.
(Sync with upstream @25072954)
Fix GCC -Wshadow warnings in LevelDB's public header files,
reported by Dustin.
Add in-memory Env implementation (helpers/memenv/*).
This enables users to create LevelDB databases in-memory.
Initialize ShardedLRUCache::last_id_ to zero.
This fixes a Valgrind warning.
(Also delete port/sha1_* which were removed upstream some time ago.)
cmumford
costan
Sanjay Ghemawat
Chris Mumford
David Grogan
Hans Wennborg