作者: 谢瑞阳 10225101483 徐翔宇 10225101535
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

262 lines
8.5 KiB

Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 years ago
Replace SSE-optimized CRC32C in POSIX port with external library. Maintaining a hardware-accelerated CRC32C implementation tailored for all modern platforms deserves a repository of its own. We extracted the implementation here into https://github.com/google/crc32c and improved it in that repository. This CL removes the SSE-optimized implementation from this codebase, and adds the ability to use the google/crc32c library, if it is present on the system. The benchmarks below show the performance impact of the change. In summary, open source builds that use the google/crc32c library can expect a 3x improvement in CRC32C throughput, whereas builds that do not use the library will see a 50% drop in CRC32C throughput. This translates in much smaller changes in overall leveldb performance. Baseline, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.064 micros/op; 36.1 MB/s fillsync : 57.861 micros/op; 1.9 MB/s (1000 ops) fillrandom : 3.887 micros/op; 28.5 MB/s overwrite : 4.140 micros/op; 26.7 MB/s readrandom : 7.433 micros/op; (1000000 of 1000000 found) readrandom : 6.825 micros/op; (1000000 of 1000000 found) readseq : 0.244 micros/op; 453.4 MB/s readreverse : 0.387 micros/op; 285.8 MB/s compact : 449707.000 micros/op; readrandom : 4.196 micros/op; (1000000 of 1000000 found) readseq : 0.228 micros/op; 485.8 MB/s readreverse : 0.320 micros/op; 345.2 MB/s fill100K : 562.556 micros/op; 169.6 MB/s (1000 ops) crc32c : 0.768 micros/op; 5085.0 MB/s (4K per op) snappycomp : 4.220 micros/op; 925.7 MB/s (output: 55.1%) snappyuncomp : 0.635 micros/op; 6155.7 MB/s acquireload : 13.054 micros/op; (each op is 1000 loads) New with crc32c, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 2.820 micros/op; 39.2 MB/s fillsync : 51.988 micros/op; 2.1 MB/s (1000 ops) fillrandom : 3.747 micros/op; 29.5 MB/s overwrite : 4.047 micros/op; 27.3 MB/s readrandom : 7.287 micros/op; (1000000 of 1000000 found) readrandom : 6.927 micros/op; (1000000 of 1000000 found) readseq : 0.253 micros/op; 437.5 MB/s readreverse : 0.411 micros/op; 269.2 MB/s compact : 440405.000 micros/op; readrandom : 4.159 micros/op; (1000000 of 1000000 found) readseq : 0.230 micros/op; 481.1 MB/s readreverse : 0.320 micros/op; 345.9 MB/s fill100K : 558.222 micros/op; 170.9 MB/s (1000 ops) crc32c : 0.214 micros/op; 18263.5 MB/s (4K per op) snappycomp : 4.471 micros/op; 873.7 MB/s (output: 55.1%) snappyuncomp : 0.833 micros/op; 4688.5 MB/s acquireload : 13.289 micros/op; (each op is 1000 loads) New without crc32c, MacBookPro13,3 with Core i7 6920HQ LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.094 micros/op; 35.8 MB/s fillsync : 52.160 micros/op; 2.1 MB/s (1000 ops) fillrandom : 4.090 micros/op; 27.0 MB/s overwrite : 4.006 micros/op; 27.6 MB/s readrandom : 6.584 micros/op; (1000000 of 1000000 found) readrandom : 6.676 micros/op; (1000000 of 1000000 found) readseq : 0.280 micros/op; 395.2 MB/s readreverse : 0.391 micros/op; 283.2 MB/s compact : 433911.000 micros/op; readrandom : 4.261 micros/op; (1000000 of 1000000 found) readseq : 0.251 micros/op; 440.5 MB/s readreverse : 0.356 micros/op; 310.9 MB/s fill100K : 584.023 micros/op; 163.3 MB/s (1000 ops) crc32c : 1.384 micros/op; 2822.3 MB/s (4K per op) snappycomp : 4.763 micros/op; 820.1 MB/s (output: 55.1%) snappyuncomp : 0.766 micros/op; 5098.6 MB/s acquireload : 12.931 micros/op; (each op is 1000 loads) ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=171667771
7 years ago
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 years ago
Replace SSE-optimized CRC32C in POSIX port with external library. Maintaining a hardware-accelerated CRC32C implementation tailored for all modern platforms deserves a repository of its own. We extracted the implementation here into https://github.com/google/crc32c and improved it in that repository. This CL removes the SSE-optimized implementation from this codebase, and adds the ability to use the google/crc32c library, if it is present on the system. The benchmarks below show the performance impact of the change. In summary, open source builds that use the google/crc32c library can expect a 3x improvement in CRC32C throughput, whereas builds that do not use the library will see a 50% drop in CRC32C throughput. This translates in much smaller changes in overall leveldb performance. Baseline, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.064 micros/op; 36.1 MB/s fillsync : 57.861 micros/op; 1.9 MB/s (1000 ops) fillrandom : 3.887 micros/op; 28.5 MB/s overwrite : 4.140 micros/op; 26.7 MB/s readrandom : 7.433 micros/op; (1000000 of 1000000 found) readrandom : 6.825 micros/op; (1000000 of 1000000 found) readseq : 0.244 micros/op; 453.4 MB/s readreverse : 0.387 micros/op; 285.8 MB/s compact : 449707.000 micros/op; readrandom : 4.196 micros/op; (1000000 of 1000000 found) readseq : 0.228 micros/op; 485.8 MB/s readreverse : 0.320 micros/op; 345.2 MB/s fill100K : 562.556 micros/op; 169.6 MB/s (1000 ops) crc32c : 0.768 micros/op; 5085.0 MB/s (4K per op) snappycomp : 4.220 micros/op; 925.7 MB/s (output: 55.1%) snappyuncomp : 0.635 micros/op; 6155.7 MB/s acquireload : 13.054 micros/op; (each op is 1000 loads) New with crc32c, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 2.820 micros/op; 39.2 MB/s fillsync : 51.988 micros/op; 2.1 MB/s (1000 ops) fillrandom : 3.747 micros/op; 29.5 MB/s overwrite : 4.047 micros/op; 27.3 MB/s readrandom : 7.287 micros/op; (1000000 of 1000000 found) readrandom : 6.927 micros/op; (1000000 of 1000000 found) readseq : 0.253 micros/op; 437.5 MB/s readreverse : 0.411 micros/op; 269.2 MB/s compact : 440405.000 micros/op; readrandom : 4.159 micros/op; (1000000 of 1000000 found) readseq : 0.230 micros/op; 481.1 MB/s readreverse : 0.320 micros/op; 345.9 MB/s fill100K : 558.222 micros/op; 170.9 MB/s (1000 ops) crc32c : 0.214 micros/op; 18263.5 MB/s (4K per op) snappycomp : 4.471 micros/op; 873.7 MB/s (output: 55.1%) snappyuncomp : 0.833 micros/op; 4688.5 MB/s acquireload : 13.289 micros/op; (each op is 1000 loads) New without crc32c, MacBookPro13,3 with Core i7 6920HQ LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.094 micros/op; 35.8 MB/s fillsync : 52.160 micros/op; 2.1 MB/s (1000 ops) fillrandom : 4.090 micros/op; 27.0 MB/s overwrite : 4.006 micros/op; 27.6 MB/s readrandom : 6.584 micros/op; (1000000 of 1000000 found) readrandom : 6.676 micros/op; (1000000 of 1000000 found) readseq : 0.280 micros/op; 395.2 MB/s readreverse : 0.391 micros/op; 283.2 MB/s compact : 433911.000 micros/op; readrandom : 4.261 micros/op; (1000000 of 1000000 found) readseq : 0.251 micros/op; 440.5 MB/s readreverse : 0.356 micros/op; 310.9 MB/s fill100K : 584.023 micros/op; 163.3 MB/s (1000 ops) crc32c : 1.384 micros/op; 2822.3 MB/s (4K per op) snappycomp : 4.763 micros/op; 820.1 MB/s (output: 55.1%) snappyuncomp : 0.766 micros/op; 5098.6 MB/s acquireload : 12.931 micros/op; (each op is 1000 loads) ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=171667771
7 years ago
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 years ago
Release 1.18 Changes are: * Update version number to 1.18 * Replace the basic fprintf call with a call to fwrite in order to work around the apparent compiler optimization/rewrite failure that we are seeing with the new toolchain/iOS SDKs provided with Xcode6 and iOS8. * Fix ALL the header guards. * Createed a README.md with the LevelDB project description. * A new CONTRIBUTING file. * Don't implicitly convert uint64_t to size_t or int. Either preserve it as uint64_t, or explicitly cast. This fixes MSVC warnings about possible value truncation when compiling this code in Chromium. * Added a DumpFile() library function that encapsulates the guts of the "leveldbutil dump" command. This will allow clients to dump data to their log files instead of stdout. It will also allow clients to supply their own environment. * leveldb: Remove unused function 'ConsumeChar'. * leveldbutil: Remove unused member variables from WriteBatchItemPrinter. * OpenBSD, NetBSD and DragonflyBSD have _LITTLE_ENDIAN, so define PLATFORM_IS_LITTLE_ENDIAN like on FreeBSD. This fixes: * issue #143 * issue #198 * issue #249 * Switch from <cstdatomic> to <atomic>. The former never made it into the standard and doesn't exist in modern gcc versions at all. The later contains everything that leveldb was using from the former. This problem was noticed when porting to Portable Native Client where no memory barrier is defined. The fact that <cstdatomic> is missing normally goes unnoticed since memory barriers are defined for most architectures. * Make Hash() treat its input as unsigned. Before this change LevelDB files from platforms with different signedness of char were not compatible. This change fixes: issue #243 * Verify checksums of index/meta/filter blocks when paranoid_checks set. * Invoke all tools for iOS with xcrun. (This was causing problems with the new XCode 5.1.1 image on pulse.) * include <sys/stat.h> only once, and fix the following linter warning: "Found C system header after C++ system header" * When encountering a corrupted table file, return Status::Corruption instead of Status::InvalidArgument. * Support cygwin as build platform, patch is from https://code.google.com/p/leveldb/issues/detail?id=188 * Fix typo, merge patch from https://code.google.com/p/leveldb/issues/detail?id=159 * Fix typos and comments, and address the following two issues: * issue #166 * issue #241 * Add missing db synchronize after "fillseq" in the benchmark. * Removed unused variable in SeekRandom: value (issue #201)
10 years ago
Replace SSE-optimized CRC32C in POSIX port with external library. Maintaining a hardware-accelerated CRC32C implementation tailored for all modern platforms deserves a repository of its own. We extracted the implementation here into https://github.com/google/crc32c and improved it in that repository. This CL removes the SSE-optimized implementation from this codebase, and adds the ability to use the google/crc32c library, if it is present on the system. The benchmarks below show the performance impact of the change. In summary, open source builds that use the google/crc32c library can expect a 3x improvement in CRC32C throughput, whereas builds that do not use the library will see a 50% drop in CRC32C throughput. This translates in much smaller changes in overall leveldb performance. Baseline, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.064 micros/op; 36.1 MB/s fillsync : 57.861 micros/op; 1.9 MB/s (1000 ops) fillrandom : 3.887 micros/op; 28.5 MB/s overwrite : 4.140 micros/op; 26.7 MB/s readrandom : 7.433 micros/op; (1000000 of 1000000 found) readrandom : 6.825 micros/op; (1000000 of 1000000 found) readseq : 0.244 micros/op; 453.4 MB/s readreverse : 0.387 micros/op; 285.8 MB/s compact : 449707.000 micros/op; readrandom : 4.196 micros/op; (1000000 of 1000000 found) readseq : 0.228 micros/op; 485.8 MB/s readreverse : 0.320 micros/op; 345.2 MB/s fill100K : 562.556 micros/op; 169.6 MB/s (1000 ops) crc32c : 0.768 micros/op; 5085.0 MB/s (4K per op) snappycomp : 4.220 micros/op; 925.7 MB/s (output: 55.1%) snappyuncomp : 0.635 micros/op; 6155.7 MB/s acquireload : 13.054 micros/op; (each op is 1000 loads) New with crc32c, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 2.820 micros/op; 39.2 MB/s fillsync : 51.988 micros/op; 2.1 MB/s (1000 ops) fillrandom : 3.747 micros/op; 29.5 MB/s overwrite : 4.047 micros/op; 27.3 MB/s readrandom : 7.287 micros/op; (1000000 of 1000000 found) readrandom : 6.927 micros/op; (1000000 of 1000000 found) readseq : 0.253 micros/op; 437.5 MB/s readreverse : 0.411 micros/op; 269.2 MB/s compact : 440405.000 micros/op; readrandom : 4.159 micros/op; (1000000 of 1000000 found) readseq : 0.230 micros/op; 481.1 MB/s readreverse : 0.320 micros/op; 345.9 MB/s fill100K : 558.222 micros/op; 170.9 MB/s (1000 ops) crc32c : 0.214 micros/op; 18263.5 MB/s (4K per op) snappycomp : 4.471 micros/op; 873.7 MB/s (output: 55.1%) snappyuncomp : 0.833 micros/op; 4688.5 MB/s acquireload : 13.289 micros/op; (each op is 1000 loads) New without crc32c, MacBookPro13,3 with Core i7 6920HQ LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.094 micros/op; 35.8 MB/s fillsync : 52.160 micros/op; 2.1 MB/s (1000 ops) fillrandom : 4.090 micros/op; 27.0 MB/s overwrite : 4.006 micros/op; 27.6 MB/s readrandom : 6.584 micros/op; (1000000 of 1000000 found) readrandom : 6.676 micros/op; (1000000 of 1000000 found) readseq : 0.280 micros/op; 395.2 MB/s readreverse : 0.391 micros/op; 283.2 MB/s compact : 433911.000 micros/op; readrandom : 4.261 micros/op; (1000000 of 1000000 found) readseq : 0.251 micros/op; 440.5 MB/s readreverse : 0.356 micros/op; 310.9 MB/s fill100K : 584.023 micros/op; 163.3 MB/s (1000 ops) crc32c : 1.384 micros/op; 2822.3 MB/s (4K per op) snappycomp : 4.763 micros/op; 820.1 MB/s (output: 55.1%) snappyuncomp : 0.766 micros/op; 5098.6 MB/s acquireload : 12.931 micros/op; (each op is 1000 loads) ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=171667771
7 years ago
Replace SSE-optimized CRC32C in POSIX port with external library. Maintaining a hardware-accelerated CRC32C implementation tailored for all modern platforms deserves a repository of its own. We extracted the implementation here into https://github.com/google/crc32c and improved it in that repository. This CL removes the SSE-optimized implementation from this codebase, and adds the ability to use the google/crc32c library, if it is present on the system. The benchmarks below show the performance impact of the change. In summary, open source builds that use the google/crc32c library can expect a 3x improvement in CRC32C throughput, whereas builds that do not use the library will see a 50% drop in CRC32C throughput. This translates in much smaller changes in overall leveldb performance. Baseline, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.064 micros/op; 36.1 MB/s fillsync : 57.861 micros/op; 1.9 MB/s (1000 ops) fillrandom : 3.887 micros/op; 28.5 MB/s overwrite : 4.140 micros/op; 26.7 MB/s readrandom : 7.433 micros/op; (1000000 of 1000000 found) readrandom : 6.825 micros/op; (1000000 of 1000000 found) readseq : 0.244 micros/op; 453.4 MB/s readreverse : 0.387 micros/op; 285.8 MB/s compact : 449707.000 micros/op; readrandom : 4.196 micros/op; (1000000 of 1000000 found) readseq : 0.228 micros/op; 485.8 MB/s readreverse : 0.320 micros/op; 345.2 MB/s fill100K : 562.556 micros/op; 169.6 MB/s (1000 ops) crc32c : 0.768 micros/op; 5085.0 MB/s (4K per op) snappycomp : 4.220 micros/op; 925.7 MB/s (output: 55.1%) snappyuncomp : 0.635 micros/op; 6155.7 MB/s acquireload : 13.054 micros/op; (each op is 1000 loads) New with crc32c, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 2.820 micros/op; 39.2 MB/s fillsync : 51.988 micros/op; 2.1 MB/s (1000 ops) fillrandom : 3.747 micros/op; 29.5 MB/s overwrite : 4.047 micros/op; 27.3 MB/s readrandom : 7.287 micros/op; (1000000 of 1000000 found) readrandom : 6.927 micros/op; (1000000 of 1000000 found) readseq : 0.253 micros/op; 437.5 MB/s readreverse : 0.411 micros/op; 269.2 MB/s compact : 440405.000 micros/op; readrandom : 4.159 micros/op; (1000000 of 1000000 found) readseq : 0.230 micros/op; 481.1 MB/s readreverse : 0.320 micros/op; 345.9 MB/s fill100K : 558.222 micros/op; 170.9 MB/s (1000 ops) crc32c : 0.214 micros/op; 18263.5 MB/s (4K per op) snappycomp : 4.471 micros/op; 873.7 MB/s (output: 55.1%) snappyuncomp : 0.833 micros/op; 4688.5 MB/s acquireload : 13.289 micros/op; (each op is 1000 loads) New without crc32c, MacBookPro13,3 with Core i7 6920HQ LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.094 micros/op; 35.8 MB/s fillsync : 52.160 micros/op; 2.1 MB/s (1000 ops) fillrandom : 4.090 micros/op; 27.0 MB/s overwrite : 4.006 micros/op; 27.6 MB/s readrandom : 6.584 micros/op; (1000000 of 1000000 found) readrandom : 6.676 micros/op; (1000000 of 1000000 found) readseq : 0.280 micros/op; 395.2 MB/s readreverse : 0.391 micros/op; 283.2 MB/s compact : 433911.000 micros/op; readrandom : 4.261 micros/op; (1000000 of 1000000 found) readseq : 0.251 micros/op; 440.5 MB/s readreverse : 0.356 micros/op; 310.9 MB/s fill100K : 584.023 micros/op; 163.3 MB/s (1000 ops) crc32c : 1.384 micros/op; 2822.3 MB/s (4K per op) snappycomp : 4.763 micros/op; 820.1 MB/s (output: 55.1%) snappyuncomp : 0.766 micros/op; 5098.6 MB/s acquireload : 12.931 micros/op; (each op is 1000 loads) ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=171667771
7 years ago
Replace SSE-optimized CRC32C in POSIX port with external library. Maintaining a hardware-accelerated CRC32C implementation tailored for all modern platforms deserves a repository of its own. We extracted the implementation here into https://github.com/google/crc32c and improved it in that repository. This CL removes the SSE-optimized implementation from this codebase, and adds the ability to use the google/crc32c library, if it is present on the system. The benchmarks below show the performance impact of the change. In summary, open source builds that use the google/crc32c library can expect a 3x improvement in CRC32C throughput, whereas builds that do not use the library will see a 50% drop in CRC32C throughput. This translates in much smaller changes in overall leveldb performance. Baseline, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.064 micros/op; 36.1 MB/s fillsync : 57.861 micros/op; 1.9 MB/s (1000 ops) fillrandom : 3.887 micros/op; 28.5 MB/s overwrite : 4.140 micros/op; 26.7 MB/s readrandom : 7.433 micros/op; (1000000 of 1000000 found) readrandom : 6.825 micros/op; (1000000 of 1000000 found) readseq : 0.244 micros/op; 453.4 MB/s readreverse : 0.387 micros/op; 285.8 MB/s compact : 449707.000 micros/op; readrandom : 4.196 micros/op; (1000000 of 1000000 found) readseq : 0.228 micros/op; 485.8 MB/s readreverse : 0.320 micros/op; 345.2 MB/s fill100K : 562.556 micros/op; 169.6 MB/s (1000 ops) crc32c : 0.768 micros/op; 5085.0 MB/s (4K per op) snappycomp : 4.220 micros/op; 925.7 MB/s (output: 55.1%) snappyuncomp : 0.635 micros/op; 6155.7 MB/s acquireload : 13.054 micros/op; (each op is 1000 loads) New with crc32c, MacBookPro13,3 with Core i7 6920HQ: LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 2.820 micros/op; 39.2 MB/s fillsync : 51.988 micros/op; 2.1 MB/s (1000 ops) fillrandom : 3.747 micros/op; 29.5 MB/s overwrite : 4.047 micros/op; 27.3 MB/s readrandom : 7.287 micros/op; (1000000 of 1000000 found) readrandom : 6.927 micros/op; (1000000 of 1000000 found) readseq : 0.253 micros/op; 437.5 MB/s readreverse : 0.411 micros/op; 269.2 MB/s compact : 440405.000 micros/op; readrandom : 4.159 micros/op; (1000000 of 1000000 found) readseq : 0.230 micros/op; 481.1 MB/s readreverse : 0.320 micros/op; 345.9 MB/s fill100K : 558.222 micros/op; 170.9 MB/s (1000 ops) crc32c : 0.214 micros/op; 18263.5 MB/s (4K per op) snappycomp : 4.471 micros/op; 873.7 MB/s (output: 55.1%) snappyuncomp : 0.833 micros/op; 4688.5 MB/s acquireload : 13.289 micros/op; (each op is 1000 loads) New without crc32c, MacBookPro13,3 with Core i7 6920HQ LevelDB: version 1.20 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 3.094 micros/op; 35.8 MB/s fillsync : 52.160 micros/op; 2.1 MB/s (1000 ops) fillrandom : 4.090 micros/op; 27.0 MB/s overwrite : 4.006 micros/op; 27.6 MB/s readrandom : 6.584 micros/op; (1000000 of 1000000 found) readrandom : 6.676 micros/op; (1000000 of 1000000 found) readseq : 0.280 micros/op; 395.2 MB/s readreverse : 0.391 micros/op; 283.2 MB/s compact : 433911.000 micros/op; readrandom : 4.261 micros/op; (1000000 of 1000000 found) readseq : 0.251 micros/op; 440.5 MB/s readreverse : 0.356 micros/op; 310.9 MB/s fill100K : 584.023 micros/op; 163.3 MB/s (1000 ops) crc32c : 1.384 micros/op; 2822.3 MB/s (4K per op) snappycomp : 4.763 micros/op; 820.1 MB/s (output: 55.1%) snappyuncomp : 0.766 micros/op; 5098.6 MB/s acquireload : 12.931 micros/op; (each op is 1000 loads) ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=171667771
7 years ago
  1. #!/bin/sh
  2. #
  3. # Detects OS we're compiling on and outputs a file specified by the first
  4. # argument, which in turn gets read while processing Makefile.
  5. #
  6. # The output will set the following variables:
  7. # CC C Compiler path
  8. # CXX C++ Compiler path
  9. # PLATFORM_LDFLAGS Linker flags
  10. # PLATFORM_LIBS Libraries flags
  11. # PLATFORM_SHARED_EXT Extension for shared libraries
  12. # PLATFORM_SHARED_LDFLAGS Flags for building shared library
  13. # This flag is embedded just before the name
  14. # of the shared library without intervening spaces
  15. # PLATFORM_SHARED_CFLAGS Flags for compiling objects for shared library
  16. # PLATFORM_CCFLAGS C compiler flags
  17. # PLATFORM_CXXFLAGS C++ compiler flags. Will contain:
  18. # PLATFORM_SHARED_VERSIONED Set to 'true' if platform supports versioned
  19. # shared libraries, empty otherwise.
  20. #
  21. # The PLATFORM_CCFLAGS and PLATFORM_CXXFLAGS might include the following:
  22. #
  23. # -DLEVELDB_ATOMIC_PRESENT if <atomic> is present
  24. # -DLEVELDB_PLATFORM_POSIX=1 for Posix-based platforms
  25. # -DHAVE_CRC32C=1 if the CRC32C library is present
  26. # -DHAVE_SNAPPY=1 if the Snappy library is present
  27. #
  28. OUTPUT=$1
  29. PREFIX=$2
  30. if test -z "$OUTPUT" || test -z "$PREFIX"; then
  31. echo "usage: $0 <output-filename> <directory_prefix>" >&2
  32. exit 1
  33. fi
  34. # Delete existing output, if it exists
  35. rm -f $OUTPUT
  36. touch $OUTPUT
  37. if test -z "$CC"; then
  38. CC=cc
  39. fi
  40. if test -z "$CXX"; then
  41. CXX=g++
  42. fi
  43. if test -z "$TMPDIR"; then
  44. TMPDIR=/tmp
  45. fi
  46. # Detect OS
  47. if test -z "$TARGET_OS"; then
  48. TARGET_OS=`uname -s`
  49. fi
  50. COMMON_FLAGS=
  51. CROSS_COMPILE=
  52. PLATFORM_CCFLAGS=
  53. PLATFORM_CXXFLAGS=
  54. PLATFORM_LDFLAGS=
  55. PLATFORM_LIBS=
  56. PLATFORM_SHARED_EXT="so"
  57. PLATFORM_SHARED_LDFLAGS="-shared -Wl,-soname -Wl,"
  58. PLATFORM_SHARED_CFLAGS="-fPIC -fvisibility=hidden"
  59. PLATFORM_SHARED_VERSIONED=true
  60. MEMCMP_FLAG=
  61. if [ "$CXX" = "g++" ]; then
  62. # Use libc's memcmp instead of GCC's memcmp. This results in ~40%
  63. # performance improvement on readrandom under gcc 4.4.3 on Linux/x86.
  64. MEMCMP_FLAG="-fno-builtin-memcmp"
  65. fi
  66. case "$TARGET_OS" in
  67. CYGWIN_*)
  68. PLATFORM=OS_LINUX
  69. COMMON_FLAGS="$MEMCMP_FLAG -lpthread -DOS_LINUX -DCYGWIN"
  70. PLATFORM_LDFLAGS="-lpthread"
  71. PORT_FILE=port/port_posix.cc
  72. ;;
  73. Darwin)
  74. PLATFORM=OS_MACOSX
  75. COMMON_FLAGS="$MEMCMP_FLAG"
  76. PLATFORM_SHARED_EXT=dylib
  77. [ -z "$INSTALL_PATH" ] && INSTALL_PATH=`pwd`
  78. PLATFORM_SHARED_LDFLAGS="-dynamiclib -install_name $INSTALL_PATH/"
  79. PORT_FILE=port/port_posix.cc
  80. ;;
  81. Linux)
  82. PLATFORM=OS_LINUX
  83. COMMON_FLAGS="$MEMCMP_FLAG -pthread -DOS_LINUX"
  84. PLATFORM_LDFLAGS="-pthread"
  85. PORT_FILE=port/port_posix.cc
  86. ;;
  87. SunOS)
  88. PLATFORM=OS_SOLARIS
  89. COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_SOLARIS"
  90. PLATFORM_LIBS="-lpthread -lrt"
  91. PORT_FILE=port/port_posix.cc
  92. ;;
  93. FreeBSD)
  94. PLATFORM=OS_FREEBSD
  95. COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_FREEBSD"
  96. PLATFORM_LIBS="-lpthread"
  97. PORT_FILE=port/port_posix.cc
  98. ;;
  99. NetBSD)
  100. PLATFORM=OS_NETBSD
  101. COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_NETBSD"
  102. PLATFORM_LIBS="-lpthread -lgcc_s"
  103. PORT_FILE=port/port_posix.cc
  104. ;;
  105. OpenBSD)
  106. PLATFORM=OS_OPENBSD
  107. COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_OPENBSD"
  108. PLATFORM_LDFLAGS="-pthread"
  109. PORT_FILE=port/port_posix.cc
  110. ;;
  111. DragonFly)
  112. PLATFORM=OS_DRAGONFLYBSD
  113. COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_DRAGONFLYBSD"
  114. PLATFORM_LIBS="-lpthread"
  115. PORT_FILE=port/port_posix.cc
  116. ;;
  117. OS_ANDROID_CROSSCOMPILE)
  118. PLATFORM=OS_ANDROID
  119. COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_ANDROID -DLEVELDB_PLATFORM_POSIX=1"
  120. PLATFORM_LDFLAGS="" # All pthread features are in the Android C library
  121. PORT_FILE=port/port_posix.cc
  122. CROSS_COMPILE=true
  123. ;;
  124. HP-UX)
  125. PLATFORM=OS_HPUX
  126. COMMON_FLAGS="$MEMCMP_FLAG -D_REENTRANT -DOS_HPUX"
  127. PLATFORM_LDFLAGS="-pthread"
  128. PORT_FILE=port/port_posix.cc
  129. # man ld: +h internal_name
  130. PLATFORM_SHARED_LDFLAGS="-shared -Wl,+h -Wl,"
  131. ;;
  132. IOS)
  133. PLATFORM=IOS
  134. COMMON_FLAGS="$MEMCMP_FLAG"
  135. [ -z "$INSTALL_PATH" ] && INSTALL_PATH=`pwd`
  136. PORT_FILE=port/port_posix.cc
  137. PLATFORM_SHARED_EXT=
  138. PLATFORM_SHARED_LDFLAGS=
  139. PLATFORM_SHARED_CFLAGS=
  140. PLATFORM_SHARED_VERSIONED=
  141. ;;
  142. *)
  143. echo "Unknown platform!" >&2
  144. exit 1
  145. esac
  146. # We want to make a list of all cc files within util, db, table, and helpers
  147. # except for the test and benchmark files. By default, find will output a list
  148. # of all files matching either rule, so we need to append -print to make the
  149. # prune take effect.
  150. DIRS="$PREFIX/db $PREFIX/util $PREFIX/table"
  151. set -f # temporarily disable globbing so that our patterns aren't expanded
  152. PRUNE_TEST="-name *test*.cc -prune"
  153. PRUNE_BENCH="-name *_bench.cc -prune"
  154. PRUNE_TOOL="-name leveldbutil.cc -prune"
  155. PORTABLE_FILES=`find $DIRS $PRUNE_TEST -o $PRUNE_BENCH -o $PRUNE_TOOL -o -name '*.cc' -print | sort | sed "s,^$PREFIX/,," | tr "\n" " "`
  156. set +f # re-enable globbing
  157. # The sources consist of the portable files, plus the platform-specific port
  158. # file.
  159. echo "SOURCES=$PORTABLE_FILES $PORT_FILE" >> $OUTPUT
  160. echo "MEMENV_SOURCES=helpers/memenv/memenv.cc" >> $OUTPUT
  161. if [ "$CROSS_COMPILE" = "true" ]; then
  162. # Cross-compiling; do not try any compilation tests.
  163. true
  164. else
  165. CXXOUTPUT="${TMPDIR}/leveldb_build_detect_platform-cxx.$$"
  166. # If -std=c++0x works, use <atomic> as fallback for when memory barriers
  167. # are not available.
  168. $CXX $CXXFLAGS -std=c++0x -x c++ - -o $CXXOUTPUT 2>/dev/null <<EOF
  169. #include <atomic>
  170. int main() {}
  171. EOF
  172. if [ "$?" = 0 ]; then
  173. COMMON_FLAGS="$COMMON_FLAGS -DLEVELDB_PLATFORM_POSIX=1 -DLEVELDB_ATOMIC_PRESENT"
  174. PLATFORM_CXXFLAGS="-std=c++0x"
  175. else
  176. COMMON_FLAGS="$COMMON_FLAGS -DLEVELDB_PLATFORM_POSIX=1"
  177. fi
  178. # Test whether CRC32C library is installed
  179. # https://github.com/google/crc32c
  180. $CXX $CXXFLAGS -x c++ - -o $CXXOUTPUT 2>/dev/null <<EOF
  181. #include <crc32c/crc32c.h>
  182. int main() {}
  183. EOF
  184. if [ "$?" = 0 ]; then
  185. COMMON_FLAGS="$COMMON_FLAGS -DHAVE_CRC32C=1"
  186. PLATFORM_LIBS="$PLATFORM_LIBS -lcrc32c"
  187. else
  188. COMMON_FLAGS="$COMMON_FLAGS -DHAVE_CRC32C=0"
  189. fi
  190. # Test whether Snappy library is installed
  191. # https://github.com/google/snappy
  192. $CXX $CXXFLAGS -x c++ - -o $CXXOUTPUT 2>/dev/null <<EOF
  193. #include <snappy.h>
  194. int main() {}
  195. EOF
  196. if [ "$?" = 0 ]; then
  197. COMMON_FLAGS="$COMMON_FLAGS -DHAVE_SNAPPY=1"
  198. PLATFORM_LIBS="$PLATFORM_LIBS -lsnappy"
  199. else
  200. COMMON_FLAGS="$COMMON_FLAGS -DHAVE_SNAPPY=0"
  201. fi
  202. # Test whether tcmalloc is available
  203. $CXX $CXXFLAGS -x c++ - -o $CXXOUTPUT -ltcmalloc 2>/dev/null <<EOF
  204. int main() {}
  205. EOF
  206. if [ "$?" = 0 ]; then
  207. PLATFORM_LIBS="$PLATFORM_LIBS -ltcmalloc"
  208. fi
  209. # Test whether -Wthread-safety is available. See
  210. # https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
  211. # -Werror is necessary because unknown attributes only generate warnings.
  212. $CXX $CXXFLAGS -Wthread-safety -Werror -x c++ - -o $CXXOUTPUT 2>/dev/null <<EOF
  213. struct __attribute__((lockable)) Lock {
  214. void Acquire() __attribute__((exclusive_lock_function()));
  215. void Release() __attribute__((unlock_function()));
  216. };
  217. struct ThreadSafeType {
  218. Lock lock_;
  219. int data_ __attribute__((guarded_by(lock_)));
  220. };
  221. int main() { return 0; }
  222. EOF
  223. if [ "$?" = 0 ]; then
  224. COMMON_FLAGS="$COMMON_FLAGS -Wthread-safety"
  225. fi
  226. rm -f $CXXOUTPUT 2>/dev/null
  227. fi
  228. PLATFORM_CCFLAGS="$PLATFORM_CCFLAGS $COMMON_FLAGS"
  229. PLATFORM_CXXFLAGS="$PLATFORM_CXXFLAGS $COMMON_FLAGS"
  230. echo "CC=$CC" >> $OUTPUT
  231. echo "CXX=$CXX" >> $OUTPUT
  232. echo "PLATFORM=$PLATFORM" >> $OUTPUT
  233. echo "PLATFORM_LDFLAGS=$PLATFORM_LDFLAGS" >> $OUTPUT
  234. echo "PLATFORM_LIBS=$PLATFORM_LIBS" >> $OUTPUT
  235. echo "PLATFORM_CCFLAGS=$PLATFORM_CCFLAGS" >> $OUTPUT
  236. echo "PLATFORM_CXXFLAGS=$PLATFORM_CXXFLAGS" >> $OUTPUT
  237. echo "PLATFORM_SHARED_CFLAGS=$PLATFORM_SHARED_CFLAGS" >> $OUTPUT
  238. echo "PLATFORM_SHARED_EXT=$PLATFORM_SHARED_EXT" >> $OUTPUT
  239. echo "PLATFORM_SHARED_LDFLAGS=$PLATFORM_SHARED_LDFLAGS" >> $OUTPUT
  240. echo "PLATFORM_SHARED_VERSIONED=$PLATFORM_SHARED_VERSIONED" >> $OUTPUT