作者: 韩晨旭 10225101440 李畅 10225102463

75 righe
2.9 KiB

  1. leveldb Log format
  2. ==================
  3. The log file contents are a sequence of 32KB blocks. The only exception is that
  4. the tail of the file may contain a partial block.
  5. Each block consists of a sequence of records:
  6. block := record* trailer?
  7. record :=
  8. checksum: uint32 // crc32c of type and data[] ; little-endian
  9. length: uint16 // little-endian
  10. type: uint8 // One of FULL, FIRST, MIDDLE, LAST
  11. data: uint8[length]
  12. A record never starts within the last six bytes of a block (since it won't fit).
  13. Any leftover bytes here form the trailer, which must consist entirely of zero
  14. bytes and must be skipped by readers.
  15. Aside: if exactly seven bytes are left in the current block, and a new non-zero
  16. length record is added, the writer must emit a FIRST record (which contains zero
  17. bytes of user data) to fill up the trailing seven bytes of the block and then
  18. emit all of the user data in subsequent blocks.
  19. More types may be added in the future. Some Readers may skip record types they
  20. do not understand, others may report that some data was skipped.
  21. FULL == 1
  22. FIRST == 2
  23. MIDDLE == 3
  24. LAST == 4
  25. The FULL record contains the contents of an entire user record.
  26. FIRST, MIDDLE, LAST are types used for user records that have been split into
  27. multiple fragments (typically because of block boundaries). FIRST is the type
  28. of the first fragment of a user record, LAST is the type of the last fragment of
  29. a user record, and MIDDLE is the type of all interior fragments of a user
  30. record.
  31. Example: consider a sequence of user records:
  32. A: length 1000
  33. B: length 97270
  34. C: length 8000
  35. **A** will be stored as a FULL record in the first block.
  36. **B** will be split into three fragments: first fragment occupies the rest of
  37. the first block, second fragment occupies the entirety of the second block, and
  38. the third fragment occupies a prefix of the third block. This will leave six
  39. bytes free in the third block, which will be left empty as the trailer.
  40. **C** will be stored as a FULL record in the fourth block.
  41. ----
  42. ## Some benefits over the recordio format:
  43. 1. We do not need any heuristics for resyncing - just go to next block boundary
  44. and scan. If there is a corruption, skip to the next block. As a
  45. side-benefit, we do not get confused when part of the contents of one log
  46. file are embedded as a record inside another log file.
  47. 2. Splitting at approximate boundaries (e.g., for mapreduce) is simple: find the
  48. next block boundary and skip records until we hit a FULL or FIRST record.
  49. 3. We do not need extra buffering for large records.
  50. ## Some downsides compared to recordio format:
  51. 1. No packing of tiny records. This could be fixed by adding a new record type,
  52. so it is a shortcoming of the current implementation, not necessarily the
  53. format.
  54. 2. No compression. Again, this could be fixed by adding new record types.