File Formats

gzip

gzip - pros and cons

• Pros

  • Fast compression and decompression speeds, ideal when spped matters
  • Widely supported

• Cons

  • Lower compression ratio than bzip2
  • Not splittable

bzip2

bzip2 - pros and cons

• Pros

  • Higher compression ratio than gzip, particularly with large files, ideal when space matters
  • Splittable

• Cons

  • Slower than gzip, especially on decompression.
  • Consumes more CPU and memory

lz4

Wikipedia - LZ4 (compression algorithm) (opens in a new tab)

lz4 - pros and cons

• Pros

  • Very fast compression and decompression speeds, compression speed is similar to lzo, and decompression speed is significantly faster than lzo
  • Splittable

• Cons

  • Less compression than gzip and bzip2

lzo

Wikipedia - Lempel–Ziv–Oberhumer (LZO) (opens in a new tab)

lzo - pros and cons

• Pros

  • Higher compression speed compared to DEFLATE compression
  • Very fast decompression
  • Allows the user to adjust the balance between compression ratio and compression speed, without affecting the speed of decompression
  • Produces files slightly larger than gzip while only requiring a tenth of the CPU use and only slightly higher memory utilization.
  • Splittable

• Cons

  • Lower compression ratio than gzip and bzip2

Snappy

Wikipedia - Snappy (compression) (opens in a new tab)

Snappy - pros and cons

• Pros

  • Very fast compression and decompression speeds
  • Widely used in Big Data
  • Default compression format for Parquet files

• Cons

  • Compression ratio is 20–100% lower than gzip
  • Not splittable

xz

Wikipedia - XZ Utils (opens in a new tab)

xz - pros and cons

• Pros

  • Higher compression rates than alternatives like gzip and bzip2, particularly for very large files.
  • Higher decompression speed than bzip2
  • Splittable

• Cons

  • Slowest
  • Most resource-intensive
  • Lower decompression speed than gzip
  • Compression can be much slower than gzip, and is slower than bzip2 for high levels of compression

Use cases

• gzip

  Use when speed is crucial, and moderate compression is acceptable. Ideal for log files and scripts.

• bzip2

  Suited for compressing large text files or when a balance between speed and compression is needed.

• xz

  Best for archiving large datasets or software distributions where compression ratio matters the most.