Parallel Gzip - Pigz

Introduction

Sometimes, we would like to compress one or several files into one zipped file or decompress a zipped file. It is very common to use tools such as gzip, zip, or 7zip to create or decompress .gz, .zip, and .7z files, respectively. However, none of these tools on Linux uses multicore and multithread during compression and decompression. When the number of files are large or the file sizes are large, compression and decompression would take a lot of time using single thread.

Pigz is one of the parallel implementation for gzip and zip. Using pigz could greatly save us the time spent on compression and decompression. In this blog post, I would like to briefly discuss how to use pigz.

Pigz

The pigz usages in the blog post are mainly targeted for Ubuntu systems. However, its usages on other Linux operating systems should be almost the same.

Pigz Usages

A typical command for compressing and decompressing a file is like the following:

However, vanilla pigz is not very friendly to compressing multiple files into one single file and custom output filepath. We would need to rely on tar, the archive tool.

Tar-Pigz Usages

Using pipe |, we could first archive multiple files or directories first to .tar file and compress using pigz to further generate .tar.gz file.

Alternatively, tar has already integrated custom compressor in its interface, which makes the command looks more clear.

Lei Mao

07-08-2020

07-08-2020