I’ve poked a bit with git and decided to have a bit of fun with git trees and git object compression.

Big Git: A 120KB git that consumes 32GB of diskspace upon clone or checkout.
99.9996% compression factor. A wonderful zip bomb 🤯.

And you can trivially make a similar sized git consume more diskspace than ever manufactured.

A Matryoshka doll of git references and object compression, expanding like crazy.
Or to quote Nine Inch Nails: A copy of a copy of a…

I’m not sure how obvious or surprising this is to your average folks. This is super obvious to most Git experts, but I talked to tech people who found it surprising and/or funny.

If you found this interesting, you may also be interested in A Self Referential Git!

Table of Contents

The Gist of It

Git compresses objects with zlib. As long as your objects are highly compressible, zlib will do wonders.

  • Step 0 is to create a file that is highly compressible.

  • Step 1 is to create a tree (directory) that is highly compressible, by containing several instances of the file, with simple file names.

  • Step i is to create a tree (directory) that is highly compressible, by containing several instances of the directory i-1 with simple file names.

  • Final step N now has N+1 layers of zlib compression, and an exponential amount of files…

Git Object view:

 ddbc38 (commit)
  |
  +-> fa1334 (tree)
      |
      +-> 5e49c (tree) dir0
       -> 5e49c (tree) dir1 [and so on...]
          |
          +-> 7614aa (tree) dir0
           -> 7614aa (tree) dir1 [and so on...]
              |
              +-> d52dd (tree) dir0
               -> d52dd (tree) dir1 [and so on...]
                  |
                  +-> 5c08a (tree) dir0
                   -> 5c08a (tree) dir1 [and so on...]
                      |
                      +-> ea8e4 (blob) file-0.bin
                       -> ea8e4 (blob) file-1.bin [and so on...]

Exploring Git Objects with git cat-file

You can use git cat-file to investigate the git:

git cat-file -p ddbc38b347441733e08053d6e17618910c9354db

tree fa13348bf59dc645af77b34d818d37d2c9d33812
parent b7c4c7b7be63cef7e944f8d0b942ec77d0d35963
author Generator <tool@synthetic.local> 1779202073 +0000
committer Generator <tool@synthetic.local> 1779202073 +0000

Synthetic structure generation step

git cat-file -p fa13348bf59dc645af77b34d818d37d2c9d33812

040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-0
040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-1
040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-2
040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-3
040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-4
040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-5
040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-6
040000 tree 5e49c320ca4abb26c91b4804352bffadb0886597    dir-7

git cat-file -p 5e49c320ca4abb26c91b4804352bffadb0886597

040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-0
040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-1
040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-2
040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-3
040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-4
040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-5
040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-6
040000 tree 7614aacc2886b51bd0042fc5dc60ba171fc2ef31    dir-7

Git Tree Compression

A git tree is basically just a list of references, either tree or object:

git cat-file -p 5c08a7a43a6cfd7975a8f0c839d0dd3c4eb9e0bf
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-0.bin
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-1.bin
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-2.bin
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-3.bin
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-4.bin
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-5.bin
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-6.bin
100644 blob ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b    file-7.bin

Or in hex for the the more hacker-inclined readers:

git cat-file tree 5c08a7a43a6cfd7975a8f0c839d0dd3c4eb9e0bf | xxd
00000000: 3130 3036 3434 2066 696c 652d 302e 6269  100644 file-0.bi
00000010: 6e00 ea8e 482b 990b 87c0 f69d 29fd 1dd6  n...H+......)...
00000020: a41d 0f1a 514b 3130 3036 3434 2066 696c  ....QK100644 fil
00000030: 652d 312e 6269 6e00 ea8e 482b 990b 87c0  e-1.bin...H+....
00000040: f69d 29fd 1dd6 a41d 0f1a 514b 3130 3036  ..).......QK1006
00000050: 3434 2066 696c 652d 322e 6269 6e00 ea8e  44 file-2.bin...
00000060: 482b 990b 87c0 f69d 29fd 1dd6 a41d 0f1a  H+......).......
00000070: 514b 3130 3036 3434 2066 696c 652d 332e  QK100644 file-3.
00000080: 6269 6e00 ea8e 482b 990b 87c0 f69d 29fd  bin...H+......).
00000090: 1dd6 a41d 0f1a 514b 3130 3036 3434 2066  ......QK100644 f
000000a0: 696c 652d 342e 6269 6e00 ea8e 482b 990b  ile-4.bin...H+..
000000b0: 87c0 f69d 29fd 1dd6 a41d 0f1a 514b 3130  ....).......QK10
000000c0: 3036 3434 2066 696c 652d 352e 6269 6e00  0644 file-5.bin.
000000d0: ea8e 482b 990b 87c0 f69d 29fd 1dd6 a41d  ..H+......).....
000000e0: 0f1a 514b 3130 3036 3434 2066 696c 652d  ..QK100644 file-
000000f0: 362e 6269 6e00 ea8e 482b 990b 87c0 f69d  6.bin...H+......
00000100: 29fd 1dd6 a41d 0f1a 514b 3130 3036 3434  ).......QK100644
00000110: 2066 696c 652d 372e 6269 6e00 ea8e 482b   file-7.bin...H+
00000120: 990b 87c0 f69d 29fd 1dd6 a41d 0f1a 514b  ......).......QK

For a small tree, we “only” get a 73% (304 to 81 byte) size reduction;

git cat-file tree 5c08a7a43a6cfd7975a8f0c839d0dd3c4eb9e0bf | wc -c
304

wc -c .git/objects/5c/08a7a43a6cfd7975a8f0c839d0dd3c4eb9e0bf
81 .git/objects/5c/08a7a43a6cfd7975a8f0c839d0dd3c4eb9e0bf

But keep in mind that if we spam a few more entries to the tree the tree will just grow bigger and bigger while barely increasing in size.

Git File Compression

For my example file we get about 99.6% file compression:

git cat-file -p ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b | wc -c
1048576

wc -c .git/objects/ea/8e482b990b87c0f69d29fd1dd6a41d0f1a514b
4419

The file is just a megabyte of incremental bytes;

git cat-file -p ea8e482b990b87c0f69d29fd1dd6a41d0f1a514b | xxd | head

00000000: 0001 0203 0405 0607 0809 0a0b 0c0d 0e0f  ................
00000010: 1011 1213 1415 1617 1819 1a1b 1c1d 1e1f  ................
00000020: 2021 2223 2425 2627 2829 2a2b 2c2d 2e2f   !"#$%&'()*+,-./
00000030: 3031 3233 3435 3637 3839 3a3b 3c3d 3e3f  0123456789:;<=>?
00000040: 4041 4243 4445 4647 4849 4a4b 4c4d 4e4f  @ABCDEFGHIJKLMNO
00000050: 5051 5253 5455 5657 5859 5a5b 5c5d 5e5f  PQRSTUVWXYZ[\]^_
00000060: 6061 6263 6465 6667 6869 6a6b 6c6d 6e6f  `abcdefghijklmno
00000070: 7071 7273 7475 7677 7879 7a7b 7c7d 7e7f  pqrstuvwxyz{|}~.
00000080: 8081 8283 8485 8687 8889 8a8b 8c8d 8e8f  ................
00000090: 9091 9293 9495 9697 9899 9a9b 9c9d 9e9f  ................

Sparse files

In theory you could just have zeros in the files.

However several file systems support Sparse files. A simple files only containing zeros won’t actually allocate any space on storage.

So if the goal is for git clone, git checkout, etc. to actually consume a lot of space, you need files to contain some pattern not supported by the sparse file handling. Not an expert on different file systems, but I think writing ones 0xff or incremental 0x01 0x02 0x03.. will allocate space on most file systems.