Release tarballs have gotten large #5698

The Zeek release tarball has grown significantly since the Zeek 3.0 release, with 8.2.0 it's now roughly 100MB, and unpacked more than half a gigabyte.

These tarballs are used by packagers to build Zeek, so reducing them would lower resource usage (network, storage, time to decompress, etc etc). Any space savings can be multiplied by number of downloads. Reducing it by ~50% seems achievable.

$ curl -s https://download.zeek.org/ | grep 'zeek-[0-9]\..*\.0.tar.gz<' | grep -v minimal | sed -E 's/<[^>]+>/ /g'
   zeek-3.0.0.tar.gz    29.3 M  
   zeek-3.1.0.tar.gz    32.1 M  
   zeek-3.2.0.tar.gz    27.4 M  
   zeek-4.0.0.tar.gz    29.4 M  
   zeek-4.1.0.tar.gz    32.4 M  
   zeek-4.2.0.tar.gz    33.0 M  
   zeek-5.0.0.tar.gz    42.7 M  
   zeek-5.1.0.tar.gz    43.5 M  
   zeek-5.2.0.tar.gz    59.3 M  
   zeek-6.0.0.tar.gz    60.1 M  
   zeek-6.1.0.tar.gz    61.6 M  
   zeek-6.2.0.tar.gz    67.8 M  
   zeek-7.0.0.tar.gz    95.7 M  
   zeek-7.1.0.tar.gz    95.9 M  
   zeek-7.2.0.tar.gz    97.2 M  
   zeek-8.0.0.tar.gz    99.6 M  
   zeek-8.1.0.tar.gz    93.3 M  
   zeek-8.2.0.tar.gz    103.2 M  

Using du on the extracted zeek-8.2.0 tarball and listing the biggest 30 directories:

$ du -h --max-depth=4 | sort -h | tail -30
9,4M    ./auxil/broker/3rdparty
10M     ./auxil/rapidjson
11M     ./auxil/broker/caf
11M     ./src/3rdparty
11M     ./testing/btest/Traces/modbus
12M     ./testing/btest/Baseline.zam
13M     ./auxil/spicy/tests
13M     ./auxil/vcpkg/versions
13M     ./doc
14M     ./auxil/libunistd
14M     ./auxil/spicy/3rdparty/libunistd
20M     ./testing/btest/Baseline/scripts.base.protocols.modbus.events
33M     ./src
41M     ./testing/btest/Traces
50M     ./auxil/prometheus-cpp/3rdparty/civetweb
52M     ./auxil/vcpkg/ports
55M     ./auxil/prometheus-cpp/3rdparty
56M     ./auxil/broker/auxil
56M     ./auxil/broker/auxil/prometheus-cpp
56M     ./auxil/prometheus-cpp
65M     ./testing/btest/Baseline
70M     ./auxil/vcpkg
72M     ./auxil/spicy/3rdparty/vcpkg
84M     ./auxil/broker
100M    ./auxil/spicy/3rdparty
121M    ./auxil/spicy
134M    ./testing/btest
136M    ./testing
375M    ./auxil
565M    .

The biggest snags:

• Including vcpkg (72M) twice: auxil/vcpgk and auxil/spicy/3rdparty/vcpk (IIUC this is only needed to compile on Windows so I don't think it should be included)
• Including prometheus-cpp (56M) twice: Via ./auxil/broker and ./auxil/prometheus-cpp and specifically it's own bundled civitweb dependency which then bundles duktape (JS engine) and LUA (multiple versions). We should not bundle LUA or Duktape in a Zeek release tarball.
• Including libunistd twice via auxil/libunistd and auxil/spicy/libunistd. If this is also only used for Windows, would probably skip inclusion of it and provide instructions where to place it?
• Some large baselines, including a 20mb modbus baseline.

  100K    ./scripts.base.protocols.dnp3.dnp3_link_only
  104K    ./scripts.base.protocols.dnp3.dnp3_udp_read
  168K    ./scripts.policy.protocols.conn.mac-logging
  388K    ./scripts.base.protocols.ftp.cwd-navigation
  1,3M    ./scripts.base.frameworks.file-analysis.irc
  1,4M    ./plugins.hooks
  2,0M    ./scripts.base.protocols.modbus.coil_parsing_big
  4,1M    ./scripts.base.protocols.modbus.policy
  4,5M    ./scripts.policy.misc.dump-events
  5,6M    ./scripts.policy.misc.dump-events-json
  20M     ./scripts.base.protocols.modbus.events
  

• Including sqlite3 a few times (note libunistd and prometheus-cpp again)

  $ ls -lha $(find . |grep -E '/sqlite[^/]*\.c$')
  -rw-rw-r-- 1 awelzel awelzel 8,8M Mai 12 20:21 ./auxil/broker/3rdparty/sqlite3.c
  -rw-rw-r-- 1 awelzel awelzel 7,8M Mai 12 20:21 ./auxil/broker/auxil/prometheus-cpp/3rdparty/civetweb/src/third_party/sqlite3.c
  -rw-rw-r-- 1 awelzel awelzel 7,4M Mai 12 20:21 ./auxil/libunistd/sqlite/source/sqlite3.c
  -rw-rw-r-- 1 awelzel awelzel 7,8M Mai 12 20:21 ./auxil/prometheus-cpp/3rdparty/civetweb/src/third_party/sqlite3.c
  -rw-rw-r-- 1 awelzel awelzel 7,4M Mai 12 20:21 ./auxil/spicy/3rdparty/libunistd/sqlite/source/sqlite3.c
  -rw-rw-r-- 1 awelzel awelzel 9,0M Mai 12 20:21 ./src/3rdparty/sqlite3.c
  

• Bundling 10M of rapidjson which seems like it'd be available on most distros as package these days (librapidjson-dev)

Biggest bang for the bug:

• Skip shipping vcpkg - only needed for Windows. Gemini produced cmake -B build -S . -DCMAKE_TOOLCHAIN_FILE=/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake, so we'd ask users to clone vcpkg themselves.

• Trim the >1MB baselines (definitely the 20MB modbus events one) to something <100KB, preferably <10KB.