shark-tools

A collection of tools for packet/flow processing based on tshark.

Enjoy is a lua module that extracts detailed connection (bidirectional flow) information from network packets.

The repository also provides a Python wrapper at py/enjoy.py that validates tshark, validates the Lua script path, selects batch vs live mode, passes protocol selection to enjoy.lua, and can save a raw capture with tshark -w.

Batch example:

python .\py\enjoy.py --mode batch --input-file .\examples\test.pcapng --protocols dns,tls,http --stdout-file .\examples\out\test.ndjson

Live example with flush interval and raw capture output:

python .\py\enjoy.py --mode live --interface "Ethernet 2" --flush 30 --protocols all --capture-output .\raw\capture.pcapng

Notes:

• --stdout-file stores the NDJSON exported by enjoy.lua.
• --capture-output maps to tshark -w and stores the raw packet capture.

py/enjoy-live.py is a live-only variant of the wrapper intended for unattended, long-running probes. Instead of keeping one tshark process attached to the interface for days, it splits capture and dissection:

1. dumpcap captures the interface into rotating files, one file per --flush seconds,
2. every completed file is processed by tshark with enjoy.lua, and its NDJSON is streamed to stdout (or --stdout-file),
3. the capture file is deleted, or moved to --capture-output when that directory is given.

Because dissection runs in short-lived per-file processes, the packet capture is never stalled by a slow or crashing dissector, and the memory held by connection tracking is released after every window.

python ./py/enjoy-live.py --interface MONITOR --flush 600 --protocols all \
    --work-dir /var/spool/enjoy --stdout-file /var/log/enjoy/enjoy.ndjson --event

Keeping the raw captures instead of deleting them:

python ./py/enjoy-live.py --interface MONITOR --flush 600 \
    --work-dir /var/spool/enjoy --capture-output /var/pcap/enjoy

The parameters are the same as for py/enjoy.py, with these differences:

Notes:

• Everything on stdout is NDJSON. With --event, the wrapper adds its own monitoring events (monitor-started, file-processed, file-failed, backlog, recovery, capture-error, monitor-stopped) next to the flow-export/eof events of enjoy.lua; each processed file contributes one flow-export/eof pair. Without --event, stdout contains flow records only. Progress and errors are always reported on stderr.
• SIGINT/SIGTERM stop the capture, process the remaining files, and exit with 0. If dissection is behind the capture, shutdown takes as long as draining the queue — raise TimeoutStopSec in the systemd unit accordingly.
• Capture files that fail to process are never deleted; they are moved to --capture-output, or to <work-dir>/failed if it is not set.
• Files left in --work-dir by a killed process are processed on the next start, so restarts do not lose data. Each running instance needs its own --work-dir.
• If dumpcap exits on its own (bad interface, missing capture privileges), the wrapper reports the error and exits with a non-zero code, which lets Restart=on-failure do its job.

This script provides a way to rotate and optionally compress Newline Delimited JSON (NDJSON) output received from a pipeline (e.g., from tshark) into manageable file chunks.

It is intended for use in long-running capture sessions where:

• File size needs to stay reasonable,
• Data should be organized over time,
• Storage efficiency is important.

✨ Features

• File rotation: Create a new file every N minutes to control file size.
• On-the-fly compression: Optionally compress each output file immediately as it is written (.ndjson.gz).
• Stream processing: Handles live input efficiently via pipeline (|).
• Safe and atomic: Ensures all files are flushed and closed properly before switching to the next file.
• Flexible: Works with any NDJSON-producing tool, not just tshark.

Capture NDJSON without compression

tshark -q -X lua_script:lua\enjoy\enjoy.lua -X lua_script1:flush=10 -i "Ethernet 2" |
    .\ps\Rotate-Json.ps1 -IntervalMinutes 1 -OutputDirectory .\examples\out\

• Rotates output files every 1 minute.
• Produces plain .ndjson files (no compression).
• Files are named based on the UTC start time, for example: capture_20240501T1210.ndjson

Capture NDJSON with compression enabled

tshark -q -X lua_script:lua\enjoy\enjoy.lua -X lua_script1:flush=10 -i "Ethernet 2" |
    .\ps\Rotate-Json.ps1 -IntervalMinutes 1 -OutputDirectory .\examples\out\ -Compress $true

• Same behavior as above but compresses output on the fly.
• Creates .ndjson.gz compressed files directly, for example: capture_20240501T1210.ndjson.gz
• Compression typically reduces file size by a factor of 10×.

• The Lua script (enjoy.lua) is responsible for formatting tshark’s output as NDJSON.
• Rotation and compression are based on the system UTC clock, ensuring consistent file naming.
• If the output directory does not exist, it will be created automatically.

The scripts provided can be used to capture communication in the local network. To deploy them, follow these steps:

• Set up port mirroring or a TAP at a suitable point in the network (typically the LAN gateway router). Below is an example configuration for a MikroTik router, mirroring all traffic from ether2 to ether5. This assumes the LAN is connected to ether2.

/interface ethernet switch set switch1 mirror-source=ether2 mirror-target=ether5

1. Prepare the monitoring interface

   Choose a Linux host for monitoring the LAN traffic. Use a dedicated interface named MONITOR configured for passive monitoring (no IP address, no outgoing traffic).

   Create the Netplan configuration file /etc/netplan/99-capture.yaml with the following content:

   network:
     version: 2
     ethernets:
       MONITOR:
         dhcp4: false
         dhcp6: false
         link-local: []
         accept-ra: false
         ipv6-privacy: false
         optional: true
         addresses: []
         wakeonlan: false

2. Create the capture script

   Create /home/USERNAME/shark/capture-tap.ps1 with the following content:

   $interface=MONITOR
   $scriptPath = $PSScriptRoot
   tshark -q -X lua_script:$scriptPath/shark-tools/lua/enjoy/enjoy.lua -X lua_script1:flush=60 -i "$interface" -b duration:600 -w $scriptPath/raw/raw.pcapng |
   & $scriptPath/shark-tools/ps/Rotate-Json.ps1 -IntervalMinutes 10 -OutputDirectory $scriptPath/log -Structured -Compress

3. Create capture termination script Create /home/USERNAME/shark/kill-tap.sh with the following content:

   #!/bin/bash
   # Try to kill the process using pkill
   /usr/bin/pkill tshark
   rm /tmp/*.pcapng
   
   # Capture the exit code of the pkill command
   exit_code=$?
   
   # Check the exit code and handle accordingly
   if [ $exit_code -eq 0 ]; then
       echo "Process terminated successfully."
       exit 0  # Success
   elif [ $exit_code -eq 1 ]; then
       echo "No matching processes found."
       exit 0  # No process to kill, but still consider as success
   else
       echo "An error occurred with pkill."
       exit 1  # Error
   fi

4. Create the systemd service

   Create /etc/systemd/system/capture-tap.service:

   [Unit]
   Description=Capture TAP PowerShell Script
   After=network.target
   
   [Service]
   ExecStartPre=/home/USERNAME/shark/kill-tap.sh
   ExecStart=/snap/bin/pwsh -File /home/USERNAME/shark/capture-tap.ps1
   Restart=always
   RestartSec=10
   User=USERNAME
   WorkingDirectory=/home/USERNAME/shark
   StandardOutput=append:/var/log/capture-tap.out
   StandardError=append:/var/log/capture-tap.err
   
   [Install]
   WantedBy=multi-user.target

   Notes:

   • Replace USERNAME with your actual Linux username.
   • Confirm the correct path to PowerShell (pwsh) using which pwsh (e.g., /usr/bin/pwsh or /snap/bin/pwsh).
   • Logs will be written to /var/log/capture-tap.out and /var/log/capture-tap.err.

5. Enable and start the service

   sudo systemctl daemon-reload
   sudo systemctl enable capture-tap.service
   sudo systemctl start capture-tap.service

6. Verify service status

   sudo systemctl status capture-tap.service