DEV Community: William Baptist

Finding Repeated CSV Identifier Values Without Changing Them in Python

William Baptist — Sun, 11 Oct 2026 07:33:56 +0000

Two records use the same product code, but that does not make the records duplicates. This read-only checker groups repeated values in one CSV column and reports their positions. It keeps leading zeros and spaces, reports empty cells separately and never deletes a row.

1. Make a practice export

CSV means comma-separated values. An identifier names something rather than an amount to calculate. Save this as MakeExamples.py in a practice folder:

from pathlib import Path


Path("Codes.csv").write_bytes(
    b'Code,Note\n'
    b'00123,first record\n'
    b'00123,second record\n'
    b'123,different text\n'
    b' 00123,leading space\n'
    b',empty code\n'
)

Run:

python3 MakeExamples.py

Use your installation's Python 3 command if it is not named python3. This creates or replaces Codes.csv, so use a folder without a file you need to keep.

The first two data records share 00123 but have different notes. The third uses 123, the fourth starts its code with a space, and the last has an empty code. The saved bytes preserve those differences without a spreadsheet converting the codes to numbers.

2. Add the reviewer

Save this as FindRepeatedCodes.py:

import csv as Csv
import io as Io
import json as Json
import sys as Sys
from pathlib import Path


def FindRepeatedCodes(FilePath, Column):
    with Path(FilePath).open("rb") as Input:
        Data = Input.read(1048577)
    if len(Data) > 1048576:
        raise ValueError("input exceeds 1 MiB teaching limit")
    Csv.field_size_limit(65536)
    Reader = Csv.reader(Io.StringIO(Data.decode("utf-8-sig"), newline=""), strict=True)
    try:
        Header = next(Reader)
    except StopIteration:
        raise ValueError("empty input") from None
    if not Header or any(not Name.strip() for Name in Header):
        raise ValueError("blank header name")
    if len(Header) != len(set(Header)):
        raise ValueError("repeated header name")
    if Column not in Header:
        raise ValueError("selected column not found")
    Index = Header.index(Column)
    Groups = {}
    Empty = []
    Records = 0
    for Records, Fields in enumerate(Reader, start=1):
        if Records > 1000:
            raise ValueError("more than 1000 data records")
        if len(Fields) != len(Header):
            raise ValueError(f"record {Records} has wrong field count")
        Position = {"Record": Records, "EndingLine": Reader.line_num}
        Value = Fields[Index]
        if Value == "":
            Empty.append(Position)
        else:
            Groups.setdefault(Value, []).append(Position)
    Repeated = [Positions for Positions in Groups.values() if len(Positions) > 1]
    return {"Records": Records, "Column": Index + 1,
            "RepeatedGroups": Repeated, "EmptyCells": Empty}


def Main():
    if len(Sys.argv) != 3:
        print("Usage: python3 FindRepeatedCodes.py input.csv COLUMN", file=Sys.stderr)
        return 2
    try:
        Report = FindRepeatedCodes(Sys.argv[1], Sys.argv[2])
    except (OSError, ValueError, Csv.Error) as Problem:
        print(f"Review stopped: {Problem}", file=Sys.stderr)
        return 2
    print(Json.dumps(Report, indent=2))
    return 1 if Report["RepeatedGroups"] or Report["EmptyCells"] else 0


if __name__ == "__main__":
    Sys.exit(Main())

The program uses Python's default comma-separated CSV dialect and accepts UTF-8 with or without an initial signature. Quotes can keep a comma or line break inside a field. strict=True stops on parser errors it detects, but it does not recognise every other CSV dialect or validate all possible quoting rules.

The header must contain nonblank, unique names. Your selected column must match exactly: Code and code are different. Every data record must have the same number of fields as the header. A wrong field count or parser error stops the whole review, with no partial JSON report.

The teaching limits are 1 MiB, roughly one million bytes, 1,000 data records and 65,536 characters per parsed field. The file read includes one extra byte to detect oversized input before decoding. Use a saved file that will not change during the read.

Groups is a dictionary that maps each original parsed value to its positions. Nothing is trimmed, lowercased or converted to a number. An exactly empty string goes into EmptyCells rather than a repeated group. A spaces-only value is not empty under this rule and can form a repeated group.

3. Read the repeated group

Run:

python3 FindRepeatedCodes.py Codes.csv Code

The output is:

{
  "Records": 5,
  "Column": 1,
  "RepeatedGroups": [
    [
      {
        "Record": 1,
        "EndingLine": 2
      },
      {
        "Record": 2,
        "EndingLine": 3
      }
    ]
  ],
  "EmptyCells": [
    {
      "Record": 5,
      "EndingLine": 6
    }
  ]
}

Records 1 and 2 share the same parsed code. They appear together in one group, even though their notes differ. Record 3's 123 and record 4's space-prefixed code are different strings from 00123, so neither joins that group. Record 5 appears only in EmptyCells.

The output uses JSON, a text format for named values and lists. It omits codes, notes and header names. Positions and counts can still reveal information, so keep the report private when it describes work or personal data.

Record starts at the first data record, excluding the header. Column starts at one. EndingLine is the last physical line consumed for the record; a quoted multiline field can make it larger than the record number plus one. Groups follow the first appearance of each value, and positions within each group follow file order.

The exit code, a small result number another script can check, is 1 when repeated groups or empty cells need review, 0 when neither is found and 2 for a command, file, decoding, parsing or limit error. A header-only file returns 0 with zero records; an empty file stops. No finding is not proof that the export is complete or suitable for import.

4. Choose the column deliberately

Run the same file with a different column:

python3 FindRepeatedCodes.py Codes.csv Note

The output is:

{
  "Records": 5,
  "Column": 2,
  "RepeatedGroups": [],
  "EmptyCells": []
}

The exit code is 0: the five notes are nonempty and different. The program checks the column you name, not which one ought to identify a record. A clean report for Note cannot stand in for checking Code.

Now try lowercase code in the command. It stops with selected column not found and exit code 2, rather than guessing a nearby header. A header containing spaces needs quoting in your command window.

5. Compare quoted values and physical lines

Try the rules together in a second file. Save this as MakeQuoted.py in the same practice folder:

from pathlib import Path


Name = Path("Quoted.csv")
with Name.open("xb") as Output:
    Output.write(
        b'Code,Note\n'
        b'00123,plain field\n'
        b'"00123","two physical\nlines"\n'
        b'" ",one space\n'
        b' ,same space without quotes\n'
        b',empty field\n'
        b'"",quoted empty field\n'
    )

Run:

python3 MakeQuoted.py
python3 FindRepeatedCodes.py Quoted.csv Code

The setup opens Quoted.csv in exclusive binary mode, so it stops if that name already exists rather than replacing the file. Use a fresh practice folder if you want to repeat it. The checker reports:

{
  "Records": 6,
  "Column": 1,
  "RepeatedGroups": [
    [
      {
        "Record": 1,
        "EndingLine": 2
      },
      {
        "Record": 2,
        "EndingLine": 4
      }
    ],
    [
      {
        "Record": 3,
        "EndingLine": 5
      },
      {
        "Record": 4,
        "EndingLine": 6
      }
    ]
  ],
  "EmptyCells": [
    {
      "Record": 5,
      "EndingLine": 7
    },
    {
      "Record": 6,
      "EndingLine": 8
    }
  ]
}

The exit code is 1. Records 1 and 2 share 00123: quote marks belong to the CSV syntax, not the parsed code. Record 2's note spans two physical lines, so that record ends on line 4. The next record ends on line 5, not line 4.

Records 3 and 4 each contain exactly one space in the selected column. Quoting that space does not change it. They form a second repeated group because the checker does not trim values. Records 5 and 6 both parse to an empty string, with and without quotes, so they appear separately in EmptyCells, not as a repeated group.

Only the selected Code fields decide the groups. The notes explain the fixture but do not take part in the comparison. Nothing in either practice file is changed by the checker.

6. Do not turn a repeat into a deletion rule

The same identifier can occur on several legitimate rows: a product can appear on multiple order lines, or an account can have multiple entries. This check compares one column, not all fields, event identifiers or business meaning. Two empty cells are reported as separate empty positions, not evidence of the same entity.

Decide from the receiving system's rules whether the selected identifier must be unique. If matching should ignore case or surrounding spaces, review that rule separately before comparing transformed values. Do not silently turn 00123 into 123 or remove spaces just to make groups line up.

The parsed value is different from its CSV spelling. Quotes around a field are syntax, so a quoted "00123" and an unquoted 00123 join the same group. This is not a byte-for-byte row comparison or a general duplicate-record detector.

Keep the original export and inspect the reported records before changing anything.

References

Finding Repeated ZIP Entry Names in Python

William Baptist — Sun, 11 Oct 2026 07:33:14 +0000

A ZIP archive can contain more than one entry with the same name. A name alone then cannot identify which entry you meant. This read-only checker reports repeated entry names and their positions without extracting or changing anything.

1. Make practice archives

A ZIP archive stores entries and a directory describing them. Two entries can use the same name even when their contents differ. Save this as MakeExamples.py in a practice folder:

import warnings as Warnings
import zipfile as ZipFile
from pathlib import Path


Names = [Path("Repeated.zip"), Path("Distinct.zip")]
if any(Name.exists() or Name.is_symlink() for Name in Names):
    raise SystemExit("Use a fresh practice folder without Repeated.zip or Distinct.zip.")
with Warnings.catch_warnings():
    Warnings.simplefilter("ignore", UserWarning)
    with ZipFile.ZipFile(Names[0], "x") as Archive:
        Archive.writestr("Note.txt", b"first")
        Archive.writestr("Note.txt", b"second")
with ZipFile.ZipFile(Names[1], "x") as Archive:
    Archive.writestr("Note.txt", b"first")
    Archive.writestr("note.txt", b"second")

Run:

python3 MakeExamples.py

The setup stops if either archive name already exists, including as a symbolic link. Mode "x" creates a new archive rather than replacing an existing file. The warning filter is only around the intentional repeated-name example: Python warns when that second Note.txt is written. Use these small practice archives rather than real files you need.

2. Save the name checker

Save this as CheckZipNames.py beside the archives:

import json as Json
import sys as Sys
import zipfile as ZipFile
from pathlib import Path


def CheckZipNames(FileName):
    with Path(FileName).open("rb") as Input:
        Input.seek(0, 2)
        if Input.tell() > 1048576:
            raise ValueError("archive exceeds 1 MiB teaching limit")
        Input.seek(0)
        with ZipFile.ZipFile(Input) as Archive:
            Entries = Archive.infolist()
            if len(Entries) > 1000:
                raise ValueError("more than 1000 entries")
            Groups = {}
            for Number, Entry in enumerate(Entries, start=1):
                Groups.setdefault(Entry.filename, []).append(Number)
    Repeated = [{"Name": Name, "Entries": Numbers}
                for Name, Numbers in Groups.items() if len(Numbers) > 1]
    return {"EntryCount": len(Entries), "RepeatedNames": Repeated}


def Main():
    if len(Sys.argv) != 2:
        print("Usage: python3 CheckZipNames.py input.zip", file=Sys.stderr)
        return 2
    try:
        Report = CheckZipNames(Sys.argv[1])
    except (OSError, ValueError, ZipFile.BadZipFile) as Problem:
        print(f"Name check stopped: {Problem}", file=Sys.stderr)
        return 2
    print(Json.dumps(Report, ensure_ascii=True, indent=2))
    return 1 if Report["RepeatedNames"] else 0


if __name__ == "__main__":
    Sys.exit(Main())

The archive limit is 1 MiB (1,048,576 bytes), measured on the opened file before the ZIP directory is parsed. This is the archive's stored size, not its total uncompressed size. Use a saved regular file that will not change while it is read.

infolist() returns an entry object for each directory entry, in archive order. The program groups each exact filename and records one-based entry positions. A dictionary keyed only by name would discard this distinction if it replaced earlier entries.

The 1,000-entry limit is checked after the library has loaded the ZIP directory. It limits the accepted report, not the work needed to parse that directory. These teaching limits are not a security sandbox or a guarantee about handling hostile archives.

3. Find the repeated name

Run:

python3 CheckZipNames.py Repeated.zip

The output is:

{
  "EntryCount": 2,
  "RepeatedNames": [
    {
      "Name": "Note.txt",
      "Entries": [
        1,
        2
      ]
    }
  ]
}

The exit code is 1. Both entries have the exact name Note.txt; their positions distinguish them even though the name does not. The checker does not choose one or compare their contents. The same report would appear if those contents matched.

JSON is a text format for named values and lists. The exit code is 0 when no exact repeated names are found, 1 when they are found and 2 when usage, reading, parsing or a teaching limit stops the check. No partial report is printed after a caught error.

4. Keep exact matching separate from extraction rules

Run:

python3 CheckZipNames.py Distinct.zip

The output is:

{
  "EntryCount": 2,
  "RepeatedNames": []
}

The exit code is 0. Note.txt and note.txt are different strings. That does not prove they would stay separate on a case-insensitive destination filesystem. This checker does not fold case, remove spaces or resolve path spellings. Directory entries are included too, so two exact directory names would be reported as a repeat.

The checker never reads entry contents, checks their CRC values, tests passwords or extracts files. A valid ZIP directory with no repeated names is not a claim that every entry is readable, the archive is complete or its paths are safe to extract. Do not extract an unfamiliar archive merely because this check returns 0.

5. Keep the report separate from the input

To save a report, use a new filename:

python3 CheckZipNames.py Repeated.zip > RepeatedNames.json

This creates or replaces RepeatedNames.json; keep that output name free. Never redirect output over the input archive, because the shell can empty it before Python opens it. Entry names can reveal personal or work information, so keep reports private when they describe real archives.

Reference

Python ZIP entry lists, creation modes and repeated names.

Checking Whether Two IPv4 Subnets Overlap in Python

William Baptist — Fri, 09 Oct 2026 19:35:23 +0000

Two subnet labels may describe some of the same addresses. Before treating them as separate ranges, compare them. This small command-line program reports overlap, equality and containment without contacting either network.

This runs on Python 3.10 or later from any command window. Only the standard library is used.

1. Choose practice ranges

An IPv4 address has four dot-separated numbers, such as 192.0.2.10. A subnet describes a range of these addresses. CIDR notation puts a slash and prefix length after the network address: 192.0.2.0/24.

A prefix fixes the first part of the address. A larger prefix length describes a smaller range. 192.0.2.128/25 is the upper half of 192.0.2.0/24. These example addresses come from a documentation range; the checker never sends packets.

Use these two pairs:

192.0.2.0/24     192.0.2.128/25
192.0.2.0/25    192.0.2.128/25

The first pair shares addresses. The second pair is adjacent without sharing any.

2. Save the checker

Save this as CheckOverlap.py in a practice folder:

import ipaddress as IpAddress
import json as Json
import sys as Sys


def CheckOverlap(FirstText, SecondText):
    First = IpAddress.IPv4Network(FirstText, strict=True)
    Second = IpAddress.IPv4Network(SecondText, strict=True)
    return {"First": str(First), "Second": str(Second),
            "Overlap": First.overlaps(Second), "Equal": First == Second,
            "FirstInsideSecond": First.subnet_of(Second),
            "SecondInsideFirst": Second.subnet_of(First)}


def Main():
    if len(Sys.argv) != 3:
        print("Usage: python3 CheckOverlap.py FIRST_CIDR SECOND_CIDR", file=Sys.stderr)
        return 2
    try:
        Report = CheckOverlap(Sys.argv[1], Sys.argv[2])
    except ValueError as Problem:
        print(f"Check stopped: {Problem}", file=Sys.stderr)
        return 2
    print(Json.dumps(Report, indent=2))
    return 1 if Report["Overlap"] else 0


if __name__ == "__main__":
    Sys.exit(Main())

IPv4Network parses IPv4 network descriptions. The examples use slash-prefix notation. The library also accepts some other forms, including IPv4 masks; this program does not implement its own format validator. IPv6 is outside this check.

Host bits are the part of an address that identifies an individual address within a subnet. strict=True rejects an address with host bits set for the chosen prefix. For example, 192.0.2.1/24 stops instead of silently turning into 192.0.2.0/24. Use the intended network address; do not remove the strict check merely to make a questionable entry pass.

3. Separate overlap, equality and containment

overlaps asks whether the ranges share any addresses. subnet_of includes equality: two identical networks are each a subnet of the other. The separate Equal field makes that case visible.

FirstInsideSecond and SecondInsideFirst depend on argument order. Overlap and equality do not. The comparison uses address-range arithmetic, not a list of every address, so a broad range does not require probing or expanding all its hosts.

The report includes network and broadcast addresses when comparing ranges. It does not count usable hosts or reserve particular addresses for a deployment.

4. Check a contained subnet

Open a command window in the folder and run:

python3 CheckOverlap.py 192.0.2.0/24 192.0.2.128/25

Use your installation's Python 3 command if it is not named python3. The output is:

{
  "First": "192.0.2.0/24",
  "Second": "192.0.2.128/25",
  "Overlap": true,
  "Equal": false,
  "FirstInsideSecond": false,
  "SecondInsideFirst": true
}

JSON is a text format for named values. The ranges overlap, but are not equal. The second is inside the first; the first is not inside the second. Reverse the command's arguments and the two containment fields swap.

python3 CheckOverlap.py 192.0.2.128/25 192.0.2.0/24

{
  "First": "192.0.2.128/25",
  "Second": "192.0.2.0/24",
  "Overlap": true,
  "Equal": false,
  "FirstInsideSecond": true,
  "SecondInsideFirst": false
}

The exit code is 1 for overlap, 0 for disjoint ranges and 2 for a usage or input error. Overlap is a review signal here, not automatically a configuration fault: nested ranges can be intentional.

5. Check adjacent and equal ranges

Run:

python3 CheckOverlap.py 192.0.2.0/25 192.0.2.128/25

The output is:

{
  "First": "192.0.2.0/25",
  "Second": "192.0.2.128/25",
  "Overlap": false,
  "Equal": false,
  "FirstInsideSecond": false,
  "SecondInsideFirst": false
}

The ranges touch at a boundary but share no address, so the exit code is 0. The first ends at 192.0.2.127, and the second begins at 192.0.2.128.

Now compare 192.0.2.0/24 with itself.

python3 CheckOverlap.py 192.0.2.0/24 192.0.2.0/24

{
  "First": "192.0.2.0/24",
  "Second": "192.0.2.0/24",
  "Overlap": true,
  "Equal": true,
  "FirstInsideSecond": true,
  "SecondInsideFirst": true
}

Overlap, Equal and both containment fields are true; the exit code is 1. A /32 describes one address and can also overlap a larger range containing it. A /0 covers all IPv4 addresses.

This check does not inspect interfaces, routing tables, firewall rules, address allocation or reachability. A successful comparison cannot prove that a network plan is safe or that a connection will work.

Keep the intended configuration unchanged while reviewing the result. If a range overlaps, check why it is present and which system owns the allocation before changing anything. Reports include network labels, so keep them private when they describe work or personal infrastructure.

References

Python IPv4 network parsing, overlap and subnet comparisons.

Distinguishing Missing and Null JSON Fields in Python

William Baptist — Fri, 09 Oct 2026 19:35:07 +0000

An export has a field in one record, null in another and no field at all in a third. Those are different cases. This read-only checker reports missing names separately from explicit null values, without treating zero or an empty string as missing.

Python 3.10 or later and a plain-text editor are all you need. The program uses only the standard library.

1. Create the practice exports

JSON is a text format for named values and lists. null is an explicit JSON value; it is not the same as leaving a name out. Save this as MakeExamples.py in a practice folder:

from pathlib import Path


Path("Records.json").write_bytes(
    b'[{"Id":"001","Note":null},{"Id":"002"},{"Id":"003","Note":""}]\n'
)
Path("Values.json").write_bytes(
    b'[{"Value":0},{"Value":false},{"Value":""},{"Value":[]},{"Value":{}}]\n'
)

Run:

python3 MakeExamples.py

Use your installation's Python 3 command if it is not named python3. This creates or replaces two files, so use a folder without files you need to keep. Records.json gives three different states for Note. Values.json contains values that a simple true/false test might accidentally treat as absent.

2. Save the checker

Save this as CheckRequiredFields.py beside the examples:

import json as Json
import sys as Sys
from pathlib import Path


def RejectConstant(Token):
    raise ValueError("non-JSON numeric token")


def UniqueObject(Pairs):
    Result = {}
    for Name, Value in Pairs:
        if Name in Result:
            raise ValueError("repeated object name")
        Result[Name] = Value
    return Result


def CheckRequiredFields(FileName, Names):
    if not Names or any(not Name.strip() for Name in Names):
        raise ValueError("choose nonblank field names")
    if len(Names) != len(set(Names)):
        raise ValueError("choose each field once")
    with Path(FileName).open("rb") as Input:
        Data = Input.read(1048577)
    if len(Data) > 1048576:
        raise ValueError("input exceeds 1 MiB teaching limit")
    Records = Json.loads(Data.decode("utf-8"), parse_constant=RejectConstant,
                         parse_float=str, object_pairs_hook=UniqueObject)
    if not isinstance(Records, list) or any(not isinstance(Item, dict) for Item in Records):
        raise ValueError("expected an array of objects")
    if len(Records) > 1000:
        raise ValueError("more than 1000 records")
    Missing = []
    Null = []
    for Number, Record in enumerate(Records, start=1):
        for Name in Names:
            if Name not in Record:
                Missing.append({"Record": Number, "Field": Name})
            elif Record[Name] is None:
                Null.append({"Record": Number, "Field": Name})
    return {"Records": len(Records), "MissingFields": Missing, "NullFields": Null}


def Main():
    if len(Sys.argv) < 3:
        print("Usage: python3 CheckRequiredFields.py input.json FIELD [FIELD ...]", file=Sys.stderr)
        return 2
    try:
        Report = CheckRequiredFields(Sys.argv[1], Sys.argv[2:])
    except (OSError, ValueError, RecursionError) as Problem:
        print(f"Check stopped: {Problem}", file=Sys.stderr)
        return 2
    print(Json.dumps(Report, indent=2, ensure_ascii=True))
    return 1 if Report["MissingFields"] or Report["NullFields"] else 0


if __name__ == "__main__":
    Sys.exit(Main())

You choose the field names on the command line. Each name must be nonblank and chosen once. Matching is exact: Note and note differ. Only direct fields in each record are checked, not names inside nested objects. A field name containing a space needs quoting in your command window.

The input must be a UTF-8 JSON array of objects, without an initial byte-order mark. The file limit is 1 MiB (1,048,576 bytes), with one extra byte read to detect oversized input, and the array limit is 1,000 records. Repeated object names and non-JSON numeric tokens such as NaN ("not a number") are rejected throughout the parsed file. Deep nesting can stop with a recursion error; the byte cap is not a complete resource-safety guarantee.

parse_float=str keeps decimal/exponent tokens from being converted to binary floating-point values during this check. The program does not save those temporary parsed values or change the file. It only uses membership and an explicit null comparison for the report.

3. Test membership before the value

Name not in Record tests whether the name exists in the dictionary, Python's collection of named values. It does not ask whether the value is true or false.

When the name is present, Record[Name] is None tests for Python's representation of JSON null. The elif keeps a missing name out of the null list and avoids trying to read a value that is not there.

The lists preserve record order and your chosen field order. Record numbers start at one. Field names appear in the report, but the values do not.

4. Check missing and null separately

Run:

python3 CheckRequiredFields.py Records.json Id Note

The output is:

{
  "Records": 3,
  "MissingFields": [
    {
      "Record": 2,
      "Field": "Note"
    }
  ],
  "NullFields": [
    {
      "Record": 1,
      "Field": "Note"
    }
  ]
}

Record 1 has Note with value null. Record 2 lacks Note. Record 3 has Note with an empty string, so neither rule flags it.

The exit code is 1 when either category is present, 0 when neither is found, and 2 when usage, reading, parsing, the input shape or a teaching limit stops the check. A stopped run prints no JSON report. Treating null as a review item is this checker's rule, not a claim that every receiving system forbids it.

5. Keep zero and empty values distinct

Run:

python3 CheckRequiredFields.py Values.json Value

The output is:

{
  "Records": 5,
  "MissingFields": [],
  "NullFields": []
}

The exit code is 0. Zero, false, an empty string, an empty array and an empty object are all present and are not null. Whether they are useful or allowed is a separate question.

Now change only the requested name:

python3 CheckRequiredFields.py Values.json value

The lowercase value is absent from all five records, so MissingFields lists records 1 through 5, NullFields stays empty and the exit code is 1. The checker reports the exact name you asked for. It cannot tell a misspelling from a genuinely required field; compare your command with the documented field names before interpreting the report.

A schema is a set of rules for expected fields and values. This is not a complete schema check. It does not validate types, formats, ranges, identifiers, nested fields or a particular importer's behaviour. An empty array also passes with zero records. Choose fields from the receiving system's documented requirements, and review its rules for null and empty values separately.

It never fills missing fields or replaces null. Those changes can invent meaning, so keep the original export and check the source before correcting anything.

Use a saved regular file that will not change while it is read. Save reports under a new filename: redirecting output over an input can empty it before Python opens it. Names, counts and positions can reveal information, so keep reports private when they describe work or personal data.

References

Building a File Fingerprint in Python

William Baptist — Thu, 08 Oct 2026 11:19:17 +0000

Calculate a file fingerprint, save it as a reference, then change the file and compare the results.

You can also follow my free guide on Tidy Desk Digital.

For a browser-based option, use my free file fingerprint checker.

A hash turns data into a fixed-length value, often called a fingerprint. We'll use SHA-256, a hash method producing 32 bytes, small units of stored data. It displays them as 64 hexadecimal characters: digits 0 to 9 and letters a to f.

You'll need Python 3, a plain-text editor and a command window. No extra packages are required.

1. Create the example file

A text editor may add a line ending even when you only type abc. Let Python write the exact three bytes instead. Save this as MakeExample.py in your practice folder:

from pathlib import Path


Path("Example.txt").write_bytes(b"abc")

Run it from a command window in that folder:

python3 MakeExample.py

Use your installation's Python 3 command if it is not named python3. This creates or replaces Example.txt, so use a practice folder without a file you need to keep. b"abc" is a bytes value; write_bytes adds no line ending. The file now contains exactly three bytes, regardless of your editor's line-ending setting.

2. Save the program

Save this as FingerprintFile.py:

import hashlib as HashLib
import os as Os
import stat as Stat
import sys as Sys


def FingerprintFile(FilePath):
    Fingerprint = HashLib.sha256()
    Total = 0
    with open(FilePath, "rb") as InputFile:
        if not Stat.S_ISREG(Os.fstat(InputFile.fileno()).st_mode):
            raise ValueError("input must be a regular file")
        while True:
            Block = InputFile.read(65536)
            if not Block:
                break
            Fingerprint.update(Block)
            Total += len(Block)
    return Fingerprint.hexdigest(), Total


def Main():
    if len(Sys.argv) != 2:
        print("Usage: python3 FingerprintFile.py input-file", file=Sys.stderr)
        return 2
    try:
        Fingerprint, Total = FingerprintFile(Sys.argv[1])
    except (OSError, ValueError) as Problem:
        print(f"Fingerprint stopped: {Problem}", file=Sys.stderr)
        return 2
    print(f"SHA-256: {Fingerprint}")
    print(f"Bytes read: {Total}")
    return 0


if __name__ == "__main__":
    Sys.exit(Main())

HashLib.sha256() starts the calculation. Each update adds the next block of bytes; adding blocks works like adding their combined contents at once. After the file ends, hexdigest returns the printable value.

The input opens in rb, read-only binary mode, without translating line endings. with closes it when reading finishes. Nothing is printed as a complete fingerprint until the full read succeeds.

Use known regular files, such as documents. Stat.S_ISREG checks the opened file's type, but a special file can block during opening before that check runs.

3. Calculate the fingerprint

Open a command window in your folder and run:

python3 FingerprintFile.py Example.txt

Use your installation's Python 3 command if it is not named python3. The output is:

SHA-256: ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad
Bytes read: 3

The exit code, a small result number another script can check, is 0 when calculation succeeds or 2 for an input/command error. This program calculates one value; it does not compare values or return a mismatch code.

4. Change the file and compare

Copy the example to another file and calculate its fingerprint. The values should match. Add a character to the copy and calculate again; its stored contents changed, so expect a different value.

Keep the reference separately from the checked file. If someone can replace both, they can calculate a new value and make the replacement match. A fingerprint from the same untrusted page as a download adds no independent reason to trust it.

The program only reads the file, so the input bytes stay unchanged.

A match checks content against a reference, not the author's identity or whether the file is safe to open. Different files can share a fixed-length value. When both stable files are available and you need a direct byte-for-byte answer, compare their contents directly.

Use a file that will not change during reading. The program does not lock it or take a frozen copy, and follows symbolic links, paths pointing to other files. Reading may affect recorded last-access times. Small read blocks do not impose file-size or time limits.

Save output to a new report filename, not over the input. Do not use this calculation as a password-storage design; protecting passwords is a different job.

Reference

Python hashlib documentation.

Checking a Saved File Fingerprint in Python

William Baptist — Thu, 08 Oct 2026 11:19:05 +0000

This is part two of Building a File Fingerprint in Python. That program calculates SHA-256, a 64-character fingerprint of a file's bytes. Here we'll reuse its module and add a checker that returns a match, mismatch or error.

Keep FingerprintFile.py from part one beside the new file, and have Python 3 and a text editor ready. Nothing else needs installing.

1. Reuse the fingerprint module

Keep FingerprintFile.py from part one's program step in your practice folder. A module is a Python source file that another program can import. We'll call its FingerprintFile function rather than repeat the file-reading code here.

The module returns a fingerprint and byte count after the complete read succeeds. Its binary read leaves stored bytes and line endings unchanged. Use known regular files, such as documents; the module's type check happens after opening, so a special file can block before that check.

2. Create a known reference case

Save this as MakeExample.py beside the module:

from pathlib import Path


Path("Example.txt").write_bytes(b"abc")

Run:

python3 MakeExample.py

Use your installation's Python 3 command if it is not named python3. This creates or replaces Example.txt, so keep the exercise in a practice folder without a file you need to preserve. Python writes exactly the three bytes abc, with no line ending. Their expected SHA-256 value is:

ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad

A line ending would add bytes and change that value. This setup makes the first test independent of your editor's line-ending setting.

3. Add the checker

Save CheckFingerprint.py beside the module:

import sys as Sys
from FingerprintFile import FingerprintFile


def CheckExpected(Value):
    if len(Value) != 64 or any(Character not in "0123456789abcdefABCDEF" for Character in Value):
        raise ValueError("expected fingerprint must be 64 hexadecimal characters")
    return Value.lower()


def Main():
    if len(Sys.argv) != 3:
        print("Usage: python3 CheckFingerprint.py input-file expected-sha256", file=Sys.stderr)
        return 2
    try:
        Expected = CheckExpected(Sys.argv[2])
        Actual, Total = FingerprintFile(Sys.argv[1])
    except (OSError, ValueError) as Problem:
        print(f"Check stopped: {Problem}", file=Sys.stderr)
        return 2
    if Actual == Expected:
        print(f"Fingerprint matches; bytes read: {Total}.")
        return 0
    print("Fingerprint differs.")
    return 1


if __name__ == "__main__":
    Sys.exit(Main())

CheckExpected accepts exactly 64 hexadecimal characters, converting uppercase letters to lowercase. Spaces, a sha256: prefix and trailing line endings are rejected before opening the file. A correctly shaped but mistyped reference can still pass this check, so choose its source carefully.

Main calculates the current value and compares it with your reference. A failed read is an error, not a mismatch: the complete contents have not been checked.

4. Run a match

Open a command window in the folder and run:

python3 CheckFingerprint.py Example.txt ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad

Use your installation's Python 3 command if it is not named python3. The output is:

Fingerprint matches; bytes read: 3.

Change b"abc" to b"abd" in MakeExample.py, run that setup script again, then rerun the checker with the original reference. The output becomes:

Fingerprint differs.

The exit code, a small result number another script can check, is 0 for a match, 1 for a complete calculation that differs, and 2 for a command, reference-format or read error. Keep 1 and 2 separate: missing input or a bad reference does not establish changed content.

To try this comparison without a command window, choose your file and paste the saved reference into my file fingerprint checker. It runs on your device and accepts files up to 16 MiB.

5. Use your own reference

Save the expected value separately from the file. If someone can replace both, they can calculate a new fingerprint and make their replacement match. A value from the same untrusted page as a download is not independent reason to trust it.

This checks contents against your reference, not the publisher's identity, malware or a digital signature, a separate method for checking who approved some data. Open or import the file in its intended application when usability matters.

Change the file to abd and the fixed abc reference fails, which is what a changed file should do.

Different files can share a fixed-length fingerprint. When both stable files are available and you need a direct byte-for-byte answer, compare their contents. Save output under a new report filename, never over the input.

Reference

Python hashlib documentation.

Building a Connection-Log Summary in Python

William Baptist — Wed, 07 Oct 2026 15:38:03 +0000

This guide builds that summary in Python. It reuses the format-checking reader from Part 1, keeps rejected lines visible and sorts accepted groups by count. The result is a starting point for investigation, not a list of attackers or proof of complete network activity.

Start with the same documented input

The practice format has four fields separated by a tab: time, source address, destination address and destination port. Each line represents one invented connection-log event, not a packet or a complete session. It is a teaching format created for this guide, not a claimed Wireshark, Zeek or firewall export format.

The first line must be Time, Source, Destination, Port, with actual tabs between the names. The time uses exactly YYYY-MM-DDThh:mm:ssZ; Z means Coordinated Universal Time (UTC), the shared reference time used here rather than a local clock. Fractional seconds, time differences such as +01:00 and leap-second values are not supported. The addresses use Internet Protocol version 4 (IPv4), the four-number address format such as 192.0.2.10. Each part is separated by a dot, with no extra leading zeros. Internet Protocol version 6 (IPv6), the newer address format that uses groups such as 2001:db8::1, is outside this example. Ports are whole numbers from 1 to 65535; that is this format's limit, not a claim that port zero never appears elsewhere.

Use only the invented sample. Its addresses come from the ranges reserved for documentation by RFC 5737, the published internet reference. No real traffic was captured or analysed.

Create practice.tsv with the following actual tab-separated text:

Time    Source  Destination Port
2026-09-30T09:00:00Z    192.0.2.10  198.51.100.20   443
2026-09-30T09:00:01Z    192.0.2.10  198.51.100.20   443
2026-09-30T09:00:02Z    192.0.2.11  203.0.113.30    53
2026-02-30T09:00:03Z    192.0.2.10  198.51.100.20   443
2026-09-30T09:00:04Z    192.0.2.11  203.0.113.30    70000

Use the complete LogReader.py module included below in the same folder. It accepts this one teaching format, preserves physical line numbers and original text in memory, and rejects unsupported data lines with a reason. It stops the overall load for a missing file, unsupported header or exceeded file/row limit.

The article includes that exact module so the example can be reproduced without a different library or parser. It is not a separately validated reader for real Zeek, Wireshark or firewall exports. Do not feed a real export into it and interpret rejection as a problem with the network.

Save the shared reader

Save this as LogReader.py. It is the same module used in Part 1; both articles include it so this guide can be run on its own.

from dataclasses import dataclass as DataClass
from datetime import datetime as DateTime
from pathlib import Path


@DataClass
class LogEntry:
    Number: int
    Original: str
    EventTime: str = ""
    Source: str = ""
    Destination: str = ""
    Port: int = 0
    Problem: str = ""
    Accepted: bool = False


def Digits(Value, Maximum):
    if not Value:
        raise ValueError("empty number")
    if any(Character not in "0123456789" for Character in Value):
        raise ValueError("number needs ordinary digits")
    Number = int(Value)
    if Number > Maximum:
        raise ValueError("number outside supported range")
    return Number


def CheckAddress(Value):
    Parts = Value.split(".")
    if len(Parts) != 4:
        raise ValueError("expected four IPv4 address parts")
    for Part in Parts:
        if len(Part) > 3 or (len(Part) > 1 and Part.startswith("0")):
            raise ValueError("use canonical dotted IPv4 addresses")
        Digits(Part, 255)


def CheckTime(Value):
    if len(Value) != 20:
        raise ValueError("expected time YYYY-MM-DDThh:mm:ssZ")
    if any(Value[Index] != Mark for Index, Mark in
           ((4, "-"), (7, "-"), (10, "T"), (13, ":"), (16, ":"), (19, "Z"))):
        raise ValueError("expected time YYYY-MM-DDThh:mm:ssZ")
    NumberText = Value[0:4] + Value[5:7] + Value[8:10] + Value[11:13] + Value[14:16] + Value[17:19]
    if any(Character not in "0123456789" for Character in NumberText):
        raise ValueError("time needs ordinary digits")
    Year = Digits(Value[0:4], 9999)
    Month = Digits(Value[5:7], 12)
    Day = Digits(Value[8:10], 31)
    Hour = Digits(Value[11:13], 23)
    Minute = Digits(Value[14:16], 59)
    Second = Digits(Value[17:19], 59)
    try:
        DateTime(Year, Month, Day)
    except ValueError:
        raise ValueError("date does not exist") from None
    try:
        DateTime(Year, Month, Day, Hour, Minute, Second)
    except ValueError:
        raise ValueError("time does not exist") from None


def ParseEntry(Entry):
    try:
        if len(Entry.Original) > 1024:
            raise ValueError("line exceeds 1024 bytes")
        if any(Character != "\t" and not 32 <= ord(Character) <= 126
               for Character in Entry.Original):
            raise ValueError("unsupported byte in practice format")
        Fields = Entry.Original.split("\t")
        if len(Fields) != 4:
            raise ValueError("expected exactly four tab-separated fields")
        CheckTime(Fields[0])
        CheckAddress(Fields[1])
        CheckAddress(Fields[2])
        Entry.Port = Digits(Fields[3], 65535)
        if Entry.Port == 0:
            raise ValueError("port must be 1 to 65535 in this format")
        Entry.EventTime, Entry.Source, Entry.Destination = Fields[:3]
        Entry.Accepted = True
    except ValueError as Error:
        Entry.Problem = str(Error)
    return Entry


def ReadLog(FileName):
    with Path(FileName).open("rb") as Input:
        Data = Input.read(1048577)
    if len(Data) > 1048576:
        raise ValueError("input exceeds 1 MiB teaching limit")
    if not Data:
        raise ValueError("empty input")
    Lines = Data.split(b"\n")
    if Lines[-1] == b"":
        Lines.pop()
    Lines = [Line[:-1] if Line.endswith(b"\r") else Line for Line in Lines]
    if Lines[0] != b"Time\tSource\tDestination\tPort":
        raise ValueError("unsupported header")
    if len(Lines) - 1 > 1000:
        raise ValueError("more than 1000 data lines")
    return [ParseEntry(LogEntry(Number, Line.decode("latin-1")))
            for Number, Line in enumerate(Lines[1:], start=2)]

Choose what one group means

One group is the exact combination of source address, destination address and destination port. Two events with the same source but different destinations belong to different groups. Two events to the same destination but different ports also stay separate.

This matters because the output is a count of entries. The practice file has no protocol, source port, byte count, session identifier or unique event identifier. It cannot tell whether two entries describe separate sessions, retries, duplicated logging or something else. It does not infer a service merely from a familiar port number.

Before adding another field to the grouping rule, decide which question you want the report to answer. A summary by source alone is a different report. Calling both reports "top connections" without explaining the difference makes the output easy to misread.

Write the summary program

Save this as SummariseLog.py beside LogReader.py:

import sys as Sys
from LogReader import ReadLog


def Main():
    if len(Sys.argv) != 2:
        print("Usage: python3 SummariseLog.py practice.tsv", file=Sys.stderr)
        return 2
    try:
        Entries = ReadLog(Sys.argv[1])
    except (OSError, ValueError) as Error:
        print(f"Input stopped: {Error}", file=Sys.stderr)
        return 2
    Groups = {}
    Good = 0
    Bad = 0
    for Entry in Entries:
        if not Entry.Accepted:
            Bad += 1
            print(f"Rejected line {Entry.Number}: {Entry.Problem}")
        else:
            Good += 1
            Key = (Entry.Source, Entry.Destination, Entry.Port)
            Groups[Key] = Groups.get(Key, 0) + 1
    print("Count Source Destination Port")
    for Key, Count in sorted(Groups.items(), key=lambda Item: -Item[1]):
        Source, Destination, Port = Key
        print(f"{Count} {Source} {Destination} {Port}")
    print(f"Accepted: {Good}; rejected: {Bad}. Counts are log entries, not sessions.")
    return 1 if Bad else 0


if __name__ == "__main__":
    raise SystemExit(Main())

The code has no comments; the article explains its work. For each accepted entry, it looks up that source, destination and port group, starting from zero if it has not been seen yet, then adds one. Rejected entries increment a separate rejected total and print their physical line numbers and reasons.

The final sorted call orders groups by descending count. Python preserves insertion order in this dictionary, and its sort keeps equal-count items in that order. Tied groups therefore appear in the order they were first accepted from the input, not in an order of threat or importance.

Run the program

Install Python 3 through the supported route for your computer. With both source files in one folder:

python3 SummariseLog.py practice.tsv

No extra packages are needed. The actual runs used Python 3.10.12 on Linux. Windows and macOS were not tested. Example names use PascalCase, such as Groups and Main; Python's own built-in names, library methods and special __name__ spelling stay unchanged.

Actual output:

Rejected line 5: date does not exist
Rejected line 6: number outside supported range
Count Source Destination Port
2 192.0.2.10 198.51.100.20 443
1 192.0.2.11 203.0.113.30 53
Accepted: 3; rejected: 2. Counts are log entries, not sessions.

Three accepted entries become two groups. Two malformed entries do not disappear: their reasons appear before the table and their count stays in the summary. The program returns 1 because the input is partly rejected. Fully accepted input returns 0, and an overall load failure returns 2 with no table.

The accepted counts sum to 3, the accepted-entry total. That is one useful consistency check. It does not reconcile the log against a network sensor or establish that the file includes every relevant event.

Check the ordering with a second invented file

A separate test file had one event in the first group, three in the second and two in the third. Actual output:

Count Source Destination Port
3 192.0.2.2 198.51.100.2 80
2 192.0.2.3 198.51.100.3 53
1 192.0.2.1 198.51.100.1 443
Accepted: 6; rejected: 0. Counts are log entries, not sessions.

Exact repeated lines were counted twice, not silently removed. Input bytes stayed unchanged.

Keeping repetitions is intentional. Without a reliable unique event reference, a repeated line may be duplicated logging or a genuinely repeated action. Decide how to handle that separately, with evidence; do not make a summary look cleaner by assuming the answer.

Investigate the result without making a threat verdict

A busy group may be ordinary business activity. A quiet group may still deserve attention. A count alone does not establish intent, data volume, unauthorised access or a scan. Compare with the actual logging format, the period covered and the context of the systems you are allowed to inspect.

This program groups the whole practice file. It does not compute a rate because it has no chosen observation interval. Ten entries over a minute and ten over a day are different questions. The next planned guide adds explicit time-window work and tests its boundaries rather than simply calling a large count a burst.

Missing evidence changes the meaning too. If many rows were rejected, the table is a summary of what the parser accepted, not the full file. Read the reasons, correct the source/export through an authorised process if appropriate, and rerun. Preserve the original rather than editing it until the rejects vanish.

Keep the exercise small and read-only

The shared reader reads at most 1 MiB plus one byte, rejecting input beyond its 1 MiB teaching limit. It accepts at most 1,000 data lines and rejects a data line longer than 1,024 bytes after reading. It loads entries and groups in memory. A file changing while it is read is not a protected snapshot. Use a stable invented input; this is not a hardened parser for hostile growing logs or a tested large-file tool.

The programs themselves only read the input. A command-window redirection can still overwrite it before they start. If you save output, choose a new report filename, never the input name. Real logs and output can reveal sensitive system activity; keep them in authorised protected storage.

This guide's useful outcome is a reproducible grouping rule with its gaps visible.

References

Python pathlib: opening files.
Python dataclasses: named record fields.
RFC 5737: documentation-only address ranges.
RFC 3339: timestamp reference. The reader handles only the exact UTC spelling described here.

No claim of real export compatibility, network completeness, malicious activity, session counts or production performance is made.

Building a Log Reader in Python

William Baptist — Wed, 07 Oct 2026 15:36:52 +0000

In this guide, you will build a small Python log reader that separates accepted entries from rejected ones and keeps their line numbers and original text. You will see what was read, what failed and why. It reads one saved practice file and never changes it.

This is Part 1 of a connected log-analysis series. Part 2 uses the same reader to count events by source, destination and port. Later guides can build time-window and comparison work on those foundations. The article is useful without a shop product or a live network tool.

Define the input before writing the reader

Use only the invented sample. Its addresses come from the ranges reserved for documentation by RFC 5737, the published internet reference. No real traffic was captured or analysed.

Create practice.tsv in a new folder. TSV means tab-separated values. The following display uses actual tabs, not groups of spaces:

Time    Source  Destination Port
2026-09-30T09:00:00Z    192.0.2.10  198.51.100.20   443
2026-09-30T09:00:01Z    192.0.2.10  198.51.100.20   443
2026-09-30T09:00:02Z    192.0.2.11  203.0.113.30    53
2026-02-30T09:00:03Z    192.0.2.10  198.51.100.20   443
2026-09-30T09:00:04Z    192.0.2.11  203.0.113.30    70000

If copying changes the tabs into spaces, the reader will reject the format rather than guess. Use a plain-text editor that preserves tabs. The two last rows are intentionally wrong: 30 February is not a real date, and 70000 is outside the supported port range.

The reader supports ordinary printable ASCII characters plus tabs, which is enough for this format's times, addresses and ports. It rejects other bytes, including terminal control characters. It does not support arbitrary user names, free-text descriptions or every international text encoding.

Build a reusable Python module

A Python module is a separate source file containing code another program can import and use. Save the following as LogReader.py. The file has no code comments; the guide explains it.

from dataclasses import dataclass as DataClass
from datetime import datetime as DateTime
from pathlib import Path


@DataClass
class LogEntry:
    Number: int
    Original: str
    EventTime: str = ""
    Source: str = ""
    Destination: str = ""
    Port: int = 0
    Problem: str = ""
    Accepted: bool = False


def Digits(Value, Maximum):
    if not Value:
        raise ValueError("empty number")
    if any(Character not in "0123456789" for Character in Value):
        raise ValueError("number needs ordinary digits")
    Number = int(Value)
    if Number > Maximum:
        raise ValueError("number outside supported range")
    return Number


def CheckAddress(Value):
    Parts = Value.split(".")
    if len(Parts) != 4:
        raise ValueError("expected four IPv4 address parts")
    for Part in Parts:
        if len(Part) > 3 or (len(Part) > 1 and Part.startswith("0")):
            raise ValueError("use canonical dotted IPv4 addresses")
        Digits(Part, 255)


def CheckTime(Value):
    if len(Value) != 20:
        raise ValueError("expected time YYYY-MM-DDThh:mm:ssZ")
    if any(Value[Index] != Mark for Index, Mark in
           ((4, "-"), (7, "-"), (10, "T"), (13, ":"), (16, ":"), (19, "Z"))):
        raise ValueError("expected time YYYY-MM-DDThh:mm:ssZ")
    NumberText = Value[0:4] + Value[5:7] + Value[8:10] + Value[11:13] + Value[14:16] + Value[17:19]
    if any(Character not in "0123456789" for Character in NumberText):
        raise ValueError("time needs ordinary digits")
    Year = Digits(Value[0:4], 9999)
    Month = Digits(Value[5:7], 12)
    Day = Digits(Value[8:10], 31)
    Hour = Digits(Value[11:13], 23)
    Minute = Digits(Value[14:16], 59)
    Second = Digits(Value[17:19], 59)
    try:
        DateTime(Year, Month, Day)
    except ValueError:
        raise ValueError("date does not exist") from None
    try:
        DateTime(Year, Month, Day, Hour, Minute, Second)
    except ValueError:
        raise ValueError("time does not exist") from None


def ParseEntry(Entry):
    try:
        if len(Entry.Original) > 1024:
            raise ValueError("line exceeds 1024 bytes")
        if any(Character != "\t" and not 32 <= ord(Character) <= 126
               for Character in Entry.Original):
            raise ValueError("unsupported byte in practice format")
        Fields = Entry.Original.split("\t")
        if len(Fields) != 4:
            raise ValueError("expected exactly four tab-separated fields")
        CheckTime(Fields[0])
        CheckAddress(Fields[1])
        CheckAddress(Fields[2])
        Entry.Port = Digits(Fields[3], 65535)
        if Entry.Port == 0:
            raise ValueError("port must be 1 to 65535 in this format")
        Entry.EventTime, Entry.Source, Entry.Destination = Fields[:3]
        Entry.Accepted = True
    except ValueError as Error:
        Entry.Problem = str(Error)
    return Entry


def ReadLog(FileName):
    with Path(FileName).open("rb") as Input:
        Data = Input.read(1048577)
    if len(Data) > 1048576:
        raise ValueError("input exceeds 1 MiB teaching limit")
    if not Data:
        raise ValueError("empty input")
    Lines = Data.split(b"\n")
    if Lines[-1] == b"":
        Lines.pop()
    Lines = [Line[:-1] if Line.endswith(b"\r") else Line for Line in Lines]
    if Lines[0] != b"Time\tSource\tDestination\tPort":
        raise ValueError("unsupported header")
    if len(Lines) - 1 > 1000:
        raise ValueError("more than 1000 data lines")
    return [ParseEntry(LogEntry(Number, Line.decode("latin-1")))
            for Number, Line in enumerate(Lines[1:], start=2)]

LogEntry keeps the physical line number, original line, parsed fields, rejection reason and accepted flag. The header is line 1. The original text is kept using a one-byte-to-one-character reading format called Latin-1, excluding the line ending. That preserves rejected bytes for review; accepted fields still allow only the ASCII characters described above. This is not a byte-for-byte archival copy; keep the original file itself.

Digits allows ordinary decimal digits only and checks the value against its allowed maximum. Python integers keep these whole numbers exact without a fixed-width overflow. CheckAddress requires four supported address parts. CheckTime checks both the spelling and whether the date and clock time exist. It accepts one deliberately narrow UTC spelling rather than guessing time zones.

ParseEntry turns a wrong data line into a rejected entry with a reason. It does not abort the whole review for one malformed event. ReadLog does stop for an unsupported header, missing file or teaching-size limit: those problems mean the overall input cannot be handled as promised. The reader loads the file before displaying results, so a stopped load produces no misleading partial report.

Add a small program to display the result

Save this beside the module as ReadLogDemo.py:

import sys as Sys
from LogReader import ReadLog


def Main():
    if len(Sys.argv) != 2:
        print("Usage: python3 ReadLogDemo.py practice.tsv", file=Sys.stderr)
        return 2
    try:
        Entries = ReadLog(Sys.argv[1])
    except (OSError, ValueError) as Error:
        print(f"Input stopped: {Error}", file=Sys.stderr)
        return 2
    Good = 0
    Bad = 0
    for Entry in Entries:
        if Entry.Accepted:
            Good += 1
            print(f"Accepted line {Entry.Number}: {Entry.EventTime} "
                  f"{Entry.Source} -> {Entry.Destination}:{Entry.Port}")
        else:
            Bad += 1
            print(f"Rejected line {Entry.Number}: {Entry.Problem}")
    print(f"Accepted: {Good}; rejected: {Bad}. Original file unchanged.")
    return 1 if Bad else 0


if __name__ == "__main__":
    raise SystemExit(Main())

Install Python 3 through the supported route for your computer. Open a command window in the folder containing both source files and run:

python3 ReadLogDemo.py practice.tsv

No extra packages are needed. LogReader.py must be beside the program so Python can import it. The actual runs used Python 3.10.12 on Linux. Windows and macOS were not tested. The command name may differ on another installation; these are the Linux instructions used here.

All names introduced by the example use PascalCase: each word starts with a capital letter, such as ReadLog and EventTime. Python's own names, library methods and the special __name__ / __main__ spellings stay exact. Changing those would change or break Python's behaviour.

Actual output:

Accepted line 2: 2026-09-30T09:00:00Z 192.0.2.10 -> 198.51.100.20:443
Accepted line 3: 2026-09-30T09:00:01Z 192.0.2.10 -> 198.51.100.20:443
Accepted line 4: 2026-09-30T09:00:02Z 192.0.2.11 -> 203.0.113.30:53
Rejected line 5: date does not exist
Rejected line 6: number outside supported range
Accepted: 3; rejected: 2. Original file unchanged.

The program returns 1 because some data lines were rejected. That is useful information for a script running it, not a verdict that a security incident occurred. A fully accepted input returns 0; a load failure or missing argument returns 2.

Test the boundary cases, not only the good row

The rebuilt Python reader and the Part 2 summary were run against 34 input/format cases each, plus missing-file and missing-argument cases for each program: 72 checks total. They covered exact field count, blank lines, impossible dates, unsupported time spellings, bad addresses, invalid ports, control/non-ASCII bytes, line and record limits, empty/header-only input, Windows-style line endings, no final newline and exact repeated events. Both programs left the input bytes unchanged in those runs.

The sample output was separately reproduced. A separate summary-order run also checked that six invented entries were sorted into counts 3, 2 and 1. These are local invented-file tests, not live-tool compatibility or a performance benchmark.

Keep the limits visible

The reader opens the input in binary mode and reads at most 1 MiB plus one byte. The extra byte lets it reject a file above the 1 MiB teaching limit without reading an unlimited file. It accepts at most 1,000 data lines and rejects a data line longer than 1,024 bytes after reading. This is a bounded teaching example, not a guarantee about a file somebody else changes while it is open. Use a stable practice file. It does not preserve a protected snapshot of a growing log.

The accepted flag means the line fits this format. It does not prove the event happened, that the logging system was honest, or that no other events are missing. Repeated entries stay separate. Their count is a count of log entries, not automatically a count of distinct actions.

The original rejected text remains in the in-memory result for a reviewer to inspect deliberately. The display program prints the line number and reason, not the raw rejected text, avoiding blindly echoing control bytes into the command window. Keep actual logs and review output private. If you save command-window output, choose a new report filename. Do not redirect output over the input file: the command window can empty that file before the program reads it.

Part 2 turns accepted entries into a connection summary and keeps the rejected count beside it. The next task is grouping useful evidence, not hiding the inconvenient lines.

References

Python pathlib: opening files.
Python dataclasses: named record fields.
RFC 5737: addresses reserved for documentation.
RFC 3339: timestamp reference. This example accepts a narrower subset, not every permitted spelling.

Use only data you have permission to read. No live capture, scanning, attack detection or file modification is performed.

Checking Work-Sharing Permission Records with Pascal

William Baptist — Wed, 07 Oct 2026 15:11:29 +0000

A developer adds a useful feature: paste a document, remove the names, ask an artificial intelligence (AI) service to produce a shorter version. The names disappear. The screen says the text is ready to send.

That last step is a different decision. Removing names does not establish that nobody can be identified from the remaining information. It also does not establish that this service, account or use has been approved for work.

The Information Commissioner's Office (ICO), the UK data protection regulator, explains that combinations of details can identify someone. Removing names and other obvious identifying details alone is insufficient. Its guidance is under review, so use the current version and appropriate advice for a real decision.

Rather than building a fictional privacy detector, this article builds a much smaller thing: a Pascal check of a written permission record for a general, made-up request. It preserves missing decisions. It never inspects a document or decides that private work is safe to send.

The case that a name-removal button cannot settle

Imagine a made-up staff note about Alex, the only night-shift supervisor at the fictional Cedar Repair shop, after an incident on 12 September. It has attached health notes. Remove Alex's name. The role, place, date and attachments remain.

In a real workplace, those details might still identify the person to someone with other information. Asking an unapproved AI service whether the real note is private enough would already share it. Do not use that as the review method.

When the task is only an outline, invent a request from scratch instead:

Create a neutral outline for a routine staff meeting. Use no personal information. Leave placeholders for facts and review by the responsible person. Do not decide what action should be taken.

That asks for structure without uploading a staff record. It still needs review before becoming work and does not approve a staff decision or provide legal advice.

Keep the permission record narrower than the product name

A service name alone cannot describe permission. The record in this example names the service, the work account and one approved task. It also records whether the approval is still current. The proposed request has the same fields plus its content category, whether attachments were reviewed and whether it connects a work folder.

The fictional permission covers only a generic meeting outline with no private work. It does not cover sensitive work through any service. A real business may approve a particular controlled use of sensitive information, but this demonstration cannot establish that approval or make the risk assessment.

The person entering GeneralOutline is making a claim about the content. The program does not read the question or files. If the content is unresolved, it asks the responsible person instead of inferring safety from a missing name.

The folder rule is also deliberately conservative: connecting any work folder requires review outside this example. It does not say every folder connection is prohibited. It says this narrow record does not authorise it.

Run the complete fictional check

Install Free Pascal through the supported route for your computer. Save this as WorkSharingReview.pas. In a command window opened in that folder, compile it with fpc -Cr -Co -Ci WorkSharingReview.pas. The switches enable additional range, arithmetic-overflow and input/output checks. On Linux, run ./WorkSharingReview.

There are no comments inside the code. It has no document entry, network connection, file upload or service integration.

program WorkSharingReview;

type
  TWorkKind = (GeneralOutline, PrivateRecord, UnknownContent);
  TSharingResult = (DoNotSend, AskResponsiblePerson, ReadyForReview);
  TPermissionRecord = record
    ServiceName: String;
    AccountName: String;
    ApprovedTask: String;
    IsCurrent: Boolean;
  end;
  TRequestRecord = record
    ServiceName: String;
    AccountName: String;
    ProposedTask: String;
    WorkKind: TWorkKind;
    AttachmentsReviewed: Boolean;
    HasConnectedWorkFolder: Boolean;
  end;

var
  Permission: TPermissionRecord;
  Request: TRequestRecord;
  Checks: Integer;
  Failures: Integer;

function ReviewRequest(Rule: TPermissionRecord;
  Proposed: TRequestRecord): TSharingResult;
begin
  if Proposed.WorkKind = PrivateRecord then
    ReviewRequest := DoNotSend
  else if (Proposed.WorkKind = UnknownContent)
    or not Proposed.AttachmentsReviewed
    or Proposed.HasConnectedWorkFolder then
    ReviewRequest := AskResponsiblePerson
  else if not Rule.IsCurrent
    or (Rule.ServiceName = '') or (Rule.AccountName = '')
    or (Rule.ApprovedTask = '')
    or (Rule.ServiceName <> Proposed.ServiceName)
    or (Rule.AccountName <> Proposed.AccountName)
    or (Rule.ApprovedTask <> Proposed.ProposedTask) then
    ReviewRequest := AskResponsiblePerson
  else
    ReviewRequest := ReadyForReview;
end;

procedure Expect(LabelText: String; Wanted: TSharingResult);
var
  Actual: TSharingResult;
begin
  Actual := ReviewRequest(Permission, Request);
  Checks := Checks + 1;
  if Actual <> Wanted then
  begin
    Failures := Failures + 1;
    WriteLn('FAIL: ', LabelText);
  end;
  Write(LabelText, ': ');
  case Actual of
    DoNotSend: WriteLn('do not send through this example');
    AskResponsiblePerson: WriteLn('ask the responsible person');
    ReadyForReview: WriteLn('record ready for review');
  end;
end;

begin
  Checks := 0;
  Failures := 0;
  Permission.ServiceName := 'Fictional Outline Service';
  Permission.AccountName := 'Fictional Work Account';
  Permission.ApprovedTask := 'generic meeting outline';
  Permission.IsCurrent := True;
  Request.ServiceName := Permission.ServiceName;
  Request.AccountName := Permission.AccountName;
  Request.ProposedTask := Permission.ApprovedTask;
  Request.WorkKind := PrivateRecord;
  Request.AttachmentsReviewed := True;
  Request.HasConnectedWorkFolder := False;
  Expect('Private staff record', DoNotSend);

  Request.WorkKind := UnknownContent;
  Expect('Names removed but content unresolved', AskResponsiblePerson);

  Request.WorkKind := GeneralOutline;
  Request.AccountName := 'Fictional Personal Account';
  Expect('Different account', AskResponsiblePerson);

  Request.AccountName := Permission.AccountName;
  Request.ProposedTask := 'staff decision';
  Expect('Different task', AskResponsiblePerson);

  Request.ProposedTask := Permission.ApprovedTask;
  Request.AttachmentsReviewed := False;
  Expect('Attachments not reviewed', AskResponsiblePerson);

  Request.AttachmentsReviewed := True;
  Request.HasConnectedWorkFolder := True;
  Expect('Connected work folder', AskResponsiblePerson);

  Request.HasConnectedWorkFolder := False;
  Permission.IsCurrent := False;
  Expect('Approval no longer current', AskResponsiblePerson);

  Permission.IsCurrent := True;
  Permission.ServiceName := '';
  Expect('Missing service in permission record', AskResponsiblePerson);

  Permission.ServiceName := Request.ServiceName;
  Permission.AccountName := '';
  Expect('Missing account in permission record', AskResponsiblePerson);

  Permission.AccountName := Request.AccountName;
  Permission.ApprovedTask := '';
  Expect('Missing task in permission record', AskResponsiblePerson);

  Permission.ApprovedTask := Request.ProposedTask;
  Request.ServiceName := 'Different Fictional Service';
  Expect('Different service', AskResponsiblePerson);

  Request.ServiceName := Permission.ServiceName;
  Expect('Made-up outline within recorded scope', ReadyForReview);
  WriteLn('Checks: ', Checks, '; failures: ', Failures);
  if Failures > 0 then
    Halt(1);
end.

What the run demonstrated

Actual output:

Private staff record: do not send through this example
Names removed but content unresolved: ask the responsible person
Different account: ask the responsible person
Different task: ask the responsible person
Attachments not reviewed: ask the responsible person
Connected work folder: ask the responsible person
Approval no longer current: ask the responsible person
Missing service in permission record: ask the responsible person
Missing account in permission record: ask the responsible person
Missing task in permission record: ask the responsible person
Different service: ask the responsible person
Made-up outline within recorded scope: record ready for review
Checks: 12; failures: 0

The twelve checks cover private content, unresolved content, a different account, a different task, unreviewed attachments, a connected folder, outdated approval, each missing permission field, a different service and the complete made-up outline record.

The last result is still ReadyForReview, not permission to send. The program only found matching entries under its narrow fictional rule. It did not verify that the recorded permission was genuinely approved, that the approval remains current, that the content category is truthful, or that the service's settings and terms satisfy the business's needs.

Exact string matching here is intentional. A slightly different service name or account remains unresolved rather than being guessed into a match. A production system would need stable account identifiers and a real permission source. Do not fix a failed comparison by making it accept anything that looks similar.

Both programs in this batch were compiled using Free Pascal 3.2.2 extracted from Ubuntu packages into a local Linux test folder. Compiler version and successful execution were recorded, with additional checks enabled. This is a computer test of invented entries, not a staff trial, a privacy assessment or testing of an AI provider. Windows and macOS were not tested.

Where the human review belongs

Keep approval with the person responsible for the business's information and tools. They need to consider the actual service, account, settings, type of work, current provider terms and relevant contracts. A work licence or a setting that stops training on submitted text is not a complete permission record by itself.

Attachments, screenshots and connected sources are part of sharing too. The interface should make them visible before any send step. Do not show a green "safe" badge merely because the question has no recognised names. A feature may help with one narrow editing task without having authority to approve the wider sharing decision.

If work has already been shared by mistake, stop sending more and tell the responsible person promptly through the approved route. Use the business's incident process. Deleting a chat is not proof that every copy is gone.

Use the free card for the discussion

Read Before you paste work into an AI tool. Its four checks cover the approved service/account, private work, made-up examples and review of the answer.

The card has no data-entry fields, scripts or saved answers. It cannot inspect a request, remove private information or approve a service. Use it to prepare a discussion about the actual rule, without copying private work into another service to ask for help.

Sources and limits

General guidance, not legal or professional advice. Check current rules, contracts and provider terms for the exact service and use.

Checking Changed Payment Details with Pascal

William Baptist — Wed, 07 Oct 2026 15:09:24 +0000

The case and all records below are fictional. The Pascal program was compiled and run on Linux with Free Pascal 3.2.2. No bank connection, supplier call or payment occurred.

An invoice arrives from a familiar supplier. It uses the expected reference and asks for payment into a different account. The bank's account-name check returns a match. The supplier does not answer your call, then someone calls back to confirm the new details.

A small invoice system could display three reassuring facts: known supplier, matching bank name, phone confirmation. None is enough to approve the changed destination.

The useful programming task is to keep those decisions separate. This article builds a small Pascal example that leaves incomplete checks on hold, gives a denial priority over other entries, and never returns an instruction to pay.

Start with the actual payment decision

Barclays advises an outgoing call using trusted contact details already held on file when checking new or amended payment instructions. It warns that incoming caller numbers can be faked. If the person is busy, arrange a time and call them yourself. A callback is not the same event.

On the call, ask the known contact to read back the details through your agreed process. Compare them privately. Do not use the telephone number in the new request to establish its own truth. If the trusted number no longer works, the payment lead needs to establish a contact route independently; the program below cannot do that.

Barclays also explains that a bank-name match cannot prove the payment request is genuine. An impersonator may have an account in a matching name. The contact check, the bank result, changing the saved supplier record and normal payment approval remain separate work.

This demonstration uses a deliberately narrow rule. Only a recorded outgoing confirmation, an account-name match and recorded approval reach human review. Any unresolved name result stays on hold. A real business may have an approved exception process, but this example does not invent one.

Give the record states names that preserve uncertainty

The program has four contact states:

NotCalled: no confirmation call is recorded.
IncomingOnly: an incoming caller spoke, but the outgoing trusted-route check is not recorded.
ConfirmedOutgoing: the record says the check was completed on the outgoing trusted-route call.
ChangeDenied: the supplier denied the change.

The last two are statements somebody entered, not facts the program independently discovered.

The three name states are deliberately simplified. NameMatches records a match. NeedsReview covers a close match, no match or an unresolved difference. NotAvailable records that the result is unavailable. A real record should retain the exact bank result separately rather than losing it in this grouping.

ApprovalRecorded means the fictional record has an explicit yes/no entry for the business's separate approval. It does not identify an approver, prove they had permission or retrieve evidence.

The result names matter too. ReadyForHumanReview does not mean approved, genuine or safe. It means this narrow set of entries is present for a person to review.

Compile the complete example

Install Free Pascal using the supported route for your computer. Save the following as PaymentReview.pas. With a normal Free Pascal installation, compile it from a command window in the same folder with fpc -Cr -Co -Ci PaymentReview.pas. These switches enable extra checks for invalid ranges, arithmetic overflow and input/output errors. On Linux, run ./PaymentReview afterwards.

The code has no comments; the article explains it. The code does not read private files, write records, contact a service or initiate a payment.

program PaymentReview;

type
  TContactCheck = (NotCalled, IncomingOnly, ConfirmedOutgoing, ChangeDenied);
  TNameCheck = (NameMatches, NeedsReview, NotAvailable);
  TReviewResult = (StopAndReport, KeepOnHold, ReadyForHumanReview);
  TChangeRecord = record
    ContactCheck: TContactCheck;
    NameCheck: TNameCheck;
    ApprovalRecorded: Boolean;
  end;

var
  Checks: Integer;
  Failures: Integer;
  Contact: TContactCheck;
  NameResult: TNameCheck;
  Approved: Boolean;
  Item: TChangeRecord;
  Expected: TReviewResult;

function ReviewChange(Change: TChangeRecord): TReviewResult;
begin
  if Change.ContactCheck = ChangeDenied then
    ReviewChange := StopAndReport
  else if (Change.ContactCheck = ConfirmedOutgoing)
    and (Change.NameCheck = NameMatches)
    and Change.ApprovalRecorded then
    ReviewChange := ReadyForHumanReview
  else
    ReviewChange := KeepOnHold;
end;

procedure Expect(LabelText: String; Actual, Wanted: TReviewResult);
begin
  Checks := Checks + 1;
  if Actual <> Wanted then
  begin
    Failures := Failures + 1;
    WriteLn('FAIL: ', LabelText);
  end;
end;

procedure ShowResult(LabelText: String; Change: TChangeRecord);
begin
  Write(LabelText, ': ');
  case ReviewChange(Change) of
    StopAndReport: WriteLn('stop and report');
    KeepOnHold: WriteLn('keep on hold');
    ReadyForHumanReview: WriteLn('record ready for human review');
  end;
end;

begin
  Checks := 0;
  Failures := 0;
  Item.ContactCheck := IncomingOnly;
  Item.NameCheck := NameMatches;
  Item.ApprovalRecorded := True;
  ShowResult('Incoming callback', Item);
  Expect('callback cannot replace outgoing check', ReviewChange(Item), KeepOnHold);

  Item.ContactCheck := ConfirmedOutgoing;
  Item.ApprovalRecorded := False;
  ShowResult('Outgoing check but no recorded approval', Item);
  Expect('missing approval stays on hold', ReviewChange(Item), KeepOnHold);

  Item.ContactCheck := ChangeDenied;
  Item.ApprovalRecorded := True;
  ShowResult('Supplier denied the change', Item);
  Expect('denial overrides other entries', ReviewChange(Item), StopAndReport);

  Item.ContactCheck := ConfirmedOutgoing;
  ShowResult('All required fictional entries present', Item);
  Expect('complete record only reaches review', ReviewChange(Item), ReadyForHumanReview);

  for Contact := NotCalled to ChangeDenied do
    for NameResult := NameMatches to NotAvailable do
      for Approved := False to True do
      begin
        Item.ContactCheck := Contact;
        Item.NameCheck := NameResult;
        Item.ApprovalRecorded := Approved;
        if Contact = ChangeDenied then
          Expected := StopAndReport
        else if (Contact = ConfirmedOutgoing)
          and (NameResult = NameMatches) and Approved then
          Expected := ReadyForHumanReview
        else
          Expected := KeepOnHold;
        Expect('decision table', ReviewChange(Item), Expected);
      end;

  WriteLn('Checks: ', Checks, '; failures: ', Failures);
  if Failures > 0 then
    Halt(1);
end.

Read the results as a record check

Actual output from the Linux run:

Incoming callback: keep on hold
Outgoing check but no recorded approval: keep on hold
Supplier denied the change: stop and report
All required fictional entries present: record ready for human review
Checks: 28; failures: 0

The first case is the fictional callback. Even with a matching name and approval entry, it stays on hold. The second case has outgoing confirmation but no approval; it also stays on hold. A denial wins over the matching name and approval. Only the last complete fictional record reaches review.

Four named checks cover those examples. The nested loops then cover all 24 combinations of four contact states, three name states and two approval values. That makes 28 checks total. The loop checks agreement with the written decision table, not a bank's behaviour. Because the expected table closely follows the function, the named examples and independent human review are important too. Passing this suite is not proof that the business rule is suitable for your organisation.

For this run, the compiler executable was extracted from Ubuntu's Free Pascal 3.2.2 packages into a local test folder. Its version was checked and both compilation and execution completed successfully with the additional checks enabled. A Windows installation, a real invoice application and a bank integration were not tested. The compile command above is the normal installed-tool command, not a claim that every computer already has fpc available.

What a real application still needs

The example stores typed choices only. It has no permission checks, evidence store, change history, input-file reader or database. Those omissions are why it must not be connected to a payment button.

A real implementation needs to keep supporting evidence available to the appropriate reviewer. Record the checker, outgoing call time, trusted contact source, confirmer's role, exact bank result and unresolved differences in protected business records. Limit who can change or approve the supplier record. Keep full bank details in the controlled payment system where required, not a public demonstration or action log.

There is another practical boundary: record completeness is not record truth. A person could select ConfirmedOutgoing without making a call. A complete screen should not silently substitute for retrieving and reviewing the evidence. The next reviewer needs to know what was checked and what the software cannot establish.

Use the free companion for the human side

Open Before you change bank details. Its four checks cover your own records, the call you make, confirming details and normal approval.

Use the fictional callback situation to practise. Leave the outgoing-call check unfinished, read the next step and agree who owns it. The page has no bank-detail entry fields. Choices stay only in the open page and clear on reload or closing. The checklist does not inspect an invoice, verify a person or account, authorise a payment or guarantee the absence of fraud.

If money may already have gone to a fraudster, contact your bank immediately through a trusted route. Do not wait for the code exercise. Recovery is not guaranteed.

Sources and limits

General guidance, not legal or financial advice. Follow current bank guidance and the business's approved payment process.

5 Fun Side Hustles for Cybersecurity Students

William Baptist — Sun, 04 Jun 2023 20:28:35 +0000

Some links in this post are affiliate links. The income figures below are rough personal estimates, not guarantees.

So many online resources seem to be vastly outdated when it comes to side hustles, especially for students. I want to show practical ways that I’ve personally used in the past few months that worked for me and will work for you. What’s great is that all of these can be a learning experience alongside being a fun way to make money.

1. Building and Selling Cybersecurity Tools

Income: $100–$500 per month

I’ve been developing and selling my own cybersecurity tools extensively. You’ll be surprised how many businesses are willing to pay for customised solutions to meet their specific problems.

Identify a need within the market and develop a tool to meet it. This requires strong programming skills and a good understanding of cybersecurity. Once you’ve developed your tool, you can sell it on platforms like GitHub Marketplace, or directly to businesses.

2. Web Hosting Security Consultant

Income: $100–$1000+ per month

Now for this method I’m going to use Fiverr as an example because that was the site I used but you can use any other number of online freelance websites or even try hosting your own site!

The key to this is to offer your services to a niche area of web hosting. For example, I offered setup services in SSL certificates for websites. I also would offer hosting as an option that was bundled with the SSL certificates.

There are two platforms that helped me achieve this named SSLs.com and Skystra with the former being the main source of income. I gained the skills in my first year of college for choosing the correct SSL certificate and setting this up for the client (naturally charging extra for the setup fee)

Skystra allowed me to scale what I needed by charging monthly so I would recommend looking into the best options for your hustle.

3. Security eCommerce

Income: $100–$10,000+ per month

Let’s be honest, traditional dropshopping sucks in 2023 but it doesn’t mean you can’t take advantage of that concept.

You can try your hand at creating a custom platform or a mobile app that caters to a niche market and integrate it with Easyship for seamless fulfillment. For example, you could make an e-commerce platform for physical security goods that are tailored to computers (Encrypted USBs, Webcam covers etc.)

4. Online Cybersecurity Tutoring

Income: $200 — $1000+ per month

As you know there is a high demand for cybersecurity knowledge and many people are looking for tutors to help them grasp complex cybersecurity concepts.

Join online tutoring platforms like Chegg and Tutor.com. To be an effective tutor, it’s best if you already have a good grasp of cybersecurity principles, patience, and excellent communication skills.

5. Writing Cybersecurity Articles

Income: $0–$200 per month

Finally, this is one I do a lot and the key to it is fun!

It’s less of a sidehustle but to me there is nothing better than sharing cool tools within articles about cybersecurity. It also looks really good on your resume if you are actively engaging and contributing to the cybersecurity community.

Get noticed, get inspired and check out my profile and see if you can spot some more articles you might like!

Advanced Wireshark Scripts for Intrusion Detection

William Baptist — Sun, 14 May 2023 19:14:38 +0000

Note: only capture and analyse traffic on networks you own or have permission to monitor.

In this article, I have delved into the depths of Wireshark scripting once again and discovered some unconventional techniques to unravel hidden threats and identify suspicious activities.

Revealing Covert Channels:

Covert channels are stealthy communication paths that bypass traditional security measures. To expose these hidden channels, you can leverage Wireshark scripting and Python’s flexibility.

The following toy script looks for one hard-coded string in HTTP payloads. It is only a demonstration of the idea:

import pyshark

#Open the captured packets file
cap = pyshark.FileCapture("packets.pcapng")

#Detect suspicious covert channels
for pkt in cap:
    if "HTTP" in pkt:
        payload = pkt.http.file_data
        if payload and payload.startswith("CovertChannel"):
            print(f"Covert Channel Detected: {payload}")

It only matches the literal text "CovertChannel", so it would never find a real covert channel. Real detection needs traffic baselines and statistical analysis. Treat this as a starting point for learning.

Uncovering DNS Tunneling:

DNS tunneling is a technique used to bypass network security by encapsulating data within DNS requests and responses. Let’s shed light on these covert channels.

The following toy script flags DNS queries containing the word "tunnel". It is a simple string match, not a real detector:

import pyshark

#Open the captured packets file
cap = pyshark.FileCapture("packets.pcapng")

#Detect potential DNS tunnels
for pkt in cap:
    if "DNS" in pkt and hasattr(pkt.dns, "qry_name"):
        if "tunnel" in str(pkt.dns.qry_name):
            print("Potential DNS Tunnel Detected!")

By parsing DNS packets and inspecting the requested domain names (questions), you can pinpoint suspicious queries that may indicate the presence of a DNS tunnel. Real DNS tunnels do not announce themselves with the word "tunnel", so real detection looks at things like query length, entropy and volume. This script only illustrates where to start.

Utilising Statistical Anomaly Detection:

Intrusion detection can be enhanced by leveraging statistical anomaly detection techniques. Wireshark scripting, combined with Python’s statistical libraries, can help us identify deviations from normal network behavior.

Consider the following script that applies anomaly detection to packet sizes:

import pyshark
import numpy as np
from scipy.stats import zscore

#Open the captured packets file
cap = pyshark.FileCapture("packets.pcapng")

#Extract packet sizes
packet_sizes = [int(pkt.length) for pkt in cap]

#Detect anomalous packet sizes
z_scores = zscore(packet_sizes)
anomalies = np.where(z_scores > 3)[0]

if len(anomalies) > 0:
    print("Anomalous Packet Sizes Detected!")

By calculating z-scores for packet sizes and comparing them to a threshold, it can identify packets that deviate significantly from the expected range. This script allows you to spot abnormal packet size patterns, potentially indicating network anomalies or malicious activities.

Port Scan Detector:

A port scanner detector is essential for network security as it helps identify unauthorised scanning activities. By detecting port scans, it enables administrators to pinpoint potential vulnerabilities in their systems and take appropriate measures to mitigate risks.

This proactive script aims to cover these bases:

import pyshark

#Open the captured packets file
cap = pyshark.FileCapture("packets.pcapng")

#Track the destination ports each source IP has contacted
scan_counter = {}

#Detect port scans
for pkt in cap:
    if "TCP" in pkt and "IP" in pkt:
        src_ip = pkt.ip.src
        dst_ip = pkt.ip.dst
        dst_port = pkt.tcp.dstport
        scan_counter.setdefault((src_ip, dst_ip), set()).add(dst_port)

#Identify potential port scanning activities
for (src_ip, dst_ip), ports in scan_counter.items():
    if len(ports) > 5:  #Adjust the threshold as per your needs
        print(f"Possible Port Scanning Detected: {src_ip} --> {dst_ip} ({len(ports)} different ports)")

The script examines each TCP packet and records which destination ports each source IP has tried on each target. A scan touches many different ports, so if one source contacts more ports than a predefined threshold, an alert is printed. Normal traffic usually touches only a few ports, though busy servers and slow scanners can still fool this simple check.

Exposing DNS Cache Poisoning Attacks:

Network’s DNS Infrastructure is obviously crucial, these attacks can lead to incorrect DNS responses. They can also lead to redirecting users to malicious websites, intercepting their communications or compromising their data.

The script below is a simple starting point. It only flags DNS requests of type ANY, which are sometimes used in reconnaissance and amplification attacks. It cannot detect cache poisoning itself, which needs analysis of DNS responses:

import pyshark

#Open the captured packets file
cap = pyshark.FileCapture("packets.pcapng")

#Track suspicious DNS requests
poisoning_attempts = []

#Detect DNS cache poisoning attacks
for pkt in cap:
    if "DNS" in pkt and hasattr(pkt.dns, "qry_type"):
        if str(pkt.dns.qry_type) == "255":  #255 is the ANY query type
            qname = str(pkt.dns.qry_name)
            if qname not in poisoning_attempts:
                poisoning_attempts.append(qname)

#Display the identified DNS cache poisoning attempts
if poisoning_attempts:
    print("Detected DNS ANY queries:")
    for attempt in poisoning_attempts:
        print(attempt)
else:
    print("No DNS ANY queries detected.")

By parsing the captured packets, the script identifies DNS packets and examines the questions section for requests of any type. If a request for an ANY type is found, the script extracts the domain name and checks if it is already in the list of detected poisoning attempts. If not, it adds the domain name to the list.

Finally, the script displays the identified DNS cache poisoning attempts, or a message indicating that no attempts were detected leaving the following result:

Detecting hidden communication within seemingly harmless network traffic allows us to expose vulnerabilities that otherwise could go unnoticed. I hope that you found some use from these scripts and remember that you will be required to extensively change them to get some use from your own network!