update gravity - improve domain validation processing speed by rrobgill · Pull Request #2710 · pi-hole/FTL

rrobgill · 2025-11-13T18:27:53Z

What does this PR aim to accomplish?:

Speed up operation of Update Gravity / pihole -g

How does this PR accomplish the above?:

Incorporates several small efficiency gains within the main domain validation loop. The individual gains per execution are comparatively minor but as they applied to every entry in blocklists and allowlists during the update process, their effects are cumulative.

Use a lookup table for valid characters, with single comparison per character instead of repeated comparisons and branches per character
Short circuit the ip address testing rather than testing every string as an IPv4 and IPv6 address:
Only perform full ipv4 address test if first character of token is numeric
Only perform full ipv6 address test if the line contains a colon,
and a colon is found within the first 5 characters of the token
Limits check for unicode BOM to once per file

Test results

On test system with the following lists:

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts	
https://media.githubusercontent.com/media/zachlagden/Pi-hole-Optimized-Blocklists/refs/heads/main/lists/all_domains.txt	
https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/adblock/ultimate.txt (ABP style, added as blocklist)
https://raw.githubusercontent.com/arapurayil/aBL/refs/heads/main/filters/main.txt (ABP style - added as allowlist)
https://raw.githubusercontent.com/DandelionSprout/adfilt/refs/heads/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareDomains.txt

Results of time pihole -g

Before (current dev):
real 0m27.348s
user 0m20.069s
sys 0m3.631s

After (this patch):
real 0m22.964s
user 0m15.607s
sys 0m3.598s

Link documentation PRs if any are needed to support this PR:

N/A

By submitting this pull request, I confirm the following:

I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
I have commented my proposed changes within the code and I have tested my changes.
I am willing to help maintain this change if there are issues with it later.
It is compatible with the EUPL 1.2 license
I have squashed any insignificant commits. (git rebase)
I have checked that another pull request for this purpose does not exist.
I have considered, and confirmed that this submission will be valuable to others.
I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
I give this submission freely, and claim no ownership to its content.

I have read the above and my PR is ready for review. Check this box to confirm

yubiuser

I see im improvements around 1-2 secs for my particular lists. I guess it depends on your hardware and list content how much improvement one sees.

DL6ER

Thank you for this pull request. I have made a few comments. The main speed increase likely comes from calling inet_pton() which is definitely a worthwhile change. Maybe also small bits from the lookup table (but definitely less). However, the two locations where direct comparisons are replaced with memcmp() calls should be undone due to the reasons I outlined.

Use lookup table (const) to test for valid characters in domain names Short circuit the ip address testing: Only perform full ipv4 address test if first character of token is numeric Only perform full ipv6 address test if the line contains a colon, and a colon is found within the first 5 characters of the token Limit test for unicode BOM to first line of file Signed-off-by: Rob Gill <rrobgill@protonmail.com>

rrobgill · 2025-12-01T20:45:00Z

Tested now with your latest changes.

The strchr() was definitely more expensive than I realised.

pihole -g is an additional 300ms faster on test machine using set of lists above with your changes applied.

…1 → 2026.02.0 (#45320) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/pi-hole/pihole](https://redirect.github.com/pi-hole/docker-pi-hole) | major | `91dc91d` → `ee34852` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/18710) for more information. Add the preset `:preserveSemverRanges` to your config if you don't want to pin your dependencies. --- ### Release Notes <details> <summary>pi-hole/docker-pi-hole (ghcr.io/pi-hole/pihole)</summary> ### [`v2026.02.0`](https://redirect.github.com/pi-hole/docker-pi-hole/releases/tag/2026.02.0) [Compare Source](https://redirect.github.com/pi-hole/docker-pi-hole/compare/2025.11.1...2026.02.0)  #### What's Changed (Docker Specific) - Pin base image by sha to catch silent rebuilds by [@yubiuser](https://redirect.github.com/yubiuser) in [#1965](https://redirect.github.com/pi-hole/docker-pi-hole/pull/1965) - Set fixed buildx version to mitigate issues with buildx version 0.31.1 by [@yubiuser](https://redirect.github.com/yubiuser) in [#1987](https://redirect.github.com/pi-hole/docker-pi-hole/pull/1987) **Full Changelog**: <pi-hole/docker-pi-hole@2025.11.1...2026.02.0> ***  #### What's Changed (FTL v6.5) - Tweak undocumented wait-for option subtly by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2707](https://redirect.github.com/pi-hole/FTL/pull/2707) - update gravity - improve domain validation processing speed by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/FTL#2710](https://redirect.github.com/pi-hole/FTL/pull/2710) - Update embedded SQLite3 to 3.51.1 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2731](https://redirect.github.com/pi-hole/FTL/pull/2731) - Update embedded dnsmasq to 2.92rc1 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2730](https://redirect.github.com/pi-hole/FTL/pull/2730) - Fix documentation - Do not use equal sign with `pihole-FTL --config` command by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/FTL#2736](https://redirect.github.com/pi-hole/FTL/pull/2736) - Add dns.cache.rrtype by [@Manakuremati](https://redirect.github.com/Manakuremati) in [pi-hole/FTL#2740](https://redirect.github.com/pi-hole/FTL/pull/2740) - Enhancements to the documentation markdown generator by [@PromoFaux](https://redirect.github.com/PromoFaux) in [pi-hole/FTL#2741](https://redirect.github.com/pi-hole/FTL/pull/2741) - Network Overview - obtain MAC and hostname from dhcp.leases by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/FTL#2727](https://redirect.github.com/pi-hole/FTL/pull/2727) - fix: make `get_domains` parameters optional by [@tien](https://redirect.github.com/tien) in [pi-hole/FTL#2278](https://redirect.github.com/pi-hole/FTL/pull/2278) - Escape unprintable characters in invalid host names by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2601](https://redirect.github.com/pi-hole/FTL/pull/2601) - Implement better allOf handling in API verifier by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2745](https://redirect.github.com/pi-hole/FTL/pull/2745) - Update build containers to Alpine 3.23 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2743](https://redirect.github.com/pi-hole/FTL/pull/2743) - Add option to hide network connection errors by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2749](https://redirect.github.com/pi-hole/FTL/pull/2749) - Harden default Content Security Policy (CSP) by [@Erasure5959](https://redirect.github.com/Erasure5959) in [pi-hole/FTL#2754](https://redirect.github.com/pi-hole/FTL/pull/2754) - Fix computation of NTP server's root delay by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2760](https://redirect.github.com/pi-hole/FTL/pull/2760) - Teleporter: Fix for custom gravity.db path by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2758](https://redirect.github.com/pi-hole/FTL/pull/2758) - Upgrade embedded Lua to 5.5 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2626](https://redirect.github.com/pi-hole/FTL/pull/2626) - Add missing \[forwarded] property in GET /api/history/database by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2750](https://redirect.github.com/pi-hole/FTL/pull/2750) - Update SQLite3 to 3.51.2 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2761](https://redirect.github.com/pi-hole/FTL/pull/2761) - Low-memory hardware optimizations by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2757](https://redirect.github.com/pi-hole/FTL/pull/2757) - Reduce startup delay by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2725](https://redirect.github.com/pi-hole/FTL/pull/2725) - home.arpa and internal TLDs may be non-local without revServer by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2772](https://redirect.github.com/pi-hole/FTL/pull/2772) #### New Contributors - [@Erasure5959](https://redirect.github.com/Erasure5959) made their first contribution in [pi-hole/FTL#2754](https://redirect.github.com/pi-hole/FTL/pull/2754) **Full Changelog**: <pi-hole/FTL@v6.4.1...v6.5>  #### What's Changed (Web v6.4.1) - Set the end date for live query update to end of epoch by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/web#3677](https://redirect.github.com/pi-hole/web/pull/3677) - Improve initial loading of Query Log by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/web#3715](https://redirect.github.com/pi-hole/web/pull/3715) Also fixes two security advisories: - <GHSA-6xp4-jw73-f4qp>: fixed with [pi-hole/web@`d328f14`](https://redirect.github.com/pi-hole/web/commit/d328f143718022d82dc94c8751121ca41be3b996) - <GHSA-8rw8-vjgp-rwj6>: fixed with [pi-hole/web@`1a0c6f4`](https://redirect.github.com/pi-hole/web/commit/1a0c6f4fe6d0116fd2846b2adaae95996b7f194d) **Full Changelog**: <pi-hole/web@v6.4...v6.4.1>  #### What's Changed (Core v6.4) - Remove wget from alpine dependencies by [@darkexplosiveqwx](https://redirect.github.com/darkexplosiveqwx) in [pi-hole/pi-hole#6484](https://redirect.github.com/pi-hole/pi-hole/pull/6484) - Remove custom FTL FirewallD zone checks from debug log by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/pi-hole#6481](https://redirect.github.com/pi-hole/pi-hole/pull/6481) - Add Alpine 3.23 to test suite by [@yubiuser](https://redirect.github.com/yubiuser) in [pi-hole/pi-hole#6488](https://redirect.github.com/pi-hole/pi-hole/pull/6488) - Debug log - Add colors to gravity tables by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/pi-hole#6485](https://redirect.github.com/pi-hole/pi-hole/pull/6485) - Use configured location for web repo when updating or repairing by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/pi-hole#6470](https://redirect.github.com/pi-hole/pi-hole/pull/6470) - Add missing `-g` to the message in gravity recovery command by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/pi-hole#6513](https://redirect.github.com/pi-hole/pi-hole/pull/6513) - Don't install unused /usr/local/share/man/man5 by [@darkexplosiveqwx](https://redirect.github.com/darkexplosiveqwx) in [pi-hole/pi-hole#6526](https://redirect.github.com/pi-hole/pi-hole/pull/6526) **Full Changelog**: <pi-hole/pi-hole@v6.3...v6.4> </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://redirect.github.com/renovatebot/renovate).

…ag to v2026 This PR contains the following updates: | Package | Update | Change | |---|---|---| | [bigbeartechworld/big-bear-pihole-unbound](https://redirect.github.com/pi-hole/docker-pi-hole) | major | `2025.11.1` → `2026.02.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/17) for more information. --- ### Release Notes <details> <summary>pi-hole/docker-pi-hole (bigbeartechworld/big-bear-pihole-unbound)</summary> ### [`v2026.02.0`](https://redirect.github.com/pi-hole/docker-pi-hole/releases/tag/2026.02.0) [Compare Source](https://redirect.github.com/pi-hole/docker-pi-hole/compare/2025.11.1...2026.02.0)  #### What's Changed (Docker Specific) - Pin base image by sha to catch silent rebuilds by [@yubiuser](https://redirect.github.com/yubiuser) in [#1965](https://redirect.github.com/pi-hole/docker-pi-hole/pull/1965) - Set fixed buildx version to mitigate issues with buildx version 0.31.1 by [@yubiuser](https://redirect.github.com/yubiuser) in [#1987](https://redirect.github.com/pi-hole/docker-pi-hole/pull/1987) **Full Changelog**: <pi-hole/docker-pi-hole@2025.11.1...2026.02.0> ***  #### What's Changed (FTL v6.5) - Tweak undocumented wait-for option subtly by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2707](https://redirect.github.com/pi-hole/FTL/pull/2707) - update gravity - improve domain validation processing speed by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/FTL#2710](https://redirect.github.com/pi-hole/FTL/pull/2710) - Update embedded SQLite3 to 3.51.1 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2731](https://redirect.github.com/pi-hole/FTL/pull/2731) - Update embedded dnsmasq to 2.92rc1 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2730](https://redirect.github.com/pi-hole/FTL/pull/2730) - Fix documentation - Do not use equal sign with `pihole-FTL --config` command by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/FTL#2736](https://redirect.github.com/pi-hole/FTL/pull/2736) - Add dns.cache.rrtype by [@Manakuremati](https://redirect.github.com/Manakuremati) in [pi-hole/FTL#2740](https://redirect.github.com/pi-hole/FTL/pull/2740) - Enhancements to the documentation markdown generator by [@PromoFaux](https://redirect.github.com/PromoFaux) in [pi-hole/FTL#2741](https://redirect.github.com/pi-hole/FTL/pull/2741) - Network Overview - obtain MAC and hostname from dhcp.leases by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/FTL#2727](https://redirect.github.com/pi-hole/FTL/pull/2727) - fix: make `get_domains` parameters optional by [@tien](https://redirect.github.com/tien) in [pi-hole/FTL#2278](https://redirect.github.com/pi-hole/FTL/pull/2278) - Escape unprintable characters in invalid host names by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2601](https://redirect.github.com/pi-hole/FTL/pull/2601) - Implement better allOf handling in API verifier by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2745](https://redirect.github.com/pi-hole/FTL/pull/2745) - Update build containers to Alpine 3.23 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2743](https://redirect.github.com/pi-hole/FTL/pull/2743) - Add option to hide network connection errors by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2749](https://redirect.github.com/pi-hole/FTL/pull/2749) - Harden default Content Security Policy (CSP) by [@Erasure5959](https://redirect.github.com/Erasure5959) in [pi-hole/FTL#2754](https://redirect.github.com/pi-hole/FTL/pull/2754) - Fix computation of NTP server's root delay by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2760](https://redirect.github.com/pi-hole/FTL/pull/2760) - Teleporter: Fix for custom gravity.db path by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2758](https://redirect.github.com/pi-hole/FTL/pull/2758) - Upgrade embedded Lua to 5.5 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2626](https://redirect.github.com/pi-hole/FTL/pull/2626) - Add missing \[forwarded] property in GET /api/history/database by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2750](https://redirect.github.com/pi-hole/FTL/pull/2750) - Update SQLite3 to 3.51.2 by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2761](https://redirect.github.com/pi-hole/FTL/pull/2761) - Low-memory hardware optimizations by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2757](https://redirect.github.com/pi-hole/FTL/pull/2757) - Reduce startup delay by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2725](https://redirect.github.com/pi-hole/FTL/pull/2725) - home.arpa and internal TLDs may be non-local without revServer by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/FTL#2772](https://redirect.github.com/pi-hole/FTL/pull/2772) #### New Contributors - [@Erasure5959](https://redirect.github.com/Erasure5959) made their first contribution in [pi-hole/FTL#2754](https://redirect.github.com/pi-hole/FTL/pull/2754) **Full Changelog**: <pi-hole/FTL@v6.4.1...v6.5>  #### What's Changed (Web v6.4.1) - Set the end date for live query update to end of epoch by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/web#3677](https://redirect.github.com/pi-hole/web/pull/3677) - Improve initial loading of Query Log by [@DL6ER](https://redirect.github.com/DL6ER) in [pi-hole/web#3715](https://redirect.github.com/pi-hole/web/pull/3715) Also fixes two security advisories: - <GHSA-6xp4-jw73-f4qp>: fixed with [pi-hole/web@`d328f14`](https://redirect.github.com/pi-hole/web/commit/d328f143718022d82dc94c8751121ca41be3b996) - <GHSA-8rw8-vjgp-rwj6>: fixed with [pi-hole/web@`1a0c6f4`](https://redirect.github.com/pi-hole/web/commit/1a0c6f4fe6d0116fd2846b2adaae95996b7f194d) **Full Changelog**: <pi-hole/web@v6.4...v6.4.1>  #### What's Changed (Core v6.4) - Remove wget from alpine dependencies by [@darkexplosiveqwx](https://redirect.github.com/darkexplosiveqwx) in [pi-hole/pi-hole#6484](https://redirect.github.com/pi-hole/pi-hole/pull/6484) - Remove custom FTL FirewallD zone checks from debug log by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/pi-hole#6481](https://redirect.github.com/pi-hole/pi-hole/pull/6481) - Add Alpine 3.23 to test suite by [@yubiuser](https://redirect.github.com/yubiuser) in [pi-hole/pi-hole#6488](https://redirect.github.com/pi-hole/pi-hole/pull/6488) - Debug log - Add colors to gravity tables by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/pi-hole#6485](https://redirect.github.com/pi-hole/pi-hole/pull/6485) - Use configured location for web repo when updating or repairing by [@rrobgill](https://redirect.github.com/rrobgill) in [pi-hole/pi-hole#6470](https://redirect.github.com/pi-hole/pi-hole/pull/6470) - Add missing `-g` to the message in gravity recovery command by [@rdwebdesign](https://redirect.github.com/rdwebdesign) in [pi-hole/pi-hole#6513](https://redirect.github.com/pi-hole/pi-hole/pull/6513) - Don't install unused /usr/local/share/man/man5 by [@darkexplosiveqwx](https://redirect.github.com/darkexplosiveqwx) in [pi-hole/pi-hole#6526](https://redirect.github.com/pi-hole/pi-hole/pull/6526) **Full Changelog**: <pi-hole/pi-hole@v6.3...v6.4> </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/bigbeartechworld/big-bear-universal-apps).

rrobgill requested a review from a team as a code owner November 13, 2025 18:27

yubiuser reviewed Nov 14, 2025

View reviewed changes

DL6ER requested changes Nov 25, 2025

View reviewed changes

Comment thread src/tools/gravity-parseList.c

Comment thread src/tools/gravity-parseList.c Outdated

Comment thread src/tools/gravity-parseList.c

Comment thread src/tools/gravity-parseList.c Outdated

rrobgill force-pushed the gravs branch from 2e00dc4 to 7ea79b1 Compare November 25, 2025 22:28

rdwebdesign requested a review from DL6ER November 26, 2025 03:12

DL6ER requested changes Dec 1, 2025

View reviewed changes

Comment thread src/tools/gravity-parseList.c Outdated

Comment thread src/tools/gravity-parseList.c Outdated

rrobgill force-pushed the gravs branch from 7ea79b1 to 69d9f68 Compare December 1, 2025 20:44

DL6ER approved these changes Dec 2, 2025

View reviewed changes

DL6ER enabled auto-merge December 2, 2025 05:42

DL6ER merged commit aa5db6d into pi-hole:development Dec 2, 2025
13 checks passed

PromoFaux mentioned this pull request Feb 15, 2026

v6.5 #2776

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

update gravity - improve domain validation processing speed#2710

update gravity - improve domain validation processing speed#2710
DL6ER merged 1 commit into
pi-hole:developmentfrom
rrobgill:gravs

rrobgill commented Nov 13, 2025 •

edited

Loading

Uh oh!

yubiuser left a comment

Uh oh!

DL6ER left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rrobgill commented Dec 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rrobgill commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yubiuser left a comment

Choose a reason for hiding this comment

Uh oh!

DL6ER left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rrobgill commented Dec 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rrobgill commented Nov 13, 2025 •

edited

Loading