unbound: merge extended blocklists into core (https://github.com/opnsense/core/issues/9136) #9301

swhite2 · 2025-10-20T09:14:54Z

TODO:

Discuss migration strategy, including post() usage to clean up config.
Improve domain tester policy matching for better traceability.

src/opnsense/mvc/app/controllers/OPNsense/Unbound/forms/dnsbl.xml

fichtner

looks good on the surface

swhite2 · 2025-10-21T10:03:26Z

One thing to note here is that with many blocklists, the matching algorithm has quite a performance penalty due to the subnet coverage calculation. This doesn't necessarily have to be a problem, as it's properly locked and is only for pre-processing, but it's slower than before.

swhite2 · 2025-10-23T14:51:34Z

Just to ease review a bit, the data path is as follows:

Input:

[blocklist:1]
description=default
rcode=NOERROR
address=0.0.0.0
source_nets=
cache_ttl=72000
excludes.default=.*localhost$
excludes.patterns=
includes.patterns=
includes.wildcards=

[blocklist:2]
description=<second>
rcode=NOERROR
address=0.0.0.0
source_nets=192.168.2.0/24,192.168.1.0/24
cache_ttl=72000
blocklists.<shortcode>=<URL> <-- only exists if one or more types set
blocklists.custom<loop.index>=<URL> <-- only exists if one or more lists set
excludes.default=.*localhost$
excludes.patterns=
includes.patterns=
includes.wildcards=

Output is a single JSON file with the following structure:

{
	"data": {
		"<domain>": [
			{"idx": <index into the first logical policy it should check},
			{"idx": ...}
			...
		],
	},
	"config": {
		"<idx>": {
			"passlist": <compiled passlist>,
			"source_nets": [
				"192.168.1.0/24",
				...
			]
			...
		}
	}
}

There may be multiple policies associated with a single domain, as source nets can differ, as well as the allowlist (passlist here) regex.
Config output is used directly in dnsbl_module.py and is returned when queried in test mode.
To keep the resulting output small, the indices are simple integers - we assume other blocklist handlers may be injected, therefore the index must be managed by the BlocklistParser class.
- We first merge all configurations, separated by index
- The handlers return their blocklists, which is an array of dictionaries, all of which may contain domains that overlap in the final blocklist.
- _merge_items_with_config merges the blocklists together and decides in what order the policies are attached to the domains -> more specific subnets have priority, and in that set the bigger passlists also have priority.

part 1: simplify blocklist handling. Introduce blocklists_iter() in BaseBlocklistHandler class which yields domain information per policy, which can then be reordered easily using the calculated priorities using source_nets in get_policies(). To prevent people from using overlapping nets or different protocol families in the same policy, we validate the input. Overlapping nets will lead to possible race conditions while matching as we cannot assume the first match found is actually the relevant one anymore. Our output data should match the expected format below: ``` { "data": { "<domain>": [ {"idx": <index into the first logical policy it should check}, {"idx": ...} ... ], }, "config": { "<idx>": { "passlist": <compiled passlist>, "source_nets": [ "192.168.1.0/24", ... ] ... } } } ``` This part only concers the parser with backwards compatibility (to keep QFeeds functional during the process) In the blocklist.conf template we changed the sequence index to use UUID's, which basically does the same thing as the index number, but is slightly better to track (used in the dnsbl.json output file as well).

Reorganize some parts so we can split this up more easily into separate components and fix one minor passlist regex issue.

Setup new unbound-dnsbl module which should replace dnsbl_module.py and tester (work in progress). In its current state this should contain all relevant classes and functions currently used in dnsbl_module.py with rudimentary unit tests.

move dnsbl_module into unbound-dnsbl directory and use library functions.

As getNodeContent() returns descriptive values as well (starting with %), there is no guarantee $bl->$key exists in the migration, check before calling applyDefault()

fix some scoping issues when dnsbl_module is used from unbound.

use /usr/local/opnsense/scripts/unbound-dnsbl/dnsbl_match.py for the tester.

…bnet coverage calculation

…ks in dynamic tooltips

part 1: simplify blocklist handling. Introduce blocklists_iter() in BaseBlocklistHandler class which yields domain information per policy, which can then be reordered easily using the calculated priorities using source_nets in get_policies(). To prevent people from using overlapping nets or different protocol families in the same policy, we validate the input. Overlapping nets will lead to possible race conditions while matching as we cannot assume the first match found is actually the relevant one anymore. Our output data should match the expected format below: ``` { "data": { "<domain>": [ {"idx": <index into the first logical policy it should check}, {"idx": ...} ... ], }, "config": { "<idx>": { "passlist": <compiled passlist>, "source_nets": [ "192.168.1.0/24", ... ] ... } } } ``` This part only concers the parser with backwards compatibility (to keep QFeeds functional during the process) In the blocklist.conf template we changed the sequence index to use UUID's, which basically does the same thing as the index number, but is slightly better to track (used in the dnsbl.json output file as well).

Reorganize some parts so we can split this up more easily into separate components and fix one minor passlist regex issue.

Setup new unbound-dnsbl module which should replace dnsbl_module.py and tester (work in progress). In its current state this should contain all relevant classes and functions currently used in dnsbl_module.py with rudimentary unit tests.

move dnsbl_module into unbound-dnsbl directory and use library functions.

As getNodeContent() returns descriptive values as well (starting with %), there is no guarantee $bl->$key exists in the migration, check before calling applyDefault()

fix some scoping issues when dnsbl_module is used from unbound.

use /usr/local/opnsense/scripts/unbound-dnsbl/dnsbl_match.py for the tester.

AdSchellevis · 2025-10-27T20:18:34Z

My changes are in, but can use an extra pair of eyes on the current state.

fichtner · 2025-10-28T15:38:22Z

src/etc/inc/plugins.inc.d/unbound.inc

    foreach ($dirs as $dir) {
        mwexecf('/bin/mkdir -p %s', "/var/unbound{$dir}");
    }
+    /* deploy python module (without test suite) */


Suggested change

/* deploy python module (without test suite) */

/* deploy python module (without test suite) */

fichtner · 2025-10-28T15:41:25Z

src/opnsense/mvc/app/models/OPNsense/Unbound/Unbound.php

+                );
+                if (count($sizes) > 1) {
+                    $messages->appendMessage(new Message(
+                        gettext("All offered networks should be equally sized to avoid overlaps"),


Suggested change

gettext("All offered networks should be equally sized to avoid overlaps"),

gettext('All offered networks should be equally sized to avoid overlaps.'),

Mostly about missing dot in sentence.

fichtner · 2025-10-28T15:41:37Z

src/opnsense/mvc/app/models/OPNsense/Unbound/Unbound.php

+                }
+                if (count($ipproto) > 1) {
+                    $messages->appendMessage(new Message(
+                        gettext("All offered networks should use the same IP protocol"),


Suggested change

gettext("All offered networks should use the same IP protocol"),

gettext('All offered networks should use the same IP protocol.'),

swhite2 · 2025-10-28T15:41:42Z

Apart from the extra review, points of interest here are:

We need to make sure we skip validation for items coming from the extended blocklists that do not fit the new validations.
The dnsbl.json file needs to be regenerated at some point. Likely needs a simple check in the Unbound code to reject the old list.

fichtner · 2025-10-28T15:42:30Z

src/etc/inc/plugins.inc.d/unbound.inc

+    foreach (new RecursiveIteratorIterator(new RecursiveDirectoryIterator($basedir)) as $filename) {
+        $target = "/var/unbound/unbound-dnsbl/".substr($filename, strlen($basedir));
+        if (str_ends_with($filename, '.py') && is_dir(dirname($target))) {
+            @copy($filename, $target);


I don't think we're making sure to clean /var/unbound/unbound-dnsbl so our future selves will be happy?

fichtner

looks ok, leaving the python parts for the other avid readers :)

…tion

swhite2 requested review from AdSchellevis and fichtner October 20, 2025 09:14

swhite2 self-assigned this Oct 20, 2025

fichtner reviewed Oct 21, 2025

View reviewed changes

src/opnsense/mvc/app/controllers/OPNsense/Unbound/forms/dnsbl.xml Outdated Show resolved Hide resolved

fichtner approved these changes Oct 21, 2025

View reviewed changes

swhite2 force-pushed the unbound_extdnsbl branch from 0e8b7b5 to db13acb Compare October 21, 2025 12:09

AdSchellevis force-pushed the unbound_extdnsbl branch from 12582c4 to 93f754e Compare October 26, 2025 10:23

AdSchellevis added a commit that referenced this pull request Oct 26, 2025

Services/Unbound/Blocklists - review #9301

5f9234a

Reorganize some parts so we can split this up more easily into separate components and fix one minor passlist regex issue.

AdSchellevis added a commit that referenced this pull request Oct 27, 2025

Services/Unbound/Blocklists - review #9301

12665a3

move dnsbl_module into unbound-dnsbl directory and use library functions.

AdSchellevis added a commit that referenced this pull request Oct 27, 2025

Services/Unbound/Blocklists - review #9301

6c2f240

move dnsbl_module into unbound-dnsbl directory and use library functions.

AdSchellevis force-pushed the unbound_extdnsbl branch from 12665a3 to 6c2f240 Compare October 27, 2025 17:30

AdSchellevis added a commit that referenced this pull request Oct 27, 2025

Services/Unbound/Blocklists - review #9301

6f0d25f

As getNodeContent() returns descriptive values as well (starting with %), there is no guarantee $bl->$key exists in the migration, check before calling applyDefault()

AdSchellevis added a commit that referenced this pull request Oct 27, 2025

Services/Unbound/Blocklists - review #9301

cb1f3ad

fix some scoping issues when dnsbl_module is used from unbound.

AdSchellevis added a commit that referenced this pull request Oct 27, 2025

Services/Unbound/Blocklists - review #9301

98a6670

use /usr/local/opnsense/scripts/unbound-dnsbl/dnsbl_match.py for the tester.

swhite2 and others added 11 commits October 27, 2025 20:43

unbound: merge extended blocklists into core (#9136)

9a98712

shorten this a bit

d8bb19e

unbound: fix OverviewController and sort module configuration elements

7c4df8b

unbound: make sure pass_regex exists

62c85b1

unbound: account for description in migration

a888870

unbound: make menu item plural

296e29d

unbound: rename whitelists to allowlists

de438e1

unbound: migration cleanup

38303db

unbound: wire up cache_ttl, re-order backend code and optimize the su…

eccf7e3

…bnet coverage calculation

unbound: format blocklist title rendering in grid and allow line brea…

d67f5de

…ks in dynamic tooltips

AdSchellevis added 6 commits October 27, 2025 20:43

Services/Unbound/Blocklists - review #9301

5a0a1ea

Reorganize some parts so we can split this up more easily into separate components and fix one minor passlist regex issue.

Services/Unbound/Blocklists - review #9301

fc7f61f

Setup new unbound-dnsbl module which should replace dnsbl_module.py and tester (work in progress). In its current state this should contain all relevant classes and functions currently used in dnsbl_module.py with rudimentary unit tests.

Services/Unbound/Blocklists - review #9301

57e6a74

move dnsbl_module into unbound-dnsbl directory and use library functions.

Services/Unbound/Blocklists - review #9301

53a12ec

As getNodeContent() returns descriptive values as well (starting with %), there is no guarantee $bl->$key exists in the migration, check before calling applyDefault()

Services/Unbound/Blocklists - review #9301

0e4f7d0

fix some scoping issues when dnsbl_module is used from unbound.

Services/Unbound/Blocklists - review #9301

a9114f0

use /usr/local/opnsense/scripts/unbound-dnsbl/dnsbl_match.py for the tester.

AdSchellevis force-pushed the unbound_extdnsbl branch from 98a6670 to a9114f0 Compare October 27, 2025 19:44

AdSchellevis requested a review from fichtner October 27, 2025 20:18

fichtner reviewed Oct 28, 2025

View reviewed changes

fichtner approved these changes Oct 28, 2025

View reviewed changes

swhite2 added 4 commits October 31, 2025 13:18

unbound: do not crash out on incompatible blocklist

6cb27f8

unbound: split unequal sized nets into multiple policies during migra…

c175502

…tion

unbound: simplify previous

2d3d987

unbound: safe init

a2133e7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

unbound: merge extended blocklists into core (https://github.com/opnsense/core/issues/9136) #9301

unbound: merge extended blocklists into core (https://github.com/opnsense/core/issues/9136) #9301

Uh oh!

swhite2 commented Oct 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

fichtner left a comment

Uh oh!

swhite2 commented Oct 21, 2025

Uh oh!

swhite2 commented Oct 23, 2025

Uh oh!

AdSchellevis commented Oct 27, 2025

Uh oh!

fichtner Oct 28, 2025

Uh oh!

fichtner Oct 28, 2025

Uh oh!

fichtner Oct 28, 2025

Uh oh!

swhite2 commented Oct 28, 2025

Uh oh!

fichtner Oct 28, 2025

Uh oh!

fichtner left a comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

	/* deploy python module (without test suite) */

	/* deploy python module (without test suite) */

	gettext("All offered networks should be equally sized to avoid overlaps"),
	gettext('All offered networks should be equally sized to avoid overlaps.'),

	gettext("All offered networks should use the same IP protocol"),
	gettext('All offered networks should use the same IP protocol.'),

unbound: merge extended blocklists into core (https://github.com/opnsense/core/issues/9136) #9301

Are you sure you want to change the base?

unbound: merge extended blocklists into core (https://github.com/opnsense/core/issues/9136) #9301

Uh oh!

Conversation

swhite2 commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

fichtner left a comment

Choose a reason for hiding this comment

Uh oh!

swhite2 commented Oct 21, 2025

Uh oh!

swhite2 commented Oct 23, 2025

Uh oh!

AdSchellevis commented Oct 27, 2025

Uh oh!

fichtner Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

fichtner Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

fichtner Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

swhite2 commented Oct 28, 2025

Uh oh!

fichtner Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

fichtner left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

swhite2 commented Oct 20, 2025 •

edited

Loading