diff --git a/docs/modules/dns.md b/docs/modules/dns.md index 4321b29..cacc2f6 100644 --- a/docs/modules/dns.md +++ b/docs/modules/dns.md @@ -42,7 +42,10 @@ records), `mr/mod_dns_query.go` (tiny DNS client, statistics, post-apply read-ba a dnsmasq built with nftset: the image ships Alpine's `dnsmasq-dnssec-nftset` (DNSSEC is compiled in but off); plain `dnsmasq` refuses such a config (`dnsmasq --test` in the init script), so the apply rolls back. -Not included on purpose: adblock lists, DHCPv6-only (no SLAAC) mode (Android cannot use it), +- **Ad blocking** (`dns.adblock`, Cd1s/mini-router#30, off by default): blocklists as dnsmasq `local=` lines, no new + process (see below). + +Not included on purpose: AdGuard Home (≈34 MiB, killed by the OOM killer with big lists), DHCPv6-only (no SLAAC) mode (Android cannot use it), DHCPv6 lease release, MX records. Cost: no new daemon. Flash: a few KB of Go code in `mr`. RAM: none at rest; stubby (~2 MB RSS) only @@ -151,6 +154,13 @@ dns: private_relay: allow # allow (default) | block: mask.icloud.com / mask-h2.icloud.com → NXDOMAIN block_dot: false # true: LAN → port 853 (DoT / DoQ) refused doh_blocklist_file: "" # /etc/mini-router/dns/doh.ips: LAN → port 443 of these IPs / CIDRs refused + adblock: + enabled: false + lists: # https only; plain domains, hosts, adblock (||name^) or dnsmasq (local=/name/) lines + - https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/multi-onlydomains.txt # ~165k + # - https://anti-ad.net/domains.txt # ~110k, Chinese sites + allow: [example.com] # never blocked, with everything under it + max_domains: 300000 # default; an update above it keeps the previous lists ``` DNS sovereignty (`dns.sovereignty`). Encrypted DNS that bypasses the router doesn't break anything by @@ -172,6 +182,25 @@ fake-ip split (ECH does not matter here: the split works on the DNS name, not on then `mr apply` (or `mr fw`) loads it. Keep it under `/etc/mini-router/dns/` so backups include it. Agents and API tokens cannot set it (a `*_file` key). +**Ad blocking** (`dns.adblock`). `mr dns adblock update` downloads the lists (https only, the router's own +connection: not through the proxy), keeps valid names (at least two labels, no IPs; element-hiding rules, exceptions +and `server=/name/#` lines are not names), drops duplicates and names under another listed name, removes the allowed +ones, and writes `/etc/mini-router/state/adblock.conf`: one `local=/name/` per domain, plus `server=/name/#` for an +allowed name under a blocked one (forwarded as usual). Both dnsmasq instances load it (`conf-file=`): a blocked name +and everything under it answer NXDOMAIN. The lists' own hosts, the NTP servers and the local domain are never blocked. + +- An update never makes things worse: a list that fails to download, is larger than 64 MiB or does not look like a + domain list (under half of its lines usable, e.g. an HTML error page) keeps the previous file; so does a result + above `max_domains`. A new file restarts dnsmasq (and the proxy's dnsmasq): DNS pauses 1–2 s; if dnsmasq does not + answer again within 30 s the previous file is put back. +- Schedule: crond runs `mr dns adblock update --cron` hourly at a minute fixed per router; it downloads only when the + file is 20 h old or was built from other settings, so switching it on, or a failed update, is (re)tried within the + hour. Never while a change waits for confirmation. `mr dns adblock update` / 立即更新 runs it now. +- `mr apply` renders an empty file when there is none (dnsmasq refuses a missing `conf-file`); an existing list is + never part of a plan or a snapshot. +- Cost: no process. RAM: dnsmasq ≥ 2.86 keeps large `local=` sets compactly (CI prints dnsmasq's RSS with 150k names; + the proxy's instance holds a second copy). Flash: ~25 bytes a name before UBIFS compression, rewritten once a day. + Both dnsmasq instances (the main one and the proxy's) answer the NXDOMAIN names. The refusals are a chain of their own (`dns_guard`, prerouting priority mangle − 1) in front of the policy marks and the proxy's tproxy: a resolver inside a proxied range is refused too, not tunnelled. Only LAN-side bridges @@ -243,11 +272,14 @@ mr dns query NAME [TYPE] [SERVER] ask dnsmasq (or SERVER ip[#port]); TYPE A mr dns leases DHCP leases (JSON) mr dns release IP [MAC] make dnsmasq drop a DHCPv4 lease mr dns querylog on [MINUTES]|off|show temporary query logging (default 10 min, max 60) +mr dns adblock status ad blocking: domains, last update, per-list results (JSON) +mr dns adblock update [--cron] download the lists now (--cron: only when due; --no-reload: no restart) ``` Web UI actions (logged-in sessions only): `dns.stats` (GET), `dns.leases` (GET), `dns.release` (POST `{ip, mac}` — the pair must exist in the lease file and the IP must be inside a LAN network), -`dns.querylog` (GET = state + last 300 lines; POST `{on, minutes}`), `dnslist` (split list editor). +`dns.querylog` (GET = state + last 300 lines; POST `{on, minutes}`), `dnslist` (split list editor), `dns.adblock` +(GET = status; POST `{update: true}` starts an update in the background; API tokens: scope operate). No handler builds a shell command; `rc-service dnsmasq restart` is the only exec, with fixed argv. ## Checks @@ -265,6 +297,11 @@ No handler builds a shell command; `rc-service dnsmasq restart` is the only exec and → 443 of a listed resolver are refused (IPv4 and IPv6), DoT to a proxied range is refused before the proxy socket could take it, the router's own 853 and the resolver's other ports still work. - `mr/mod_dns_sovereignty_test.go`: defaults, validation, blocklist parsing (bad lines named, not echoed). +- `mr/mod_dns_adblock_test.go`: every list format, subdomain folding and allow, validation, render (empty file only + when missing), cron line; updates against a TLS test server: unchanged lists do not restart dnsmasq, `--cron` + skips a fresh file and rebuilds after a settings change, failed / HTML / oversized lists keep the file, a dnsmasq + that does not come back gets the previous file. `tools/ci.d/dns.sh` runs the real update against a local HTTPS + server (150k names) and the real dnsmasq answers them. - stubby 0.4.3 (Alpine 3.24) accepts the rendered stubby.yml (`stubby -C … -i`, checked once in an arm64 Alpine container; not part of CI because it needs network access). @@ -286,6 +323,10 @@ No handler builds a shell command; `rc-service dnsmasq restart` is the only exec 或全部走 DoT(需同时开启 stubby;NTP 域名会用“引导 DNS”明文解析,保证开机对时)。 - **DHCP 选项**:网络 › DHCP / IPv6 RA › DHCP 选项,按网络设置下发的 DNS、NTP、搜索域和其他编号选项。 - **IPv6**:同页 › IPv6 通告 (RA)。一般保持 SLAAC;需要固定段地址时选“有状态 DHCPv6 + SLAAC”。 +- **去广告**:网络 › DNS › 去广告 → 点“+ HaGeZi Multi NORMAL”(或填别的列表地址),打开“启用”,“保存并应用”, + 再点“立即更新”。约半分钟后状态显示拦截的域名数。某个网站被误拦:把它的域名加进“白名单”再保存。 +- **防止设备绕过路由器的 DNS**:网络 › DNS › 上游与缓存 › 防绕过。Firefox 金丝雀默认开;需要时打开“拦截 DoT / + DoQ”,或填一个 DoH 服务器 IP 列表文件。 - **看缓存命中率 / 排查解析**:网络 › DNS › 统计 / 查询日志。查询日志默认关闭,开启会重启 dnsmasq (约 1 秒),到时自动关闭,重启路由器也不会保留。 diff --git a/examples/lab.d/30-dns.yaml b/examples/lab.d/30-dns.yaml index 7855d95..6074357 100644 --- a/examples/lab.d/30-dns.yaml +++ b/examples/lab.d/30-dns.yaml @@ -65,3 +65,8 @@ dns: private_relay: block # iCloud Private Relay off on this network block_dot: true # LAN -> port 853 refused doh_blocklist_file: @REPO@/mr/testdata/doh.ips # LAN -> 443 of these resolvers refused + adblock: # ad blocking with dnsmasq lists, no new process (tools/ci.d/dns.sh serves these) + enabled: true + lists: ["https://127.0.0.1:8453/ads.txt", "https://127.0.0.1:8453/hosts.txt"] + allow: [cdn.ads.example] + max_domains: 200000 diff --git a/mr/api_token.go b/mr/api_token.go index 85bef4a..512caa4 100644 --- a/mr/api_token.go +++ b/mr/api_token.go @@ -72,7 +72,7 @@ var tokenActions = map[string]string{ "mon.now": "read", "mon.history": "read", "mon.devices": "read", "mon.conns": "read", "mon.procs": "read", "mon.dmesg": "read", "sys.services": "read", "sys.time": "read", "sys.doctor": "read", "sys.events": "read", "sys.edge": "read", // operate: runtime actions that do not change the config - "net.redial": "operate", "wifi.kick": "operate", "wifi.scan": "operate", "dns.release": "operate", + "net.redial": "operate", "wifi.kick": "operate", "wifi.scan": "operate", "dns.release": "operate", "dns.adblock": "operate", "proxy.delay": "operate", "proxy.select": "operate", "service": "operate", "diag": "operate", // apply: config changes, through plan → apply → confirm with automatic rollback "plan": "apply", "validate": "apply", "apply": "apply", "confirm": "apply", "revert": "apply", "rollback": "apply", diff --git a/mr/main.go b/mr/main.go index d4e795c..f4476d7 100644 --- a/mr/main.go +++ b/mr/main.go @@ -58,6 +58,7 @@ module commands (JSON output unless noted): mr wan status | health WAN state; multi-WAN health mr wifi status | stations | survey | scan PHY | kick MAC [IFNAME] mr dns stats | leases | query NAME [TYPE] [SERVER] | release IP [MAC] | querylog on [MIN]|off|show + | adblock status | update ad blocking lists (dns.adblock) mr proxy status | check | delay [NODE|GROUP] | select GROUP NODE | parse [--secrets] [FILE] | fetch [--secrets] URL|SUB mr mon now | history | devices | conns [JSON] | procs | dmesg mr sys run ACTION [TARGET] | backup [-secrets] FILE|- | restore [-confirm SECS] FILE | keys diff --git a/mr/mod_dns.go b/mr/mod_dns.go index 3a0cdf2..87f2c6c 100644 --- a/mr/mod_dns.go +++ b/mr/mod_dns.go @@ -85,8 +85,9 @@ type DNS struct { Records []Record `yaml:"records"` // local DNS records (home servers) Split []Split `yaml:"split"` // per-domain upstream (DNS 分流) - // keep LAN devices on the router's DNS (mod_dns_sovereignty.go) + // keep LAN devices on the router's DNS (mod_dns_sovereignty.go); ad blocking (mod_dns_adblock.go) Sovereignty DNSSovereignty `yaml:"sovereignty"` + Adblock Adblock `yaml:"adblock"` } // DoT is the stubby (DNS-over-TLS) forwarder. It listens on 127.0.0.1:Port only. @@ -267,7 +268,7 @@ func renderDnsmasq(c *Config) string { for _, l := range recordLines(c) { b.WriteString(l + "\n") } - for _, l := range dnsLocalOnly(c) { + for _, l := range append(dnsLocalOnly(c), adblockLines(c)...) { b.WriteString(l + "\n") } w("dhcp-leasefile=%s", LeaseFile) @@ -484,6 +485,7 @@ func init() { if _, _, err := dohBlocklist(c, true); err != nil { return err } + adblockRender(c, out) out.Add("/etc/dnsmasq.conf", 0644, renderDnsmasq(c)) if c.Services.Stubby.Enabled { out.Add(StubbyConf, 0644, renderStubby(c)) @@ -514,6 +516,8 @@ func init() { return "dnsmasq" case StubbyConf: return "stubby" + case adblockConf: + return "dnsmasq" case "/etc/resolv.conf": return "-" } @@ -532,6 +536,7 @@ func init() { "dns.leases": apiDNSLeases, "dns.release": apiDNSRelease, "dns.querylog": apiDNSQueryLog, + "dns.adblock": apiDNSAdblock, }, Commands: map[string]func(c *Config, args []string) error{ "dns": dnsCommand, @@ -747,6 +752,7 @@ func dnsValidate(c *Config, v *Validator) { } validRecords(c, v) sovereigntyValidate(c, v) + adblockValidate(c, v) for _, h := range c.DNS.AddnHosts { if !rePath.MatchString(h) { v.Add("dns.addn_hosts: absolute path required, got %q", h) diff --git a/mr/mod_dns_adblock.go b/mr/mod_dns_adblock.go new file mode 100644 index 0000000..1705ffa --- /dev/null +++ b/mr/mod_dns_adblock.go @@ -0,0 +1,560 @@ +package main + +// DNS ad blocking (dns.adblock, Cd1s/mini-router#30), off by default. No new process: `mr dns adblock +// update` downloads the lists (https only), keeps valid domain names, drops duplicates and names under +// another listed name, removes the allowed ones, and writes one `local=/name/` line per domain to +// adblockConf, which both dnsmasq instances (main, proxy) load. A blocked name and everything under it +// answers NXDOMAIN; an allowed name under a blocked one is forwarded as usual (`server=/name/#`). +// +// Updates never make things worse: a list that fails to download or does not look like a domain list +// (fewer than half of its lines usable) keeps the previous file, and so does a result above +// max_domains. After a new file dnsmasq (and the proxy's instance) restart — DNS pauses for a second +// or two — and if dnsmasq does not answer again, the previous file is put back. +// +// Schedule: crond runs `mr dns adblock update --cron` every hour. It only downloads when the file is +// 20 hours old or was built from other settings (lists, allow, max_domains), so the first download +// after switching it on, and a retry after a failure, come within the hour. Never while a change waits +// for confirmation. The router's own traffic does not go through the proxy. +// +// Cost: dnsmasq keeps roughly 100 bytes per domain and instance (HaGeZi Multi NORMAL, ~165k: ~16 MB, +// twice with the proxy's instance), the file ~25 bytes per domain on flash (UBIFS compresses it). + +import ( + "bufio" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "hash/fnv" + "io" + "net" + "net/http" + "net/url" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "time" +) + +type Adblock struct { + Enabled bool `yaml:"enabled"` + Lists []string `yaml:"lists"` // https URLs: plain domains, hosts, adblock (||name^) or dnsmasq lines + Allow []string `yaml:"allow"` // never blocked, with everything under them + MaxDomains int `yaml:"max_domains"` // default 300000 +} + +const ( + adblockDefaultMax = 300000 + adblockListBytes = 64 << 20 // per download + adblockMaxLists = 8 + adblockMaxAllow = 1000 + adblockAge = 20 * time.Hour +) + +// variables so tests can point them elsewhere +var ( + adblockConf = "/etc/mini-router/state/adblock.conf" + adblockStatus = RunDir + "/adblock.json" + adblockLock = RunDir + "/adblock.lock" + adblockHTTP = func() *http.Client { + return &http.Client{Timeout: 2 * time.Minute, CheckRedirect: func(r *http.Request, via []*http.Request) error { + if r.URL.Scheme != "https" || len(via) >= 5 { + return errors.New("redirect to " + r.URL.Scheme + " refused") + } + return nil + }} + } + adblockNow = time.Now + // adblockReload restarts the dnsmasq instances that load the file and waits until dnsmasq answers + adblockReload = func() error { + if err := restartDnsmasq(); err != nil { + return err + } + if _, err := run("rc-service", "mr-proxy-dns", "status"); err == nil { + run("rc-service", "mr-proxy-dns", "restart") + } + if !waitFor(time.Now().Add(30*time.Second), func() bool { + _, _, err := dnsQuery(dnsLocal, "localhost", dnsTypeA, dnsClassIN, time.Second) + return err == nil + }) { + return errors.New("dnsmasq does not answer") + } + return nil + } +) + +const adblockEmpty = "# mr dns adblock: no list yet (`mr dns adblock update`)\n" + +// adblockLines: what dnsmasq.conf (and the proxy's) gets while ad blocking is on. +func adblockLines(c *Config) []string { + if !c.DNS.Adblock.Enabled { + return nil + } + return []string{"conf-file=" + adblockConf} +} + +// adblockRender: the file must exist before dnsmasq starts with conf-file= (dnsmasq refuses a missing +// one); an existing file is left to the updater and never becomes part of a plan or a snapshot. +func adblockRender(c *Config, out *Out) { + if !c.DNS.Adblock.Enabled { + return + } + if _, err := os.Stat(adblockConf); err != nil { + out.Add(adblockConf, 0644, adblockEmpty) + } +} + +func adblockValidate(c *Config, v *Validator) { + a := &c.DNS.Adblock + if a.Enabled && len(a.Lists) == 0 { + v.Add("dns.adblock.lists: at least one list while enabled") + } + if len(a.Lists) > adblockMaxLists { + v.Add("dns.adblock.lists: at most %d", adblockMaxLists) + } + for i, l := range a.Lists { + if !adblockURLOK(l) { + v.Add("dns.adblock.lists[%d]: an https:// URL (no user, no spaces, at most 512 characters), got %q", i, l) + } + } + if len(a.Allow) > adblockMaxAllow { + v.Add("dns.adblock.allow: at most %d", adblockMaxAllow) + } + for i, d := range a.Allow { + if adblockName(d) == "" { + v.Add("dns.adblock.allow[%d]: a domain name like example.com, got %q", i, d) + } + } + if a.MaxDomains != 0 && (a.MaxDomains < 1000 || a.MaxDomains > 1000000) { + v.Add("dns.adblock.max_domains: 1000-1000000, got %d", a.MaxDomains) + } +} + +func adblockURLOK(s string) bool { + u, err := url.Parse(s) + return err == nil && len(s) <= 512 && u.Scheme == "https" && u.Hostname() != "" && u.User == nil && + !strings.ContainsAny(s, " \t\r\n\"'\\") && u.Fragment == "" +} + +// adblockName normalizes one name from a list: lower case, no leading "*." / "." or trailing ".", +// a valid DNS name with at least two labels that is not an IP address. "" = not usable. +func adblockName(s string) string { + s = strings.ToLower(strings.TrimSuffix(strings.TrimPrefix(strings.TrimPrefix(s, "*."), "."), ".")) + if !strings.Contains(s, ".") || !validDNSName(s) || net.ParseIP(s) != nil || s == "localhost.localdomain" { + return "" + } + return s +} + +// adblockLineNames: the names one list line blocks, nil when the line is not usable. Formats: a plain +// name (or *.name), hosts ("0.0.0.0 name …"), adblock ("||name^", "||name^$important"), dnsmasq +// ("local=/name/", "address=/name/…", "server=/name/"). +func adblockLineNames(l string) []string { + var raw []string + f := strings.Fields(l) + switch { + case strings.HasPrefix(l, "||"): + name, rest, ok := strings.Cut(l[2:], "^") + if !ok || (rest != "" && rest != "$important" && rest != "|") { + return nil + } + raw = []string{name} + case strings.HasPrefix(l, "local=/") || strings.HasPrefix(l, "address=/") || strings.HasPrefix(l, "server=/"): + key, rest, _ := strings.Cut(l, "=/") + parts := strings.Split(rest, "/") + // server=/name/1.2.3.4 forwards and server=/name/# is an exception: only an empty server= blocks + if len(parts) < 2 || (key != "address" && parts[len(parts)-1] != "") { + return nil + } + raw = parts[:len(parts)-1] + case len(f) >= 2 && net.ParseIP(f[0]) != nil: + raw = f[1:] + case len(f) == 1: + raw = f + default: + return nil + } + var out []string + for _, r := range raw { + n := adblockName(r) + if n == "" { + if r == "localhost" || r == "localhost.localdomain" || r == "local" || r == "broadcasthost" || + strings.HasPrefix(r, "ip6-") { // the standard lines at the top of a hosts file + continue + } + return nil + } + out = append(out, n) + } + if len(out) == 0 { + return nil + } + return out +} + +// adblockParse reads one list into set; good / total count the lines that are not comments. +func adblockParse(r io.Reader, set map[string]bool) (good, total int, err error) { + sc := bufio.NewScanner(r) + sc.Buffer(make([]byte, 64<<10), 1<<20) + for sc.Scan() { + l := strings.TrimSpace(sc.Text()) + if l == "" || l[0] == '#' || l[0] == '!' || l[0] == '[' { + continue + } + // a comment after the entry ("0.0.0.0 ads.example # tracker"); "example.com##.ad" is an + // element-hiding rule, not a name, and stays unusable + if i := strings.Index(l, " #"); i >= 0 { + l = strings.TrimSpace(l[:i]) + } else if i := strings.Index(l, "\t#"); i >= 0 { + l = strings.TrimSpace(l[:i]) + } + total++ + if strings.Contains(l, "#") && !strings.HasPrefix(l, "address=/") { // address=/name/# = the null address + continue + } + names := adblockLineNames(l) + if names == nil { + continue + } + good++ + for _, n := range names { + set[n] = true + } + } + return good, total, sc.Err() +} + +// under reports whether d is name or a name below it. +func under(d, name string) bool { return d == name || strings.HasSuffix(d, "."+name) } + +// adblockBuild: the names to block (sorted, none under another), and the allowed names that sit +// under a blocked one (forwarded as usual). allow also removes everything under it. +func adblockBuild(set map[string]bool, allow []string) (block, fwd []string) { + parentIn := func(d string, m map[string]bool) bool { + for p := d; ; { + i := strings.IndexByte(p, '.') + if i < 0 { + return false + } + p = p[i+1:] + if m[p] { + return true + } + } + } + allowed := map[string]bool{} + for _, a := range allow { + allowed[a] = true + } + kept := map[string]bool{} + for d := range set { + if allowed[d] || parentIn(d, allowed) || parentIn(d, set) { + continue + } + kept[d] = true + block = append(block, d) + } + sort.Strings(block) + for _, a := range dedup(allow) { + if parentIn(a, kept) { + fwd = append(fwd, a) + } + } + sort.Strings(fwd) + return block, fwd +} + +// adblockImplicitAllow: names the router itself needs: the lists' hosts (or updates would block +// themselves), the NTP servers, the local domain. +func adblockImplicitAllow(c *Config) []string { + var out []string + for _, l := range c.DNS.Adblock.Lists { + if u, err := url.Parse(l); err == nil && adblockName(u.Hostname()) != "" { + out = append(out, adblockName(u.Hostname())) + } + } + for _, h := range ntpHostnames(c) { + if n := adblockName(h); n != "" { + out = append(out, n) + } + } + if n := adblockName(c.DHCP.Domain); n != "" { + out = append(out, n) + } + return out +} + +// adblockConfigHash: what the file was built from (the header records it; --cron rebuilds on a change). +func adblockConfigHash(c *Config) string { + a := &c.DNS.Adblock + h := sha256.Sum256([]byte(strings.Join(a.Lists, "\n") + "\x00" + strings.Join(a.Allow, "\n") + "\x00" + strconv.Itoa(adblockMax(c)))) + return hex.EncodeToString(h[:6]) +} + +func adblockMax(c *Config) int { + if n := c.DNS.Adblock.MaxDomains; n > 0 { + return n + } + return adblockDefaultMax +} + +// adblockHeader is the file's first line: "# mr dns adblock: domains=N time=UNIX config=HASH". +type adblockHeader struct { + Domains int + Time int64 + Config string +} + +func adblockReadHeader() adblockHeader { + var h adblockHeader + f, err := os.Open(adblockConf) + if err != nil { + return h + } + defer f.Close() + line, _ := bufio.NewReader(f).ReadString('\n') + for _, kv := range strings.Fields(strings.TrimPrefix(line, "# mr dns adblock:")) { + k, v, _ := strings.Cut(kv, "=") + switch k { + case "domains": + h.Domains, _ = strconv.Atoi(v) + case "time": + h.Time, _ = strconv.ParseInt(v, 10, 64) + case "config": + h.Config = v + } + } + return h +} + +type adblockListResult struct { + URL string `json:"url"` + Domains int `json:"domains"` // usable lines + Lines int `json:"lines"` + Error string `json:"error,omitempty"` +} + +type adblockResult struct { + Time int64 `json:"time"` + OK bool `json:"ok"` + Domains int `json:"domains"` + Changed bool `json:"changed"` + Lists []adblockListResult `json:"lists"` + Error string `json:"error,omitempty"` +} + +// adblockFetch downloads and parses one list into set. +func adblockFetch(ctx context.Context, hc *http.Client, u string, set map[string]bool) adblockListResult { + r := adblockListResult{URL: u} + req, err := http.NewRequestWithContext(ctx, "GET", u, nil) + if err != nil { + r.Error = err.Error() + return r + } + req.Header.Set("User-Agent", "mini-router") + resp, err := hc.Do(req) + if err != nil { + r.Error = ddnsErrText(err) + return r + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + r.Error = "HTTP " + resp.Status + return r + } + one := map[string]bool{} + lr := &io.LimitedReader{R: resp.Body, N: adblockListBytes + 1} + good, total, err := adblockParse(lr, one) + switch { + case err != nil: + r.Error = ddnsErrText(err) + case lr.N <= 0: + r.Error = fmt.Sprintf("larger than %d MiB", adblockListBytes>>20) + case good == 0 || good*2 < total: + r.Error = fmt.Sprintf("not a domain list (%d of %d lines usable)", good, total) + } + r.Domains, r.Lines = good, total + if r.Error == "" { + for d := range one { + set[d] = true + } + } + return r +} + +// adblockUpdate is `mr dns adblock update [--cron] [--no-reload]`. +func adblockUpdate(c *Config, cron, reload bool) (*adblockResult, error) { + if !c.DNS.Adblock.Enabled { + return nil, errors.New("dns.adblock is off") + } + lk := flock(adblockLock, false) + if lk == nil { + if cron { + return nil, nil + } + return nil, errors.New("an update is already running") + } + defer lk.Close() + if err := pendingBlocks(); err != nil { + if cron { + return nil, nil + } + return nil, err + } + hdr, cfg := adblockReadHeader(), adblockConfigHash(c) + if cron && hdr.Config == cfg && adblockNow().Sub(time.Unix(hdr.Time, 0)) < adblockAge { + return nil, nil + } + res := &adblockResult{Time: adblockNow().Unix()} + defer func() { + b, _ := json.Marshal(res) + writeAtomic(adblockStatus, b, 0644) + }() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + set, hc := map[string]bool{}, adblockHTTP() + var failed []string + for _, u := range c.DNS.Adblock.Lists { + r := adblockFetch(ctx, hc, u, set) + res.Lists = append(res.Lists, r) + if r.Error != "" { + failed = append(failed, u+": "+r.Error) + } + } + if len(failed) > 0 { + res.Error = "kept the previous lists: " + strings.Join(failed, "; ") + logf("dns: adblock: %s", res.Error) + return res, errors.New(res.Error) + } + var allow []string + for _, a := range append(append([]string{}, c.DNS.Adblock.Allow...), adblockImplicitAllow(c)...) { + if n := adblockName(a); n != "" { + allow = append(allow, n) + } + } + block, fwd := adblockBuild(set, allow) + if len(block) > adblockMax(c) { + res.Error = fmt.Sprintf("kept the previous lists: %d domains, above max_domains %d", len(block), adblockMax(c)) + logf("dns: adblock: %s", res.Error) + return res, errors.New(res.Error) + } + var b strings.Builder + fmt.Fprintf(&b, "# mr dns adblock: domains=%d time=%d config=%s\n", len(block), res.Time, cfg) + b.WriteString("# generated by `mr dns adblock update` from dns.adblock — do not edit\n") + for _, a := range fwd { + b.WriteString("server=/" + a + "/#\n") + } + for _, d := range block { + b.WriteString("local=/" + d + "/\n") + } + os.MkdirAll(filepath.Dir(adblockConf), 0755) + old, _ := os.ReadFile(adblockConf) + body := func(s string) string { _, rest, _ := strings.Cut(s, "\n"); return rest } + res.Changed = body(string(old)) != body(b.String()) + if err := writeAtomic(adblockConf, []byte(b.String()), 0644); err != nil { + res.Error = err.Error() + return res, err + } + if res.Changed && reload { + if err := adblockReload(); err != nil { + writeAtomic(adblockConf, old, 0644) + adblockReload() + res.Error = "dnsmasq did not come back with the new lists (" + err.Error() + "); the previous ones are back" + logf("dns: adblock: %s", res.Error) + return res, errors.New(res.Error) + } + } + res.OK, res.Domains = true, len(block) + logf("dns: adblock: %d domains from %d lists%s", len(block), len(c.DNS.Adblock.Lists), map[bool]string{false: " (unchanged)"}[res.Changed]) + return res, nil +} + +// adblockState: what `mr dns adblock status` and the web UI show. +func adblockState(c *Config) map[string]any { + h := adblockReadHeader() + st := map[string]any{"enabled": c.DNS.Adblock.Enabled, "domains": h.Domains, "updated": h.Time, + "current": h.Config != "" && h.Config == adblockConfigHash(c)} + if b, err := os.ReadFile(adblockStatus); err == nil { + var last adblockResult + if json.Unmarshal(b, &last) == nil { + st["last"] = last + } + } + if lk := flock(adblockLock, false); lk == nil { + st["running"] = true + } else { + lk.Close() + } + return st +} + +// adblockCronLine: hourly (the update itself decides whether to download), at a minute fixed per router. +func adblockCronLine(c *Config) []string { + if !c.DNS.Adblock.Enabled { + return nil + } + h := fnv.New32a() + h.Write([]byte(c.System.Hostname + "|adblock")) + return []string{"# dns ad blocking (dns.adblock)", fmt.Sprintf("%d * * * * %s dns adblock update --cron", h.Sum32()%60, mrBin)} +} + +func adblockCommand(c *Config, args []string) error { + if len(args) == 0 { + return errors.New("usage: mr dns adblock status | update [--cron] [--no-reload]") + } + switch args[0] { + case "status": + b, _ := json.MarshalIndent(adblockState(c), "", " ") + fmt.Println(string(b)) + return nil + case "update": + cron, reload := false, true + for _, a := range args[1:] { + switch a { + case "--cron": + cron = true + case "--no-reload": + reload = false + default: + return fmt.Errorf("unknown option %q", a) + } + } + res, err := adblockUpdate(c, cron, reload) + if err == nil && res != nil && !cron { + fmt.Printf("%d domains blocked%s\n", res.Domains, map[bool]string{false: " (unchanged)"}[res.Changed]) + } + return err + } + return fmt.Errorf("unknown: mr dns adblock %s", args[0]) +} + +// apiDNSAdblock: GET = status; POST {"update": true} starts an update in the background. +func apiDNSAdblock(r apiReq) apiResp { + c, err := loadConfig(ConfigPath, SecretsPath) + if err != nil { + return errResp(500, "%v", err) + } + if r.method != "POST" { + return apiResp{body: adblockState(c)} + } + var in struct { + Update bool `json:"update"` + } + json.Unmarshal(r.body, &in) + if !in.Update { + return errResp(400, "update: true expected") + } + if !c.DNS.Adblock.Enabled { + return errResp(409, "ad blocking is off (dns.adblock.enabled)") + } + if err := pendingBlocks(); err != nil { + return errResp(409, "%v", err) + } + self, args := selfCmd("dns", "adblock", "update") + startDetached(self, args...) + return apiResp{body: map[string]any{"ok": true, "started": true}} +} diff --git a/mr/mod_dns_adblock_test.go b/mr/mod_dns_adblock_test.go new file mode 100644 index 0000000..883877c --- /dev/null +++ b/mr/mod_dns_adblock_test.go @@ -0,0 +1,279 @@ +package main + +import ( + "errors" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "reflect" + "sort" + "strings" + "sync/atomic" + "testing" + "time" +) + +// Every list format; comments, element-hiding rules, exceptions and IPs are not names. +func TestAdblockParse(t *testing.T) { + list := `# comment +! adblock comment +[Adblock Plus 2.0] +Ads.Example. +*.wild.example +.dot.example +||abp.example^ +||imp.example^$important +||third.example^$third-party +@@||exception.example^ +0.0.0.0 hosts.example other.example # two names +127.0.0.1 localhost +::1 ip6-localhost +local=/dm1.example/ +address=/dm2.example/0.0.0.0 +address=/dm3.example/# +server=/dm4.example/ +server=/fwd.example/1.1.1.1 +server=/exc.example/# +example.com##.banner +1.2.3.4 +com +bad_label-.example +` + set := map[string]bool{} + good, total, err := adblockParse(strings.NewReader(list), set) + var got []string + for d := range set { + got = append(got, d) + } + sort.Strings(got) + want := []string{"abp.example", "ads.example", "dm1.example", "dm2.example", "dm3.example", "dm4.example", "dot.example", + "hosts.example", "imp.example", "other.example", "wild.example"} + if err != nil || !reflect.DeepEqual(got, want) { + t.Errorf("names %v, want %v (%v)", got, want, err) + } + if good != 10 || total != 20 { + t.Errorf("good %d of %d", good, total) + } +} + +func TestAdblockBuild(t *testing.T) { + set := map[string]bool{"example.com": true, "ads.example.com": true, "x.y.example.com": true, "tracker.net": true, + "sub.tracker.net": true, "cdn.site.org": true, "img.cdn.site.org": true, "site.org": false} + block, fwd := adblockBuild(set, []string{"good.example.com", "site.org", "nothing.example"}) + // site.org is allowed: cdn.site.org and below are dropped; example.com covers its subdomains; + // good.example.com sits under a blocked name: forwarded as usual + if want := []string{"example.com", "tracker.net"}; !reflect.DeepEqual(block, want) { + t.Errorf("block %v, want %v", block, want) + } + if want := []string{"good.example.com"}; !reflect.DeepEqual(fwd, want) { + t.Errorf("fwd %v, want %v", fwd, want) + } +} + +func TestAdblockValidateRender(t *testing.T) { + c := testConfig(t) + c.DNS.Adblock = Adblock{Enabled: true} + errs := strings.Join(c.Validate(), "\n") + if !strings.Contains(errs, "dns.adblock.lists: at least one list while enabled") { + t.Errorf("no lists accepted:\n%s", errs) + } + c.DNS.Adblock = Adblock{Enabled: true, MaxDomains: 5, Allow: []string{"ok.example", "bad name", "com"}, + Lists: []string{"http://lists.example/a.txt", "https://user:pw@lists.example/a", "https://lists.example/a b", "https://lists.example/ok.txt"}} + errs = strings.Join(c.Validate(), "\n") + for _, want := range []string{"lists[0]: an https:// URL", "lists[1]: an https:// URL", "lists[2]: an https:// URL", + `allow[1]: a domain name`, `allow[2]: a domain name`, "max_domains: 1000-1000000"} { + if !strings.Contains(errs, want) { + t.Errorf("want %q in:\n%s", want, errs) + } + } + if strings.Contains(errs, "lists[3]") || strings.Contains(errs, "allow[0]") { + t.Errorf("a good entry refused:\n%s", errs) + } + + d := t.TempDir() + old := adblockConf + adblockConf = filepath.Join(d, "adblock.conf") + t.Cleanup(func() { adblockConf = old }) + c = testConfig(t) + if strings.Contains(renderDnsmasq(c), "conf-file") || len(adblockCronLine(c)) != 0 { + t.Error("ad blocking off: no conf-file line, no cron line") + } + c.DNS.Adblock = Adblock{Enabled: true, Lists: []string{"https://lists.example/a.txt"}} + mustValid(t, c) + line := "conf-file=" + adblockConf + "\n" + if !strings.Contains(renderDnsmasq(c), line) { + t.Error("dnsmasq.conf lacks the adblock conf-file") + } + if px, err := proxyDnsmasq(c, nil); err != nil || !strings.Contains(px, line) { + t.Errorf("proxy dnsmasq lacks the adblock conf-file (%v)", err) + } + files := func() map[string]string { + fs, err := Render(c) + if err != nil { + t.Fatal(err) + } + m := map[string]string{} + for _, f := range fs { + m[f.Path] = f.Data + } + return m + } + // missing: rendered empty (dnsmasq refuses a missing conf-file); present: left to the updater + if got, ok := files()[adblockConf]; !ok || got != adblockEmpty { + t.Errorf("missing file not rendered: %q", got) + } + os.WriteFile(adblockConf, []byte("local=/x.example/\n"), 0644) + if _, ok := files()[adblockConf]; ok { + t.Error("an existing list is part of the render (plans, snapshots)") + } + if serviceFor(adblockConf) != "dnsmasq" { + t.Error("restart mapping") + } + if l := adblockCronLine(c); len(l) != 2 || !strings.HasSuffix(l[1], " * * * * "+mrBin+" dns adblock update --cron") { + t.Errorf("cron line %v", l) + } + if cr := strings.Join(renderCronLines(c), "\n"); !strings.Contains(cr, "dns adblock update --cron") || !cronWanted(c) { + t.Errorf("crontab: %s", cr) + } +} + +type adblockEnv struct { + c *Config + hits *atomic.Int64 + reloads *int + lists map[string]string + fail *error +} + +func newAdblockEnv(t *testing.T) *adblockEnv { + t.Helper() + e := &adblockEnv{hits: &atomic.Int64{}, reloads: new(int), fail: new(error), lists: map[string]string{ + "/a.txt": "# list a\nads.example\nsub.ads.example\ntracker.example\n", + "/b.txt": "0.0.0.0 hosts.example\n0.0.0.0 cdn.site.example site.example\n", + }} + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + e.hits.Add(1) + body, ok := e.lists[r.URL.Path] + if !ok { + http.Error(w, "nope", 500) + return + } + w.Write([]byte(body)) + })) + t.Cleanup(srv.Close) + d := t.TempDir() + o1, o2, o3, o4, o5, o6 := adblockConf, adblockStatus, adblockLock, adblockHTTP, adblockReload, adblockNow + t.Cleanup(func() { + adblockConf, adblockStatus, adblockLock, adblockHTTP, adblockReload, adblockNow = o1, o2, o3, o4, o5, o6 + }) + adblockConf, adblockStatus, adblockLock = filepath.Join(d, "state", "adblock.conf"), filepath.Join(d, "run", "adblock.json"), filepath.Join(d, "run", "adblock.lock") + adblockHTTP = srv.Client + adblockReload = func() error { *e.reloads++; return *e.fail } + now := time.Unix(1800000000, 0) + adblockNow = func() time.Time { return now } + e.c = testConfig(t) + e.c.DNS.Adblock = Adblock{Enabled: true, Lists: []string{srv.URL + "/a.txt", srv.URL + "/b.txt"}, Allow: []string{"cdn.site.example"}} + mustValid(t, e.c) + return e +} + +func TestAdblockUpdate(t *testing.T) { + e := newAdblockEnv(t) + res, err := adblockUpdate(e.c, false, true) + if err != nil || !res.OK || res.Domains != 4 || !res.Changed || *e.reloads != 1 { + t.Fatalf("first update: %+v %v, %d reloads", res, err, *e.reloads) + } + data, _ := os.ReadFile(adblockConf) + want := "server=/cdn.site.example/#\nlocal=/ads.example/\nlocal=/hosts.example/\nlocal=/site.example/\nlocal=/tracker.example/\n" + if !strings.HasPrefix(string(data), "# mr dns adblock: domains=4 time=1800000000 config="+adblockConfigHash(e.c)+"\n") || !strings.HasSuffix(string(data), want) { + t.Errorf("file:\n%s", data) + } + st := adblockState(e.c) + if st["domains"] != 4 || st["current"] != true || st["last"].(adblockResult).Lists[1].Domains != 2 { + t.Errorf("state %+v", st) + } + // same lists: rewritten (header time), no restart + if res, err := adblockUpdate(e.c, false, true); err != nil || res.Changed || *e.reloads != 1 { + t.Errorf("unchanged update: %+v %v, %d reloads", res, err, *e.reloads) + } + // cron: fresh and same settings: nothing fetched + h := e.hits.Load() + if res, err := adblockUpdate(e.c, true, true); res != nil || err != nil || e.hits.Load() != h { + t.Errorf("cron on a fresh file fetched (%v %v)", res, err) + } + // cron: other settings (allow) rebuild at once + e.c.DNS.Adblock.Allow = append(e.c.DNS.Adblock.Allow, "tracker.example") + if res, err := adblockUpdate(e.c, true, true); err != nil || res == nil || res.Domains != 3 || *e.reloads != 2 { + t.Errorf("cron after a settings change: %+v %v", res, err) + } + // cron: a day later + adblockNow = func() time.Time { return time.Unix(1800000000+21*3600, 0) } + if res, _ := adblockUpdate(e.c, true, true); res == nil { + t.Error("cron did not update a day-old file") + } + good, _ := os.ReadFile(adblockConf) + + // a list that fails or is not a domain list keeps the previous file + e.lists["/b.txt"] = "\nrate limited\n\n" + if _, err := adblockUpdate(e.c, false, true); err == nil || !strings.Contains(err.Error(), "not a domain list (0 of 3 lines usable)") { + t.Errorf("HTML page: %v", err) + } + delete(e.lists, "/b.txt") + if _, err := adblockUpdate(e.c, false, true); err == nil || !strings.Contains(err.Error(), "HTTP 500") { + t.Errorf("HTTP 500: %v", err) + } + if now, _ := os.ReadFile(adblockConf); string(now) != string(good) { + t.Error("a failed update changed the file") + } + if st := adblockState(e.c); st["last"].(adblockResult).OK || st["last"].(adblockResult).Error == "" { + t.Errorf("failure not in the status: %+v", st["last"]) + } + // above max_domains + e.lists["/b.txt"] = "x.example\n" + e.c.DNS.Adblock.MaxDomains = 1000 + big := "" + for i := 0; i < 1001; i++ { + big += "n" + strings.Repeat("x", i%7) + time.Duration(i).String() + ".big.example\n" + } + e.lists["/a.txt"] = big + if _, err := adblockUpdate(e.c, false, true); err == nil || !strings.Contains(err.Error(), "above max_domains 1000") { + t.Errorf("max_domains: %v", err) + } + // dnsmasq does not come back: the previous file is put back and dnsmasq restarted again + e.lists["/a.txt"] = "new.example\n" + *e.fail = errors.New("dnsmasq does not answer") + r0 := *e.reloads + if _, err := adblockUpdate(e.c, false, true); err == nil || !strings.Contains(err.Error(), "the previous ones are back") { + t.Errorf("reload failure: %v", err) + } + if now, _ := os.ReadFile(adblockConf); string(now) != string(good) || *e.reloads != r0+2 { + t.Errorf("reload failure: file restored %v, %d reloads", string(now) == string(good), *e.reloads-r0) + } + // off: refused + e.c.DNS.Adblock.Enabled = false + if _, err := adblockUpdate(e.c, false, true); err == nil { + t.Error("update while off") + } +} + +// The list hosts, the NTP servers and the local domain are never blocked: the router needs them. +func TestAdblockImplicitAllow(t *testing.T) { + c := testConfig(t) + c.System.NTP = []string{"pool.ntp.org", "192.0.2.123"} + c.DNS.Adblock.Lists = []string{"https://cdn.jsdelivr.net/gh/x/y.txt"} + got := adblockImplicitAllow(c) + for _, want := range []string{"cdn.jsdelivr.net", "pool.ntp.org"} { + if !contains(got, want) { + t.Errorf("%s missing from %v", want, got) + } + } + block, _ := adblockBuild(map[string]bool{"jsdelivr.net": true, "ntp.org": true}, got) + if len(block) != 2 { + t.Errorf("parents of needed names stay blocked, the names are forwarded: %v", block) + } + _, fwd := adblockBuild(map[string]bool{"jsdelivr.net": true, "ntp.org": true}, got) + if !contains(fwd, "cdn.jsdelivr.net") || !contains(fwd, "pool.ntp.org") { + t.Errorf("fwd %v", fwd) + } +} diff --git a/mr/mod_dns_leases.go b/mr/mod_dns_leases.go index f782156..dd21722 100644 --- a/mr/mod_dns_leases.go +++ b/mr/mod_dns_leases.go @@ -353,7 +353,8 @@ const dnsUsage = `mr dns stats dnsmasq cache / upstream mr dns query NAME [TYPE] [SERVER] ask dnsmasq (or SERVER ip[#port]); TYPE A AAAA CNAME PTR SRV TXT mr dns leases DHCP leases (JSON) mr dns release IP [MAC] make dnsmasq drop a DHCPv4 lease -mr dns querylog on [MINUTES]|off|show temporary query logging (default 10 min, max 60, gone after reboot)` +mr dns querylog on [MINUTES]|off|show temporary query logging (default 10 min, max 60, gone after reboot) +mr dns adblock status | update [--cron] [--no-reload] ad blocking lists (dns.adblock)` func dnsCommand(c *Config, args []string) error { if len(args) == 0 { @@ -365,6 +366,8 @@ func dnsCommand(c *Config, args []string) error { return nil } switch args[0] { + case "adblock": + return adblockCommand(c, args[1:]) case "stats": st, err := queryDnsmasqStats(dnsLocal) if err != nil { diff --git a/mr/mod_proxy_render.go b/mr/mod_proxy_render.go index 9335789..6713b39 100644 --- a/mr/mod_proxy_render.go +++ b/mr/mod_proxy_render.go @@ -355,7 +355,7 @@ func proxyDnsmasq(c *Config, sets []proxyRuleSet) (string, error) { for _, h := range d.AddnHosts { w("addn-hosts=%s", h) } - for _, l := range dnsLocalOnly(c) { + for _, l := range append(dnsLocalOnly(c), adblockLines(c)...) { w("%s", l) } w("# local names (DHCP leases, static hosts, the router) come from the main dnsmasq") diff --git a/mr/mod_sys_cron.go b/mr/mod_sys_cron.go index a1d8601..c9a2285 100644 --- a/mr/mod_sys_cron.go +++ b/mr/mod_sys_cron.go @@ -16,6 +16,10 @@ package main // */ * * * * /usr/sbin/mr ddns sync --cron // M H * * * /usr/sbin/mr edge renew --cron // +// and while dns.adblock is on, the hourly list check (mod_dns_adblock.go; downloads once a day): +// +// M * * * * /usr/sbin/mr dns adblock update --cron +// // `mr sys run` checks the action against the live config again, logs it (syslog + change log) and // runs it. The time spec is a strict 5-field cron expression (numbers, *, a-b, /step, lists; no // names, no @reboot) with safety limits: every task runs at most once per hour (fixed minute), a @@ -196,7 +200,7 @@ func validateSchedules(c *Config, v *Validator) { } func cronWanted(c *Config) bool { - if ddnsInterval(c) > 0 || edgeOn(c) { + if ddnsInterval(c) > 0 || edgeOn(c) || c.DNS.Adblock.Enabled { return true } for _, s := range c.Schedules { @@ -230,7 +234,7 @@ func renderCronLines(c *Config) []string { if n := ddnsInterval(c); n > 0 { lines = append(lines, "# ddns (services.ddns)", fmt.Sprintf("*/%d * * * * %s ddns sync --cron", n, mrBin)) } - return append(lines, edgeCronLine(c)...) + return append(append(lines, edgeCronLine(c)...), adblockCronLine(c)...) } // renderCrontab: /etc/crontabs/root with the managed block (ok=false: leave the file alone). diff --git a/rootfs/www/ui/dns.js b/rootfs/www/ui/dns.js index 0180747..2a052a5 100644 --- a/rootfs/www/ui/dns.js +++ b/rootfs/www/ui/dns.js @@ -290,6 +290,46 @@ const statsTab = ()=>{ return box; }; +// 去广告(dns.adblock, #30):dnsmasq 列表,不新增进程 +const ADBLOCK_LISTS = [["HaGeZi Multi NORMAL(约 16 万,推荐)","https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/multi-onlydomains.txt"], + ["HaGeZi Multi LIGHT(约 4.5 万)","https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/light-onlydomains.txt"], + ["anti-AD(约 11 万,中文网站)","https://anti-ad.net/domains.txt"]]; +const adblockTab = ()=>{ + const d = S.cfg.dns; const box = h("div"); const stat = h("div"); + const ab = lazy(d, "adblock", ()=>({enabled:false, lists:[], allow:[], max_domains:0})); + const setLists = v=>{ ab.o.lists = v; ab.attach(); }; + const ta = h("textarea",{rows:4, placeholder:"每行一个 https 地址", oninput:e=>setLists(e.target.value.split(/\s+/).filter(Boolean))}, (ab.o.lists||[]).join("\n")); + const presets = h("div",{class:"row",style:"flex-wrap:wrap;gap:6px;margin-top:6px"}, ADBLOCK_LISTS.map(([l,u])=>h("button",{class:"btn sm",onclick:()=>{ + const cur = ab.o.lists||[]; if (!cur.includes(u)) { setLists([...cur, u]); ta.value = ab.o.lists.join("\n"); } + }},"+ "+l))); + const load = async ()=>{ + let j; try { j = await api("dns.adblock"); } catch(e){ stat.replaceChildren(h("span",{class:"mut"},e.message)); return; } + const last = j.last || null; + stat.replaceChildren(...[ + h("div",{}, h("b",{}, j.domains ? j.domains.toLocaleString()+" 个域名已拦截" : "还没有列表"), + j.updated ? h("span",{class:"mut"}," · 更新于 "+new Date(j.updated*1000).toLocaleString()) : null, + j.running ? h("span",{class:"tag warn",style:"margin-left:6px"},"更新中…") : null, + j.enabled && j.domains && !j.current ? h("span",{class:"mut"}," · 设置已改,一小时内按新设置重新下载") : null), + last && last.lists ? h("div",{class:"mut",style:"margin-top:6px;font-size:12px"}, last.lists.map(x=>h("div",{class:"mono"}, + (x.error ? "✗ " : "✓ ")+x.url+":"+(x.error ? x.error : x.domains.toLocaleString()+" 行可用")))) : null, + last && last.error ? h("div",{style:"color:var(--bad);margin-top:6px"}, "上次更新失败(仍用之前的列表):"+last.error) : null].filter(Boolean)); + }; + const upd = h("button",{class:"btn",onclick:async()=>{ + try { await api("dns.adblock",{update:true}); toast("已开始更新,约半分钟"); setTimeout(load, 4000); setTimeout(load, 15000); setTimeout(load, 40000); } + catch(e){ toast(e.message,4000); } + }},"立即更新"); + box.replaceChildren( + card("去广告", [h("div",{class:"dns-note"},"广告、跟踪、恶意域名直接返回“不存在”。不新增进程:列表变成 dnsmasq 的规则,每小时检查一次、满一天才重新下载;下载失败或内容不对会继续用旧列表。更新时 DNS 停顿 1–2 秒。列表经路由器自己的网络下载(不走代理)。"), + onEdit(form( + ...field("启用", inBool(ab.o,"enabled",()=>ab.attach())), + ...field("列表地址", h("div",{}, ta, presets), "纯域名、hosts、AdGuard(||域名^)或 dnsmasq 格式"), + ...field("白名单", inList(ab.o,"allow",{placeholder:"example.com, cdn.example.org"}), "这些域名及其子域名永不拦截"), + ...field("最多域名数", inNum(ab.o,"max_domains"), "0 = 300000;超过就不更新(保护内存)")), ()=>ab.attach())]), + card("状态", [stat, h("div",{class:"row",style:"margin-top:10px"}, upd, h("span",{class:"mut"},"启用并“保存并应用”后才能更新"))])); + load(); + return box; +}; + registerPage("network", "dns", "DNS", 50, ()=>tabs([ - ["basic","上游与缓存", basicTab], ["records","本地记录", recordsTab], ["split","分流", splitTab], ["stats","统计 / 查询日志", statsTab]])); + ["basic","上游与缓存", basicTab], ["records","本地记录", recordsTab], ["split","分流", splitTab], ["adblock","去广告", adblockTab], ["stats","统计 / 查询日志", statsTab]])); })(); diff --git a/tools/ci.d/dns.sh b/tools/ci.d/dns.sh index cb5d13b..366bdea 100644 --- a/tools/ci.d/dns.sh +++ b/tools/ci.d/dns.sh @@ -5,7 +5,10 @@ # 2. the rendered lab dnsmasq.conf runs for real in a throwaway network + mount namespace: # the lab's local records answer (`mr dns query`), the CHAOS statistics answer (`mr dns stats`), # and `mr dns release` makes dnsmasq drop a lease (DHCPRELEASE, lease file never edited); -# the Firefox canary and (lab) iCloud Private Relay names answer NXDOMAIN from the router itself. +# the Firefox canary and (lab) iCloud Private Relay names answer NXDOMAIN from the router itself; +# ad blocking: `mr dns adblock update` fetches the lab's lists from a local HTTPS server (plain and +# hosts format, 150k names) and the real dnsmasq answers them NXDOMAIN, an allowed name under a +# blocked one is forwarded; dnsmasq's RSS and start time with the list are printed. set -eu : "${OUT:?}" "${ROOT:?}" @@ -40,6 +43,40 @@ for d in $(sed -n 's/^interface=//p' "$CONF"); do done ip addr add "$ADDR" dev "$BR" echo "$(($(date +%s) + 3600)) 02:00:5e:00:00:01 $LEASEIP ci-host 01:02:00:5e:00:00:01" > /tmp/dhcp.leases +# the lab's ad blocking lists over HTTPS (a throwaway CA the mr client trusts through SSL_CERT_FILE) +mkdir -p /tmp/lists +openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 1 -subj /CN=127.0.0.1 \ + -addext subjectAltName=IP:127.0.0.1 -keyout /tmp/lists.key -out /tmp/lists.crt 2>/dev/null +{ + echo "# a plain domain list" + echo "ads.example" + echo "track.ads.example" + echo "*.wild.example" + echo "example.com##.banner" + seq 1 150000 | awk '{printf "ad%d.bulk%d.example\n", $1, $1 % 997}' +} > /tmp/lists/ads.txt +printf '127.0.0.1 localhost\n::1 ip6-localhost\n0.0.0.0 hosts-ad.example # tracker\n0.0.0.0 a.example b.example\n0.0.0.0 c.example\n' > /tmp/lists/hosts.txt +python3 - /tmp/lists <<'PY' > /dev/null 2>&1 & +import functools, http.server, ssl, sys +h = functools.partial(http.server.SimpleHTTPRequestHandler, directory=sys.argv[1]) +s = http.server.ThreadingHTTPServer(("127.0.0.1", 8453), h) +ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER); ctx.load_cert_chain("/tmp/lists.crt", "/tmp/lists.key") +s.socket = ctx.wrap_socket(s.socket, server_side=True) +s.serve_forever() +PY +lpid=$! +i=0 +until out=$(SSL_CERT_FILE=/tmp/lists.crt mr dns adblock update --no-reload 2>&1); do + i=$((i + 1)) + [ "$i" -lt 20 ] || { echo "FAIL: mr dns adblock update: $out"; exit 1; } + sleep 0.2 +done +kill $lpid +echo "$out" | grep -q '^150006 domains blocked$' || { echo "FAIL: adblock update: $out (150000 bulk + ads.example + wild.example + 4 from hosts)"; exit 1; } +grep -q '^server=/cdn.ads.example/#$' /etc/mini-router/state/adblock.conf || { echo "FAIL: allowed name not forwarded"; exit 1; } +if grep -q 'track.ads.example\|localhost\|example.com' /etc/mini-router/state/adblock.conf; then echo "FAIL: adblock file keeps a subdomain / localhost / an element-hiding rule"; exit 1; fi +echo "ok: $out" +t0=$(date +%s%N) dnsmasq --conf-file="$CONF" --keep-in-foreground --pid-file= --user=root --group=root --log-facility=/tmp/dnsmasq.log & pid=$! trap 'kill $pid 2>/dev/null || true' EXIT @@ -50,6 +87,7 @@ until mr dns query localhost A >/dev/null 2>&1; do [ "$i" -lt 50 ] || fail "dnsmasq never answered" sleep 0.1 done +echo "adblock: dnsmasq answered $((($(date +%s%N) - t0) / 1000000)) ms after start; $(grep VmRSS /proc/$pid/status | tr -s ' \t' ' ') with 150006 blocked names" check() { out=$(mr dns query "$1" "$2") || fail "query $1 $2" echo "$out" | grep -qF "$3" || fail "$1 $2: want $3, got: $out" @@ -70,6 +108,15 @@ for n in use-application-dns.net mask.icloud.com mask-h2.icloud.com; do echo "$out" | grep -q 'NXDOMAIN, 0 answers' || fail "$n: want NXDOMAIN, got: $out" echo "ok: $n -> NXDOMAIN" done +for n in ads.example x.track.ads.example ad777.bulk777.example deep.wild.example hosts-ad.example b.example; do + out=$(mr dns query "$n" A 2>&1) || true + echo "$out" | grep -q 'NXDOMAIN, 0 answers' || fail "blocked $n: want NXDOMAIN, got: $out" +done +for n in cdn.ads.example img.cdn.ads.example notlisted.example; do + out=$(mr dns query "$n" A 2>&1) || true + if echo "$out" | grep -q NXDOMAIN; then fail "$n is not blocked but answers NXDOMAIN"; fi +done +echo "ok: adblock: listed names and their subdomains NXDOMAIN; allowed and unlisted names forwarded" mr dns stats | grep -q '"cache_size": 8000' || fail "mr dns stats: $(mr dns stats 2>&1)" echo "ok: mr dns stats" grep -q " $LEASEIP " /tmp/dhcp.leases || fail "test lease not loaded" diff --git a/tools/ci.d/sys.sh b/tools/ci.d/sys.sh index d009c34..a682f33 100644 --- a/tools/ci.d/sys.sh +++ b/tools/ci.d/sys.sh @@ -73,7 +73,8 @@ grep -qx 'crond' "$L/etc/mini-router/gen/services" || fail "lab: crond not enabl if grep -qx 'crond' "$H/etc/mini-router/gen/services"; then fail "home: crond enabled without schedules"; fi grep -qx "export TZ='CET-1CEST,M3.5.0,M10.5.0/3'" "$L/etc/conf.d/crond" || fail "lab: crond zone" block=$(sed -n '/^# --- begin mini-router schedules/,/^# --- end mini-router schedules/p' "$C" | grep -v '^#') -[ "$(echo "$block" | wc -l)" = 7 ] || fail "lab crontab: want 5 jobs + the DDNS check + the certificate check, got: $block" +[ "$(echo "$block" | wc -l)" = 8 ] || fail "lab crontab: want 5 jobs + the DDNS check + the certificate check + the adblock check, got: $block" +echo "$block" | grep -qE '^[0-9]+ \* \* \* \* /usr/sbin/mr dns adblock update --cron$' || fail "lab crontab: hourly adblock check missing" echo "$block" | while read -r m hr d mo w cmd; do if [ "$cmd" = "/usr/sbin/mr ddns sync --cron" ]; then [ "$m $hr $d $mo $w" = "*/10 * * * *" ] || fail "DDNS check time spec: $m $hr $d $mo $w" @@ -83,6 +84,10 @@ echo "$block" | while read -r m hr d mo w cmd; do echo "$m $hr $d $mo $w" | grep -Eq '^[0-9]{1,2} [2-5] \* \* \*$' || fail "certificate check time spec: $m $hr $d $mo $w" continue fi + if [ "$cmd" = "/usr/sbin/mr dns adblock update --cron" ]; then + echo "$m $hr $d $mo $w" | grep -Eq '^[0-9]{1,2} \* \* \* \*$' || fail "adblock check time spec: $m $hr $d $mo $w" + continue + fi echo "$m $hr $d $mo $w" | grep -Eq '^[0-9]+ [0-9*/,-]+ [0-9*/,-]+ [0-9*/,-]+ [0-9*/,-]+$' || fail "bad time spec: $m $hr $d $mo $w" echo "$cmd" | grep -Eqx '/usr/sbin/mr sys run (reboot|restart [a-z0-9.-]+|reconnect [a-z0-9_-]+|wol [A-Za-z0-9:-]+)' || fail "unexpected crontab command: $cmd" done diff --git a/tools/mock/fixtures/dns.adblock.json b/tools/mock/fixtures/dns.adblock.json new file mode 100644 index 0000000..add6079 --- /dev/null +++ b/tools/mock/fixtures/dns.adblock.json @@ -0,0 +1,9 @@ +{ + "enabled": true, + "domains": 164218, + "updated": 1790248800, + "current": true, + "last": {"time": 1790248800, "ok": true, "domains": 164218, "changed": true, "lists": [ + {"url": "https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/multi-onlydomains.txt", "domains": 164218, "lines": 164218} + ]} +} diff --git a/tools/mock/fixtures/dns.adblock.post.json b/tools/mock/fixtures/dns.adblock.post.json new file mode 100644 index 0000000..a81eceb --- /dev/null +++ b/tools/mock/fixtures/dns.adblock.post.json @@ -0,0 +1 @@ +{"ok": true, "started": true}