Markus Stubbig

Practical OPNsense
Markus Stubbig

Practical OPNsense
Building Enterprise Firewalls with Open Source
Third version, May 2021

c 2021 Markus Stubbig

This book is available in print. ISBN: 978-3-75430-256-9

Editing by Kelda Neely

Cover, diagrams and screenshots by author unless otherwise credited.

All rights reserved. This book or any portion thereof may not be
reproduced or used in any manner whatsoever without the written
permission of the publisher except for the use of brief quotations in a book
review.
Trademark notice: Product or corporate names may be trademarks or
registered trademarks, and are used only for identification and explanation
without intent to infringe.
Contents

Preface xii

Introduction 17

I For Beginners 21

1 Lab Network 23
Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Lab Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2 Platform 31
Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
VirtualBox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3 Installation 45
Operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Post-installation tasks . . . . . . . . . . . . . . . . . . . . . . . . 48

v
4 Initial Setup 51
Initial setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Secondary setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Final testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5 IP Version 6 61
Crash course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Addresses and routes . . . . . . . . . . . . . . . . . . . . . . . . . 63
Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

II For Intermediates 67

6 Firewall 69
OPNsense as a firewall . . . . . . . . . . . . . . . . . . . . . . . . 70
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Firewall rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Best practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Additional filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 81
Order of processing . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

7 Transparent Firewall 85
Pros and cons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Filter operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Ruleset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

vi
 Uncover transparent firewall . . . . . . . . . . . . . . . . . . . . . 91
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 91
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

8 Network Address Translation 93

Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
NAT Reflection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 104
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

9 Management Interface 105

Two-factor authentication . . . . . . . . . . . . . . . . . . . . . . 111
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

III For Experts 115

10 IPsec VPN 117

Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Connection setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Address translation . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Dead Peer Detection . . . . . . . . . . . . . . . . . . . . . . . . . 126
IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
VPN throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 131
Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

11 OpenVPN 137
Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Differences to IPsec . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

vii
 Site-to-Site tunnel . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Client-server tunnel . . . . . . . . . . . . . . . . . . . . . . . . . 146
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 153
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

12 High Availability 155

Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Lab network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Address translation . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Best practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Quicker failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
IP version 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 170
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

13 NetFlow 173
The content of a flow . . . . . . . . . . . . . . . . . . . . . . . . . 173
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Insight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 178
IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

14 Web Proxy 181

Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Explicit proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Proxy cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
TLS Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Transparent proxy . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 198
Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

viii
 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

15 Central Authentication 201

Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Microsoft Server . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Directory-as-a-Service . . . . . . . . . . . . . . . . . . . . . . . . 211
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 222
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

IV For Hackers 225

16 Multi-WAN 227
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Load distribution in the WAN . . . . . . . . . . . . . . . . . . . . 229
Lab environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 239
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

17 DSL router 241

DSL types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
PPPoE Dial-in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
LAN adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
DNS and DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
IPv4 with Address Translation . . . . . . . . . . . . . . . . . . . . 249
IPv6 with prefix delegation . . . . . . . . . . . . . . . . . . . . . 249
Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 253
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254

ix
18 Intrusion Detection 255
IPS and IDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Network integration . . . . . . . . . . . . . . . . . . . . . . . . . 256
Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Activate IDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Activate IPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Transparent IDS . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 264
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266

19 Command Line 267

configd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Configuration changes . . . . . . . . . . . . . . . . . . . . . . . . 269
Undo changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

20 Performance Tuning 273

Lab setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Virtual network adapter . . . . . . . . . . . . . . . . . . . . . . . 275
Routing throughput . . . . . . . . . . . . . . . . . . . . . . . . . 278
IPsec throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
Increasing performance . . . . . . . . . . . . . . . . . . . . . . . 283
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

V For Admins 293

21 Best Practice 295

Factory reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
Benchmark throughput . . . . . . . . . . . . . . . . . . . . . . . . 296
SSH login without password . . . . . . . . . . . . . . . . . . . . . 298
Password reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301

x
 Contents

22 Configuration 305
Dropbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
Google Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

23 Life Hacks 315

Access from Windows . . . . . . . . . . . . . . . . . . . . . . . . 316
Span port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
Telegram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
Firewall rules with category . . . . . . . . . . . . . . . . . . . . . 320
Quick search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321

24 Application Programming Interface 323

How does the API work? . . . . . . . . . . . . . . . . . . . . . . . 323
Read Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
Write Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
What does the API cover? . . . . . . . . . . . . . . . . . . . . . . 330
API browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
Technical background . . . . . . . . . . . . . . . . . . . . . . . . 333
Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334

Bibliography 335

Index 339

A Editing Files in FreeBSD 347

B Pattern Matching 351

C Bonus Material 357

xi
Preface

Preface
OPNsense is 6 years old and has gradually grown up. There’s been no
quarrels with the code donor pfSense, instead OPNsense invests in better
security and code quality. Additionally, the web interface is now a linguist
and can handle ten languages.
OPNsense’s popularity is growing steadily: respectable computer magazines
favorably review the firewall and large IT companies offer services around
OPNsense. In Google Trends, OPNsense is getting closer and closer to its
predecessor pfSense.

The third edition of the book is dedicated to OPNsense’s success. All

chapters are tested with version 21.1. As expected, many restrictions have
been dropped because the developers of OPNsense always stay up-to-date
and react to security vulnerabilities in the shortest time possible.

Have fun reading and trying things out – and be ready for wonderful
surprises.

Preface of the first and second edition

OPNsense started its life as a bitchy little sister of pfSense who wanted to
be superior: better code, better security, better licensing, better targets –
and even better open source then its siblings!

With these grandstanding words OPNsense separated from pfSense in 2014.

The OPNsense developers started with a spring-cleaning of the pfSense
source code. They presented the first version of OPNsense at the beginning
of 2015: they tidied up all the code and added a modern web GUI without
changing the functionality.

After all the effort, did OPNsense actually make the cut and find friends?
If so, who are they? As it turns out, well-structured and documented
source code, as seen in OPNsense, is apparently a significant attribute for
an open-source firewall! And several celebrities from the security world
have complimented OPNsense, first and foremost of these being the chief
developer of monowall.

xii
 Overview

Probably every pfSense administrator has taken a brief look at OPNsense

and reviewed its differences. The OPNsense web interface appears in a
responsive design, while the known features from pfSense are accessible
only from swiveling menus. This improvement adds to the already positive
impression of OPNsense.

This book will show you how to operate OPNsense and the many fea-
tures which are all possible with this open-source firewall.

Enjoy reading and trying things out – and be ready for wonderful sur-
prises (and even a bit of cursing).

Overview
Part 1, For Beginners, sets up the network environment with physical de-
vices or on a virtual platform. All machines get an operating system and
a quick configuration, followed by essential functions, like routing and IPv6.

In part 2, For Intermediates, the firewalls fulfill some pressing tasks, which
must be present in every network. As a packet filter and address translator,
they will connect and isolate the attached subnets.

Part 3, For Experts, dives into enterprise-grade topics and establishes site-to-
site VPN tunnels and firewall clusters for high-availability. An in-depth look
inside the data flow provides good-old NetFlow. And the included proxy
server can even sniff inside TLS connections.

Outside the closed lab environment OPNsense acts in part 4, For Hack-
ers, as DSL router, load-balancer for multiple Internet links and even as
Sheriff for data trespassers.

Part 5, For Admins, provides many small hints that make daily work with
the firewall more fluent and straightforward. After that, OPNsense uploads
its configuration file to the cloud and stores it revision-safe on a DropBox
or Google Drive. Finally, let’s check out the programming interface of
OPNsense.

xiii
Resources

Resources
https://opnsense.org
The homepage of OPNsense offers a good start into the topic and links to
the official documentation, to the forum, and the download area.

https://github.com/opnsense
The source code is hosted at GitHub, where anybody can review the code
and its development process. It also offers the build tools and tutorials on
how to compile the code yourself.

https://docs.opnsense.org/
OPNsense for reading: manuals for user and developer, how-to documents
with many screenshots and step-by-step tutorials. Almost as comprehensive
as a full book.

https://forum.opnsense.org/
The forum is the first place to find small tutorials, discussions, and support
from the community. The language is not limited to English, and many
posts are in German.

Conventions
Constant Width regular shows the output of a command.

Typewriter font is used for configuration and keywords, and must be typed
exactly as shown.

Constant Width bold shows commands that expect some kind of output.

Accentuations indicate unique words or lines within a command or its

output.
a-very-long-command-string --with --many \
--many "options"
Commands with many arguments can take more space than fits into one
line. For a clear overview, these commands are printed in several lines

xiv
 Legal

indented by two spaces. At the end of the line is a backslash to indicate

that the command continues on the next line.

Legal
All terms mentioned in this book that are known to be trademarks or service
marks have been appropriately capitalized. The author cannot attest to
the accuracy of this information. Use of a term in this book should not be
regarded as affecting the validity of any trademark or service mark.

While every precaution has been taken in the preparation of this book,
the publisher and author assume no responsibility for errors or omissions,
or for damages resulting from the use of the information contained herein.

xv
Introduction

16
Introduction

OPNsense is an open-source network operating system for routers and

firewalls. It is based on HardenedBSD, derived from FreeBSD UNIX, and
contains such applications as Squid, pf, StrongSwan, and OpenVPN with a
consistent web interface. OPNsense runs on physical hardware, as a virtual
machine, or in the cloud.
Although it offers a wide range of functions, it has not yet become a well-
known brand. Even though it hits it out of the ballpark for its functionality
and usability. OPNsense combines the charm of UNIX with the functional
range of a professional firewall at a very low budget.

OPNsense is:

Evolving. And that’s said in a positive light because it means there is

room to grow. In addition, implementing features is sometimes out of the
ordinary: the provider-centric QinQ-tagging and VXLAN are included, but
IPv6 does need focus.

Open Source. The advantage of an open-source solution isn’t always

its price. There is no license cost involved, but its use requires time and
resources from an IT department to set up; and when it is finally set up, the
software may be poorly documented and not provide vendor support.
Thus, the main advantage of open source is the ability to detect unwanted
code. At the time of writing, it is rumored that the NSA will force vendors
to install backdoors in their security software. For a consumer who installs
a firewall system, that is almost impossible to discover. But it is a big
drawback when these firewalls are used in your own network.

17
Introduction

On the contrary, security experts can review open-source products and have
a good chance of finding malicious code. Furthermore, it is challenging
for vendors to install a backdoor in the source code if it is available for
everybody to read and analyze.

Try before Buy. You can (and should) evaluate OPNsense thoroughly
before spending money on infrastructure. That’s the same idea as with a
shareware application. Who is happy about limited functionality or a demo
license that expires after 30 days?
In this context, try means evaluating with sample scenarios and buy stands
for deployment in the local environment.

Hardware-independent. OPNsense is a software that requires some sort

of hardware or virtual infrastructure. Since there are many options, all of
which are acceptable, choosing the right one isn’t easy. In terms of require-
ments and desired characteristics, for example, which piece of electronics is
needed to saturate a 34 Mbps link with a VPN tunnel doing strong encryp-
tion? In the past, software-based network solutions could not keep up with
the performance of a hardware device. The main reason for this was the
terrible cooperation between software drivers and underlying hardware.
The choice of network adapters, mainboards, CPUs, and memory is virtually
unlimited, which makes it impossible for the software to get the maximum
performance out of every combination.
Nowadays, regular servers or embedded systems have surprisingly good
performance. Even with a non-optimized software and small packet sizes,
it is possible to break the bandwidth level of 100 Mbps.
The Dutch corporation Deciso [1] and its Netboard A10, A20 and the DEC
product series tackled the question about which hardware component is
best for the job. Optimization, adaptation and marketing have led to a
respectable firewall appliance.

UNIX. Within OPNsense is a customized HardenedBSD. Access to the

operating system is possible but protected by a password. Login is permitted
from the console menu or by an SSH connection. This flexibility allows you
to customize, enhance, or install additional tools. Be careful though, these
changes might also lead to unstable behavior.

18
 Introduction

Best Of. Although OPNsense doesn’t reinvent the wheel, it does imple-
ment many services from the UNIX and Linux domain. The software has
attained rock-solid stability after years of development. The web proxy
comes from Squid, the SSH server is a subsidiary of OpenSSH and the
firewall ruleset is the packet filtering engine pf from BSD.
Is use of OPNsense, an open-source software, theft of intellectual property?
Not at all! It simply proves that open source works. When license terms
are met, it is perfectly legal to integrate 3rd party software. It is highly
recommended, especially in the security world, that application developers
do not invent another crypto algorithm; but rather use stable free libraries.

History
The history of OPNsense is coupled with monowall and pfSense. At the
beginning of 2003, monowall started as a firewall which used FreeBSD
as its operating system. One year later, pfSense forked monowall with
the goal of being better. This approach worked well and in 2006 pfSense
outperformed its predecessor regarding functionality and popularity.
The concept behind pfSense and its development was successful. In the
following years, they released one version after the other.

The rivalry apparently ended in January 2014 when monowall published

its ultimate stable version. The project announced its end later in February
2015 and ceased development on its firewall software.
Later in 2014, the US enterprise Electric Sheep Fencing LLC offered com-
mercial support for pfSense and finally took over the firewall distribution.
This acquisition resulted in a license change, which made it difficult for
developers to get the source code.
This political change to pfSense and the downfall of monowall was the
main reason that some Dutch and German developers started their firewall
distribution as a fork of pfSense. Their aims were code quality, security,
transparency, and tight integration with the community. The title OPNsense
was intended as a reminder to its origin as pfSense.
The first version of OPNsense came out in January 2015 and was pfSense
code dressed up in a nice suit. The developers then began significant work
below the surface to replace version after version of pfSense code by their

19
Introduction

own code. Currently (2021), both firewall distributions hardly have any
code lines in common.
OPNsense publishes new releases on a precise semiannual basis. This
regular interval makes update schedules easy, and the community likes
this strategy. Indeed, critical security patches have been released in the
meantime, when necessary, so there is no need to wait for the next major
release.

OPNsense and pfSense compete with similar goals to win trust of its users.
Which distribution will win has not been decided yet: OPNsense with a
fresh start and clear goals or pfSense with long-time confidence, stability,
and a reputable name.

20
 Part I

For Beginners

21
Chapter 1

Lab Network

An OPNsense firewall would be useless without a surrounding network to

protect it. In practical terms, it is best to set up a separate lab network.
Within this environment, it is safe to experiment with the firewall and its
features without affecting any productive services.

All topics within this book have a practical background. Each chapter
begins with the basics to establish an understanding or to refresh dusty
knowledge. The examples and exercises are meant to be played with and
rebuilt.
The chapters are based on the exact same network diagram, which rep-
resents a small corporate network with two remote sites and redundant
wide area network (WAN) connections. Depending on the complexity of
the topic, it might be enough to use only a small part of the lab network.
Some chapters might have a different setup or an additional device. In that
case, this will be explained in detail at the beginning of the section.

Resources
Using the exact same setup for the lab network removes the need to modify
the infrastructure between chapters, i.e. there is no need to re-cable or
modify the virtual environment. This saves time and prevents error. And
after a few chapters, the lab network will become more and more familiar,
since the names of firewalls, clients, network adapters, and IP addresses

23
Chapter 1. Lab Network

remain unchanged. The complete network diagram is printed in Figure 1.1.

The following chapters will use only part of this network for demonstration
purposes.
There is no need to have physical access to the devices. It is even possible
to operate the lab remotely.

The choice for hardware components depends on the expected throughput.

More memory, CPU cores, and disk space will lead to a more powerful
firewall that can handle higher bandwidth and more user sessions.
The official specifications for memory and disk capacity take into account
bandwidth and activated features. The minimal requirements are low
enough to run the full lab on a laptop computer or on cheap hardware. For
example, an OPNsense firewall requires 2 GB memory and a 4 gigabyte
hard disk.

Despite the ability to run a lab on minimal requirements, more CPU cores,
memory space, and disk space are always encouraged. Table 1.1 gives
the specifications for different configurations. The numbers are based on
official documentation [2].

Specification Processor CPU cores Memory Disk size

Minimum 1 GHz 2 2 GB 4 GB
Reasonable 1 GHz 2 4 GB 40 GB
Recommended 1,5 GHz 2-4 8 GB 120 GB

Table 1.1: Hardware requirements for OPNsense

Some chapters use isolated networks, others need Internet access. The
path to the Internet is always through the firewall RT-core. The firewall
expects Internet access behind its network adapter em0. This is done with
a NAT adapter in a virtual environment. In a physical setup, connect the
network adapter to the DSL router. Any scenario that leads to the Internet
is welcome.

24
 Resources

CL-1
Debian 10
10.1.1.0/24 eth0 10.5.1.15
fd00:1::/64 eth1 00:15:16:11:00:15
Site-1 vnet1
10.1.1.25 00:15:16:11:01:25
DMZ 10.4.1.0/24
fd00:4::/64
vnet4
10.1.1.1 10.4.1.1 00:15:16:07:04:07 10.4.1.2 10.1.1.2
00:15:16:01:01:01 00:15:16:01:04:01 10.4.1.7 00:15:16:02:04:02 00:15:16:02:01:02

em1 em2 eth1 em2 em1

RT-1
OPNsense 21.1
em0 10.5.1.1
00:15:16:01:00:01
em4 em3 labsrv em3
RT-2
192.0.2.1 198.51.100.1 CentOS 8 198.51.100.2 OPNsense 21.1
00:15:16:01:06:01 00:15:16:01:07:01 eth0
eth0– 10.5.1.7 00:15:16:02:07:02 em0
em0– 10.5.1.2
00:15:16:07:00:07
00:15:16:02:00:02

198.51.100.6 WAN-1
00:15:16:06:07:06 198.51.100.0/24
em1 2001:db8:1::/64
WAN-2 vnet7
192.0.2.0/24 RT-core
2001:db8:2::/64 OPNsense 21.1
vnet6 em0 10.5.1.6
em2 00:15:16:06:00:06
192.0.2.6
00:15:16:06:06:06

192.0.2.3
00:15:16:03:06:03
em2
RT-3 CL-2
OPNsense 21.1
Windows 10
em0 10.5.1.3
10.5.1.25
00:15:16:03:00:03 00:15:16:12:00:25
em1 lan0
10.2.1.3 10.2.1.25
00:15:16:03:02:03 00:15:16:12:02:25

Site-2 10.2.1.0/24 fd00:2::/64 vnet2

Figure 1.1: The lab network is a template for all chapters

25
Chapter 1. Lab Network

Virtualization
It is possible to fully virtualize all devices used in the lab network. Each
firewall becomes a separate virtual machine (VM) with virtual network
cables connecting neighboring VMs. The interconnecting networks between
the VMs are VMnetX (at VMware) and vboxnetX (at VirtualBox). A physical
network adapter in the host system is only required when mixing the lab
with real gear.

Firewall Interface VMnet/vboxnet IPv4 IPv6

RT-1 em0 Management 10.5.1.1 fd00:5::1
em1 VMnet1 10.1.1.1 fd00:1::1
em2 VMnet4 10.4.1.1 fd00:4::1
em3 VMnet7 198.51.100.1 2001:db8:1::1
em4 VMnet6 192.0.2.1 2001:db8:2::1
RT-2 em0 Management 10.5.1.2 fd00:5::2
em1 VMnet1 10.1.1.2 fd00:1::2
em2 VMnet4 10.4.1.2 fd00:4::2
em3 VMnet7 198.51.100.2 2001:db8:1::2
RT-3 em0 Management 10.5.1.3 fd00:5::3
em1 VMnet2 10.2.1.3 fd00:2::3
em2 VMnet6 192.0.2.3 2001:db8:2::3
RT-core em0 Management 10.5.1.6 fd00:5::6
em1 VMnet7 198.51.100.6 2001:db8:1::6
em2 VMnet6 192.0.2.6 2001:db8:2::6
labsrv eth0 Management 10.5.1.7 fd00:5::7
eth1 VMnet4 10.4.1.7 fd00:4::7
CL-1 eth0 Management 10.5.1.15 fd00:5::15
eth1 VMnet1 10.1.1.25 fd00:1::25
CL-2 eth0 Management 10.5.1.25 fd00:5::25
eth1 VMnet2 10.2.1.25 fd00:2::25

Table 1.2: All firewalls with network adapters and VMnet/vboxnet

Table 1.2 lists which interface belongs to which virtual network. Though not
technically required, the network adapters of the VMs can use a predefined
MAC address to help interpret the output. This technique makes it easy to

26
 Hardware

recognize a device when reading a command output or when comparing it

with the samples in the book.
The Management network segment provides access to the firewall’s web
interface. The best-suited connection types are bridged and host-only. In
bridged mode, the VM is exposed to the surrounding network via the host’s
physical network adapter. In host-only mode, the VM is accessible only by
the host (and other VMs on the same host).
All labs are tested and validated with VMware Workstation 15.5, VMware
ESXi 7.0 and VirtualBox 6.1.

Hardware
OPNsense performs well on all devices with x86_64-type architecture. The
brand of the network adapter is not that important since the lab setup
is intended to demonstrate features and not to provide the best possible
performance. Check the FreeBSD hardware compatibility list [3] to see
compatible gear.

Networks
The network architecture between the firewalls is based on Ethernet. Every
subnet is its own broadcast domain. It is therefore important not to mix ca-
bles of different segments. Two methods are available to correctly separate
the subnets: by switches or by VLANs.

Separate by switches
Each network segment uses a separate network switch or hub. The switches
are not interconnected.
A smaller 5-port device is acceptable since the subnets are rather small. Any
brand and model should be sufficient for the job.

Separate by VLANs
All cables are connected to a single switch. Cables and switch ports that
belong to the same network segment are members in a common virtual LAN

27
Chapter 1. Lab Network

(VLAN). For example, all switch ports leading to the WAN-2 core network
will be a member of VLAN 6.
The switch device must have enough ports to provide connectivity for all
the firewalls. The switch is not required to provide routing between VLANs.
A regular VLAN capable layer 2 switch is adequate.

It is even possible to mix both modes. For instance, both WAN segments
could connect to one switch and the site subnets could connect to another
switch. The requirements correspond to the method Separate by VLANs.

Firewall
The OPNsense firewall uses the current stable version 21.1 as a 64-bit
image. If different versions or additional devices are included, then the lab
is modified by replacing the device in question.
Each firewall has one additional network adapter for management access.
That way, an SSH client will still reach the firewall even if some configura-
tion change has failed and the regular interfaces become inaccessible. If
the hardware does not provide an extra interface for device management,
it is acceptable to skip this option.
The lab firewalls are serially numbered. The device number is echoed in
the IPv4, IPv6, and MAC addresses. This allows easy device identification
in a command output listing.
The name of each network adapter is printed next to the device icon. The
full IPv4 address is added below. Information about IPv4 subnet and IPv6
prefix is presented at the network line icon.

Addressing
The subnetworks of the imaginary remote sites use private IPv4 addresses
and unique local IPv6 addresses. Each site has a client computer, which is
used only to validate a feature or generate traffic. The required command
set is limited to ping, traceroute, netstat and a web browser. Even the
choice of the operating system is irrelevant – the demo lab picks Debian,
CentOS and Windows due to their popularity.

28
 Lab Server

The area between the sites is the core network. Devices in this network use
the address ranges 192.0.2.0/24 and 198.51.100.0/24, which are reserved
for documentation (RFC 5737).
The addressing of IPv6 also uses two unequal prefixes to visually sim-
plify the differences: prefix fd00::/16 is used in site networks and prefix
2001:db8::/32 is used in the core network.

The address ranges are intended exactly for this purpose and do not collide
with public prefixes. Furthermore, the addressing is kept straightforward.
All ranges are structured uniformly and have only “regular” netmasks of
/24 (IPv4) or a prefix of /64 (IPv6).
Table 1.3 summarizes the IPv4 and IPv6 ranges attached to the VMnet net-
works. Additional addresses (e.g. for PPPoE, tunnel, CARP) are derived
from the same ranges.

vnet Purpose IPv4 IPv6

VMnet1 Site 1 10.1.1.0/24 fd00:1::/64
VMnet2 Site 2 10.2.1.0/24 fd00:2::/64
VMnet4 DMZ 10.4.1.0/24 fd00:4::/64
Management 10.5.1.0/24 fd00:5::/64
VPN (internal) 10.6.0.0/16 fd00:6::/64
VMnet6 WAN light gray/pink 192.0.2.0/24 2001:db8:2::/64
VMnet7 WAN dark gray/purple 198.51.100.0/24 2001:db8:1::/64
VPN 203.0.113.0/24 2001:db8:3::/64

Table 1.3: All virtual networks with IP ranges

Lab Server
The lab server provides infrastructure services. It can run on physical
hardware or as a virtual machine. If the OPNsense firewall is evaluated for
a client/server protocol, the lab server will be the counterpart. It can accept
requests from the firewalls on NTP, DNS, Syslog, FTP/TFTP, NetFlow and
HTTP. The deployed lab server runs on CentOS 8.

29
Chapter 1. Lab Network

Utilization
Each chapter uses a subset of the full lab network. Lesser devices provide
better control, simpler examples and briefer command output. This limi-
tation leads to a better overview. Feel free to insert additional firewalls to
dive deeper into features.

30
Chapter 2

Platform

The next step is all about setting up the lab components. It begins with the
creation or purchase of the equipment, followed by installation and finally
networking.
As mentioned in Chapter 1, the lab can run on physical hardware or find
its home entirely in a virtual environment. This makes a big difference in
the structure, but is irrelevant for the example scenarios in the following
chapters.
The installation procedure is the same for all methods: it first begins by
creating the virtual networks, which are separated either by a virtual switch
or a port group. The next step is to set up the virtual machines (VMs) and
finally the new VMs put their network adapters into the local VM networks.
The choice of virtualization software depends on your personal preferences.
The following explanations apply to VMware ESXi, Workstation and Player,
and VirtualBox.

This chapter cannot substitute as a reference manual for VMware or Vir-

tualBox! The installation of the VMs requires a basic knowledge of the
respective products. The descriptions only cover the installation of the new
VM and not why the individual steps are necessary or advantageous.

31
Chapter 2. Platform

Preparation
The firewall installation starts with a live image in ISO format. The website
of OPNsense [4] always offers the latest release for download.
The lab in this book uses version 21.1 and the ISO image
OPNsense-21.1-OpenSSL-dvd-amd64.iso

First, in a hardware lab, burn the ISO file to a DVD and start the server
or laptop from this medium. Devices with a 32-bit CPU are no longer
supported – the latest version for 32-bit devices is 20.1.

VMware
VMware has a wide range of products, but the main ones for the lab are
ESXi, Player, and Workstation. The lab begins with the setup of the virtual
networks.

Workstation Pro
VMware Workstation Pro is a software application for Windows and Linux
that supports virtual machines.
The configuration takes place in the Virtual Network Editor. If not already
present, create the virtual networks VMnet1 to VMnet7 there. Choose type
Host-Only and do not use DHCP. The subnet IP is insignificant since it is
not addressed in the lab.
Next, create the virtual machines. The procedure is always the same:
1. Start VMware Workstation
2. File → New Virtual Machine. . .
3. Type of configuration? Custom (advanced)
4. Hardware compatibility: pick the latest (e.g. Workstation 15.x)
5. Installer disk image file: select the ISO file from the previous section
Preparation
6. Virtual machine name: RT-1
Location: no preference

32
 VMware

7. Number of processors: 1
Number of cores per processor: 2
8. Memory for this virtual machine: 2 GB (or more)
9. Network connection: use host-only networking (the wizard allows
only one single network card, the others will follow later)
10. SCSI Controller: LSI Logic
11. Virtual disk type: SCSI
12. Disk: Create a new virtual disk
13. Maximum disk size (GB): 6
Store virtual disk as a single file
14. Disk File: no preference

The newly created machine will need a cleanup: a firewall does not need
a floppy drive, sound card, USB controller, and printer. However, it does
need more network adapters.
VM → Settings provides an insight into the soul of the virtual machine.
That’s the right place to add and delete objects until the settings fit. New
network adapters are always of type Custom with an assignment to the cor-
responding VMnet. The fixed MAC address is hidden behind the Advanced
button at Network Adapter.

The version used is VMware Workstation 15.5.6 on Windows.

Workstation Player
The software VMware Workstation Player is a reduced feature version of
VMware Workstation Pro. It is free of charge for non-commercial use.
Dialogues and procedures are similar, so the settings in the previous section
also apply here.
The properties of the virtual networks cannot be changed, but the default
setting is acceptable.
The creation of a VM begins with the button Create a New Virtual Machine.
Then some questions will follow concerning the installer image, the name,
and location of the VM, and finally, the size of the hard disk.

33
Chapter 2. Platform

All other details are adjusted outside the wizard. The same parameters
apply as for VMware Workstation.
The Linux version of the VMware Player is not suitable for the demo lab
because the dialog window does not show the selection of VMnet networks.
All host-only network adapters of the OPNsense firewall are in the same
network. Manual adjustments of VMnet networks and manipulation inside
the .vmx file of a VM do not yield any improvement.
The best alternative under Linux is VirtualBox (see section VirtualBox on
page 37).

The version used is VMware Workstation Player 16 on Windows.

ESXi
VMware ESXi is a Type 1 hypervisor. It does not run as an application on an
operating system, but works directly on the physical hardware. A graphical
web client creates and manages the virtual networks and machines. Inter-
nally, the virtual firewalls communicate with each other via virtual switches,
as illustrated in Figure 2.1 [5].

First, create a switch within the ESXi world. This switch later carries the
virtual networks with segmentation by VLANs. The principle corresponds to
the physical environment from section Hardware on page 41 in virtual form.

The virtual network environment starts in the navigator of the web client
under Networking in the register Virtual switches. If the lab is isolated
inside the ESXi, no physical network adapter is required. For everything
else, the following configuration requires the unused network card eth1,
which is listed as vmnic1 in ESXi.

• Click on Add standard virtual switch

• vSwitch Name: vSwitch1
Uplink 1: vmnet1 (that’s the unused network adapter in the server.
Leave the field empty if the lab does not need to communicate with
the outside world)
• Click on Add

34
 VMware

vFirewall
Virtual
Machine RT-2
em1 em2 em3 em0
Virtual
Network
adapter
vNIC1 vNIC2 vNIC3 vNIC0
Port group Port group Port group Port group
Port groups VMnet1 VMnet4 VMnet7 VM Network

Virtual
switches vSwitch1 vSwitch0
vSwitch0

vmnic1 vmnic0

Physical NICs
eth1 eth0
Figure 2.1: The interaction of the virtual components

After the new switch vSwitch1 is created, it will not yet have any VLANs
or ports. VLANs correspond to a port group at ESXi and are created and
assigned at the register Port groups. Click the button Add port group to
set up the first group. The VLAN number is important if the VMs are to
communicate with a physical network. VLANs 1401 to 1407 are used for
the demo lab.
• Name: VMnet1
• VLAN ID: 1401
• Virtual Switch: vSwitch1
This step is identical for the other networks VMnet2 to VMnet7. The virtual
network environment is then complete and should resemble the list in
Figure 2.2 on the following page. The scenario for creating virtual machines
starts in the navigator under Virtual Machines. The button Create / Register
VM triggers the wizard. It will ask several questions; answer them as
follows.

35
Chapter 2. Platform

Figure 2.2: Port groups divide the virtual networks

1. Select creation type: Create a new virtual machine

2. Select a name and guest OS:

• Name: RT-1
• Compatibility: latest
• Guest OS family: Other
• Guest OS version: FreeBSD (64-bit)

3. Select storage: pick the right datastore

4. Customize settings:

• CPU: 1
• Cores per Socket: 2
• Memory: 2 GB
• Hard disk 1: 6 GB
• SCSI Controller 1: LSI Logic Parallel

36
 VirtualBox

• USB Controller 1: (remove)

• Network Adapter 1: VM Network
• CD/DVD Drive 1: Host Device

Since OPNsense is installed four separate times, it is advisable to copy the

ISO file to the ESXi server datastore and mount from it. For the virtual
machine CD/DVD drive, select Datastore ISO File. Once using the file
browser, upload the .iso file, and then select it for the VM to use. The new
VM will then start from this DVD image. The live system will be booted and
will need to be installed as instructed in Chapter 3.

The newly created VM is still missing some network cards. Use the proper-
ties of the VM to add the network adapters and place them in the correct
VMnet. Table 1.2 on page 26 lists the affiliation of firewall interface to the
virtual network.

The version used is VMware ESXi 7.0b.

VirtualBox
VirtualBox is an application for Windows, Linux, and macOS that creates
and hosts virtual machines.
VirtualBox has a clear portfolio of products. It does not offer more than
one virtualization application, but it does provide several configuration
methods. The Oracle VM VirtualBox Manager is included in the normal
installation. It is easy to use but requires an X11 interface under Linux.
Alternatively (or complementary) phpVirtualBox supports the setup [6].
It is a web-based manager that provides the look and feel of the Oracle
manager inside the browser.
VirtualBox also offers a command line, so that the lab setup can be com-
pletely scripted.

vboxnet
The journey begins with the creation of the virtual networks vboxnet1 to
vboxnet7.

37
Chapter 2. Platform

Oracle VM VirtualBox Manager

With VirtualBox, the configuration of VMs and networks is controlled by
the same program.

1. File → Host Network Manager. . .

2. Click on the icon for Create for each Host-only Ethernet adapter
3. Click on the checkbox for DHCP Server (right side) to disable the
DHCP service for every network adapter

phpVirtualBox
The advantage of phpVirtualBox is that it provides a look and feel identical
to the operation of the VirtualBox Manager GUI. Therefore, the virtual
network set up is the same as the Oracle Manager setup described in the
previous section.

CLI
The command line expects a lot of typing work for one-time use. The
procedure differs between Linux and Windows:

Linux

1. Log-in to the Linux host system as vbox user

2. Create each vboxnet with the command
VBoxManage hostonlyif create

Windows

1. Start a command prompt and navigate to the path

%ProgramFiles%\Oracle\VirtualBox
2. Create each virtual host-only network with the command:
vboxmanage.exe hostonlyif create

38
 VirtualBox

Note
VirtualBox on Windows assigns the new network adapters a name of
“VirtualBox Host-Only Ethernet Adapter”, followed by a number.

The accuracy of the virtual networks’ setup can only be checked later when
the network is live and in use.

Virtual machines
The next step is to create the virtual machines. The process is similar for all
VMs, so the examples only show the steps from the first device.

Oracle VM VirtualBox Manager The setup in the management software

of VirtualBox is done via a wizard. The following description also applies
to phpVirtualBox.

1. Machine → New. . .

• Type is BSD, Version is FreeBSD (64-bit)

• Memory size: 2 GB (or more)
• Hard disk: 6 GB (or more)

2. After creating the VM, adjustments are still important so that the
network adapters play along in the right networks (Figure 2.3).

• Mount the DVD with the image from section Preparation (see
page 32)
• Network
– Declare adapter as Host-only Adapter
– Attach to vboxnetX (Linux) or “VirtualBox Host-Only Ether-
net Adapter X” (Windows)
– Under Advanced change the type to Intel PRO/1000 MT
Server. This is one of the most powerful network adapters
(see Chapter 20), but the others also work well.
• Adjust MAC address if desired (not mandatory)

39
Chapter 2. Platform

• Four NICs can be set up via the GUI. All others (RT-1 is the
only VM with a fifth NIC) via the command line, see section CLI
below.

Figure 2.3: Network properties of firewall RT-1 for VirtualBox

CLI The command line setup expects commands that correspond to the
GUI mouse clicks.

1. The journey via the command line path begins with the creation of a
virtual machine using the example of RT-1.

VBoxManage createvm --name "RT-1" --register

VBoxManage modifyvm RT-1 --memory 2048
VBoxManage modifyvm RT-1 --ostype "FreeBSD_64"

40
 Hardware

2. Attach a DVD drive with inserted DVD image from section Preparation.
VBoxManage storagectl RT-1 --name "IDE Controller" --add ide
VBoxManage storageattach RT-1 --medium \
OPNsense-21.1-OpenSSL-dvd-amd64.iso \
--storagectl "IDE Controller" \
--port 0 --device 1 --type dvddrive

3. Create a new hard disk and connect it.

VBoxManage storagectl RT-1 --name "SATA Controller" --add sata
VBoxManage createhd --filename "RT-1/RT-1.vdi" --size 6144 \
--format VDI --variant Fixed
VBoxManage storageattach RT-1 --storagectl "SATA Controller" \
--medium "RT-1/RT-1.vdi" --port 0 --type hdd

4. Setup and attach network card nic5 in host RT-1. This NIC must be
created via the CLI because the GUI is limited to four adapters.
1 VBoxManage modifyvm RT-1 --nic5 hostonly
2 VBoxManage modifyvm RT-1 --hostonlyadapter5 "vboxnet6"
3 VBoxManage modifyvm RT-1 --hostonlyadapter5 \
4 "VirtualBox Host-Only Ethernet Adapter #6"
5 VBoxManage modifyvm RT-1 --nictype5 82540EM
6 VBoxManage modifyvm RT-1 --macaddress5 001516010601

The instruction in line 2 is for VirtualBox on Linux and line 4 uses the
naming convention on Windows.
The first virtual firewall is now created and ready for installation. The
procedure for the remaining devices is identical, except for the network
adapters. To determine if everything is adequately connected, see Chapter 4
when the firewalls are assigned their IP addresses.

The used version is VirtualBox 6.1.

Hardware
The networks between the firewalls need a strict separation because many
protocols look for the best way on their own. And this runs – especially
with IPv6 – through unwanted paths if network separation is unclean.

41
Chapter 2. Platform

In the network diagram, each firewall adapter is visually connected to a

horizontal line which represents a network segment. Each segment is its
own switch or VLAN on a shared switch, as described in Chapter 1.

For example, the network segment with IPv4 range 192.0.2.0/24 contains
several firewalls and its network adapters:

• RT-1:em4
• RT-3:em2
• RT-core:em2

The adapters are connected to a 5-port switch. The switch has no other
connections or uplinks.
When using VLAN tagging, the cables of the firewall devices are connected
to a managed switch, for example on the first three switch ports. Listing 2.1
provides the configuration for a Cisco Catalyst switch. The OPNsense
firewalls do not notice this VLAN mapping. The configuration and setup of

vlan 1406
name WAN-2_192.0.2.0
!
interface range GigabitEthernet1/0/1 - 3
switchport mode access
switchport access vlan 1406

Listing 2.1: Network separation with switch ports and VLANs

the cable can only be validated later when the interfaces of the firewall are
equipped with IP addresses as outlined in Chapter 4, i.e. when the ping
command can detect errors.

Embedded systems
OPNsense is based on BSD but is only offered for AMD64 architectures.
Images for other platforms, such as ARM or MIPS, are not available.
The boot process on a regular PC or server with a keyboard, screen, and
DVD drive is simple: insert DVD and boot. The installation starts in Chap-
ter 3.

42
 Hardware

The installation becomes more difficult with embedded hardware because

the procedure depends on the physical components. Embedded devices are
minimalistic and therefore usually do not require a DVD drive or a display
port, and the operating system is stored on a flash medium.

A practical example is the APU 1D4 -Board [7] of the Swiss manufacturer
PC-Engines. It is equipped with three Gigabit network adapters, SD card,
serial console, and two USB slots.
The OPNsense repositories do provide a precompiled flash image, but the
recommended installation method is via a USB stick. The APU board boots
from this medium and starts the installation of the operating system. The
installation image for devices with serial console includes the keyword
serial in its file name.
1. First retrieve the image file from the OPNsense download server.
wget https://pkg.opnsense.org/releases/21.1/ \
OPNsense-21.1-OpenSSL-serial-amd64.img.bz2

2. When downloading via the Internet, it is advisable to validate the

result. This prevents transmission errors and intentional manipulation.
The repository provides a file signature that can be verified using the
onboard tools of a Linux system.
wget https://pkg.opnsense.org/releases/21.1/OPNsense-21.1.pub
wget https://pkg.opnsense.org/releases/21.1/ \
OPNsense-21.1-OpenSSL-serial-amd64.img.bz2.sig
openssl base64 -d -out /tmp/image.sig \
-in OPNsense-21.1-OpenSSL-serial-amd64.img.bz2.sig
openssl dgst -sha256 -verify OPNsense-21.1.pub -signature \
/tmp/image.sig OPNsense-21.1-OpenSSL-serial-amd64.img.bz2

3. Next unzip the file.

bunzip2 OPNsense-21.1-OpenSSL-serial-amd64.img.bz2

4. The file image

OPNsense-21.1-OpenSSL-serial-amd64.img
must then be copied to an empty USB stick so that it is bootable. The
physdiskwrite [8] tool performs this task on any Windows PC with
Administrator privileges.

43
Chapter 2. Platform

physdiskwrite.exe -u -d 1 \
OPNsense-21.1-OpenSSL-serial-amd64.img

Warning
physdiskwrite writes the image file to the specified drive with
the -d option without further prompt. A wrong specification can
lead to data loss.

5. A serial connection to the APU board is required. When the PC does

not have a serial connector, use a USB-to-serial converter dongle,
even if its initial setup is a hassle. Console software, such as TeraTerm
or PuTTY, need a speed of 115200 bps for communication with the
APU board.
6. Finally, the APU board boots from the newly created USB stick. Press
the F12 key to show the bootloaders start options. The USB stick is
listed as USB MSC Drive.

OPNsense is now running, but not yet installed. The installation on the SD
card starts in Chapter 3.

44
Chapter 3

Installation

The virtual networks have been set up and the machines are ready to go.
However, just like any regular computer, an operating system has to be
installed at the beginning. OPNsense does this the classic way: booting
from a DVD and starting the setup wizard.
This chapter deals with the installation of OPNsense on the (virtual) ma-
chine, as well as a minimal configuration for network connectivity.

Operating system
The system boots from the DVD or a USB stick and finally displays login: to
report the completed startup process of the live system. So far, the live DVD
has not installed anything. It has only started an OPNsense system from
the DVD. The installation begins after login with the user installer and the
password “opnsense”.
The installer asks the usual questions: 1) Where to install the operating
system? 2) How big should the partition be? and 3) Should there be a
stronger password for the login account?
Answer these questions with the default values. If the virtual machine
matches the recommendations in Chapter 2, the OPNsense installer will
find a hard drive that appears as da0 in the system. The last step initiates a
reboot and starts the fresh OPNsense installation from the disk.
The following section describes the installation steps that may differ from
the standard installation.

45
Chapter 3. Installation

A combination of two disks is prevalent for mitigating failures of a single

disk. Unfortunately, the installer does not automatically use both disks for
a mirrored disk setup. BSD uses the term GEOM mirror when referring to
mirroring multiple disks, so the installer expects the user to select Setup
GEOM mirror.
Next, the installer needs to know the primary disk, which is da0. It will
then query about the mirrored disk, which is da1. The GEOM mirror is
complete, and the installation can continue with Guided installation. The
subsequent question about the installation target is no longer a single hard
disk, but the entry mirror/OPNsense, as shown in Figure 3.1.

Figure 3.1: Installation of OPNsense on mirrored hard disks

If two different media are involved, only a mirrored drive will improve the
failover protection. For virtual machines, the provided hard disks should
not be located in the same datastore.

Note
The implementation of the following tips requires access to the web
GUI or the command line of the OPNsense firewall, which will be
presented in section Initial Setup of Chapter 4.

46
 Storage

Storage
If the OPNsense firewall runs on regular hard disks, no further adjustments
are necessary. A closer look is recommended if the storage device is a USB
stick, CompactFlash, SD card, or any kind of flash-based memory.
Flash memory allows a limited number of write cycles. Even with this
limitation, the flash memory and the firewall will have a longer life if fewer
cycles are written. If any type of flash-based memory is used as a storage
medium for OPNsense, there are optimization solutions for operating the
firewall as long as possible:

• Round-Robin Database. OPNsense stores system information, usage

of network adapters, traffic data, and then stores them in local files for
later access. For a firewall with high life expectancy, the Enables the
RRD graphing backend option under Reporting → Settings should
be unchecked.

• Memory file system. Variable and temporary files are also located in
the local file system under /var and /tmp. These directories could be
implemented as RAM disks. Thus their contents are no longer on the
storage medium but in memory.
Go to System → Settings → Miscellaneous, find the area Disk /
Memory Settings and enable the memory file system.

Warning
Since the files are in volatile memory, their contents will disap-
pear after a reboot (or power failure).

This change requires a subsequent restart of the system.

• Logging. A firewall with many clients writes many log messages that
perform write cycles to the storage medium. To protect the medium,
log entries can remain in the memory or get discarded immediately.
The procedure is the same as for the Memory file system. If the log
entries are of no interest, switch off logging completely. The verbatim
option is called Disable writing log files to the local disk and is hidden
in System → Settings → Logging.

47
Chapter 3. Installation

If the log files are required for verification or proof, this option should
not be used. OPNsense also allows you to send log entries to a remote
syslog server.

Post-installation tasks

OPNsense is installed, but a few details need adjustment to create a smooth

workflow. Those adjustments include VMware tools, keyboard layout, and
sound.

VMware tools

In the virtual environment, VMware tools usually gain better performance

and are even necessary for management and backup. Fortunately, OPNsense
offers the VMware tools as a plug-in os-vmware. The tools are installed
from the web interface under System → Firmware → Plugins. Without
a graphical user interface, the setup runs from the command line and is
described in section Packages from Chapter 19 on page 272.

Keyboard layout

The keyboard uses the US layout by default. Other countries have a dif-
ferent layout which might have an adverse effect on many commands. It
also becomes difficult to type IPv6 addresses with colons and prefixes with
slashes when the used keyboard has these keys at a different position.

If your keyboard differs from the US layout, you can switch to a different
one. The command shell behind the menu item 8) Shell can be used to
define the keyboard mapping inside the file /etc/rc.conf. For example,
a German keyboard requires a change to the variable keymap with a text
editor (see Appendix A) or via the single command:

echo ’keymap="de"’ >> /etc/rc.conf

48
 Post-installation tasks

System sounds
OPNsense emits squeaky tones after system startup and before shutdown.
If this startup tone is unwanted or does not meet your personal musical
taste, it can be switched off in the System Sounds area of System →
Settings → Miscellaneous. Select and save with the option Disable the
startup/shutdown beep.
A system restart is not necessary but shows the quiet reboot of OPNsense.

49
Chapter 3. Installation

50
Chapter 4

Initial Setup

The OPNsense firewalls are now installed and require initial configuration.
This chapter starts with an introduction to the menu structure of the com-
mand line and basic setup for the web interface. After completing the initial
configuration, the firewalls are equipped with IP addresses to contact each
other.
This chapter uses a high-level approach to complete the basic configuration.
For a detailed step-by-step guide with many illustrations, please refer to the
OPNsense website.

The OPNsense console menu is an acceptable tool for assigning IP ad-

dresses and troubleshooting in case the web interface is not available. The
web GUI is much more versatile and convenient for configuring network
adapters and IP addresses. Therefore, the OPNsense console menu is only
used for setting up the management adapter and the second mandatory
adapter.
Figure 4.1 on the next page shows the OPNsense console menu and its
choices.

Use a US keyboard layout to enter the IPv4 or IPv6 address, otherwise

it may be difficult to enter the address correctly. It is therefore recom-
mended to switch to a more convenient keyboard layout as described in
section Keyboard layout on page 48.

51
Chapter 4. Initial Setup

Figure 4.1: The console interface of OPNsense

Initial setup

The initial setup provides enough information to allow remote login. The
firewall can then be easily accessed with a web browser to begin the detailed
configuration work. The assignment of network adapters and IP addresses
is a question-and-answer process.

Defaults

Without further request, OPNsense uses the first network adapter as its
LAN interface, which connects to a trusted network. The second adapter
immediately becomes the WAN interface, with contact to the Internet or
another hostile network. Table 4.1 summarizes the default settings.

Interface Adapter IPv4 IPv6

LAN em0 192.168.1.1/24 Track WAN IPv6
WAN em1 DHCP DHCPv6

Table 4.1: Default settings of both network adapters

OPNsense permits access to the web interface only via the LAN interface.
For this reason, the first network adapter of any firewall in the laboratory
network becomes the management interface.

52
 Initial setup

Assigning the network adapter

OPNsense uses the FreeBSD naming convention to refer to the network
adapters, which is based on vendor and driver. An Intel network card is
called em0 or em1. If VirtualBox uses the paravirtualized adapter, the first
network card will later be called vtnet0.
Table 4.2 lists the most common FreeBSD names.
Driver Description Usage
vtnet VirtIO Ethernet driver VirtualBox virtio-net
em Intel(R) PRO/1000 GigE VMware and VirtualBox
vmx VMware VMXNET VMware Workstation and ESXi
le AMD Am7900 LANCE and VMware Workstation as
Am79C9xx ILACC/PCnet PCnet-PCI II
pcn AMD PCnet/PCI FastEthernet VMware Workstation as
PCnet-FAST III
re RealTek PCI/PCIe Ethernet APU 1D4 (see
Embedded systems, p. 42)

Table 4.2: Default names of network adapters for OPNsense/FreeBSD

If the assignment of the network adapters is reversed or does not fit a

desired plan, the menu option 1) Assign interfaces changes the order. The
first question-answer round begins and expects the specification of LAN
and WAN interfaces. The following abbreviated text output shows the use
of em0 as a LAN interface.
Valid interfaces are:
em0 00:15:16:01:00:01 Intel(R) PRO/1000 Network Connection
em1 00:15:16:01:01:01 Intel(R) PRO/1000 Network Connection
em2 00:15:16:01:04:01 Intel(R) PRO/1000 Network Connection
em3 00:15:16:01:07:01 Intel(R) PRO/1000 Network Connection
em4 00:15:16:01:06:01 Intel(R) PRO/1000 Network Connection

You now have the opportunity to configure VLANs. If you don’t

require VLANs for initial connectivity, say no here and use the
GUI to configure VLANs later.

Do you want to set up VLANs now? [y/N]: n

Enter the WAN interface name or ’a’ for auto-detection: em1

53
Chapter 4. Initial Setup

Enter the LAN interface name or ’a’ for auto-detection

NOTE: this enables full Firewalling/NAT mode.
(or nothing if finished): em0

The interfaces will be assigned as follows:

WAN -> em1

LAN -> em0

Do you want to proceed? [y/N]: y

Assigning IP addresses

Now the network adapters are ready to use and require an IP address.
The first IP address of the firewall must be assigned via the console menu.
For all other network adapters, the simpler web interface method will be
available later. The setup procedure is the same for all interfaces.
It starts in the console menu with 2) Set interface IP address. The subse-
quent series of questions refer to interface number 1 (LAN) and expects the
management address to be specified, as listed in Table 1.2 on page 26. All
questions about DHCP can be confidently answered with No. All settings
for IPv6 remain empty for the time being because this will be the main
topic in Chapter 5.

Secondary setup

The secondary setup requires no further steps on the command line because
the new OPNsense machine should be accessible via the management
address. If the machine is accessible from a local computer, point a web
browser to the IPv4 address 10.5.1.1 of firewall RT-1 and wait for the usual
login page to appear. The initial login credentials of username root and
password opnsense will provide access to the web interface for the next
configuration tasks.

54
 Secondary setup

Security
The first step is to choose a better password. Navigate to Lobby → Pass-
word and enter a strong password that can withstand a dictionary attack.
OPNsense versions 17.7 and later include this step during installation of
the operating system.
Next, review the software version; a firewall is a security-critical net-
work element and always requires the most recent version. At System
→ Firmware → Updates OPNsense shows which versions are available
and may offer an update. Armed with a strong password and up-to-date
software, you can now start the routine work.

Miscellaneous
At System → Settings → General OPNsense requires the usual information
about the hostname, domain, DNS server, and time zone. Thoroughly
examine the other areas below System → Settings; it will provide tools
needed for logging, notification, hardware customization, and accessing
the firewall via the web, SSH, and console.
The system setup is pretty standard in the market and does not differ signifi-
cantly from comparable firewalls of other manufacturers.
For orientation, Table 4.3 on the following page shows where the common
parameters are located in the OPNsense web interface.

Network card
Initially, OPNsense uses only the first two network adapters that appear in
the web menu at Interfaces as LAN and WAN. All other network cards must
be created manually at Interfaces → Assignments.
Firewall RT-1 has three additional adapters available, which are activated
next to the string New interface in Figure 4.2 on the next page by clicking
on the orange plus button. The new interface has the tedious name OPT1
and is available for its configuration. The adapter is now listed by name
under Interfaces. In this area, all network adapters receive their static IPv4
address and subnet mask.
This section is also the right place to manipulate Maximum Transmission
Unit (MTU), MAC address, speed, and duplex.

55
Chapter 4. Initial Setup

Topic is located at:

Login System → Access → Users
DNS System → Settings → General
Domain System → Settings → General
Hardware System → Settings → Miscellaneous
Hostname System → Settings → General
Console System → Settings → Administration
Kernel System → Settings → Tunables
Logging System → Settings → Logging
Monitoring Services → Monit → Settings
Reboot Power → Reboot
Language System → Settings → General
SSH System → Settings → Administration
System Sounds System → Settings → Miscellaneous
Updates System → Firmware → Settings
Web GUI System → Settings → Administration
Certificates System → Trust → Certificates
Time Services → Network Time → General
Time zone System → Settings → General

Table 4.3: OPNsense is configurated at various sections

Figure 4.2: OPNsense assigns new network interfaces

56
 Routing

Routing
A firewall has a routing table that contains the known networks. The rout-
ing table is automatically supplied with the IP addresses of the installed
network adapters. The firewall learns all other IP networks by manual
entry.
At first, the IPv4 routing table of firewall RT-1 looks quite empty. The
section System → Routes → Status displays all entries of the routing table.

OPNsense uses a two-step process to set up a new static route:

1. Create a gateway. The gateway is a neighboring device, which knows

the destination network. A gateway can be used for any number of
routes.
Basically, a gateway has an IP address and is accessible by one of the
local interfaces. OPNsense can monitor this gateway for availability,
which is later used in Chapter 16 for load balancing. OPNsense
manages the gateways at System → Gateways → Single.

2. Create a route. A route is a path to an IP network. As soon as a

firewall knows the way to a remote network, it will send the data
packets for this network to the connected gateway.
OPNsense places static routes at System → Routes → Configuration.

Using the example of RT-1, OPNsense will learn the path to the remote
network 10.2.1.0/24 via firewall RT-3. That makes RT-3 the gateway with
the IPv4 address 192.0.2.3. Both firewalls communicate over the shared
network segment of interface WAN2. Figure 4.3 on the following page
shows the route to the desired destination network.

The firewall will learn all other networks in the same manual way. The new
static routes are added to the local routing table. Now RT-1 knows all IP
networks of the demo network.
All other networks are reachable by a default route. A device uses its default
route when it has absolutely no idea how to reach that particular target
network. OPNsense creates a default route if the Default Gateway option is
active for a gateway.

57
Chapter 4. Initial Setup

Figure 4.3: A static route to the target network in site-2

Final testing
Now all components are ready for an accessibility test. The best tool for
this purpose is the ping command. Unfortunately, a firewall blocks all
connection requests resulting in a definite ping failure.
Thus, for a simple connection test in the lab network, the firewall needs
to temporarily open its gates. There is an excellent option for this task,
which is called Disable all packet filtering. Enable this feature at Firewall
→ Settings → Advanced to disable the filtering function of the firewall.

Warning
This option does exactly what its name says: it deactivates all filters
completely and also disables the address translation. This action turns
a strict firewall into a permissive router. The method is not suitable for
tests in productive environments.

The ping command is available at the command line and in the web GUI at
Interfaces → Diagnostics → Ping. Specify the target IP address, pick the
correct source address, and hit the ping button to start the test.

58
 Summary

Note
The ping of the command line distinguishes between IPv4 and IPv6
addresses. For an IPv6 address, the command changes to ping6.

Warning
Do not forget to enable packet filtering after the connection testing is
complete.

Summary
This chapter has given a rough outline of the first setup of the OPNsense
firewall without going into detail. After completing the system configuration
and setting up the network adapters with the intended IP addresses, all
laboratory devices have connectivity with each other.

59
Chapter 4. Initial Setup

60
Chapter 5

IP Version 6

The IP protocol in version 4 has quite a large address space. However, a

mix of waste and demand used up the last free IPv4 addresses in 2015.
This day was predicted 20 years ago, so the development of an improved
successor version having the number 6 started to take off (a streaming
protocol had already reserved number 5). The first tests started shortly
after that and ended successfully in 2005. Since then, IPv6 has found
its way into the operating systems of firewalls, routers, computers, and
smartphones.

Crash course
This book is not a manual for migrating to IP version 6, so let’s go through
the basics and differences briefly.
The addresses have grown larger in size. An IPv4 address is 32 bits in
length and the IPv6 address is up to 128 bits. To make the address readable,
it is split into eight 16-bit blocks and written as a hexadecimal number. The
blocks are separated by a colon, which is distinguishable from the period
used with IPv4 addresses. A conversion from IPv4 to IPv6 is not intended.
IPv6 addresses take on a new meaning, which is roughly explained in
Table 5.1 on page 63.
A complete IPv6 address might look like this:

2001:0db8:0002:0000:0000:0000:0000:0001

61
Chapter 5. IP Version 6

The leading zeros may be omitted. This follows the same logic as IPv4
addresses and in a mathematical equation; the above address is reduced as
follows:
2001:db8:2:0:0:0:0:1

The next reduction concerns multiple blocks, which contain only zeros.
These must be replaced by two colons so that the address is further reduced
as follows:
2001:db8:2::1

The subnet mask from IPv4 appears as a prefix length in IPv6. The meaning
is the same: it separates the routing prefix, subnet, and interface identifier.
The final 64 bit of the address are used as Interface ID and represent the
network adapter. One of the many global registries provides the first 64 bits.
They offer address ranges to Internet Service Providers (ISP). The ISP use
these ranges to supply their customers with address prefixes. The end
customer receives a prefix between /48 (generous ISP, see Figure 5.1) and
/56 (economic ISP).

2001 0db8 Interface Identifier

Registry /12
ISP /32
Customer /48
Site /64

Figure 5.1: Hierarchical assignment of IPv6 prefixes

This strict hierarchical allocation of addresses is the primary goal of IPv6

and helps backbone providers keep the routing tables of their devices
efficient and small.
In IPv6 networks, every end device is supposed to use a public address.
This means that every device can be addressed globally. However, global
addressing does not mean that every device is also reachable because
firewalls prevent incoming access to corporate and home networks – just

62
 Lab setup

as with IPv4. The devices in the lab network use reserved addresses for
documentation and Unique Local addresses, which are similar to the private
addresses of IPv4. The choice of addressing prevents lab equipment from
colliding with productive routers, even if the networks are accidentally
connected.

Type IPv6 scope

Global unicast address 2000::/3
Unique local unicast fc00::/7
Link-local unicast fe80::/10
Multicast ff00::/8
Documentation 2001:db8::/32

Table 5.1: Allocated IPv6 ranges

This concludes a brief explanation of IPv6 and its architecture. Due to this
new concept, developers must adapt all protocols or develop new ones.
This includes DNS, DHCP, all routing protocols, ICMP or ARP, which gets
additional tasks with IPv6 as Neighbor Discovery Protocol .

Lab setup
The structure of this chapter is adventurous because it skips IPv4 altogether
(except for the management interface). The lab focuses on IPv6 addresses
and IPv6 routes.
The result of this lab is the basis for all further chapters that include IPv6,
for instance, the firewall (Chapter 6) or NetFlow (Chapter 13).
In this lab scenario, all firewalls and clients participate in a basic configura-
tion for IPv6.

Addresses and routes

The configuration is divided into a two-step process. First, each network
adapter gets one IPv6 address and second, the firewall gets IPv6 routes to
remote networks. The configuration for this does not differ significantly
from IPv4.

63
Chapter 5. IP Version 6

Each interface receives its static IPv6 address under Interfaces and the
respective network adapter name. The prefix length is always /64 in order
to avoid unnecessary complexity. The full list of addresses is printed in the
lab description of Chapter 1 in Table 1.2 on page 26.

Figure 5.2: Network adapter with IPv4 and IPv6 addresses

In addition, the setup of the routes in IPv6 networks does not differ from
the procedure of IPv4 in section Routing of Chapter 4. Figure 5.2 illustrates
the two IP protocols on the same network adapter em0 of firewall RT-1.
Figure 5.3 shows the setup of a static IPv6 route.

Figure 5.3: Static route to a remote IPv6 network

64
 Clients

Clients
All favorite client operating systems provide good support for IPv6. The
configuration of IPv6 addresses and routes is usually convenient using a
graphical method. Even dusty Windows XP has limited support for IPv6,
albeit outside the GUI. A modern Windows computer can add the IPv6
address via the well-known Network and Sharing Center. Figure 5.4 shows
a network connection with both IPv4 and IPv6 addresses.

Figure 5.4: Windows 10 with static IPv6 address

The configuration in Linux differs between distributions. Only the command

line has similar commands in all distributions. The appropriate IPv6 settings
for client CL-1 are as follows:
ip addr add fd00:1::25/64 dev eth0
ip route add ::/0 via fd00:1::1

Now CL-1 is ready for IPv6. Name resolution is best configured using the
distribution-specific method.

65
Chapter 5. IP Version 6

Connections
Clients can reach each other as soon as all firewalls are configured with
their IPv6 addresses and routes. Client CL-1 checks the connection to the
other computers and firewalls using the ping6 command. traceroute6
even shows which way the IPv6 packet takes to get to its destination:
root@CL-1:~# traceroute6 -In fd00:2::25
traceroute to fd00:2::25 (fd00:2::25), [...], 80 byte packets
1 fd00:1::1 0.315 ms 0.305 ms 0.316 ms
2 2001:db8:2::3 1.040 ms 1.089 ms 1.066 ms
3 fd00:2::25 1.697 ms 1.965 ms 1.944 ms

The traceroute command uses ICMP packets to visualize the path. These
must be allowed in the intermediate firewalls. Alternatively, the IP filter
takes a short nap; this method may still be known from the initial setup in
section Final testing on page 58 of Chapter 4.

Summary
IP Version 6 is no longer some strange phenomenon in modern networks.
Today, routers and firewalls must also understand the language of IPv6 to
protect end devices. OPNsense is well positioned for this and knows several
methods for assigning IPv6 addresses, of which the lab network only uses
static addresses.
Thus, all participants are at eye level with IPv6 and ready for any upcoming
challenges.

66
 Part II

For Intermediates

67
Chapter 6

Firewall

A firewall is not a single device – it is a concept. The primary goal of this

concept is to provide security between computer networks to control access
and withstand attacks as long as possible.

The security concept is usually implemented with packet filters, application

gateways (proxies), demilitarized zone (DMZ), encryption, and logging.
Address translation (NAT, see Chapter 8) might be part of this strategy, but
its security gain is discussed controversially.
Simply put, routers connect computer networks, firewalls secure them.
It is also common to increase the protection to a higher level: systems for
detecting and preventing intrusions scan the internal network for packets
that should not be there due to security rules.
Also popular is the honeypot, which rebuilds a regular network with fake
hosts just like a movie set that looks like a real street scene. The honeypot
distracts the attacker from the real targets and allows the security team to
study attack patterns.

In general, the term firewall used in OPNsense is not associated with

the security concept but is synonymous with a packet filter. Therefore, the
following chapter describes a single OPNsense device and its rules as a
firewall.

69
Chapter 6. Firewall

OPNsense as a firewall
A packet filter has a set of rules that classify IP packets. Each rule has one
or more conditions that the package must meet for further processing. As
soon as a packet matches a rule, it is treated according to the rule action.
That is, it is either forwarded or dismissed.
This description mainly applies to all packet filters. Most firewall vendors
differ in the way the ruleset is configured and how detailed the rules can be.

OPNsense uses the first match principle to process the ruleset. It checks
the IP packet against each rule from top to bottom until a rule matches. If
there is no matching rule, the packet filter will discard the packet.

Note
The basic filter logic of OPNsense looks like this: a rule always belongs
to an interface and filters packets in the inbound direction.

If an unusual situation requires different processing, firewall rules can also

filter in the outbound direction. The Direction field of each rule reverses
the processing and filters packets that want to leave the firewall via the
specified interface. Furthermore, the Floating rules can also filter in an
outgoing direction or act on multiple interfaces.
The packet filter of OPNsense is connection-oriented. The reply packets of
a connection do not require a separate rule – they are automatically allowed.

The first match principle makes the order of the rules important. A blocking
rule at the beginning of the ruleset makes the subsequent rules ineffective.
It is best to sort the rules according to the following scheme:

1. Anti-spoofing rules. These rules match packets with bogus addresses

(see Section Anti-spoofing on page 78).

2. Special rules. The upper part of the ruleset should contain rules with
detailed information about the source IP, target IP, and ports.

3. General rules. The lower part of the ruleset should host the general
rules. These rules filter by source networks, destination networks
with or without port numbers.

70
 Lab setup

4. Cleanup rules. Next, put the rule which silently drops everything that
should not appear in the log file.

5. Final-deny-any-log. As the name implies, this rule blocks everything

and logs the denied packets. By default, OPNsense drops unmatched
packets, but this rule is easy to switch on and off. That is a quick
helper for troubleshooting.

Lab setup
A firewall ruleset can grow long and become complicated. For that reason,
a small number of devices is sufficient to investigate the possibilities of the
OPNsense firewall.
The host RT-1 acts as a firewall and filters between its local network at
adapter em1, the DMZ network at em2, and the Internet behind em3
(Figure 6.1 on the following page).
The host RT-core gets degraded to a regular client, which tries to reach
firewall RT-1 via the WAN network. The target is the host labsrv located in
the DMZ and protected by RT-1.

Firewall rules
The main objective of OPNsense is to filter packets. The web UI locates
the filtering policy at Firewall → Rules. Each network adapter has its own
ruleset. All rulesets are independent of each other. The LAN interfaces
have two default rules, which allow all IPv4/IPv6 traffic to pass. The other
interfaces have no rules preinstalled and subsequently block all connection
attempts.

If another rule is to allow IP communication, the setup starts at the in-

terface through which the connection request enters the firewall. The new
rule belongs to the ruleset of this interface.
For example, the firewall RT-1 should allow web access from the Internet to
the HTTP service of the lab server. The connection request of a client will
arrive at the WAN1 interface of RT-1. Hence, the right place for the new
rule is in the ruleset of WAN1.

71
Chapter 6. Firewall

Site-1 10.1.1.0/24 fd00:1::/64

DMZ 10.4.1.0/24
fd00:4::/64

10.1.1.1 10.4.1.1 10.4.1.7

em1 em2 eth1
RT-1 labsrv
OPNsense CentOS 8
em0 – 10.5.1.1 eth0 – 10.5.1.7

em3
198.51.100.1

WAN-1
198.51.100.6 198.51.100.0/24
2001:db8:1::/64
em1
RT-core
OPNsense
em0 – 10.5.1.6

Figure 6.1: Lab setup with RT-1 acting as packet filter

The following applies to the content of a rule: create rules to be as precise

as possible and as general as necessary. This advice means that the rule
for web access to the server labsrv should contain more details than just
allowing port 443 (https) into the DMZ.

Note
OPNsense does not issue a warning if a rule is configured correctly but
lies in the ruleset of the wrong interface. Troubleshooting will show
that no traffic is matching this specific rule.

72
 Logging

The exact rule is composed of:

• Action: Pass
• Interface: WAN1
• Direction: in
• TCP/IP Version: IPv4
• Protocol: TCP
• Destination: 10.4.1.7
• Destination port range: HTTPS
• Description: public https access to labsrv

The position of this new rule belongs to the upper section of the ruleset
because its content is specific for access to the lab server.
That way, the ruleset of all network adapters grows over time. Finally, they
include all required settings to enforce the security policy of the surrounding
network.

Block or Reject?
OPNsense can either block or reject connection attempts. Both actions
deny the request, block silently discards the packet. Reject also dis-
cards the packet, but it will inform the sender, by an ICMP packet, that
the connection was denied.
In principle, block is used for hostile networks (e.g. Internet), so that
no response packet is generated that could potentially disclose informa-
tion or amplify to a DDoS attack. For friendly networks (LAN, Wi-Fi)
reject is best used, so that the clients know immediately about the
denied connection attempt. That way, the user does not have to spend
time waiting.

Logging
A firewall rule can create a record in the log file, if it allows, rejects, or
denies a new connection attempt. The option Log packets that are handled

73
Chapter 6. Firewall

by this rule applies per rule. When the firewall has many rules with a
logging function or deals with lots of connections, the log area can become
overgrown.

Logging only makes sense if something meaningful is done with the log
messages. Otherwise, they are data garbage or a flash killer (see Chapter 3).
The usual cases are:

• Troubleshooting. Enable logging in case of a problem, analyze the

logs, solve the issue, and disable logging.

• Anti-spoofing. Usually, the anti-spoofing rule does not match any

packets. Otherwise, the log message contains valuable information to
find the root cause.

• Critical network segments. For example, servers in the DMZ do not

establish new connections on their own. A filter rule with log function
will document connection attempts. These log entries indicate a
malfunction of the server or a compromised server.

• Evidence. Log entries are good companions when it comes to docu-

menting or proving network traffic. Whether these entries are recog-
nized as evidence depends on the environment.

Even the help text of the log function advises against excessive logging:

“Hint: the firewall has limited local log space. Do not turn on
logging for everything. For those with heavy logging, consider
using a remote syslog server.”

It is best to use logging sparingly and only for critical rules.

Throughput
As the number of rules increases, the throughput rate of the firewall de-
creases. That is because OPNsense has to consult the ruleset or the connec-
tion table for each incoming packet and act accordingly. Also, some packets
require an address translation or an entry in the logbook.

74
 Best practice

It is best to optimize the ruleset to speed up processing. For example,

frequently used rules should always be at the beginning so that new packets
meet their rule faster and are forwarded quickly.

What is the critical size of a ruleset when the firewall throughput degrades?
It is not possible to give a definite answer to this question, it depends on the
complexity of rules and the use of aliases. In general, a ruleset is considered
large if the number of rules is in the mid four-digit range.

Best practice
A ruleset can outgrow itself over the years and become confusing. Having
a structured approach from the very start is the best way to establish an
organized ruleset.

• Keep it simple. A complicated ruleset only works until the next change
or until an error occurs. And sometimes, when reviewing the firewall
after several months, even your own rules may look strange if they
are too elaborate.

• Documentation. Each rule contains a field for descriptions. That is

the perfect place to briefly explain the rule’s background. The content
of this rule does not belong in the description field, because that is
mentioned in other areas.

• Consolidate devices and services. Combining multiple devices in a

group is possible. The same applies to numerous services, which are
combined in a port group. OPNsense calls this group Alias. They
are located at Firewall → Aliases. Firewall rules can include these
Aliases. That way, a single firewall rule applies to multiple servers
and multiple port numbers. The result is a shorter ruleset, making it
even more readable.

• Source network. If a firewall rule allows traffic, choose a specific

source network. If a firewall rule denies traffic, pick any as a source.
That will effectively deny remote networks, which may reach the
firewall over this network adapter.

75
Chapter 6. Firewall

This distinction is even more important when the firewall controls

networks that are not directly connected to it.

• IPv4+IPv6. The packet filter of OPNsense is ready for IPv6. A firewall

rule can filter IPv4 and IPv6 traffic in a single rule. It is not required
to set up two rules for two IP protocols – just set the TCP/IP Version
to IPv4+IPv6. Also, make sure that the Aliases contain both versions
of the IP address. For example, the Alias record for the lab servers
contains 10.4.1.7 and fd00:4::7.

• Final-deny-any-rule. Use this last explicit rule to block any traffic and
create a log record for every denied packet. The logging function is
enabled on demand and during troubleshooting.

• Filtering at the source. If multiple firewalls are in use, filtering should

be done as close to the source as possible. That way, traffic is avoided
and does not utilize the network.

• Inbound filter. Rulesets remain manageable if they filter in an in-

coming direction. OPNsense can also filter outbound traffic using the
Direction field or the Floating rules, but it might become confusing if
the policy spreads over several places.

• Audit. Check the ruleset once per year. Deactivate any unused rules
and delete them later. And how do you recognize a rule that has not
matched any traffic?
The button Inspect of each rule at Firewall → Rules has counters that
enumerate the transmitted data and number of packets. When the
counter is set at 0 for both Packets and Bytes, nothing has been done
and the rule is ready for a cleanup.
Earlier versions of OPNsense show this information in the Section
Rules at Firewall → Diagnostics → pfInfo.

Additional filter
The described methods of packet filtering are usually adequate for enforcing
the security policy in an organization’s network. However, OPNsense has a
couple of tricks up its sleeve to cover special use cases.

76
 Additional filter

Time-based rules
A regular firewall rule is enabled all the time without intermission. A time-
based rule has a “timetable” which tells when to filter packets and when
not to. This schedule can consist of simple specifications, e.g. 8 to 11 a.m.,
or it can implement a complex schema comprised of time, date, and day of
the week.

Figure 6.2: This firewall rule is valid only within the selected schedule

The example in Figure 6.2 shows a time-based firewall rule. Its purpose is to
permit traffic to the email server only during the daytime so that employees
will not access their emails at night. The schedule is configured at Firewall
→ Settings → Schedules and includes working days from 6 a.m. to 10 p.m.
(Figure 6.3 on the next page).

77
Chapter 6. Firewall

Figure 6.3: This schedule includes times from 6 a.m. to 10 p.m.

Note
The schedule only works reliably with a synchronized clock. By default,
OPNsense regularly picks the exact time from a public NTP server under
Services → Network Time → General.

A firewall rule (Figure 6.2) for the LAN interface allows access to this server
only within the times specified in the schedule. Outside this time range,
this rule does not apply, and the email request fails implicitly or due to a
final-deny-any rule.

Anti-spoofing
Intentionally faking the source address of an IP packet is called spoofing.
It differs from Network Address Translation (see Chapter 8) in which the
source or destination address is manipulated on purpose.
It is impossible to identify a fake address just by looking at the packet
headers. The authenticity of an address depends on the environment and
its context. For example, a packet with the source IP 10.1.1.25 is generally
a valid packet from client CL-1 at site 1. However, if this packet enters
at the WAN1 interface of firewall RT-1, then the address is incorrect and
probably fake.

78
 Additional filter

Figure 6.4: This firewall rule detects fake source addresses

Fake addresses are a trick to bypass simple packet filters or provoke certain
response packets.
OPNsense has a built-in method to detect spoofed packets. It inserts a
hidden rule for every network adapter. This rule blocks incoming packets
with a source address that belongs to one of the other adapters. This
method offers acceptable protection against address forgery. A separate
anti-spoofing rule is only necessary if multiple IP networks are reachable
behind a network adapter.

Figure 6.5: A firewall alias includes all subnetworks behind the LAN adapter

79
Chapter 6. Firewall

The configuration of this rule is straightforward. Each firewall rule has an

option to invert the source network. It then denies all incoming packets on
the LAN adapter that do not originate from the LAN network. Figure 6.4
shows the settings of this anti-spoofing rule with the necessary aliases from
Figure 6.5.

GeoIP
Filtering IP connections based on the country of origin is called GeoIP. An
IP address does not belong to a country, but it gets assigned to an ISP or
organization inside a country. The magic of GeoIP is basically a mapping
between IP ranges and countries. OPNsense uses the GeoIP databases
of MaxMind [9] for the geo-functions. The provider indicates a 99.8%
accuracy in the assignment of a country’s IP address.

Beginning December 2019, access to the MaxMind database requires free

registration with the provider. Access the MaxMind web portal under My
License Key to generate a new license key (e.g. LDswU7Agprdv9JFY). The
generated key must be entered in the OPNsense firewall at Firewall →
Aliases → GeoIP settings as a downloadable URL, which also contains the
license key. The proper URL for the example key is:
https://download.maxmind.com/app/geoip_download?edition_id= \
GeoLite2-Country-CSV&license_key= LDswU7Agprdv9JFY &suffix=zip

Click on Apply and wait for OPNsense to load all IP ranges from MaxMind,
which effectively creates aliases of the type GeoIP.

Note
By default, OPNsense accepts a maximum number of 200,000 entries
in the firewall table. MaxMind provides significantly more records, so
enlarge OPNsense’s default to 800,000, for example, under Firewall
Maximum Table Entries at Firewall → Settings → Advanced.

A firewall rule for blocking at the country level prohibits all connections
with addresses in one or several countries. If IP addresses are mistakenly
assigned to this country, or if individual servers need to be excluded from
the blocking, the ruleset requires a prior exception rule.

80
 Technical background

Figure 6.6: The firewall enforces access based on country

Figure 6.6 shows an example of blocking all addresses to the Netherlands

with the exception of the website operated by a popular daily newspaper.
The order of the firewall rules matters: exceptions first, then geo-blockades,
and finally the general rules.

Technical background
The OPNsense firewall implements the packet filter pf from FreeBSD. Orig-
inally it was developed in 2001 for OpenBSD, but it quickly found its way
to the sources of FreeBSD. Since 2004, pf is a fixed part of the distribution.
Working with pf is performed with the command pfctl on the command
line. pfctl loads new rules, displays the active configuration, checks the
syntax, or deactivates the filter function.

The web interface of OPNsense translates the firewall rules into the syntax
of pf, so there is no need to hassle with the packet filter directly.
OPNsense does not provide a way to create new rules from the command
line. For troubleshooting, it helps to be familiar with the basic options from
pfctl. Table 6.1 shows important commands and its meaning.
When it comes to logging, pf uses a software interface called pflog. Then,
pf sends the header of a packet to pflog and lets this tool do the log house-
keeping. Unfortunately, pflog uses its own binary format, which implicates
tcpdump for the display.

81
Chapter 6. Firewall

Command Description
pfctl -d Disables the packet filter and address translation
pfctl -e Enables the packet filter and NAT
pfctl -s rules Shows the active ruleset from pf
pfctl -s nat Shows the address translation rules
pfctl -s states Shows all connections listed in the state table.
The web UI has this information available at
Firewall → Diagnostics → States Dump

Table 6.1: pfctl controls pf on the command line

OPNsense makes it easy for its users and converts the log messages from
pflog to Syslog messages using the filterlog service. The syslog service then
places the messages into the file:
/var/log/filter.log

Access this file with clog, a pipe, and the well-known commands tail,
grep or less. For example:
clog /var/log/filter.log | grep 10.5.1.7

Order of processing
The packet filter of OPNsense processes the policy in a particular order.
Each packet first passes through the NAT rules (see Chapter 8), followed by
the system rules, and finally the custom rules of the user.

1. One-to-One-NAT rules (Firewall → NAT → One-to-One)

2. Outbound NAT rules (Firewall → NAT → Outbound )

3. Inbound NAT rules, e.g. forwarded ports (Firewall → NAT → Port

Forward )

4. Predefined internal rules, e.g. anti-lockout rule, bogon networks for

IPv4, DHCP, mandatory ICMP types for IPv6

5. Custom floating rules (Firewall → Rules → Floating)

82
 Troubleshooting

6. Custom rules, which apply to a group of network adapters

7. Custom rules, which apply to a single network adapter, e.g. MGMT,

LAN (Firewall → Rules)

8. Automatic VPN rules

The individual areas do not have to contain any rules. If the firewall does
not terminate VPN tunnels, then the automatic VPN rules are empty with-
out blocking traffic.

Depending on the application, more rules might sneak in so that the ap-
plication does not “get stuck” somewhere in the ruleset. These include
OpenVPN with RADIUS authentication and IPsec clients.
Only pf knows about the complete ruleset and its order of execution. It
provides the content in the file /tmp/rules.debug which has all the details
about the individual steps.

Troubleshooting
A new rule might be completely ineffective and not allow established data
traffic, but why? OPNsense provides insight into internal processes, log
files, and network traffic for this situation. The following tips are intended
to provide orientation if you have trouble finding the root cause:

1. Logic check. Is the rule part of the correct network adapter? Do the
rule details include the desired selection of source, ports, protocol,
and destination?

2. Order. Is the rule at a suitable position in the ruleset? The filter lists
are processed from top to bottom. Perhaps another rule is first called
so that the rule in question does not match at all. For the duration
of the troubleshooting, the new rule, with exceptions, may go to the
beginning of the ruleset.

3. Special case: Floating rules. The floating rules are processed before
the normal interface rules if the default option Quick is selected.
Thus, if two rules contradict each other, the floating rule wins.

83
Chapter 6. Firewall

4. Address modification. Does a NAT rule (see Chapter 8) modify the

addresses inside the packets so that it is rejected from the receiver?

5. Routing. In general, the routing table determines the outbound

interface of the packets. In particular, the Gateway field of a rule
can overwrite this decision (see Chapter 16). Is the packet using the
correct outgoing interface?

6. Logbook. OPNsense offers several formats for browsing firewall logs.

The search function at Firewall → Log Files → Live View filters the
log entries by time, interface, IP addresses, ports, and protocols.

If the logbook has no entries about the new connection, the packets may
not reach the local firewall at all. For definite proof, it is possible to observe
the transmitted packets at Interfaces → Diagnostics → Packet Capture. This
function of OPNsense starts the analysis tool tcpdump for recording and
displaying transmitted packets in real-time.

Summary
The packet filter pf of FreeBSD has quite a clear syntax, but OPNsense deals
with the modifications inside the rule files. It covers these details behind
the web interface. The administrator can then work with rules, groups, and
aliases at a high level to implement a security policy.
The possibilities are manifold: OPNsense filters by source, destination,
protocol, ports, and schedule. And that applies to both IPv4 and IPv6.
OPNsense can even use the rules to influence the Quality of Service param-
eters of the transmitted packets or to change the gateway of the outgoing
packets (see Chapter 16).

84
Chapter 7

Transparent Firewall

A transparent firewall, or firewall bridge, filters packets similar to a regular

firewall: it allows good IP packets and denies bad packets. However, the
method of processing is different. The transparent firewall acts on layer 2
and is placed transparent in the data path. Routing functions remain
disabled, and the firewall host is invisible to ping and traceroute. The
firewall only needs its own IP address for management purposes. Simply
stated, a transparent firewall is an Ethernet switch with a filtering function.

Pros and cons

The transparent firewall does not need an IP address. Existing servers and
routers do not need any changes to their addresses when the firewall is
placed in a network segment. This form of firewall is therefore the ideal
security device of network areas that should not be changed. The firewall
begins its work as an Ethernet switch without a filtering policy. It then
starts rule-by-rule to secure the surrounding network areas.
The firewall bridge has another advantage when it comes to multicast. A
permitted multicast stream merely flows through it. The tedious work-
arounds with IGMP proxy are not necessary.
A transparent firewall operates on layer 2 of the OSI model and is therefore
not limited to IP packets. It is possible to transport Ethernet packets which
contain other protocols. This is especially true in data centers where there
might be protocols like IPX, FCoE or MPLS. A regular firewall cannot handle

85
Chapter 7. Transparent Firewall

these protocols. Unfortunately, OPNsense is limited to IP packets. For all

other protocols, it is either permit or reject.

The invisibility of the firewall is also its biggest drawback during trou-
bleshooting. Regular network elements are visible and have an IP address.
It is easier to solve an issue when devices are known and appear in the
command output.
Also, the transparent firewall participates on layer 2 of the OSI model and
cannot join a dynamic routing protocol, which operates at layer 3. The
same applies to VPN tunnels and Quality of Service measures that run at IP
level.

Lab setup

The host labsrv is accessible for the remote stations RT-core and RT-3, which
are connected by the shared IP network. RT-1 is located in front of the
server as a transparent firewall. It protects access to labsrv with a sample
set of rules. The network adapters of RT-1 are bridged so that the data
traffic can flow through transparently. Figure 7.1 shows the setup of the
lab devices.

RT-1
OPNsense
em2
10.4.1.3 RT-3
OPNsense
10.4.1.7 em2 em4
eth1 bridge0
DMZ WAN-2 RT-core
labsrv 10.4.1.0/24
fd00:4::/64
10.4.1.0/24
fd00:4::/64
10.4.1.6 OPNsense
CentOS 8 vnet4 vnet6 em2

Figure 7.1: Lab setup with RT-1 as transparent firewall

86
 Configuration

Configuration
The network bridge is quickly built from the GUI. The firewall has a few
more options that are important for later operations.

1. Create network bridge. The network bridge is an additional interface.

It is created at Interfaces → Other Types → Bridge. The member
interfaces of the new bridge are network adapters LAN and WAN2.
The name is predefined and is bridge0 for the first bridge.
The new network bridge must not participate in the Spanning Tree
Protocol so that it does not send messages and therefore remains
invisible.

2. Create network adapter. The new network port bridge0 is waiting for
duty at Interfaces → Assignments. When added to the firewall RT-1,
it is assigned the name OPT4 and is immediately present in the list of
interfaces.

3. Configure network adapter. The interface of the bridge behaves

exactly like its member adapters (see Chapter 4). Go to Interfaces →
OPT4 and find a better name for the bridge. Do not configure an IP
address.
Next, the member interfaces need some modification. Both adapters
DMZ and WAN2 must give up their IP configuration to become
transparent. Just change the Configuration Type to None and save
the modifications.
Note
If the firewall is configured through one of the network adapters
discussed above, the conversion to a network bridge will per-
manently interrupt the connection. It is best to use a dedicated
management adapter (see Chapter 9) that does not belong to the
productive interfaces.

4. NAT rules. The firewall is now an Ethernet bridge and must not
perform IP address translation (see Chapter 8). The default setting of
OPNsense regarding NAT is to create NAT rules automatically. They
must be manually disabled at Firewall → NAT → Outbound.

87
Chapter 7. Transparent Firewall

If the firewall performs address translation for other network adapt-

ers, the NAT rules must be created manually. The ruleset for NAT
must not contain the member adapters of the network bridge!

5. Filtering bridge. The firewall rules at OPNsense apply to the packets

of the physical network adapters. The bridge is a logical adapter that
also needs to participate in filtering.
OPNsense can also filter on a bridge but needs some modifications
at System → Settings → Tunables first. The two leading settings of
Table 7.1 move the filter function from the hardware adapter to the
network bridge.

Setting Value Description

net.link.bridge.pfil_member 0 Stop filtering on the physical
interfaces of a bridge.
net.link.bridge.pfil_bridge 1 Start filtering on the network
bridge.
net.link.bridge.pfil_onlyip 0 What happens to non-IP packets?
By default, OPNsense forwards
them without filtering.

Table 7.1: Kernel modifications to filter on the network bridge

6. Ruleset. Filtering starts right here. After the change in step 5, only
the bridge can filter packets. The rules of their member interfaces
DMZ and WAN2 are not used anymore and should be removed – for
a better overview.
Note
The rules of the network bridge work in both directions, inbound
and outbound.

The new ruleset should always have a source and destination IP

address configured and do not pick any carelessly.

7. That’s all. Save the interfaces and ruleset and apply them.

The new bridge is ready for operation. The WAN-2 network is now di-
rectly connected to the DMZ network 10.4.1.0/24. As a consequence,

88
 Filter operation

both networks share the same IP subnet. The original IP addresses of the
WAN-2 network are omitted, and the participants RT-3 and RT-core need
corresponding IP addresses from the DMZ network (see Table 7.2).

Firewall Interface VMnet/vboxnet IPv4 IPv6

RT-1 em2 VMnet4 10.4.1.1 fd00:4::1
RT-3 em2 VMnet6 10.4.1.3 fd00:4::3
RT-core em2 VMnet6 10.4.1.6 fd00:4::6
labsrv eth1 VMnet4 10.4.1.7 fd00:4::7

Table 7.2: New addresses for the shared network of DMZ and WAN-2

Filter operation
The firewall RT-1 connects its network adapters DMZ and WAN2 to a new
bridge interface bridge0. This network bridge allows the communication of
all participants on the Ethernet level. The hosts labsrv, RT-3, and RT-core
are now all in the same broadcast domain created by the RT-1 bridge. This
setup resembles a standard switched Ethernet segment without any filtering
operation.
The firewall can examine all packets passing through the Ethernet bridge.
And it can look deeper into the packet and filter for IPv4/IPv6 addresses
and port numbers as well.
The firewall inspects in both directions: rules apply simultaneously in the
incoming and outgoing interfaces. A blocked port 80 without a source or
destination network is therefore valid in both directions. If the direction
is essential, IP addresses in the filter rule must describe the situation
accurately.

Ruleset
The structure of the ruleset and the processing of the individual lines remain
the same with the transparent firewall. The most important difference is
that the rules of a firewall bridge apply in both directions since it is no
longer possible to distinguish between inbound and outbound.

89
Chapter 7. Transparent Firewall

For instance, SSH access from 10.4.1.3 (RT-3) to 10.4.1.7 (labsrv) is shown
in the last line of Figure 7.2. The other rules shown have at least selected a
destination so that web accesses are only allowed into the DMZ.

Figure 7.2: Firewall rules with exact source and destination information

Connection test
Let’s validate the setup: ping, ssh, and curl generate connection requests
which are handled by the rules. The external host RT-core tests the new
firewall rules:
curl --head http://10.4.1.7 # allowed, TCP port 80
ssh -v 10.4.1.7 # denied, TCP port 22
ping 10.4.1.7 # denied, ICMP echo

The connection attempt via SSH fails because only a single rule handles
SSH traffic, but this one is limited to RT-3. As such, traffic from RT-core
matches the default rule and gets discarded.

When testing IPv6 connectivity, RT-core again uses curl, but this time
with an IPv6 address as target:
curl --head "http://[fd00:4::7]/"

The connection should work since access to TCP port 80 is allowed for both
IP protocols originating at RT-1.

90
 Uncover transparent firewall

Note
When using an IPv6 address in the URL, use enclosing quotation marks
and square brackets. Otherwise, curl simply reports No match.

Uncover transparent firewall

OPNsense is invisible in the traffic path when acting as a transparent
bridge. The neighboring devices cannot detect that their data packets are
being processed by a firewall because the filters do not change either the
MAC or the IP address. For this reason, the address translation in section
Configuration (see page 87) must also be deactivated.
If a filter rule should reject a request, then the firewall must actively
terminate the connection. OPNsense will not reveal its identity in this case
either, and it will respond to the denied packet on behalf of the remote peer.
This hide-and-seek gives the client the impression that the server has been
disconnected, even though the firewall is in the middle and was the source
of the packet.
It is possible to uncover a transparent firewall, but only if that device reveals
information about itself. These include unwanted ICMP responses (e.g.
ping), neighbor advertisements using LLDP, or loss in throughput. In the
most trivial case, the two network adapters of the firewall hardware agree
on different settings for speed and duplex, so that the remote peers can at
least suspect a device in the path.

Technical background
The transparent firewall uses the same filtering methods as its explicit
variant (see Chapter 6). OPNsense assembles the network bridge using a
software component, which is developed by FreeBSD as the kernel module
if_bridge. This module offers Ethernet packet forwarding between mem-
ber interfaces and Traffic Shaping, which is used to reduce throughput.
The command line has only a single command ifconfig to set up a net-
work bridge, assign its members, and provide IP addresses. An additional
command – like brctl under Linux – is not required in FreeBSD.

91
Chapter 7. Transparent Firewall

Summary
OPNsense makes a great impression as a transparent firewall. Troubleshoot-
ing is somewhat peculiar because the firewall is invisible when running
traceroute and observing its output.
A transparent firewall is best suited for environments which cannot change
the addressing information but require a protective element to secure the
network better.

92
Chapter 8

Network Address Translation

A router or firewall can modify the IP address of packets passing through

them. This manipulation is done on purpose to implement the chosen
network design. It is commonly called Network Address Translation (NAT).

These changed addresses are mainly transparent for the client and server.
Most applications also work with NAT. As long as the targeted server sends
its data to the requesting client, nobody questions the addressing.
NAT is present in virtually every IPv4 network design which is connected to
the Internet. Including NAT in the design does not mean there is a lack of
imagination but a lack of public IPv4 addresses. IPv6 has more than enough
addresses and therefore does not rely on NAT. In particular scenarios, e.g.
failover with several ISPs, NAT has also crept into IPv6 designs.

NAT is omnipresent: every DSL router uses address translation so that

all its clients can access the Internet via a single public IPv4 address. The
hotspot function of a smartphone uses the same technology. And a large
corporate firewall uses a pool of IP addresses for address translation to
enable Internet access for all clients.
Troubleshooting NAT is confusing because IPv4 addresses are translated
back and forth. Complex environments even change the addresses multiple
times. For example, the WAN router translates an internal source address
into a service provider’s private address, which then converts this address
into a public IPv4 two hops later.

93
Chapter 8. Network Address Translation

Lab setup
This chapter is an excellent demonstration of why the site networks have
private addresses, and why the core networks use public addresses. Before
the client’s packets pass the WAN, the firewalls must convert them into
appropriate addresses.

Each site network has an entry point to the Internet, which is simulated by
WAN-1 and WAN-2. The firewall RT-core is used to check the NAT configu-
ration because it will only respond to public IP addresses and discard access
from private IPv4 addresses without notice.
Figure 8.1 shows the lab setup with all NAT gateways.

Site-1 10.1.1.0/24
10.1.1.1
em1 DMZ 10.4.1.0/24
em2 10.4.1.1 10.4.1.7 eth1
RT-1 labsrv
em4
192.0.2.1
em3
198.51.100.1
WAN-1 WAN-2
198.51.100.0/24 192.0.2.0/24

198.51.100.6 192.0.2.3
em1 em2

RT-core RT-3
OPNsense OPNsense
Internet
em1
10.2.1.3
Site-2 10.2.1.0/24

Figure 8.1: Lab setup with NAT gateways

94
 Scenarios

Scenarios
OPNsense can now demonstrate its ability to translate source and destina-
tion addresses with and without masquerading and port forwarding.
To make the situation in the core network realistic, the RT-core firewall
will not accept packets with private IP addresses on its WAN adapters.
This method represents the IP routing on the Internet and immediately
shows if the other firewalls have configured their NAT rules correctly. OPN-
sense offers the option Block private networks in the interface settings.
Alternatively, a firewall rule can filter and log requests to private addresses.

One-to-One NAT
In this first scenario, the lab server should be accessible via the public WAN
networks. For this purpose, RT-1 provides an additional IPv4 address in
each WAN network. Clients can reach the lab server using the new ad-
dresses. The addresses must match the IP network and be known to the
clients, e.g. via DNS. Figure 8.2 shows the modification of IPv4 addresses
in incoming data packets by RT-1 for the network adapter of WAN-1.

198.51.100.1 10.4.1.1
em3 em2
RT-1
198.51.100.7 10.4.1.7
NAT
permit 10.4.1.7

Figure 8.2: NAT modifies the IP address of packets in transit

The implementation in OPNsense is a bit more complicated because the

firewall needs virtual IPs, NAT rules, and firewall rules.

The configuration requires three steps:

1. Virtual IPs. RT-1 is responsible for the two new IPv4 addresses, so
the IPs must be configured first. OPNsense implements multiple IP

95
Chapter 8. Network Address Translation

addresses using virtual IPs, which are created at Interfaces → Virtual

IPs → Settings.
The settings from Table 8.1 contain details on the new addresses.
RT-1 will accept requests for addresses 198.51.100.7 and 192.0.2.7
and handle them according to two steps; namely, the NAT rule and
firewall rule.
Attribute WAN-1 WAN-2
Mode IP Alias IP Alias
Interface WAN1 WAN2
Address 198.51.100.7/32 192.0.2.7/32
Description for 1:1-NAT for 1:1-NAT

Table 8.1: OPNsense requires virtual IPs to implement One-to-One NAT

2. NAT rule. Choose what to do with the virtual IPs at Firewall → NAT
→ One-to-One. The plan is to use them as a public alias for the IP
address of the lab server. Table 8.2 shows the settings of the NAT
rules.
Attribute WAN-1 WAN-2
Interface WAN1 WAN2
Type BINAT BINAT
External network 198.51.100.7 192.0.2.7
Source / Invert 2 2
Source 10.4.1.7 10.4.1.7
Destination / Invert 2 2
Destination any any
NAT reflection Enable Enable

Table 8.2: One-to-One rules for accessing the lab server

What the heck is NAT Reflection? Jump forward to section NAT

Reflection on page 103 to get a brief explanation.

3. Firewall rule. The addresses of the incoming packets are changed

appropriately, but the packet filter still prevents access. A firewall
rule is missing. The configuration in Table 8.3 allows web access via

96
 Scenarios

both WAN network adapters. The destination address is the internal

IPv4 of the lab server. The firewall rule does not know anything about
address translation and deals with internal addresses only.

Attribute WAN-1 WAN-2

Action Pass Pass
Disabled 2 2
Interface WAN1 WAN2
TCP/IP Version IPv4 IPv4
Protocol TCP TCP
Source / Invert 2 2
Source any any
Destination / Invert 2 2
Destination labsrv (Alias) labsrv (Alias)
or 10.4.1.7 (IP) or 10.4.1.7 (IP)
Destination port range HTTP HTTP

Table 8.3: Firewall rules permit web access to labsrv

Validate the setup using the text-based web client curl, which is available
on all devices. Both IPv4 addresses of the lab server can be accessed from
the firewalls RT-core and RT-3 for testing purposes.
curl --head http://192.0.2.7
curl --head http://198.51.100.7
Access to the public IP address should also be possible from the internal
CL-1 client in the northern site network. Section NAT Reflection explains
why this works. The successful response contains several headers and the
status HTTP/1.1 200 OK.

The lab server logs the request, including the sender’s address, in the
log file of the Apache web server.
root@labsrv ˜> tail /var/log/httpd/access_log
198.51.100.6 - - [20/Feb/2021:20:33:27 +0100] "HEAD / HTTP/1.1" \
200 - "-" "curl/7.74.0"
192.0.2.3 - - [20/Feb/2021:20:32:55 +0100] "HEAD / HTTP/1.1" \
200 - "-" "curl/7.74.0"
10.1.1.25 - - [20/Feb/2021:20:34:16 +0100] "HEAD / HTTP/1.1" \
200 - "-" "curl/7.64.0"

97
Chapter 8. Network Address Translation

Simple outbound translation

The TCP or UDP port number is used to easily distinguish between internal
clients and public addresses. In addition to the translation of network ad-
dresses, port numbers are also considered for manipulation. The concept is
called Port and Address Translation, IP masquerading, or simply Outbound
NAT. This scenario applies to most home networks.

Masquerade
This unique form of NAT not only translates the IP addresses but also
remaps the port numbers.

In this setup, firewall RT-1 will translate site-1 network 10.1.1.0/24 to

its own WAN address from network adapter em3. Technically, it is doing
source NAT with masquerading.
OPNsense comes with an automatic mode at Firewall → NAT → Outbound.
It tries to make the right decision and generates the required NAT rules.
This mode works well, but it is better to use the manual mode to create and
understand custom rules. To do this, change the radio button to Manual
outbound NAT rule generation so that the web GUI will accept new rules
for address translation.
Attribute Value
Interface WAN-1
TCP/IP Version IPv4
Protocol any
Source invert 2
Source address 10.1.1.0/24
Source port any
Destination invert 2
Destination address any
Destination port any
Translation / target Interface address

Table 8.4: Site-1 shares the source IP address of RT-1 for outbound traffic

Table 8.4 shows the settings of the required rule. The individual lines read
like an if-then condition: if IPv4 packets are sent from the source network

98
 Scenarios

10.1.1.0/24 to the WAN-1 network, then replace the source address by the
address of the WAN-1 network adapter.
By specifying the Interface address, the configuration does not expect a
predefined IP address. Choosing this option is a best practice because
the interface IPv4 address might be dynamic and not known in advance.
OPNsense therefore always uses the address of the specified interface as
the new source address for the address translation.

Advanced outbound translation

Internet Service Providers usually offer multiple public IPv4 addresses for
business customers. OPNsense can use this address pool for outbound
address translation.

10.2.1.3 192.0.2.3
em1 em2
RT-3
10.2.1.0/24 192.0.2.48 ̶̵ .55
NAT

Figure 8.3: Firewall RT-3 translates in multiple IP addresses

This scenario (see Figure 8.3) requires RT-3 to translate all site-local clients
to a WAN address taken from an address pool. This pool contains eight
public IPv4 addresses from 192.0.2.48 to .55. The rule of thumb is to use
one public address for each 256 internal IP addresses. The sample pool is
therefore sufficient for a maximum of 2,048 clients.
The configuration is similar to the scenario from section Simple outbound
translation and is therefore described briefly.

1. Virtual IPs. The firewall in charge RT-3 requires a virtual IP for the
subnet 192.0.2.48/29. The mode of operation is Other or Proxy ARP .

2. NAT rule. Create a new rule at Firewall → NAT → Outbound. The

settings match the values from Table 8.4. The only exception is
the source IPv4 network 10.2.1.0/24, which will be translated in a

99
Chapter 8. Network Address Translation

random address from IP scope 192.0.2.48/29. The drop-down box

next to Translation / target should contain this value, due to the
newly created virtual IP from the previous step.
How should OPNsense sequence and use the addresses from the pool?
That’s decided in the field Pool Options. Choose between Random,
Round Robin, Source Hash, or Bitmask.

3. Firewall rule. If not already present, a new filter rule must allow
access from the LAN to the WAN-2 network. The firewall rule is
created independently of an address translation.

The client CL-2 validates the address translation in site-2. It merely tries
to reach the IP address of the firewall RT-core. The provided website at
http://192.0.2.6/ will only respond to the request if RT-3 is configured
correctly.

Port forward
In this last scenario, RT-3 translates the destination address in the IP header
for incoming requests from the WAN network. This method is commonly
referred to as port forward. The principle is similar to One-to-One-NAT;
however, the port forwarding is different and only handles packets that
match the selected TCP or UDP port number.
In this example, the additional IPv4 address 192.0.2.25 takes over the
public representation of the CL-2 computer in the southern site network.
The TCP port 443 (https) is “forwarded” to the target computer CL-2 by the
firewall RT-3. The configuration process should already be known from the
previous scenarios:

1. Virtual IP. RT-3 requires a virtual IP of type IP Alias for the address
192.0.2.25/24 on network adapter WAN2.

2. Port forward. For two TCP ports, OPNsense also needs two entries
at Firewall → NAT → Port Forward. Table 8.5 shows the selected
settings for the HTTPS port.
The option Filter rule association looks interesting. OPNsense offers
to create a firewall rule to allow data traffic.

100
 IPv6

Attribute Value
Interface WAN2
TCP/IP Version IPv4
Protocol TCP
Source / Invert 2
Source 192.0.2.25
Destination port range HTTPS
Redirect target IP 10.2.1.25
Redirect target port HTTPS
Filter rule association Pass

Table 8.5: Port forward for target host CL-2

3. Firewall. The automatically generated filter rule permits access from

anywhere. If only specific IP networks are allowed to access the target
computer, a custom firewall rule is the better choice.
Go back to the port forwarding rules and change the option Filter rule
association to None. Next set up a filter rule at Firewall → Rules to
permit traffic to the 192.0.2.25 IP address of CL-2.

Note
The descriptions show that most NAT configurations require a virtual
IP. A virtual IP is not needed under the condition that the IP addresses
of the local network adapters are used.

IPv6
NAT is a workaround from the address shortage of IPv4. There is no need
for NAT in IPv6 due to the huge address range. However, ten years after the
introduction of IPv6 an address translation was added. This IPv6-to-IPv6
Network Prefix Translation (NPTv6) does not translate individual addresses
but whole IPv6 prefixes.
NPTv6 has advantages when load balancing with multiple Internet provid-
ers over IPv6 networks. In that scenario, the outgoing packet is always
translated into the appropriate IPv6 address of the respective ISP.

101
Chapter 8. Network Address Translation

An example should illustrate the translation of the internal network prefix

fd00:2::/64 behind RT-3 into the prefix 2001:db8:22::/64.
Figure 8.4 visualizes the source address modification of an IPv6 client in
site 2.

fd00:2::3 2001:db8:2::3
em1 em2
RT-3
fd00:2::0015:16ff:fe11:0125 2001:db8:22::0015:16ff:fe11:0125
NPTv6

Figure 8.4: IPv6-to-IPv6 Network Prefix Translation

The IPv6 network 2001:db8:22:: is supposed to be the public representation

of the clients’ IPv6 addresses behind RT-3. The other routers in the lab still
need a static route so that the response packets return to RT-3.

The question then arises: why not use the 2001:db8:2::/64 prefix from
the local WAN interface? The public IPv6 subnet for address translation
must not match the prefix of a local network card. For NPTv6 to work,
an additional prefix has to be added to the firewall. If the same prefix
was used, FreeBSD would have problems with neighbor detection. Also,
duplicate addresses could occur if other devices are present in the prefix of
the WAN.

Figure 8.5 shows the configuration under Firewall → NAT → NPTv6 to

translate from internal prefix to external prefix.
Finally, a regular firewall rule controls which clients and protocols can flow
through the NPTv6 gateway. The filter rules always refer to the original
address before NPTv6 changes them.

Note
Unlike the NAT rules for IPv4, the Network Prefix Translation of IPv6
does not require a virtual IP address for its translations.

102
 NAT Reflection

Figure 8.5: OPNsense translates IPv6 prefixes using NPTv6

NAT Reflection

The NAT Reflection technique is a solution to a restriction in FreeBSD. If a

port forward or One-to-One translation exists on a WAN adapter, OPNsense
will handle incoming packets that enter the firewall through the WAN
adapter. Incoming packets from the LAN adapter are not considered because
the NAT rules do not apply to them.
In the One-to-One NAT scenario, firewall RT-1 allows access from the WAN-1
and WAN-2 networks to the address of the lab server. The configured
address translation does not apply to the other networks LAN and DMZ of
RT-1 (see Figure 8.1).
The effect is weak in a lab network because the members of site 1 use the
actual IPv4 address 10.4.1.7 of the server. External clients access the public
address 198.51.100.7 to reach the same target host.
In practice, this is a serious situation because client devices are mobile.
They are sometimes connected to the corporate network and sometimes to
the Internet. The user (or a script) would have to change the destination
address at each “network change”. NAT Reflection solves this problem with
additional rules. With these rules, clients in internal networks may also
communicate with the public addresses of the lab server.

103
Chapter 8. Network Address Translation

Note
NAT Reflection is just a workaround. The more elegant approach is
to resolve the address of the server via DNS. Inside the LAN the DNS
query for labsrv.example.net returns the internal address 10.4.1.7.
When called from the WAN network, the DNS server replies with the
public address 198.51.100.7.

Technical background
OPNsense operates the address translation with the packet filter pf of
FreeBSD (see Chapter 6). For pf a manipulation of IPv4 addresses is just
another line in the ruleset, which is prefixed with the keywords nat or rdr.
The keyword nat handles the one-to-one implementation and rdr performs
port forwarding. The IPv6 prefix translation has the code word binat at pf.

A nat rule applies only to the network adapter, which is mentioned in

the rule. If an IP packet enters the ruleset from a different adapter, differ-
ent rules apply. This fact explains why the workaround from section NAT
Reflection is required.

Summary
OPNsense can manipulate packets in transit and exchange the addresses
in the IP header. The possibilities are manifold: an address translation can
affect only the IPv4 header of selected TCP/UDP ports (Port Forward). It
can also translate all packets to a selected destination address (One-to-One).
OPNsense also masters the standard function of every home router, i.e. to
“hide” your own IP network behind the public IPv4 address (Outbound
NAT).
Caution: an address translation always requires a firewall rule that allows
access. When non-local IP addresses are used, the configuration also needs
a virtual IP.
OPNsense does not even stop at IPv6 and lets NPTv6 rewrite entire prefixes.
This feature is required for Chapter 16 Multi-WAN.

104
Chapter 9

Management Interface

The firewall’s web interface is accessible via the Internet protocol. The
firewall needs a reachable IP address for this. If this address belongs to the
devices’ “normal” IP addresses, it is called in-band management. When this
interface gets disconnected or operates under a high load, the management
access response is slow or fails.

em1
em2 OpenSSH

em3 bsnmpd

RT-1
lighttpd
em4 em0

Firewall and routing Management

Figure 9.1: Out-of-band management

105
Chapter 9. Management Interface

Avoid this situation and access the firewall over a separate network adapter
with an additional IP address (see Figure 9.1). The management port works
out-of-band and:

• must not transport user data,

• receives an IP address, which is separated from the end-user networks,

• uses a firewall ruleset that permits only management protocols,

• allows outbound connections only in predefined IP ranges.

This method for interface hardening only makes sense if absolutely no

management access is possible via other interfaces.
OPNsense fulfills these requirements but does not provide a ready-to-use
configuration. It is up to the admin to create the management interface
and the firewall rules.
The following sections are a practical approach to create and secure man-
agement access to an OPNsense firewall.

Create a management interface

The network adapter for configuring the OPNsense firewall should be a
separate interface. In a virtual environment, this does not pose a challenge
because additional interfaces can be added easily by a few clicks in the
hypervisor’s configuration interface. With a physical firewall, the manage-
ment interface occupies an RJ45 port, which must be included when sizing
the hardware.
In the lab network, the first network adapter of any device is always re-
served for management access.

Secure management interface

By default, the HTTP web interface service and the SSH server listen on all
available IP addresses of the firewall. This freedom quickly comes to an
end when the respective process is bound to an interface under System →
Settings → Administration. The option Listen Interfaces is initially set to
All (recommended). When switching to the management interface MGMT
(Figure 9.2), the OPNsense machine no longer accepts management access

106
on the other network adapters’ IP addresses, even if firewall rules allow
this.

Figure 9.2: Accessing the web interface is only permitted through

the management adapter

The same method for securing access exists for the SSH service. The setup
can be found in the web menu at the bottom of the same page.

Define management subnets

Firewall rules scale much better if they contain aliases. The sample alias
mgmt_net contains all IP networks that may access the web interface of the
OPNsense firewall. A list of legitimate TCP/UDP ports is present in alias
mgmt_apps. Figure 9.3 on the next page shows these aliases for multiple
networks and ports. The alias for IP networks may contain IPv4 subnets
and IPv6 prefixes as well.
If the management network is divided into additional IP areas, the firewall
needs a static route to each subnet. Enter the missing routes at System →
Routes → Configuration. That way, OPNsense will reply to the requests
from the management network back to the management network and not
to the Internet via the default route.

107
Chapter 9. Management Interface

Figure 9.3: Firewall aliases for management networks and ports

Firewall rules
The ruleset of the management adapter at Firewall → Rules → MGMT
requires only two rules. Rule number one allows access from all IP networks
listed in the alias mgmt_net that use a port number of alias mgmt_apps
and are directed to the IP address of the firewall. The second rule blocks
and logs all other connection attempts, as shown in Figure 9.4.

Figure 9.4: The management interface’s firewall ruleset

Depending on the initial setup of the local OPNsense machine, the ruleset
of the network adapter MGMT still contains preconfigured entries, which
allow all inbound connections. These rules have the text “Default allow
* to any rule” in the description. They do not belong in the policy of a
management network and must be deleted.

108
Also, avoid the anti-lockout rule. Disable this rule at Firewall → Settings
→ Advanced (Figure 9.5). Once the Disable administration anti-lockout
rule option is selected, OPNsense will remove this particular firewall rule
from all network adapters.

Figure 9.5: Anti-lockout rule

Separate from end-user traffic

Further, it is best to separate the management network from the normal
data traffic transported by the firewall. Traffic entering through the regular
network adapters must not reach the management adapter.
A floating rule that applies to several interfaces, is ideally suited for this
purpose. Figure 9.6 on the following page shows the configuration of this
additional protection. Ultimately, this rule should block all network traffic
that enters the firewall through the network adapters WAN1, WAN2, LAN,
and DMZ and wants to leave through the management adapter MGMT.

Bandwidth limitation
Denial of service attacks (DoS) usually flood the victim with requests and
packets until they can no longer perform their routine tasks. These attacks
are even possible from inside the management network when a virus or
worm has crept in.

109
Chapter 9. Management Interface

Figure 9.6: A floating firewall rule protects from end-user traffic

The firewall can mitigate the effect by limiting the number of incoming
packets to a reasonable level. This bandwidth limitation falls under the term
Traffic Shaping. OPNsense offers this feature under Firewall → Shaper.
For the mitigation of DoS attacks, the firewall should transport a fixed
maximum, e.g. 10 Mbit/s, of data through its management interface.
The configuration is split in two sections. First, set a maximum bandwidth
in the Pipes section. Second, assign this restriction to a network adapter at
section Rules. Table 9.1 lists the used exemplary values.
Now all incoming and outgoing data streams of the MGMT adapter are
limited to 10 Mbit/s. For powerful firewalls, this value may be too low, and
for weak embedded systems even 10 Mbit/s might seem dangerously high.

110
 Two-factor authentication

Attribute Value
[Section Pipes]
Bandwidth 10
Bandwidth Metric Mbit/s
Mask (none)
Description 10mbps
[Section Rules]
Sequence 1
Interface MGMT
Proto ip
Source any
Destination any
Target 10mbps (from Pipes)
Description mgmt

Table 9.1: The management adapter’s bandwidth limitation

Two-factor authentication

It is advisable to strengthen the logon to the web interface with two-factor

authentication. This adds a six-digit PIN created by a token generator to
the normal password. The PIN is only valid for one minute and is generated
by a separate device or smartphone app.
Consequently, a login is only possible if the admin provides the two factors
knowledge (password) and possession (token generator) during authentica-
tion. OPNsense supports various apps for time-based, one-time passwords
with a PIN length of six or eight digits.
The login page of the web interface does not change for the second factor:
the additional PIN is simply put before the static password. The authenti-
cation process of OPNsense separates PIN and password and checks both
factors independently.

Set up the two-factor authentication in the firewall at System → Access →

Servers with the Add button. The new server is of type Local + Timebased
One Time Password and in this example uses the common token length of
6 digits.

111
Chapter 9. Management Interface

Note
The term “server” is misleading, the dial-in process does not require
another server, but rather, validates the PIN locally.

The next step is to associate each local OPNsense user account with a token
generator. Using the root user as an example, this user receives a crypto-
graphic key at System → Access → Users in the line OTP seed with the
checkbox Generate new secret (160 bit). After saving, OPNsense generates
the string and presents it as a QR code in Figure 9.7.
The admin needs a token generator, which is software or a smartphone
app that generates time-based PINs. OPNsense’s website lists Google Au-
thenticator and FreeOTP as supported apps for Apple iOS and Android.
Both apps operate in the same way: launch the app, create a new entry,
and finally scan the QR code. Then, based on the shared OTP seed, the
smartphone creates the required six-digit PIN for the root user.

Figure 9.7: Transfer the secret OTP seed by scanning the QR code
with a smartphone

112
 Summary

Note
Validate two-factor authentication setup at System → Access → Tester.

In the last step, the web interface must use the second factor for authen-
tication. Select the new server and disable the Local Database in the
Authentication section under System → Settings → Administration.

Figure 9.8: Two-factor authentication via smartphone app and password

From now on a successful login is only possible with a PIN and password.
In Figure 9.8 the admin authenticates via FreeOTP and his password. The
combined password is displayed in plain text for demonstration purposes
only. Usually, a browser hides the password entry behind asterisks or dots.

Summary
It is best to manage a firewall using a separate network adapter that is
specially protected and separated from user traffic. That approach increases
the security of the firewall and makes it less vulnerable to attacks or a
denial-of-service attempt.

113
Chapter 9. Management Interface

114
 Part III

For Experts

115
Chapter 10

IPsec VPN

If two remote networks want to communicate with each other, they usually
use a private link. Whether this is a leased line or dial-up connection, the
communication is either expensive or slow.
The alternative is to use the Internet as a transit infrastructure between the
networks. The Internet, as a public network, is used as a transport network;
the inter-connection is only virtually available as a Virtual Private Network
(VPN).

IPv4
Ethernet ï 10.1.1.25 IPv4
Ethernet ï 10.1.1.25
TCP/ Data TCP/ Data
ð 10.2.1.25 UDP ð 10.2.1.25 UDP

IPv4 IPv4
ï 198.51.100.1 ESP ï 10.1.1.25 TCP/ Data ESP ESP-
ð 192.0.2.3 header ð 10.2.1.25 UDP trailer auth.

RT-1 encrypted
RT-3
Site-1 authenticated Site-2
10.1.1.0/24 10.2.1.0/24
fd00:1::/64 fd00:2::/64

Figure 10.1: An IP packet gets more headers when traveling through a VPN

Private networks always have private addresses that are not allowed on
the Internet. So, how can an Internet router be convinced to transport
packages between two private locations? The answer is by a tunnel, which
is the technical basis for a VPN. The responsible tunnel router at site 1

117
Chapter 10. IPsec VPN

encapsulates the outgoing IP packets into an additional IP header. The

router will then pass the encapsulated packet to the router at the other
end of the tunnel at site 2. The inner IP header has the private addresses
of sites 1 and 2, and the outer header has the public addresses of the two
tunnel routers. The tunnel router of site 2 removes the outer header from
the received packet and forwards its contents as a full packet to the target
computer. Figure 10.1 shows two small networks, which are connected by
a VPN tunnel.

The Internet routers only see the outer header and transport the VPN
packets, just like all other packets. For them, a VPN packet is nothing
special since they cannot see the inner structure of the packet.
The members of the private networks are not aware of the tunnel. For them,
the remote stations in the other network are directly addressable.

Security
The transmitted packets flow through the public Internet as they pass
through several Internet Service Providers. It is impossible for the operator
of the VPN to influence the path of the VPN. For the security of the VPN
connection, there is a useful recipe against curious readers: encryption.

Ethernet IP TCP/ Data

UDP
Plain text

Ethernet IP ESP IP TCP/ Data ESP ESP

header UDP trailer auth.

encrypted
authenticated

Figure 10.2: A single IP packet with and without IPsec headers

In principle, it is quite simple: before a packet enters the Internet, the VPN
router encrypts the content. And after it leaves the Internet and reaches

118
 Lab setup

the private network, the content is decrypted again. Figure 10.2 shows the
IP packet with and without IPsec protection.
The technical implementation is more complicated because cryptography
is not just switched on. In practice, there are cryptographic procedures,
algorithms, one-way hashing functions, key exchange, and the magic ques-
tion of authentication. All these methods are intended to help make the
encrypted packets unreadable to an unauthorized person.

Lab setup
The two WAN networks represent the Internet, which does not transport
private IP addresses. The site networks need to communicate with each
other using OPNsense firewalls and VPN tunnels. A VPN tunnel spans from
RT-1 to RT-3, and another VPN tunnel goes from RT-1 to RT-2. The structure
is shown in Figure 10.3.

RT-1 em1 10.1.1.1 Site-1 10.1.1.0/24

OPNsense
em0
em0– 10.5.1.1 fd00:1::/64 DMZ
em4 em3 10.4.1.0/24
VPN fd00:4::/64
192.0.2.1 198.51.100.1 tunn
el 198.51.100.2

198.51.100.6 WAN-1
198.51.100.0/24 em3
em1
VPN

2001:db8:1::/64
em2
RT-core 10.4.1.2
tunn

OPNsense
em0
em0– 10.5.1.3
em2 RT-2
el

192.0.2.6 OPNsense
em0
em0– 10.5.1.2

WAN-2
192.0.2.0/24 192.0.2.3
2001:db8:2::/64
em2
10.2.1.0/24
RT-3 10.2.1.3
Site-2 fd00:2::/64
OPNsense em1
em0
em0– 10.5.1.3

Figure 10.3: VPN tunnels connect the site networks over the Internet

119
Chapter 10. IPsec VPN

In practice, there could be a device somewhere in the data path between

the VPN endpoints, which modifies the IP addresses of the VPN packets
using NAT (see Chapter 8). When this happens, IPsec will react strongly
and refuse authentication of the remote peer. To implement this scenario,
RT-core will sit in the data path between RT-1 and RT-3 and will eventually
manipulate the source address of the packets like a DSL router does for its
connected clients. Outgoing packets from RT-3 receive a different sender
address when passing RT-core and reach RT-1 with a wrong identity.

Firewall RT-1 is connected to the Internet via several lines. In practice, it is

best to create two redundant VPN tunnels, which protect from the failure
of a single Internet connection. How does a VPN gateway recognize an
unreachable partner? The detection of dead peers (Dead Peer Detection,
DPD) is a prerequisite for switching from the primary VPN connection to
the backup tunnel.

Connection setup

The setup of a VPN tunnel is more complicated than the three-way hand-
shake of the TCP protocol. That’s because both parties must agree on an
encryption algorithm, authentication, IP networks, and session timeout.
The following configuration process is done on firewall RT-1. For the
other two firewalls, only the IPsec interface, and the IP prefixes have to be
adapted.

1. Internet Key Exchange (IKE), phase 1.

In this first step of the negotiation, the two firewalls authenticate
each other and form a control connection (Security Association, SA).
This agreement is only possible if the selected settings for the Diffie-
Hellman group (DH), key algorithm, hash procedure, and key match.
The goal of phase 1 is a secured and encrypted connection between
the VPN partners, through which the values for the actual VPN tunnel
can be securely negotiated in phase 2.
The IPsec tunnel setup in the web GUI is at VPN → IPsec → Tunnel
Settings.

120
 Connection setup

Note
The rule of thumb for the choice of crypto algorithms is this: the
firewall hardware requires more computing power and stronger
encryption as the number of the algorithms increases.

2. Internet Key Exchange (IKE), phase 2.

In the second phase, the firewalls negotiate a data connection. If
the proposals of the VPN gateways match, they will use that data
connection to transport IP packets of the users.
In this phase, the VPN tunnel also receives information about which
local IP networks are sent into the tunnel and which IP prefixes are
reachable behind the tunnel.
The settings of phase 2 always correspond to phase 1. With OPNsense
they appear by clicking on the button Show Phase-2 entries. Then a
button with a plus sign appears to create a new Phase-2.

3. Enable IPsec. Now OPNsense sends and accepts IPsec packets on

selected interfaces. It will also start negotiating remote peers in the
background.

The example configuration for RT-1 uses the settings from Table 10.1 on
the next page for phase 1 and Table 10.2 for phase 2.
When the configuration is saved with the button Apply changes, the firewall
tries to enter phase 1 with its peer 192.0.2.3 and negotiate the encryption
algorithm and key length.

Next, set up the VPN on firewall RT-3. Step 1 uses the IPv4 address
192.0.2.1 of RT-1. The same applies to the Local Network and Remote
Network entries from step 2, which are mirrored in the configuration of
RT-3.

121
Chapter 10. IPsec VPN

Attribute Value
Connection method default
Key Exchange version auto
Internet Protocol IPv4
Interface WAN-2
Remote gateway 192.0.2.3
Description RT-3 via IPv4
Authentication method Mutual PSK
My identifier My IP address
Peer identifier Peer IP address
Pre-Shared Key e.g. YOGHURT
Encryption algorithm AES / 256
Hash algorithm SHA256
DH key group 14 (2048 bit)
NAT Traversal Disable
Dead Peer Detection 10 seconds, 3 retries

Table 10.1: Simple site-to-site tunnel from RT-1

Attribute Value
Mode Tunnel IPv4
Local Network (Type) LAN subnet
Remote Network (Type) Network
Address 10.2.1.0/24
Protocol ESP
Encryption algorithms AES / auto
Hash algorithms SHA256

Table 10.2: What travels through the site-to-site tunnel between RT-1 and RT-3?

Firewall
The IPsec tunnels of the RT-1 and RT-3 gateways are now set up, and both
firewalls try to reach each other. However, the packet filter (see Chapter 6)
might block the traffic. OPNsense is so friendly and creates the necessary
rules automatically. This only works if the IP address of the remote gateway

122
 Connection setup

is known in advance. With unknown VPN partners, the automation creates

strange filter rules, which could completely prevent the tunnel negotiation
(see Technical background on page 131).
If in doubt, manually create firewall rules to allow incoming VPN packets.
IPsec VPN tunnels use the UDP protocol with port 500 to negotiate the con-
nection. The bearer packets then use the protocols Encapsulated Security
Payload (ESP), Authentication Header (AH), or merely the UDP port 4500,
if NAT-traversal (see Address translation on page 124) has discovered a
NAT gateway somewhere.
The rules for allowing these ports and protocols are found at Firewall →
Rules and allows the physical interface to receive the encrypted packets. In
the demonstration, these are WAN1 and WAN2.

OPNsense also filters within the VPN tunnel. As soon as IPsec is acti-
vated, the firewall suddenly has another interface IPsec, which resembles
the VPN traffic.

Figure 10.4: A firewall ruleset to filter inside the VPN tunnel

The filter rules apply to internal IP addresses and filter packets flowing
through the tunnel. The use of this ruleset is common if the remote network
is unknown or belongs to an untrusted party, e.g. a partner company.

Note
An empty ruleset on the IPsec interface blocks incoming data traffic that
arrives through the VPN tunnel. Outgoing packages are not affected.

For example, firewall RT-1 could only permit requests to specific IP ad-
dresses at its LAN network. In the simplest case, the ruleset allows all

123
Chapter 10. IPsec VPN

packets from the remote network to pass through the IPsec connection, as
configured in Figure 10.4.

Depending on the scenario, it might make sense to filter the network traffic
flowing through the VPN tunnel in an outgoing direction. Then, only specific
connections may enter the tunnel. At this point, the necessary filter rules
belong to the LAN adapter.

Status
If all parts of the VPN connection are configured correctly, the web GUI
should honor VPN → IPsec → Status Overview with connection informa-
tion. Figure 10.5 shows the status of a negotiated VPN connection.

Figure 10.5: Status information of a connected VPN tunnel

A successfully established VPN tunnel shows up as INSTALLED and provides

positive values for incoming and outgoing bytes below Stats.

Address translation
A router in the path might change the IP address of packets traveling be-
tween the VPN peers (NAT, see Chapter 8). Then IPsec faces a problem
because it cannot distinguish between an intended IP change and a mali-
cious manipulation of the packet content. As a result, the VPN connection
will fail.

124
 Address translation

Network address translation is mandatory in many IPv4 environments.

Therefore, IPsec gateways use NAT traversal to work around this situation.
The gateway encapsulates its outgoing IPsec packets in regular UDP data-
grams that can pass any NAT routers or firewalls. The original IPsec packet
remains unchanged, so all checksums and signatures are valid.

Next, the lab firewall RT-core performs address translation. The device
overwrites the source address of packets from RT-3 with its IP address of
the em1 interface. The configuration is shown in Figure 10.6.

Figure 10.6: Address translations are troublemakers for IPsec

When the new NAT rule on RT-core is active, the VPN tunnel between RT-1
and RT-3 will not transport any more data. That’s because no packets from
the authenticated IPv4 address arrive at RT-1.
Firewall RT-1 must identify its partner by a criterion other than the sender
address. It is best to recognize the VPN partner by an identifier. In the
simplest case, this identifier is the IP address of the remote gateway. Even
a combination of letters is allowed. The identifier provides a bit more
security when it comes to preventing unknown VPN peers from guessing
the Pre-Shared Key.

Which changes to the VPN configuration are necessary to detect modi-

fied IP addresses? In the Tunnel Settings configuration, navigate to Phase 1
proposal (Authentication). If a NAT gateway disrupts the VPN negotiation,
the peer identifier Distinguished Name is suitable.
The settings in Figure 10.7 will make RT-1 recognize the remote gateway
192.0.2.3 by the string RT-3 instead of the source IP address. That way,

125
Chapter 10. IPsec VPN

Figure 10.7: OPNsense recognizes the VPN peer by its identifier

RT-1 can identify the sender of the IKE packets, even if the IPv4 address
of the incoming packets has been changed on its way. RT-1 will also cease
introducing itself with its IP address, and begin using an identifier.
The NAT detection is located at the bottom of the tunnel settings and is
called NAT Traversal. The option Enable triggers VPN partners to detect an
address translation between them. With Force, a NAT gateway is assumed,
and the VPN packets are unconditionally wrapped in UDP datagrams.

To which IPv4 address should firewall RT-1 send the IKE packets to es-
tablish a VPN tunnel? Since the address of the remote gateway is unknown,
RT-1 can only wait for connection requests – that’s what the connection
method Respond only is all about.

The remote firewall RT-3 needs to start the IPsec connection (connec-
tion type Start immediate). This approach is recommended because RT-3
knows the IP address of firewall RT-1. Compared to the previous configura-
tion, RT-3 requires the same settings, whereby the Distinguished name is
configured in reverse order.

Dead Peer Detection

The health check of the VPN tunnel has an appropriate name. Each firewall
sends a sign of life to its VPN partner and waits for a response to detect a

126
 IPv6

failed remote peer. If these heartbeats stop for a particular time, the tunnel
is terminated on purpose. Without Dead Peer Detection (DPD), the faulty
tunnel continues to run senselessly until its lifetime is exceeded and a new
tunnel is negotiated. And that could be several hours later.
Fast error detection is necessary so that the firewall can react quickly. The
IKE process immediately starts a new tunnel setup, or the routing protocol
takes care of an alternative path to the target network.
In short, DPD is a kind of ipsec-ping that can act on failure. The configura-
tion expects a time interval between two pings and a maximum number of
retries (Figure 10.8).
If no response arrives within the time span, the firewall discards this tunnel
and negotiates a new one.

Figure 10.8: Dead Peer Detection uncovers failed VPN gateways

IPv6
IPsec is a security protocol from the IPv6 realm. By popular request, it was
also specified for IPv4. Initially, every IPv6 device was obligated to commu-
nicate via IPsec, but during development this requirement was relaxed and
IPsec was downgraded from must to should.
An OPNsense firewall is capable of securing the communication between
two IPv6 sites with IPsec. From the perspective of IPsec and its configura-
tion, the only two changes are the IP addresses of the VPN partners and
the definition of the prefixes. In brief: replace the IPv4 addresses by IPv6
addresses in the configuration.

127
Chapter 10. IPsec VPN

A new VPN tunnel on firewall RT-1 can have the same values for authentica-
tion, algorithms, and key exchange. After setting up the IPv6 tunnel on RT-1
and RT-3, the clients from site 1 (fd00:1::/64) can reach their neighbors in
site 2 (fd00:2::/64). Below is a packet trace from CL-1 to CL-2 showing the
participating firewalls on the way:
root@cl-1 ˜> traceroute6 -In fd00:2::25
traceroute to fd00:2::25 (cl-2), 30 hops max, 80 byte packets
1 2001:db8:2::1 5.277 ms 1.285 ms 1.241 ms
2 * * *
3 fd00:2::25 1.853 ms 2.044 ms 2.118 ms

Even IPv6 networks may have an address translation somewhere, although

this is almost always unnecessary. As a consequence, it is safe to deactivate
NAT Traversal in advance.
The status overview also includes the IPv6 tunnels. The output is almost
the same, but with the longer IPv6 addresses as tunnel endpoints. Fig-
ure 10.9 shows the negotiated tunnel between two gateways over the
WAN-2 network.

Figure 10.9: An IPsec tunnel carries local and remote IPv6 prefixes

VPN throughput
A firewall needs more performance when it must encrypt every outgoing
packet and decrypt every incoming packet. The achievable throughput rates
when using IPsec are always below the regular routing performance.
The stronger the crypto algorithms, the more time and performance the
firewall needs to work on the packets. The IPsec bandwidth depends on the

128
 Troubleshooting

hardware and software: the overall throughput can increase and decrease
with the operating system, the FreeBSD kernel, and the implementation of
the algorithms.
It also depends on the method of how the performance of the IPsec gate-
way is measured. Short measurements or large IPsec buffers might show
significantly higher bandwidths because packet loss does not occur yet.
Unfortunately, only a few manufacturers disclose in their datasheets which
test methodology they use to determine the IPsec performance. For market-
ing purposes, a high peak value may be more important than the average
value of a long-term measurement.
The correct measurement of the IPsec performance of a firewall is complex
and takes place in Chapter 20.

Troubleshooting
Finding an error in the VPN environment is difficult because many hurdles
can prevent a tunnel from forming correctly. OPNsense assists with VPN →
IPsec → Advanced Settings and multiple debug options.
The most basic tools ping and traceroute can not help too much. They
can merely confirm that the tunnel is interrupted somewhere, even though
DPD has already investigated that.

For structured troubleshooting, one VPN firewall must initiate the tun-
nel setup (connection method Start immediate), and the remote gateway
must react to it (connection method Response only). By default, both sides
will start the negotiations. To begin the investigation, switch to passive
mode on the remote gateway.
Troubleshooting now focuses on the responder, who will reject the VPN
connection for a reason. And this reason will become more or less clear by
analyzing the log file located at VPN → IPsec → Log File.

Error pattern
The passive firewall will report an error in its log file during VPN negotia-
tion. The following messages show typical error patterns for problems or
differences in the configuration.

129
Chapter 10. IPsec VPN

Peer identifier mismatch

Oct 15 21:10:14 RT-1 charon[63249]: 14[CFG] <6> looking for pre-shared \
key peer configs matching 198.51.100.1...192.0.2.3[FW-3]
Oct 15 21:10:14 RT-1 charon[63249]: 14[IKE] <6> no peer config found
Oct 15 21:10:14 RT-1 charon[63249]: 14[ENC] <6> generating \
INFORMATIONAL_V1 request 2982182528 [ HASH N(AUTH_FAILED) ]

The remote peer is recognized by the identifier Distinguished name. The

local firewall is contacted by a partner whose ID (FW-3) or IPv4 address
(192.0.2.3) is unknown.
Solution: go to VPN → IPsec → Tunnel Settings and change the peer
identifier to the string used by the partner.

Pre-Shared Key mismatch

Oct 15 21:16:43 RT-1 charon[63249]: 14[ENC] <7> invalid ID_V1 payload \
length, decryption failed?
Oct 15 21:16:43 RT-1 charon[63249]: 14[ENC] <7> could not decrypt payloads
Oct 15 21:16:43 RT-1 charon[63249]: 14[IKE] <7> message parsing failed
Oct 15 21:16:43 RT-1 charon[63249]: 14[ENC] <7> generating \
INFORMATIONAL_V1 request 1932693488 [ HASH N(PLD_MAL) ]

If the encryption key on the VPN endpoints is different, the VPN process
will detect invalid packet content during decryption and discard it.
Solution: compare and correct the keys on both endpoints.

IKE negotiation failed

Oct 15 21:18:49 RT-1 charon[63249]: 14[CFG] <8> received proposals: \
IKE:AES_CBC_192/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1536
Oct 15 21:18:49 RT-1 charon[63249]: 14[CFG] <8> configured proposals: \
IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1536
Oct 15 21:18:49 RT-1 charon[63249]: 14[IKE] <8> no proposal found

Both firewalls compare their available crypto algorithms in phase 1, and

the responder selects the most robust algorithm from the similarities. If
there are no similarities, the negotiations fail, and the responder will report
the failure.
Solution: the offers of Phase 1 proposal (Algorithms) must have common
values for encryption, hash function, DH group, and key length on both
firewalls.

130
 Technical background

ESP negotiation failed

Oct 15 21:31:06 RT-1 charon[81227]: 10[CFG] <con1|3> received \
proposals: ESP:3DES_CBC/HMAC_MD5_96/NO_EXT_SEQ
Oct 15 21:31:06 RT-1 charon[81227]: 10[CFG] <con1|3> configured \
proposals: ESP:AES_CBC_192/HMAC_SHA1_96/NO_EXT_SEQ
Oct 15 21:31:06 RT-1 charon[81227]: 10[IKE] <con1|3> no matching \
proposal found, sending NO_PROPOSAL_CHOSEN

The firewalls also choose the strongest common encryption for phase 2, but
this time the encryption is used to secure the transferred end-user packets.
The negotiation fails if no similarities are found. Then the responder reports
a similar message as in the IKE negotiation, but with a note about ESP,
which is an indicator for phase 2.
Solution: the offers of Phase2 proposal (SA/Key Exchange) must also have
a common basis for encryption, hash function, DH group, and key length
on both firewalls.

Technical background
OPNsense relies on the software strongSwan [10] for the cryptographic im-
plementation of the VPN tunnels. After the predecessor FreeS/WAN closed
its doors for further development in mid-2004, a total of three projects
forked the code. Besides Openswan and Libreswan, the implementation of
strongSwan is probably the best known and most powerful software. The
OPNsense firewall uses the stable version 5.9.1 of strongSwan.

In the underlying FreeBSD operating system, strongSwan runs two pro-

cesses: starter takes care of starting, stopping, and configuring the IKE
daemon. And charon is responsible for negotiating with the VPN partners
via the IKE protocol. The IPsec packets are sent by the kernel itself, without
the need of an additional software process.
If a VPN tunnel fails to operate and the setup is correct, it is worth taking
a look at the strongSwan configuration file ipsec.conf. The provider’s
website [10] explains every possible directive in the Wiki section and gives
hints on usage, compatibility, and required version.
The automatically generated VPN rules are integrated into the normal fire-
wall rules. The website at Firewall → Rules lists them at the beginning of

131
Chapter 10. IPsec VPN

the user-defined rules. The Order of processing is described in Chapter 6

on page 82.
Note
If the automatically generated rules cause problems, they can be easily
switched off at VPN → IPsec → Advanced Settings.

Outlook
VPN is a versatile topic and can fill an entire book. This section briefly
discusses other notable enhancements. The focus is not limited to the
IPsec protocol suite and also includes alternative approaches to setting up a
virtual private network.

IKEv2
The original version of IKE goes back to 1998 and has shown some weak-
nesses in practice. The updated version IKEv2 was launched in 2005 and
was fine-tuned until 2014. IKEv2 is more economical in messaging and
makes DoS attacks more difficult. However, the biggest advantage is the
improved interaction with NAT gateways because IKEv2 already has NAT
traversal built in. If the IKEv2 partners detect an address manipulation
during the negotiation, they encapsulate everything in UDP segments to
operate through the NAT gateway.
IKEv2 has been on the market for a few years, but is not as widespread as
its predecessor. If possible, both VPN peers should use the newer variant. If
in doubt, OPNsense provides the Auto option, which leaves the selection to
the VPN gateways. Then both partners prefer the newer version, but switch
back to IKEv1 to comply with older implementations.

Mobile clients
The tunnels constructed so far are permanent connections between two
VPN gateways. Endpoints of a VPN connection can also be mobile clients
that only sporadically require an encrypted connection to the private net-
work. These include laptops, smartphones, or home offices equipped with
a low-cost VPN router.

132
 Outlook

OPNsense provides a dial-in service for mobile clients at VPN → IPsec →

Mobile Clients. The OPNsense firewall is the VPN server, and the remote
devices are the VPN clients. These devices require a VPN software that is
compatible with IPsec. Start the software when VPN access is required. The
authentication process is similar to that of a permanent VPN tunnel and
can also be strengthened using a username and password or certificates. In
larger environments, user accounts reside in a central directory server that
OPNsense consults via LDAP or the RADIUS protocol (see Chapter 15).
As soon as the VPN client is connected to the firewall, it receives an addi-
tional IP address. The client will use that address to communicate with
the other end devices in the remote network through its own VPN tunnel.
Whether the VPN clients can reach each other depends on the firewall policy
of the IPsec interface.
A VPN server for mobile clients with user login and server authentication is
shown as an example in Figure 10.10.
Application servers VPN server VPN clients

RT-1
WAN-1

LAN Internet
WAN-2
Authentication IPsec VPN
server

Plain text Encrypted

Figure 10.10: OPNsense acts as a VPN dial-in gateway for mobile clients

Tinc VPN
Tinc is an open-source VPN software with a focus on full mesh. Tinc
establishes a VPN tunnel between two gateways to connect the attached
private networks, just like any other VPN software. What seems rather
unspectacular here becomes interesting as the number of VPN gateways
increases. With Tinc, all gateways connect to each other, even if there is
no direct tunnel configured between them. The result is a full mesh. If one

133
Chapter 10. IPsec VPN

of the VPN tunnels fails, the software still finds the shortest route to the
destination by choosing one of the remaining VPN nodes.
Tinc wants to be simple, robust, and safe. The simple attribute refers to the
VPN connection setup. There is no exchange of certificates or negotiations
about the best crypto algorithms.
The VPN network is robust because it consists of many tunnels. All Tinc
nodes permanently check their mutual reachability, build new tunnels, and
optimize their path tables for short distances. Manual intervention by the
administrator is not necessary.
The demand for security must not be missing in any VPN software. Tinc
offers, with preselected strong algorithms, the use of stable libraries and
openly deals with program errors and security holes.
Tinc does not encrypt its packets with IPsec but with a customized security
protocol. This protocol uses the OpenSSL library and is based on AES-256
for encryption and SHA-256 for hashing function.
Tinc is not included with OPNsense but is available as a plug-in via System
→ Firmware → Plugins. When installed, Tinc integrates into the web GUI
at VPN → Tinc.
For the firewalls in Figure 10.11, Tinc would automatically connect RT-2 to
RT-3 to allow communication between the connected networks.

RT-1
OPNsense
Con
f ig u r
ed tu
nn el
Con
f ig u r

Internet
ed tu

RT-2
el
nn

OPNsense
n
tun
el

c
mi
na
Dy

RT-3
OPNsense

Figure 10.11: Tinc built a full-mesh network to connect VPN gateways

134
 Outlook

ZeroTier

ZeroTier describes itself as a global Ethernet switch to which any end device
can connect. The participants plug to this switch and then communicate
similarly to a local network.
The Ethernet switch is imaginary and serves just as an aid for a better
understanding. In fact, the participants connect via VPN to a predefined
VPN server operated by ZeroTier. The VPN server connects its clients to
each other, so that the devices can also communicate among themselves. It
creates an encrypted peer-to-peer network.
This transport network is responsible for the secure communication and
authentication of all participants. The transport network’s additional task
is to find the routing path efficiently.
Above this network layer, ZeroTier emulates Ethernet. That way, applica-
tions can use familiar protocols to communicate. For the participants, the
VPN appears as an Ethernet switch. The technology behind it resembles
IPsec and VXLAN.
The data packets between the ZeroTier clients are encrypted and only un-
derstood by other clients, not by the ZeroTier servers. The implemented
algorithm comes from the area of elliptic curves and offers a key length of
256 bit. The client software automatically takes care of key exchanges and
certificate management.

What does this global Ethernet switch have to do with OPNsense? OPN-
sense offers the ZeroTier client as a plug-in and thus has a connection to the
global switch. From the perspective of an OPNsense firewall, the ZeroTier
switch is another network adapter with an IP address, routes, and firewall
rules. This adapter can be used to connect locations, mobile clients, or offer
management access.
ZeroTier implements its own software client ZeroTier One, which is offered
in source code and as a precompiled binary for many operating systems
and smartphones.

The setup of a ZeroTier VPN begins with registration on the ZeroTier

web page. This registration gives the customer access to the web-based
management area to set up a new Ethernet switch. ZeroTier provides a net-
work identifier which the clients use for dialing in. The network identifier

135
Chapter 10. IPsec VPN

distinguishes the customers from each other and prevents customer A from
reaching the devices in customer B’s network.

The example in Figure 10.12 uses the ZeroTier cloud as a fully meshed back-
bone network, which automatically finds the shortest connections between
locations.

Site A
RT-A RT-C
OPNsense OPNsense
Site C

Site B
RT-B RT-D
OPNsense OPNsense
Ethernet Site D
switch
Overlay
network

Underlay
network ZeroTier
Controller

Internet
ZeroTier
Endpoint

Figure 10.12: The ZeroTier Ethernet switch connects OPNsense firewalls

Summary
IPsec is a protocol family with the explicit goal of connecting two remote
networks via an insecure medium. The main priorities are transmission
security and authentication.
OPNsense masters the art of IPsec. The various possibilities of encryption,
authentication, and transportation make initial setup complicated. However,
it also brings flexibility if OPNsense needs to communicate with IPsec peers
of other manufacturers.
The implementation of OPNsense interconnects site networks. It also offers
mobile clients a secure dial-in from anywhere to the company network for
secure access to servers and applications.

136
Chapter 11

OpenVPN

OpenVPN is an open-source software for setting up a Virtual Private Net-

work. Through this VPN tunnel, individual devices or entire networks can
communicate with each other.

OpenVPN is the community’s answer to the complexity of IPsec. While

the target of both protocols is the same, OpenVPN is easier to configure
and easier to operate through packet filters and NAT gateways. OpenVPN
is not an RFC standard, but a ready-to-use software that is available for
most operating systems and smartphones. OpenVPN does not implement
the encryption routines itself but relies on the popular OpenSSL library.
OpenVPN uses the virtual network adapters TUN and TAP as device drivers.

Operation
The functionality of OpenVPN corresponds to that of IPsec, i.e. a VPN gate-
way establishes an encrypted channel to another gateway. Through this
tunnel, both endpoints can communicate securely with each other and also
protect the communication of other devices.

The tunnel setup uses the client-server principle: an OpenVPN process

assumes the role of the client and establishes the connection. The other
process acts as a server and waits for new requests. For the operation of
the VPN, it is irrelevant which gateway started the connection.

137
Chapter 11. OpenVPN

Authentication is one of the significant factors for the privacy of transmitted

data. OpenVPN prefers the use of certificates. In simple environments, a
static key or the use of a username with a password is acceptable.

The VPN tunnel has two different modes of operation:

• Routing mode. The tunnel is a point-to-point connection (at OSI
layer 3) and only transports IP packets. The OpenVPN process runs
on a router that forwards IP packets based on its routing table.
• Bridge mode. The tunnel is an Ethernet connection (OSI layer 2) and
transmits Ethernet frames like an Ethernet switch. LAN ports and VPN
tunnels are bridged together, and the OpenVPN process transmits the
frames. That way, Ethernet networks can be connected via WAN links
without the clients’ need to contact a router.

Authentication
OPNsense supports several methods of authentication, which differ in
complexity and security.

Username
The simplest and most insecure form of authentication is to login with a
username and password. That’s because users write down their passwords
or choose obvious terms. This method of authentication can be combined
with another type to achieve a multi-factor authentication.

Pre-shared key
The OpenVPN peers trust each other if they share the same cryptographic
key and use it for authentication. Before the first authentication, the key
must be distributed to all participating OpenVPN firewalls via a secure
connection.
The authentication by pre-shared key is easy to achieve. The security of
the data transmitted depends on the confidentiality of the key. When a
pre-shared key is compromised, an attacker can decrypt all intercepted
packets.

138
 Differences to IPsec

Certificates
When authentication is based on certificates, each communication peer
requires a certificate. Client and server exchange their certificates and check
each other’s validity.

It takes time and energy to implement this method because certificates have
to be created, distributed, and may be revoked. Despite these challenges,
certificate-based authentication is the most secure form of authentication.

Differences to IPsec
OpenVPN and IPsec are both considered mature and secure. OpenVPN
is open-source software and therefore visible to everyone. IPsec is not a
software, but a collection of protocols used by vendors for their software.
The software may be under a proprietary license that does not disclose the
source code.

IPsec is already integrated into most operating systems, while OpenVPN is

a 3rd-party product. The software supports all popular operating systems
but first needs to be installed.
Table 11.1 on the following page compares IPsec to OpenVPN, as OPNsense
offers both technologies.

The throughput refers to the IPsec implementation of strongSwan and

the OpenVPN package used by OPNsense. Also, the support of the CPU
instruction set AES-NI as a key player, which significantly improves the
performance of OpenVPN.

OpenVPN is an implementation in user-space, while strongSwan uses the

IPsec capabilities of the kernel. This difference makes OpenVPN more
easily portable to other operating systems and does not require a kernel
module. However, the frequent context changes have a negative effect on
performance because the OpenVPN process runs in user space, while the
network drivers for emX and tunX are located in kernel space.

139
Chapter 11. OpenVPN

Feature IPsec OpenVPN

Configuration advanced simple
Installation provided by OS additional software
Authentication PSK, certificate, PSK, certificate,
username/password1 username/password
Encryption AES 128/192/256, AES 128/192/256,
algorithm Blowfish 128/192/256, Blowfish 128,
3DES, DES, CAST128 3DES, DES, DESX, CAST5
Camellia 128/192/256 Camellia 128/192/256
SEED, RC2
Auth Digest SHA 1/256/384/512, SHA 1/224/256/384/512
algorithm MD5, AES-XCBC MD4, MD5, RIPEMD,
BLAKE2, SHAKE,
whirlpool
Security level high high
Throughput high medium
Interaction
with firewall simple to filter complex to filter
with NAT with NAT traversal no issues
Protocol support
IPv4 yes yes
IPv6 yes yes
non-IP yes (with GRE) yes (Bridge mode)
Multipoint tunnel yes (with DMVPN) yes
Compatible with
proxy server no yes
Key rotation automatic automatic (certificate)
manual (PSK)

Table 11.1: IPsec and OpenVPN compared

1
The authentication with username and password is only available for mobile clients.

140
 Lab setup

Lab setup
The two firewalls RT-1 and RT-2 create a VPN tunnel to enable data commu-
nication between their site networks. In this scenario, RT-2 with its DMZ
network becomes a separate site that has no direct connection to the site 1
network.
The VPN service of RT-1 also offers dial-in to the OpenVPN client. Fig-
ure 11.1 shows the connectivity of the VPN gateways and a client in site 2.
The tunnels’ IP addresses derive from 10.6.0.0/16 and fd00:6::/64.
Site-1 10.1.1.0/24
fd00:1::/64 10.1.1.1
RT-2
OPNsense
em1
10.4.1.2
RT-1 S i te -to -s i te
V P N tu n n e l em2
OPNsense
em4 em3 em3
192.0.2.1 198.51.100.1 198.51.100.2
C l i e n t-s e rv e r V P N tu n n e l

WAN-2
192.0.2.0/24 WAN-1 198.51.100.0/24
2001:db8:2::/64 2001:db8:1::/64
192.0.2.3

em2
labsrv
RT-3 CentOS 8 10.4.1.7
OPNsense eth1
em1
10.2.1.3 DMZ
Site-2 10.2.1.0/24 10.4.1.0/24
fd00:2::/64 fd00:4::/64

CL-2
Windows 10

Figure 11.1: Lab setup for OpenVPN tunnels

Site-to-Site tunnel
In the first scenario, OpenVPN connects site 1 to the DMZ network through
a site-to-site VPN. This tunnel spans from firewall RT-1 to RT-2 and connects
the attached networks.

141
Chapter 11. OpenVPN

OPNsense distinguishes between server and client in the OpenVPN config-

uration. The client initiates the tunnel setup and the server reacts to it.
In this scenario, RT-1 assumes the role of server and RT-2 starts its VPN
activity as a client.

The IP packets from several computers will later flow through the tun-
nel. The confidentiality of many data connections rely on the tunnel’s
security. Depending on the environment, a pre-shared key or a certificate
can achieve the required security level. The following instructions use a
shared key.
The tunnel begins and ends at the public IP addresses of the VPN gateways.
Within the tunnel, OpenVPN uses additional addresses that may originate
from a private IP range. These addresses can later be used to quickly check
whether the tunnel is working or for permanent status checks of a monitor-
ing system.

The OPNsense web GUI locates the configuration of OPNsense at VPN

→ OpenVPN → Servers. The Peer to Peer mode is the best option for the
intended scenario. The remaining values are listed in Table 11.2, followed
by additional details and descriptions to assist in the configuration.

Attribute Value
Server Mode Peer to Peer (Shared Key)
Protocol UDP
Device Mode tun
Interface WAN1
Local port 1194
IPv4 Tunnel Network 10.6.12.0/24
IPv6 Tunnel Network fd00:6:12::/64
Redirect Gateway 2
IPv4 Local Network 10.1.1.0/24
IPv6 Local Network fd00:1::/64
IPv4 Remote Network 10.4.1.0/24
IPv6 Remote Network fd00:4::/64
Address Pool 2


Table 11.2: RT-1 acts as VPN server for a site-to-site tunnel

142
 Site-to-Site tunnel

OpenVPN accepts any port number. Without further specification, OpenVPN

uses the default port 1194 of the UDP protocol.
The drop-down list of Encryption algorithm and Auth Digest Algorithm is
overwhelming. When in doubt, use AES with SHA. This recommended
combination establishes a remarkably high security level even with a small
key length (AES-128-CBC and SHA-256).
OPNsense creates a new cryptographic key if the option Automatically gen-
erate a shared key is checked. The generated key is available as soon as all
settings are confirmed with the button Save.
OPNsense uses the values from the Tunnel Settings to populate the routing
tables. The entries at IPv4 Remote Network are entered into the local rout-
ing table. The opposite VPN router requires the specification of IPv4 Local
Network for its routing decisions. The same applies to IPv6 networks. If
the tunnel transports several networks, these must be separated by commas.

Firewall RT-1 is now a fully-fledged VPN gateway, which expects a re-

mote peer. The renewed view into the configuration shows the created key
in the text field Shared Key. The content must find its way unchanged into
the settings of the RT-2 client. It is best to choose a secure communication
channel for the copy.

Security

If no encryption is specified, OpenVPN uses the Blowfish algorithm. This

algorithm may be fast, but it does have security weaknesses. The vendor no
longer recommends the Blowfish algorithm and prefers AES with a 256-bit
key.
Even though Blowfish’s weakness does not apply to all environments, the
AES algorithm is still a better choice. The crypto performance of AES
is comparable to Blowfish despite a much longer key, as shown by the
throughput measurements in Chapter 20 on page 285.
Change the algorithm to AES at VPN → OpenVPN → Server or Clients in
the field Encryption algorithm. The recommended selection is AES-256-
CBC. The OpenVPN service and the connected tunnels will restart when
the configuration is modified.

143
Chapter 11. OpenVPN

Client
The remote peer of the OpenVPN server is the firewall RT-2. Configure the
settings of this host to establish a VPN connection with RT-1 at VPN →
OpenVPN → Clients. To do this, the client must provide the IP address of
the server. The specifications for local and remote IP networks are inverted
to RT-1 and listed in Table 11.3. The pre-shared key must be identical on

Attribute Value
Server Mode Peer to Peer (Shared Key)
Protocol UDP
Device mode tun
Interface WAN1
Remote server
Host or address 198.51.100.1
Port 1194
Shared Key insert the key from RT-1
IPv4 Tunnel Network 10.6.12.0/24
IPv6 Tunnel Network fd00:6:12::/64
IPv4 Remote Network 10.1.1.0/24
IPv6 Remote Network fd00:1::/64

Table 11.3: RT-2 acts as VPN client for a site-to-site tunnel

both ends of the VPN tunnel. If RT-1 generates this key, copy the generated
key string to RT-2 via a secure channel.

Ruleset
So far, the configured tunnel fails to establish because the packet filter
blocks all OpenVPN connection attempts. Unlike the IPsec VPN, which auto-
matically creates its firewall rules, OpenVPN requires explicit permission in
the WAN interface ruleset. A simple rule that allows access to the network
adapter WAN1 via UDP port 1194 is sufficient. The access should be limited
to the source address of the VPN partner if it has a fixed IP address.
As soon as the missing firewall rule is created, the participants can agree on
a VPN connection. However, the fun does not last long because the tunnel
is not allowed to transport end-user traffic. Also, the firewall now has a

144
 Site-to-Site tunnel

special ruleset for the OpenVPN tunnels. This ruleset popped up at Firewall
→ Rules → OpenVPN. It filters or allows data connections that want to
flow through the tunnel. The initial empty ruleset at the OpenVPN interface
blocks all incoming data packets that arrive through the VPN tunnel. The
logic of this additional filter function is similar to IPsec and is described in
section Firewall from Chapter 10 on page 122.

Connectivity
OPNsense provides feedback on the tunnel’s status at VPN → OpenVPN
→ Connection Status. It includes the inner and outer IP addresses of the
remote gateway and the number of bytes transferred (see Figure 11.2).

Figure 11.2: OPNsense confirms the established OpenVPN tunnel

The display does not show the transported IP networks. These are available
in the routing table, which OPNsense presents at System → Routes →
Status or via the CLI command netstat -nr. The keyword ovpn in the
search field hides anything from the routing table that is not relevant to
OpenVPN.
However, the most reliable test is an end-to-end connectivity check. A quick
test is also possible with ping and traceroute.
root@cl-1:~# traceroute -I 10.4.1.7
traceroute to 10.4.1.7 (10.4.1.7), 30 hops max, 60 byte packets
1 10.1.1.1 (10.1.1.1) 0.613 ms 0.543 ms 0.551 ms
2 10.6.12.2 (10.6.12.2) 2.878 ms 3.079 ms 3.323 ms
3 labsrv (10.4.1.7) 3.430 ms 3.506 ms 3.584 ms

145
Chapter 11. OpenVPN

root@cl-1:~# traceroute6 -I fd00:4::7

traceroute to fd00:4::7 (fd00:4::7), 30 hops max, 80 byte packets
1 fd00:1::1 (fd00:1::1) 0.303 ms 0.272 ms 0.279 ms
2 fd00:6:12::2 (fd00:6:12::2) 2.506 ms 2.718 ms 2.843 ms
3 labsrv (fd00:4::7) 2.985 ms 3.110 ms 3.186 ms

Client-server tunnel
If OpenVPN operates in a client-server mode, the remote peers can identify
themselves by certificate or username with a password (or both). From a
security standpoint, it is best to use certificates; however, the administrative
overhead is more complicated than a shared key.
If the deployment of certificates is too extensive, the second best option is a
combination of usernames and pre-shared keys.
The following scenario uses a compromise between security and man-
ageability: the server introduces itself with a certificate and the clients
authenticate themselves with a username and password. Also, both sides
trust each other only with a pre-shared key (two-factor authentication).
The VPN server RT-1 will validate its clients during login. First, the server
takes a close look at the offered key string. If it matches its own key, the
server compares the received username with its user list and checks the
password. If the OPNsense firewall has no local user accounts, it can query
a remote directory server via RADIUS or LDAP (see Chapter 15). If client
and server agree, the tunnel setup is complete and starts transporting en-
crypted packets.
Table 11.4 shows the relevant settings. When the interface is set to any,
OpenVPN accepts its clients via all network adapters with IPv4 and IPv6.
The selection of the IP version is located in the Protocol field. UDP4 makes
OPNsense listen to the local IPv4 address, and UDP6 makes the OpenVPN
process listen only to its IPv6 address. Choose UDP to allow both versions
of the Internet protocol. It is also possible to limit access to protocols or
network adapters with firewall rules. The same logic applies to the protocol
TCP.

The client will learn which networks are accessible through the VPN tunnel.
When connected, the client will insert the list of networks into its local
routing table.

146
 Client-server tunnel

Attribute Value
Server Mode Remote Access (User Auth)
Backend for
authentication Local Database
Protocol UDP
Device Mode tun
Interface any
Local port 1195
TLS Authentication Automatically generate a shared
2
TLS authentication key
IPv4 Tunnel Network 10.6.1.0/24
IPv6 Tunnel Network fd00:6:1::/64
IPv4 Local Network 10.1.1.0/24
IPv6 Local Network fd00:1::/64
Address Pool 2


Table 11.4: VPN server RT-1 provides dial-in for clients

Note
OpenVPN can operate in device mode tun and tap. The tun mode is
preferred when connecting mobile devices.

For example, the OpenVPN app on Apple’s iPhone or iPad only offers the
tun mode. It is not possible to communicate with an OpenVPN server using
a tap adapter.
Click on Save and the OpenVPN process of RT-1 listens for connection
requests from clients. Any OpenVPN client with a valid key may connect.
In the lab network, client CL-2 running Windows 10 is ready to dial-in.

Client
As mentioned above, the OpenVPN software must be installed separately.
The installer is available on the OpenVPN [11] website. The software also
includes the TUN/TAP drivers.
OPNsense offers an excellent tool for setting up the VPN client. OPNsense
assembles the appropriate configuration for the selected device at VPN →

147
Chapter 11. OpenVPN

Figure 11.3: OPNsense prepares the configuration for its clients

OpenVPN → Client Export (Figure 11.3). The final result is a ZIP archive
which contains the configuration files and all required certificates.

On a Windows client, extract the contents of the ZIP file in the directory:
%ProgramFiles%\OpenVPN\config

The configuration file provides instructions for the OpenVPN process. The
sample configuration from Listing 11.1 starts a client-server connection
(line 7). The communication runs via port 1195 of the UDP protocol (line 9).
If the server is reachable via IPv6, this line must contain the IPv6 address
of the server:
remote 2001:db8:2::1 1195 udp

The server will present a valid certificate that is signed by the trusted CA in
line 12. The client authenticates itself with a user password (line 11).

148
 Client-server tunnel

1 dev tun
2 tun-ipv6
3 persist-tun
4 persist-key
5 cipher AES-256-CBC
6 auth SHA256
7 client
8 resolv-retry infinite
9 remote 192.0.2.1 1195 udp
10 lport 0
11 auth-user-pass
12 pkcs12 Dial_In_Server_RT_1_opnsense_lab.p12
13 tls-auth Dial_In_Server_RT_1_opnsense_lab-tls.key 1

Listing 11.1: Configuration file Dial_In_Server_RT1.ovpn for OpenVPN

OpenVPN will log errors and major events during tunnel setup. While
troubleshooting, it is best to increase the verbosity with the additional
configuration line:
verb 3

A higher number will generate more log messages: numbers up to 5 are

suitable for troubleshooting, values from 6 to 11 are intended for software
debugging.
The Windows OpenVPN installer comes with a small GUI icon for the
taskbar. It provides a comfortable way to start the connection (Figure 11.4)
and monitor the tunnel setup.

Figure 11.4: OpenVPN GUI running on Windows 10

149
Chapter 11. OpenVPN

After successful authentication, the OpenVPN client receives an IPv4 and

an IPv6 address, which are routed via the TUN interface (line 1) for the
duration of the tunnel connection. The routing table now also contains
other IP routes that the server announced when establishing the connection.
The server remembers each client connection currently established and
displays it in Figure 11.5.

Figure 11.5: OPNsense shows all active OpenVPN clients

Finally, the OpenVPN server RT-1 requires a ruleset that allows or restricts
client connections through the tunnel. The procedure corresponds to section
Ruleset on page 144 for the site-to-site tunnel.

Troubleshooting
The OpenVPN service can describe the problem in great detail in its log
messages. OPNsense makes the OpenVPN process report to the Syslog
service. The latest messages are displayed on the web interface at VPN →
OpenVPN → Log File.
During troubleshooting, it is helpful to view the log files from both end-
points of the OpenVPN connection. If the messages narrow down the
problem but do not describe it sufficiently, it is best to increase OpenVPN
verbosity.

150
 Troubleshooting

The larger the number at Verbosity level, the more detailed OpenVPN
becomes. The default setting of OPNsense is 1, which is a good fit for
normal operation. Values from 3 to 5 are helpful during troubleshooting.
Remember to reset the log level after the troubleshooting has finished.
The following sections describe typical problem scenarios during an Open-
VPN tunnel setup.

Compression
2021-02-19T21:49:28 openvpn[45170] Bad LZO decompression \
header byte: 42

One side of the connection compresses the packets before they are sent,
while the other party expects uncompressed content.
Solution: both firewalls must agree on the compression by picking either
with or without.

Encryption algorithm differs

2021-03-30T20:39:50 openvpn[88870] 192.0.2.3:45950 WARNING: \
’cipher’ is used inconsistently, local=’cipher AES-192-CBC’, \
remote=’cipher AES-128-CBC’

The OpenVPN participants of a VPN connection use a different encryption

algorithm. The VPN connection may come up, but does not transport any
traffic due to mismatching cipher algorithms.
Solution: either specify the same algorithm on both sides or let OpenVPN
negotiate the ciphers by using the option --ncp-ciphers in the tunnel
settings at Advanced configuration.

Peer refused the connection

2021-02-19T21:56:38 openvpn[96267] TCP: connect to \
[AF_INET]198.51.100.1:1194 failed: Connection refused

The local firewall starts a connection attempt and is directly rejected by its
peer. This failure can have several causes. If the IP address of the remote
gateway is specified correctly, the remote device may not have started the
OpenVPN process or may be listening on another UDP port. Alternatively,
there might be a rule missing in the ruleset that allows communication, or
the policy prohibits the connection.

151
Chapter 11. OpenVPN

Authentication failed
2021-02-19T22:10:24 openvpn[24756] Authenticate/Decrypt \
packet error: cipher final failed

Both VPN peers use a shared key, but authentication has failed. The key is
either different or was modified during transmission.
Solution: both OpenVPN firewalls must have exactly the same key file
inserted in the OPNsense configuration.

Digest algorithm differs

2021-02-19T22:04:11 openvpn[24756] Authenticate/Decrypt packet \
error: packet HMAC authentication failed

The OpenVPN firewalls use a different authentication digest algorithm.

OpenVPN uses SHA-1 by default, however, this algorithm is currently
considered insecure and it is advisable to switch to SHA-2 or SHA-3.
Solution: the digest algorithm must be identical on both gateways.

Certificates
If there is already a Public Key Infrastructure (PKI) in place, it is an excel-
lent option to use the existing Certificate Authority (CA) to generate the
certificates for clients and servers. If no CA is available, an OPNsense fire-
wall could take over this role. OPNsense includes the necessary functions
to create a CA and issue certificates at System → Trust → Authorities.

Warning
It is best to place the CA on a well-secured server (or OPNsense firewall)
that is not connected to the Internet. If the private key of the CA is
compromised, the security of the OpenVPN network is lost.

The promotion to certificate authority is described in section Certificate

Authority of Chapter 14 on page 194. The administration of the new CA is
done on the web interface.
Let’s continue with a key pair for firewall RT-1, which is signed by the
CA. To establish a tunnel with this firewall, the remote peer requires a

152
 Technical background

client certificate. The CA creates both certificates at System → Trust →

Certificates. For both certificates, click on the Add button and pick the
method Create an internal Certificate. The difference lies in the type: for
RT-1 Server Certificate is the right choice, while the remote peer needs a
Client Certificate.
All certificates are ready to use, but the documents for RT-2 are still in the
file system of RT-1. They are provisioned manually via the web interfaces
of RT-1 and RT-2. Use the web UI of RT-1 to export the CA certificate, user
certificate, and user key. The export button offers to store the file on the
local computer. In the GUI of RT-2, the process is reversed: import the
certificates and keys as text into the system via copy-and-paste.
Finally, both firewalls have the same CA certificate and their respective pair
of user certificate and key. The temporary files on the local hard disk are
no longer needed and should be deleted.

Technical background
OpenVPN is a turnkey VPN software. OPNsense always comes with a cur-
rent version of OpenVPN. The firewall offers two cryptographic libraries.
By default, OpenVPN uses the OpenSSL library. In the past, OpenSSL
has attracted negative attention due to several critical security gaps. As a
consequence, OPNsense offers the alternative library LibreSSL . Change the
library at Firmware Flavour under System → Firmware → Settings. When
saved, the button Check for updates causes reinstallation of all software
components using the selected library.

OPNsense generates a configuration file for OpenVPN in the directory

/var/etc/openvpn/ based on the selected settings from the web UI. Next,
OPNsense will start (or stop) the OpenVPN process, but without the as-
sistance of UNIX-typical concepts like SysVinit, Upstart, or SystemD. The
identification of the process takes place via the process ID. To restart
the daemon, OPNsense will first send a TERM signal to stop the daemon,
followed by a regular process launch. That way, OpenVPN will read the
modified configuration file and continue to service tunnels.
OPNsense gathers information about the connected clients, status, and
statistics from the running process via a UNIX socket. The web GUI will

153
Chapter 11. OpenVPN

receive this information as a comma-separated list and modifies it for

presentation on the website.

Summary
The OPNsense website offers a full-fledged configuration tool that includes
virtually all OpenVPN options. This versatility allows OpenVPN to establish
tunnels against firewalls from other vendors. And if something is missing,
the Advanced configuration section is the right place for extraordinary
settings.
Authentication with certificates is fully integrated into the configuration
process. This collaboration should give self-assurance to use certificates
more frequently to secure VPN tunnels.
Setting up a site-to-site tunnel is a little strange because OPNsense imple-
ments it as a client-server setup.
A special goodie is the feature Client Export, which builds complete con-
figuration packages for various OpenVPN clients and makes them readily
available for download and use.

154
Chapter 12

High Availability

A firewall device will eventually fail someday. And then it will not fulfill its
most fundamental task of protecting computer networks.
Firewalls crash as much as other electronic gear. This constraint is an
accepted fact and the reason why high-end firewall products have additional
power supplies, fans, CPUs, or uplinks. Cheap consumer devices lack these
add-ons so the best workaround is to deploy multiple devices that act as a
group. This cluster of firewalls protects from the failure of a single device
and provides high availability service.
Within the group, the devices agree that one firewall is doing the work and
the other is just observing. Keeping track of the primary device is essential
because the passive device must take over the operation if the primary
firewall fails.

Basics
All firewall devices that join the redundancy group must comply with the
standard Common Address Redundancy Protocol (CARP).
When CARP is enabled, the firewall will listen on its network adapters for
vital signs of other CARP devices. The first member of the group considers
itself the master and sends signs of life every second to the network. The
second member of the group receives the keepalive packets and stays
passive in a backup mode, which means: observe and wait.

155
Chapter 12. High Availability

Note
For CARP to work correctly, it is irrelevant if the group contains fire-
walls, routers, servers, or other gateways. This chapter uses these
terms interchangeably.

As soon as the backup device misses three keepalive packets from the pri-
mary device, it must accept that the master has failed and considers itself
the new master. It will then take over all tasks of the failed device as fast as
possible so that end users will not recognize any outage.
Who is going to tell all computers on the attached network about the new
firewall? Nobody does, because the new firewall also took over the IP
address and MAC address of the redundancy group. For the other network
device, nothing has changed at all – except for a brief interruption.
The life signs, heartbeats, or keepalives, are IPv4 packets sent to multicast
address 224.0.0.18. These packets contain the virtual IPv4 address of the
redundancy group that all CARP routers share. Also, each redundancy
group has its unique group number, which makes it possible to operate
several CARP groups in the same network segment.

The description of CARP and its functionality are very similar to the re-
dundancy protocol Virtual Router Redundancy Protocol (VRRP). That’s
because most parts of CARP are a copy of VRRP. The license terms of the
BSD operating system do not allow the use of VRRP, so the developers just
coded a similar version and published it as CARP.

Lab network
The demonstration lab network contains three firewalls, two of which (RT-1
and RT-2) build a CARP cluster. Figure 12.1 shows the setup as a network
diagram.
All subscribers of site 1 use as a standard gateway, neither of which is the
IPv4 address of RT-1, nor that of RT-2, but rather the additional address
10.1.1.5 belonging to the CARP group. This setup belongs to the LAN side
of the devices. On the WAN side, the firewalls create an additional CARP
address along with their known IP addresses. For the failure protection to

156
 Lab network

work correctly, neighboring devices are required to communicate only with

the virtual addresses of the CARP group.
On the WAN side, both CARP devices communicate with firewall RT-core,
which becomes the target of many connection requests from location 1.

CL-1
Debian 10
10.5.1.15
10.1.1.0/24
Site-1 fd00:1::/64
lan0
10.1.1.25

RT-1
OPNsense
10.1.1.1 em1 em0
em0– 10.5.1.1 em1 10.1.1.2

10.4.1.1 10.4.1.2
em2 em2
SYNC 10.4.1.0/24 fd00:4::/64
em3 em3
198.51.100.1 198.51.100.2 RT-2
OPNsense
em0
em0– 10.5.1.2

198.51.100.6 WAN-1
198.51.100.0/24
em1 2001:db8:1::/64

RT-core
OPNsense
em0
em0– 10.5.1.6

Figure 12.1: The firewalls RT-1 and RT-2 build a CARP cluster

The filter ruleset of all firewalls are less restrictive and permit everything
so that network traffic can flow between all devices without hassle. Once
all aspects of high availability are in place, the rules should be adjusted to
enforce the local security policy.
The network segment SYNC is required and will be explained later in this
chapter.

157
Chapter 12. High Availability

CARP group
OPNsense becomes a CARP router in the web UI at Interfaces → Virtual IPs
→ Settings. Start the setup with the button Add and pick a unique group
number and a virtual address with a suitable subnet mask. Table 12.1 lists
the settings for the primary firewall RT-1.

Attribute LAN WAN

Mode CARP CARP
Interface LAN WAN1
Address 10.1.1.5/24 198.51.100.12/24
Virtual IP Password any any
VHID Group 1 1
Advertising Frequency 1/0 1/0

Table 12.1: Settings of the virtual IPv4 address

The advertising frequency has two meanings. First, it indicates the time (in
seconds) between two heartbeat packets. Second, it reveals the priority of
the sending firewall, whereby a lower value leads to a higher priority. An
advertising frequency of 1 ensures that its sender is selected master. The
other firewall needs a higher value to become the backup device.
The password protects the CARP group from unintended members and
must be identical on all authorized devices.
When ready, hit the Save button and RT-1 will begin sending heartbeat
packets on its LAN and WAN interfaces. Configure RT-2 similarly, with the
exception of a higher advertising frequency, 1/10 for instance.
Several seconds later, both CARP candidates should have agreed on a boss
(master) and an assistant (backup). In this example, RT-1 has won and
assumed the master role (Figure 12.2).
Firewall RT-2 has similar settings, except for the status of BACKUP (Fig-
ure 12.3). If RT-2 also shows MASTER, then both firewalls are in the master
position and will fight for the virtual IP address. This misunderstanding
should not happen during normal operation since it will lead to unexpected
session disconnect on the client’s computers.
Both firewalls will become MASTER if they miss the heartbeat packets of the
other device. In that case, use the ping command to check if the machines
can reach each other by the physical IPv4 address.

158
 Lab network

Figure 12.2: Firewall RT-1 becomes CARP Master

When MASTER and BACKUP reach an agreement, traffic can flow from
client CL-1 through the firewalls to target host RT-core. Use traceroute to
visualize the path through the network. The following listing shows that
RT-1 is responsible for forwarding the traffic.

Figure 12.3: Firewall RT-2 becomes CARP Backup

159
Chapter 12. High Availability

root@cl-1:~# traceroute -I 198.51.100.6

traceroute to 198.51.100.6 (198.51.100.6), 60 byte packets
1 10.1.1.1 (10.1.1.1) 0.301 ms 0.293 ms 0.321 ms
2 198.51.100.6 (198.51.100.6) 0.886 ms 0.902 ms 0.922 ms

Now that everything is running smoothly, what happens when a device

suddenly fails? Let’s power off the master firewall RT-1 to simulate an
outage and see what happens.
RT-2 no longer receives vital signs and promotes itself to master several
seconds later. The same traceroute command on client CL-1 still reaches
its target but shows a different hop in the path.
root@cl-1:~# traceroute -I 198.51.100.6
traceroute to 198.51.100.6 (198.51.100.6), 60 byte packets
1 10.1.1.2 (10.1.1.2) 0.608 ms 0.604 ms 0.644 ms
2 198.51.100.6 (198.51.100.6) 1.425 ms 1.451 ms 1.452 ms

Stateless
No client will notice the outage of firewall RT-1, unless it is sending or
receiving data. So what happens during a file transfer?
The exact behavior of an interruption depends on the application and time-
outs. A download on CL-1 from a public web server will stop during failover
but continue after approximately four seconds.

At the moment, the configuration of all lab devices is somewhat unrealistic:

the firewalls pass all traffic and also ignore the state of client sessions. Usu-
ally, corporate networks have limiting devices that use address translation
(NAT) and strict firewall rules, which remember the state of each client
connection.

Address translation
A router with Internet access usually has a packet filter enabled. Most likely
it also modifies the IP address of transmitted packets (NAT, see Chapter 8)
to replace private addresses by public IPs.
Let’s move the lab network a little closer to reality and make the CARP
devices translate addresses from the internal network 10.1.1.0/24 to a

160
 Address translation

correct public IPv4 address. Figure 12.4 shows the setup of NAT on firewall
RT-1.

Figure 12.4: Firewall RT-1 translates IPv4 addresses

It is important to note that the OPNsense host should not use the IPv4
address of its WAN adapter (198.51.100.1), but uses the common CARP ad-
dress (198.51.100.12) instead. The CARP address is present on all firewalls
within the CARP group, so the address is still reachable if a single firewall
fails.
It is best if RT-1 translates all outgoing packets into the CARP address,
which will move to RT-2 during failover. That way, answering packets will
flow back through RT-2 to the client.

Now the firewalls have to keep exact records: which packet must the
packet filter accept, which IP is connected with which port, and how is the
packet translated?

161
Chapter 12. High Availability

If the CARP master RT-1 fails, first, a data transfer on CL-1 will stop and
break when the failover to the backup firewall is complete. The root cause
of this behavior is the empty firewall table and empty NAT table in the
backup device.

State tables
A state table is generally a security enhancement. It keeps track of all
active sessions that the firewall has permitted. The packet filter and address
translation process consult the session table to determine if the packet in
question belongs to an established connection and may pass, or if the packet
needs additional processing.
When using multiple firewalls, this leads to a problem because each firewall
device maintains its local state table and is unaware of any other sessions.
In a CARP setup, the master router handles all sessions and keeps them in
its state table. The table of the backup router remains empty because this
router has not seen a single connection yet.

Synchronization of sessions
OPNsense approaches the problem of differing session tables with the help
of the operating system. For over ten years, FreeBSD includes the protocol
pfsync for synchronizing packet filters. pfsync is independent of CARP, and
together they make a good team.
Now the CARP master shares its session table knowledge to the backup
router. The master sends all changes in its table to a pre-defined IPv4
address or multicast address. The backup router receives the messages and
updates its local table accordingly. The objective is that all devices within
the CARP group have the same content in the firewall table and NAT table.
It is best to have a separate network segment dedicated to the state synchro-
nization. That’s because synchronization is a real-time business. A session
table dated ten seconds ago is not good for sessions that were established
within the last nine seconds.

Synchronization is configured at System → High Availability → Settings.

The section State Synchronization expects a network adapter to perform
the synchronization. This scenario uses the SYNC adapter and the IP range

162
 Address translation

Figure 12.5: RT-1 relays all state changes to RT-2

10.4.1.0/24. The Synchronize Peer IP is the IPv4 address of the partnering

router; RT-1 (10.4.1.1) will send all updates to the IPv4 address 10.4.1.2,
which will finally reach RT-2 (Figure 12.5). The configuration of RT-2 is
vice versa.
The Synchronization Settings are on the same web page and let the admin-
istrator decide which modifications are sent to the partner firewall. Section
Synchronization of configuration will review the settings.
The exchange of table content runs over the interface em2, which provides
a direct but independent connection between RT-1 and RT-2. In case there
are two firewalls, it is best to use the target IPv4 address of the partner.
When using multiple firewalls, leave the address field empty so that pfsync
will use the predefined multicast address 224.0.0.240. Packets destined to
that address will reach all receivers in the same network segment.

163
Chapter 12. High Availability

Now both CARP devices state each other’s table contents. In principle, it
would be sufficient if the backup router gets updates from the master, but
synchronization runs in both directions.

Now, when it comes to an unplanned outage of the primary firewall RT-1,

its partner RT-2 will take over the CARP role and start routing the client
connections. That failover process takes a few seconds, but then any file
transfer on CL-1 will continue because its session information is already
present in the state table of RT-2 before the disaster happens.

root@ RT-2 :˜ # /sbin/pfctl -s state | grep 10.1.1.25

all tcp 37.58.58.140:80 <- 10.1.1.25:39684 ESTABLISHED:ESTABLISHED
all tcp 198.51.100.12:46502 (10.1.1.25:39684) -> 37.58.58.140:80 \
ESTABLISHED:ESTABLISHED

Synchronization of configuration

OPNsense can do more than keeping the session tables in sync. The firewall
is also able to replicate configuration changes from the primary host to the
backup machine. Any changes to a firewall cluster need to be configured
only on a single device. This consolidation saves time and prevents typing
errors.

Luckily, there is no need to invent a new protocol for this task. OPN-
sense will execute the configuration change on both nodes at the same
time. Communication with the remote firewall utilized plain HTTP. For
that reason, setting up the Synchronization Settings (XMLRPC Sync) at
System → High Availability → Settings requires a username, password,
and the IP address of the secondary node.

Best practice

There are different methods for the redundant design of firewalls, which
offer a smooth deployment and improve availability.

164
 Best practice

Asymmetric routing
If a packet takes a different path on the way to the server than on the
way back, the routing is asymmetric. In theory, this is not a problem, but
in practice, stateful firewalls, NAT gateways, or IDS systems prevent a
successful connection.
It is easy with CARP to accidentally end up in asymmetric routing. This
asymmetry even occurs in the laboratory network when RT-1 is master for
the LAN side, and RT-2 is master for the WAN side.

CL-1

RT-1 RT-2

outbound inbound

RT-core

Figure 12.6: CARP might introduce asymmetric routing

In Figure 12.6 client CL-1 sends a packet to its default gateways, which is
accepted by RT-1 as CARP master. The packet arrives at its destination via
RT-core. On the return path, the packet reaches RT-core. It will pass the
packet to the CARP cluster, and RT-2 receives it because it is master on the
WAN side. However, RT-2 does not know anything about this connection
because it has not seen the initial packet, which was routed by RT-1. If RT-2
is just acting as a stateless router, it will relay all packets solely based on
the routing table. But if RT-2 behaves stateful, it will dismiss packets that
have no record in the session table. This behavior prevents asymmetric
routing and successful communication of CL-1.

165
Chapter 12. High Availability

OPNsense can prioritize the CARP process so that the same firewall is
master for all CARP groups. This election will render the routing symmetric.
Assign priorities using the CARP advertising frequency, which is explained
in section CARP group on page 158.

Master election
The CARP router with the highest advertising frequency will win the election
and become master. The frequency is calculated from the Base and Skew
values. A higher frequency sends keepalive packets more often. CARP then
calculates the interval between two heartbeats with Formula 12.1. In a
nutshell: the higher the numbers, the lower the advertising frequency. A
lower advertising frequency ensures the firewall will not win the election
to become the master router.
Skew
Interval = Base + (12.1)
256
If both candidates use the pre-configured values of Base=1 and Skew=0,
then the router with the largest IP address will win. In this scenario, RT-2
becomes the master because its IPv4 10.1.1.2 is numerically larger than the
IP 10.1.1.1 of its opponent RT-1. The same applies to the WAN subnet.
If RT-1 is the preferred firewall device and is chosen as master, then RT-1
must offer a better advertising frequency. Keep the Base value on both
firewalls low (e.g. 1 second) for a quick failover and increase the Skew
value of RT-2 to a high value of 100.
This modification will downgrade RT-2, and the master role swings from
RT-2 to RT-1.

Synchronization
It is essential to choose the correct devices as source and destination for
synchronization to prevent overwriting new content with old values.
The synchronization of state tables works bidirectionally. On both firewalls,
insert the IP address of the partner node in the field Synchronize Peer IP.
This task will even synchronize client sessions that are flowing over the
backup firewall by accident.

166
 Quicker failover

The synchronization of configuration changes works unidirectional. The

primary firewall executes the changes on the secondary firewall – and not in
reverse. The field Synchronize Config to IP on RT-1 must be the IP address
of RT-2. On RT-2 this field must remain empty. Any unintended changes on
RT-2 stay there and will not endanger the stable operation of the primary
firewall RT-1.

Quicker failover
Other protocols for redundancy and gateway availability achieve failover
times below one second. The keepalive interval is within the range of
several hundred milliseconds with a timeout of one second.
This indulgence timing is not possible with OPNsense. The predefined
interval between two heartbeat packets is at the same time the minimum
value: one second. Higher values are possible, but the timeout is always
three times longer. By default, this is about 3–4 seconds.
A quicker failover is possible with CARP, but the software implementation
of FreeBSD chooses a minimum value of one second. CARP can do a
sub-second failover on OpenBSD, which is not the chosen platform below
OPNsense.

Load balancing
CARP has one designated firewall that is actively working on the client
sessions. Distributing the network load to several firewall devices is possible
with a workaround.
This “poor man’s load balancing” (Figure 12.7 on the next page) assigns an
additional CARP group to every network adapter. This group must elect the
CARP backup of the primary group as master for the new group.
Following the example above, RT-1 is master and RT-2 is backup of the first
CARP group. Within the second CARP group, RT-1 is backup, and RT-2 is
master. While group 1 serves IPv4 address 10.1.1.5, group 2 could provide
address 10.1.1.6. The trick is to make half of the client computers use
10.1.1.5 as the default gateway, and the other half use 10.1.1.6 as their
gateway.

167
Chapter 12. High Availability

Use default Use default

gateway gateway
10.1.1.5 10.1.1.6

Site-1
10.1.1.0/24
10.1.1.5 10.1.1.6
10.1.1.1 10.1.1.2
MASTER for .5 BACKUP for .5
em1 BACKUP for .6 MASTER for .6 em1
Virtual
routers
RT-1 RT-2
MASTER for .12 BACKUP for .12
em3 BACKUP for .21 MASTER for .21 em3
198.51.100.1 198.51.100.2
198.51.100.12 198.51.100.21

Figure 12.7: Load-sharing with CARP

Whether a firewall becomes master or backup depends on its advertising

frequency, which is calculated from Base and Skew. The initial values are 1
and 0. The settings in Table 12.2 will promote RT-1 as master for group 1
and backup for group 2.

Firewall Interface VHID Base Skew Role Address

Group
RT-1 LAN 1 1 0 Master 10.1.1.5
RT-1 LAN 2 1 100 Backup 10.1.1.6
RT-1 WAN 7 1 0 Master 198.51.100.12
RT-1 WAN 8 1 100 Backup 198.51.100.21
RT-2 LAN 1 1 100 Backup 10.1.1.5
RT-2 LAN 2 1 0 Master 10.1.1.6
RT-2 WAN 7 1 100 Backup 198.51.100.12
RT-2 WAN 8 1 0 Master 198.51.100.21

Table 12.2: Advertising Frequency for load-sharing with CARP

168
 IP version 6

The reverse settings apply to RT-2, which is backup for group 1 and master
for group 2. Figure 12.8 shows the election result.

Figure 12.8: Firewalls RT-1 and RT-2 share the network traffic

The exact numbers for the time offset are not critical, as long one firewall
uses a higher value than the other.
Now both firewalls share the network traffic. The ratio of each device is
not easily controlled: in a best case scenario, both gateways handle 50% of
all packets, in a worst case RT-1 gets 99% of the sessions and RT-2 is almost
idle with the remaining one percent.

IP version 6
CARP is ready for IPv6. But it needs more than that because the table
synchronization has issues with the new IP version. The protocol pfsync is
not yet migrated to IPv6. The pfsync protocol is not yet ready for IPv6 and
the web interface refuses to accept an IPv6 address at Synchronize Peer
IP. If the synchronization is performed over a separate network segment,

169
Chapter 12. High Availability

nobody will notice the IPv4 connection, even if the remaining network is
running purely IPv6.
Fortunately, the synchronization of configuration (XMLRPC Sync) is more
advanced and appears completely “IPv6 ready”.

Technical background
The web UI of OPNsense discloses many features and protocols that are
used below the surface: CARP, pfsync, and XMLRPC.
CARP takes the user-defined Virtual IP and generates an additional IP ad-
dress for the selected network adapter. This freely selectable IP address
has the fixed MAC address 00:00:5e:00:01:NN, where NN represents the
group number. This number is a unique identifier and permits multiple
CARP groups to act in the same network segment without violating each
other’s operation.
The vital signs from the CARP master are small packets targeted to the
multicast address 224.0.0.18 or ff02::12 when using IPv6. This address
allows all subscribers in the same network to receive the multicast packets.

The command line tool ifconfig shows all settings and election results.
For instance, the LAN adapter of RT-2 reports about IP and CARP:
root@RT-2:˜ # ifconfig em1
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> [...]
options=810098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
ether 00:15:16:02:01:02
inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
inet 10.1.1.5 netmask 0xffffff00 broadcast 10.1.1.255 vhid 1
inet 10.1.1.6 netmask 0xffffff00 broadcast 10.1.1.255 vhid 2
inet6 fe80::215:16ff:fe02:102%em1 prefixlen 64 scopeid 0x2
inet6 fd00:1::2 prefixlen 64
inet6 fd00:1::5 prefixlen 32 vhid 15
carp: BACKUP vhid 1 advbase 1 advskew 10
carp: MASTER vhid 2 advbase 1 advskew 0
carp: BACKUP vhid 15 advbase 1 advskew 10
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

CARP would not be complete without its synchronization buddy pfsync.

The handling is very similar, and it also uses ifconfig as a configuration
tool. However, pfsync is independent of an existing network adapter and

170
 Summary

even creates a new adapter with the appropriate name pfsync0. RT-2 can
easily display adapter’s settings:
root@RT-2:˜ # ifconfig pfsync0
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1500
pfsync: syncdev: em2 syncpeer: 10.4.1.1 maxupd: 128 defer: off
groups: pfsync

No separate software process is required for the high-availability stuff –

because the kernel performs these tasks on its own.

Summary
High availability of firewalls with CARP and pfsync is a stable and straight-
forward way of lowering the downtime of services. The trick is to use
multiple firewalls: two identical devices build the firewall group of which
the first device runs the business and the second device takes over as soon
as its partner fails.
The additional configuration effort is minimal, and XMLRPC Sync will
replicate the configuration changes to all members of the firewall cluster.

171
Chapter 12. High Availability

172
Chapter 13

NetFlow

Firewalls are hardworking security devices and operate in the background.

Firewalls report some network traffic statistics but are limited by transmit-
ted bytes and discarded packets.
A firewall becomes much more verbatim when it hears the magic word
NetFlow. This term ignites OPNsense and turns it into a chatterbox: all of
the transmitted data stream is logged and the flow information is sent to a
NetFlow collector.
The collector receives messages from all firewalls, thus providing an excel-
lent source of information for statistics, analysis, or capacity planning.

The content of a flow

A NetFlow package contains the profiles of several data streams. Each
record provides accurate information about the IP connection. Each Net-
Flow version 5 flow includes:
• Source and destination IPv4 address
• Source and destination port (for TCP or UDP)
• Amount of transmitted bytes and packets
• Beginning and end of the connection
• Ingress and egress interface of the router/firewall
• QoS information

173
Chapter 13. NetFlow

• Autonomous system (for BGP routing)

• TCP flags
• IP Protocol (TCP, UDP, ICMP)

The firewall collects this flow record for each connection. It does not matter
if it is a short-lived DNS request of 200 bytes or a large download of 750
megabyte – everything is noted, packaged, and sent to the collector.

NetFlow is based on the connectionless UDP protocol, so the firewall will

not get angry or stuck if the collector rejects its packets. At the same time,
there is no feedback if these packages are lost in transmission.

Lab setup
As a professional firewall operating system, OPNsense includes a NetFlow
exporter. Before enabling NetFlow, it needs some background information
about the installation: at which local interfaces should traffic be logged?
Who is the NetFlow collector? And which NetFlow version does the receiver
expect?

Site-1
RT-1
OPNsense
10.1.1.0/24 em0
em0– 10.5.1.1
Internet
fd00:1::/64
em1
10.1.1.1 198.51.100.6
CL-1 em1 em3
10.1.1.25 198.51.100.1
em2
10.4.1.1 WAN-1 RT-core
198.51.100.0/24 OPNsense
DMZ 2001:db8:1::/64 em0
em0– 10.5.1.6
10.4.1.0/24
labsrv eth1 fd00:4::/64
CentOS 8
eth0 10.5.1.7 10.4.1.7

Figure 13.1: Firewall RT-1 reports via NetFlow

For a first example, the firewall RT-1 is ready to report the flows from its
LAN interface to the lab server 10.4.1.7 via NetFlow version 5. Figure 13.1
shows the setup of the lab network.

174
 Lab setup

Figure 13.2 explains the different roles of the NetFlow concept. A NetFlow
collector is not limited to a single exporter. Depending on the hardware,
thousands of firewalls can send their NetFlow data to the collector.

CL-1 RT-1
Internet

NetFlow RT-core
exporter

NetFlow
collector

Figure 13.2: NetFlow exporter and collector interact together

OPNsense places the NetFlow setup in the web UI to Reporting → Net-

Flow. Inside the small lab network, it is acceptable to monitor all adapters.
The firewall remembers all connections that pass through the interfaces.
It will report that information as a NetFlow packet to the collector host.
Figure 13.3 shows the initial setup.

Figure 13.3: Initial setup of OPNsense as NetFlow exporter

175
Chapter 13. NetFlow

When the setup is initially complete, nothing happens. There is nothing to

report due to lack of traffic flow through the firewall.
Luckily, generating traffic is not a problem, a few web requests from client
CL-1 and the exporter of RT-1 informs the collector about its client’s activi-
ties. Now the setup of the NetFlow exporter on the OPNsense firewall is
complete. The heart of a NetFlow installation lies in the collector, which
has to process the mass of information sensibly.

Collector
The NetFlow collector is a software that receives, understands, and stores
the information contained in NetFlow packets. A NetFlow analyzer is almost
always associated with this software and comes to several conclusions with
the collected data.
The lab environment uses a small Linux application, which receives the
flow packets and stores them on the hard drive. The software nfdump [12]
is capable of this task and runs on the lab server making it a true NetFlow
collector. For the Linux distribution CentOS 8 there is even a precompiled
package. The installation is a minimal task:
yum install epel-release
yum install nfdump
The package comes without a start script, so a little extra effort on the
command line is required.
mkdir -p /var/netflow
/usr/bin/nfcapd -D -4 -p 2055 -S 0 -l /var/netflow
These commands create a new directory for NetFlow data below /var.
Then it starts the process in the background (-D), which listens on the local
IPv4 address (-4) on the default UDP port (-p). Newly arriving packets are
first kept in memory and then are written to disk after a maximum of five
minutes (-l). The files on the hard drive have a fixed format:
nfcapd.YYYYMMDDHHMM
Each file contains flow information of exactly five minutes. The data is
written in binary format. Use the command nfdump to nicely sort and
summarize the output data:
nfdump -r /var/netflow/nfcapd.202010090935 -o line -a -O bytes

176
 Troubleshooting

Troubleshooting
OPNsense provides a Cache section in the NetFlow configuration area.
On this web page, the Cache gathers all interfaces that report traffic for
NetFlow. It also includes any traffic that has not been sent to the NetFlow
collector.

Insight
If there is no NetFlow collector available in the surrounding network, OPN-
sense can act as a NetFlow collector. This integrated collector is called
Insight. The technical procedure is the same as NetFlow, with one ma-
jor difference: the exporter and collector operate on the same computer.
The checkbox Capture local in the NetFlow configuration enables local
processing. Once this is active, the web GUI adds a destination address
of 127.0.0.1, which represents the local machine. At the same time, the
collector process starts in the background. It presents the collected data in
the section Reporting → Insight of the web UI.

Figure 13.4: OPNsense can act as NetFlow collector

Insight gives a pretty good view of the stored flow data. A time-series
graph shows the throughput of the network adapter. Several pie charts (Fig-
ure 13.4) summarize highly used port numbers and requested IP addresses.

177
Chapter 13. NetFlow

Next to this picturesque view is the section Details. It provides access to

the NetFlow data with filter and search capabilities.

Technical background
OPNsense does not reinvent the wheel to support NetFlow. It just uses
the kernel module ng_netflow, which has been part of FreeBSD for many
years. The module ng_netflow listens on the selected network adapters and
records all IP communication. As soon as a connection terminates, it puts
that information into a NetFlow packet and relays that packet to the local
socket 127.0.0.1:2055 for analysis.
There the packet meets the software samplicator [13]. It accepts the
package and forwards it to the IPv4 addresses configured as Destinations in
the web interface. The only task of samplicate is to distribute the pending
packets to several IPv4 addresses.
If the NetFlow packets are analyzed in a remote collector, OPNsense’s
work is complete. If the local collector Insight is in charge of the analysis,
samplicate passes the packets on to the socket 127.0.0.1:2056. There the
process flowd greedily waits for NetFlow information to make it more
attractive for the web interface. The results are the colorful pie charts and
time series which are visible in the section Reporting → Insight.

IPv6
Version 9 of NetFlow is capable of reporting about IPv6 connections. Un-
fortunately, the implementation of OPNsense has one small limitation: the
web interface only accepts IPv4 addresses for the communication between
exporter and collector. The contents of this IPv4 communication are Net-
Flow packets containing IPv4 and IPv6 flows.
The external NetFlow collector from section Collector on page 176 does not
have this limitation. The service nfcapd handles the different versions of
NetFlow transparently and stores the data independently of its version. The
frontend command nfdump also does not require any modification when
showing IPv6 flows.
The internal NetFlow collector Insight also has no objections to IPv6 ad-
dresses and treats them in the web interface just like IPv4 connections.

178
 Summary

Summary
NetFlow is a well-known method for reporting TCP/IP connections in
computer networks. OPNsense comes with an industry standard NetFlow
exporter and even a collector for visualizing simple statistics. These tools
allow NetFlow to be used in all environments.
The view of traffic information gives a good insight into the daily work of
the firewall and addresses several questions such as: which services and
servers do the users communicate with? And how busy are the network
adapters? In addition to graphical evaluations, the web interface also
provides detailed tables with search functions and can thus even help
during troubleshooting.

179
Chapter 13. NetFlow

180
Chapter 14

Web Proxy

A proxy is a service that acts as an intermediary between the client and

server. The client talks to the proxy and the proxy talks to the server. For
the user, the proxy is invisible, so that the communication seems to take
place directly with the server.

Proxy www.google.de
server

Internet
opnsense.org

github.com
Cache

Figure 14.1: The proxy server is a mediator between client and server

The proxy is mostly used as an intermediary between the client’s computer

and the Internet (Figure 14.1). In this way, the proxy provides all clients
with access to the web. At the same time, it takes care of security and policy
restrictions.

181
Chapter 14. Web Proxy

Proxy servers are available for many services: email, FTP, DNS, SIP and
application independent proxies called SOCKS. OPNsense includes a proxy
server for HTTP(s) and FTP. Further proxies for SIP, DNS, and IGMP are
available as a plug-in.

In the good old days, the advantage of a proxy server was its cache for
locally storing and providing web content because the Internet bandwidth
was low. For example, 64 kilobits per second could be transferred between
client and server, but up to 100 Mbps were possible between client and
proxy.
Nowadays, the caching of web content does not bring any noticeable speed
gain. There are two main reasons for this:

• Home networks are connected to the Internet at Ethernet-like speeds.

Internet providers even offer connection rates of up to a Gigabit. The
throughput between client and server, as well as between client and
proxy, is very similar.

• The ratio between static and dynamic content of web pages has
changed in favor of dynamics. Google’s website displays different
content when a registered user or new visitor accesses it. In this
context, what web content would the proxy store?

However, these changes do not make the proxy service obsolete, because
the web has new demands for the network infrastructure.

Antivirus
All data flows through the proxy, providing a central location to check
the data for viruses. As soon as a virus is detected, the proxy blocks the
download and presents the user with a warning website.

Authentication
Access to the web is only permitted with user login. For this reason, only
selected users can enjoy the Internet.

182
 Lab setup

Accounting
The proxy knows exactly which computer is accessing which website. When
the user has to authenticate before browsing the web, the proxy service will
add the username next to the visited web pages. The log file is therefore a
powerful data source for evaluation, statistics, and verification.

Warning
The username and even the IP address are personal data and may not
be stored without the consent of the person concerned. Check the legal
situation first!

Filter
A restricted website protects an employee or family from dangerous or
immoral content. Blocking is done by IP address, web page, web do-
main, or category. The first three groups may be blocked individually,
but this is tedious and time consuming because it involves creating filter
lists. On the other hand, using categories simplifies blocking: websites are
grouped into categories according to their content, e.g. Shopping. If the
category Shopping is allowed, many more domains like www.amazon.com
or www.ebay.com are automatically approved. If the category Shopping is
on the blacklist, the proxy forbids access to all known websites for shopping.
OPNsense itself does not assign the website category. The firewall makes
use of publicly available category lists.

Lab setup
A proxy server is no fun without real web browsing, so the lab network
needs access to the Internet. Firewall RT-core will become the gateway to
the Internet, as described in the lab set up in Chapter 1.
The proxy service runs on RT-1 and, for the high availability cluster, on
RT-2. Computer CL-1 from site-1 will access the Internet through the proxy
service.
Figure 14.2 on the following page shows the network diagram with IP
addresses and interfaces.

183
Chapter 14. Web Proxy

WAN-2
Site-1 RT-1 192.0.2.0/24
10.1.1.0/24 2001:db8:2::/64
fd00:1::/64
10.1.1.1 192.0.2.6
Internet
em1 em4
CL-1 192.0.2.1
10.1.1.25 em2
em1
10.1.1.2 198.51.100.2
em1 em3
198.51.100.6
WAN-1
RT-core
198.51.100.0/24
2001:db8:1::/64
RT-2
Figure 14.2: Lab setup with a proxy server and Internet access

Explicit proxy
The explicit proxy server expects to be addressed by the clients. The appli-
cation must therefore be proxy-enabled, and the user (or administrator)
must have configured the application accordingly. The well-known web
browsers have excellent proxy support in their settings. They just require
the IP address and port number of the server that provides the proxy service.
That’s all on the client side.

URL Update filter lists

Squid
http://
2 blacklists.tar.gz

1
3
Internet
4
Proxy server
Figure 14.3: Clients communicate with the servers only through the web proxy

184
 Explicit proxy

The proxy on the server side is an OPNsense firewall with a running proxy
service. Figure 14.3 illustrates how the OPNsense proxy software works. A
client requests the web proxy and reaches the service Squid (step 1). Squid
will look into its local cache first to see whether the web page is already
present, or at least some parts of it (step 2). If the web page is not in the
cache and the policy allows access, Squid retrieves the web content from
the Internet (step 3). The content is placed in the cache and passed on to
the client (step 4). The filter lists shown are used in the section Filter by
category on page 186.
At Services → Web Proxy → Administration OPNsense enables the proxy
service with the single checkbox next to Enable proxy. Confirm the changes
with the button Apply, and the first OPNsense firewall RT-1 will accept proxy
requests. Without specifying a TCP port number, OPNsense picks 3128.
A client in site-1 addresses the proxy via 10.1.1.1:3128 in its application
settings.

URL filter
The proxy now processes all web requests without restriction. Unlimited
access is good, but it could be better. It makes sense to filter inconvenient
web pages so that the client cannot reach them.
The filter function is hidden behind the middle tab of Access Control List.
The Blacklist field expects all web pages that should no longer be available
for clients. There is no need to specify the web address precisely. The URL
filter also accepts parts of the address – much like the search function in a
word processor.
For example, the website http://www.example.net should no longer be
accessible via the proxy. The entry example.net blocks not only the desired
website but also all other subdomains and even websites that have this text
somewhere in the URL, e.g.
http://open-source-firewall.com/search.html?q=example.net

Note
Blacklist entries should be as precise as possible and as general as
necessary. This principle prevents the blocking of legitimate web
addresses that match the blacklist due to an inaccurate entry.

185
Chapter 14. Web Proxy

To cut a long story short: the syntax of the blacklist utilizes regular expres-
sions. These expressions are used for matching patterns in strings. Their
syntax is peculiar, but they are extremely flexible and powerful. The black-
list of the web proxy, as well as the search function of many applications,
use the Perl Compatible Regular Expressions (PCRE) for pattern recognition
so that some knowledge will also help in other areas.
A small introduction to the regular expressions with examples is given in
Appendix B.

Filter by category
It is a time-consuming task to organize long lists of web pages. OPNsense
can group web pages with similar content into categories and apply its filter
to all records in the selected category.
It is true that the categories also contain long lists of web pages, but they
are assembled by someone else. Set up the category at the web proxy
administration section below Remote Access Control Lists. There are no
entries by default, so it is up to the administrator to add the URL of a
supplier.

Commercial suppliers provide accurate and comprehensive filter lists. For

a demonstration of the URL filter, it is enough to pick a supplier which
offers its listings free of charge for personal use. Install a new filter list in
the above-mentioned section by pressing the plus icon. The settings for
the supplier Shalla [14] are listed in Table 14.1. Finally, hit Save changes
followed by Download ACLs to fetch the blacklists, as shown in Figure 14.3
on page 184.
Categories, and their included websites, is a powerful tool for filtering. With
just a few configuration lines you can block tens of thousands of websites.
By default, all categories are included on the blacklist – consequently, the
proxy service will block all known websites.
Which kinds of categories are available? The edit button of the just created
filter list shows all entries of the nearly 70 categories, including banking,
sports, dating, shopping, etc. Removing individual categories from this
restrictive list allows the category to be accessed.
Category lists are always in motion because new websites are added and
existing entries are improved. The button Schedule with Cron makes OPN-

186
 Explicit proxy

Attribute Value
Enabled 2

Filename Shalla
URL http://www.shallalist.de/Downloads/shallalist.tar.gz
Username blank
Password blank
Categories blank
SSL ignore cert 2
Description Shalla

Table 14.1: OPNsense fetches the category listings from supplier Shalla

sense use its scheduler to update the records regularly. It is best to run this
update during the night so that the user is not disturbed during their web
surfing activities.

Restrictions
The selected rules, filters, and categories apply to all clients using the proxy
service. There is no selection by subnet, user, group, or IP range.
When a category is classified as bad, it blocks traffic from all clients to
websites that fall into that category.

OPNsense does not disclose which URLs and web domains fall into which
category. If a legitimate website is blocked, use the search function on the
provider’s website [14] to find out why this website is classified that way.

Blacklists and whitelists

A policy decides if a category is permitted or denied. The policy is based on
one of two possible principles:

• Anything is allowed except what is on the blacklist.

• Everything is forbidden except what is on the whitelist.

The first principle is easy to implement because it is the default setting for
OPNsense. In its original state, the proxy will not block a single web page.

187
Chapter 14. Web Proxy

The client browser will only receive an error message in its browser if the
requested web address matches an entry of the blacklist from section URL
filter on page 185.

Figure 14.4: This proxy allows only a few web pages

The second principle is way more restrictive and blocks everything by

default. The configuration in OPNsense contains a blacklist, which in-
cludes every possible web address. The regular expression for this is a
period, which represents all letters, numbers and special characters (see
Appendix B).
The whitelist in OPNsense is a list of exceptions. It will contain all web
addresses that are excluded from the blacklist. It also accepts regular ex-
pressions.
The example web proxy in Figure 14.4 works according to the whitelist
principle. It only allows web pages that offer dictionaries for translating
texts.

Troubleshooting
If the proxy does not work as expected, take a look at the log file. The web
proxy keeps its log records organized at Services → Web Proxy → Access
Log.

188
 Proxy cluster

A typical log line contains a lot of useful information and is somewhat

self-explanatory. OPNsense logs each request to a website in a special
format:
2021-02-10T20:35:35.860000 284 10.1.1.25 TCP_MISS/200 1175 \
GET http://example.com/ - HIER_DIRECT/93.184.216.34 text/html

Table 14.2 gives a short explanation of the individual fields.

SquidLog Description
2021-02-10T20:35:35 Timestamp
284 How long was the web cache working on the
transaction (in ms)
10.1.1.25 Client IP address
TCP_MISS/200 Result of the transaction (MISS reveals, that
the cache did not contain the web page)
1175 The amount of data transferred to the client
GET HTTP request method
http://example.com/ Requested URL
- Username (if the proxy requires authenti-
cation)
ORIGINAL_DST/ Cache hierarchy information
93.184.216.34
text/html HTML type and format

Table 14.2: Web Proxy log format explained

Proxy cluster
Multiple OPNsense firewalls with activated proxy service and identical
ruleset can increase the availability of the Internet access or can balance
the load. If an OPNsense pair is already in use as the default gateway (see
Chapter 12), the proxy service can also use this infrastructure.
A proxy cluster consisting of a primary firewall (active) and a secondary
device (passive) achieves higher availability as a single system. In the lab,
the firewall RT-1 is the preferred proxy, and RT-2 comes into play when RT-1
fails. The high availability relies on the protocol CARP (see Chapter 12).

189
Chapter 14. Web Proxy

10.1.1.1 em1

Primary
10.1.1.5 RT-1 Internet

CARP
Backup
10.1.1.5
Site-1
10.1.1.0/24
10.1.1.2 em1

RT-2
Figure 14.5: Active–passive cluster with CARP and Squid

Figure 14.5 shows the interaction between the two devices. Both firewalls
of site-1 share the highly available IPv4 address 10.1.1.5. Priorities and
other details of CARP are skipped to keep it simple.
In addition, the proxy cluster requires bidirectional State Synchronization
via the LAN adapters of RT-1 and RT-2. Also, OPNsense provides Configu-
ration Synchronization (XMLRPC Sync) to synchronize the list of allowed
categories, blacklist, and proxy settings, but only from RT-1 to RT-2. That
way, RT-1 becomes the firewall for the configuration and both firewalls
become the workers. Unfortunately, the download task to fetch the ACLs
is not synchronized and needs a manual click on the button Download ACLs.

It is no surprise that both firewalls also need an extra line in the filter-
ing ruleset, which permits XMLRPC communication between them. The
new rule allows TCP port 80 from the partner firewall on its LAN adapter.
Next, the leading firewall RT-1 requires configuration to become an explicit
web proxy. The resulting configuration must be identical on both devices to
achieve the same application behavior.

190
 Proxy cluster

Note
If the proxy service is configured to listen on a specific network adapter,
it accepts requests from clients at all available addresses (IPv4, IPv6,
CARP, virtual IP) of that adapter.

The clients from site-1 communicate with their new proxy cluster by its
virtual IP address. The web browser uses the address 10.1.1.5 and the TCP
port 3128 in the proxy settings.
Clients still address the same proxy, but by its new IP address. That way, if
the firewall RT-1 crashes, the IPv4 10.1.1.5 moves to the backup device and
the proxy on RT-2 accepts the web requests from all clients.
Now it becomes clear why the proxy configuration should be identical on
both firewalls: when RT-2 has different filter lists than RT-1, the Internet
will behave strangely for the user should RT-1 crash.

Functional test

Figure 14.6: Public web page discloses a proxy server

If Internet access is available, it might be provided by a proxy server. There

are public web pages which try to disclose an intermediate proxy server, like
http://www.whatismyproxy.com. Request this web page from a browser
or the command line and it will analyze the request details to detect a proxy.
Figure 14.6 has found the lab proxy server on firewall RT-1.

191
Chapter 14. Web Proxy

More important than the functionality of a single firewall is the protection

from failure. After a system breakdown of RT-1, the CARP protocol waits for
3 seconds before it triggers a failover. A failover immediately promotes the
secondary firewall RT-2 to be the proxy boss. The regular user will hardly
notice this procedure during web surfing. A web download stops for the
duration of the failover and continues afterward.

TLS Inspection
If web pages are secured with Transport Layer Security, then the server and
client first negotiate a secure connection between each other before they
exchange web content. The encrypted communication is visible by the https
in a web address. This technique is a security gain for both participants, as
routers, firewalls, and proxies no longer have access to the content. The
advantage is at the same time a disadvantage because the firewall cannot
understand the clear text messages and is unable to check the content for
virus or malware.

Note
The encryption protocol Secure Socket Layer (SSL) was continued
around the turn of the millennium under the abbreviation TLS. That
makes TLS the successor of SSL. In this chapter, both abbreviations are
used interchangeably.

The general solution lies in a compromise: the firewall or proxy decrypts

the HTTPS connection, but only for its investigation. After inspection, the
firewall will encrypt the content and relay it to its destination, so that the
client is not confused.
The technical process behind it is complicated, as Figure 14.7 and the
following explanation illustrate.

1. The client initiates an https connection to a web server and expects

encrypted communication.

2. The proxy receives that request and asks the client to wait. It will
then masquerade as a client and set up a separate https connection
to the target server.

192
 TLS Inspection

Server Original
certificate server
signed by proxy
certificate

TLS TLS

Client Proxy Server

Figure 14.7: https connection setup with proxy

3. The server will eventually receive the connection request. It cannot

tell if the request comes from the client or the proxy, nor does it care.
The server will always process the request and reply.

4. The proxy has now established an https connection with the server.

5. Next, the proxy will relay the data from the server back to the client.
In fact, the client only has an encrypted connection with the proxy
without the user knowing it.

6. Normally the client web browser will notice this trick due to a cer-
tificate mismatch. It is up to the proxy to manipulate the presented
certificate so that the client browser will not notice.
Technically, the proxy holds two TCP connections per web request: the
first connection is between client and proxy, and the other one is between
proxy and server. Before the data from the server connection flows into the
client connection, the proxy can see the plaintext data. Now it is possible
to inspect the data for virus, malware, ransomware, in addition to content
detection and category filters.

This is where the journey ends because certificates strengthen the trust of
https connections, and the proxy server does not own the server’s crypto
material. Consequently, the proxy server must establish the https communi-
cation with the client through another certificate. And that manipulation
must be noticed by the client browser, which immediately presents its user
with a certificate warning.

193
Chapter 14. Web Proxy

How does the TLS inspection work unnoticed? The client browser must
somehow trust the TLS proxy so that it will not issue that nasty certificate
warning. The trick lies in the certificate store: the certificate of the Cer-
tificate authority (CA) that the proxy uses must be present on the client
computer. As soon as that certificate is inserted as “trusted root certificate,”
the client will have confidence in all web pages that are served by the proxy.
The certificate trick is required on every client computer that uses the proxy
server.

Certificate Authority
For the TLS proxy to work without warning, it needs its own Certificate
Authority (CA). It can be a self-created root CA (see Chapter 24), or an
intermediate CA by an existing root CA.

For the lab environment, the OPNsense firewall RT-1 becomes a new root
CA. The certificates are located at System → Trust → Authorities. The
orange button Add may create a new certificate or import an existing one.
When the Method field is set to Create an internal Certificate Authority, it
will ask all the necessary questions that make up a certificate. Table 14.3
shows sample answers for the intended Certificate Authority.

Attribute Value
Descriptive name opnsense.lab
Method Create an internal Certificate Authority
Lifetime (days) 3650
Country Code DE (Germany)
State or Province NRW
City Cologne
Organization Practical OPNsense
Email Address opnsense@example.net
Common Name opnsense-ca

Table 14.3: OPNsense is promoted to internal Certificate Authority

The Save button will store the changes, generate cryptographic keys, and
sign them. That results in a self-signed CA hosted on the OPNsense firewall.

194
 TLS Inspection

Export that CA certificate using the export CA cert button and store its
content in a file named opnsense.lab.crt. The clients later require that
file.

Configuration
The conversion to a TLS-sniffing proxy is surprisingly simple. Go to Services
→ Web Proxy → Administration and navigate to the section Forward Proxy.
Tick the field Enable SSL inspection and select the CA certificate from
Table 14.3 in the list at CA to use. Activate the eavesdropping proxy by
clicking the Apply button. Figure 14.8 shows the final configuration.

Figure 14.8: OPNsense runs its Web Proxy in TLS mode

Client
In contrast to the regular web proxy, a modification at the client is es-
sential for the TLS examination to work without permanent certificate
warnings at the client browser. That will only succeed if the certificate
opnsense.lab.crt from section Certificate Authority finds its way into the
client browser.

195
Chapter 14. Web Proxy

For Internet Explorer and Google Chrome a double click on the file is suffi-
cient, after that a wizard takes over this task. During the import process,
it must be ensured that the certificate is placed in the store Trusted Root
Certification Authorities.
Firefox starts the process in the browser’s options. In the Privacy & Security
area, the certificates section is at the bottom of the page. The View Cer-
tificates button leads to a dialog box with an Import button. Firefox then
needs the certificate file opnsense.lab.crt for the import process. In the
last window, trust the certificate to identify websites explicitly. Now the
client trusts the TLS proxy.

Functional test
Nothing is easier than verification because that final check consists of
surfing to the Google website from the prepared client browser. Google’s
search engine has been using https for some time, so the TLS proxy is
triggered.
Figure 14.9 shows the web page visible inside Firefox. The certificate that
reaches the client is conspicuous. It shows the web page www.google.com,
but it is issued by the certification authority of the OPNsense proxy. For the
web browser, it is a valid certificate, because web address, issuing CA, and
(not visible in the screenshot) expiration date are genuine.

Figure 14.9: Google’s search engine – viewed through a TLS proxy

196
 Transparent proxy

Transparent proxy
The normal web proxy is no secret to the user, but the transparent proxy
remains as invisible as possible. This type of proxy does not have to be
configured in the application because it works unnoticed by the user and the
application. The transparent proxy “snaps” the HTTP stream and forwards
it to its proxy service. Then the regular work begins: filtering, caching,
and fetching web pages for the client. This process requires some technical
tricks in order for it to work. For users, it is an unwanted forced proxy.
The initial setup is hardly different from the explicit proxy, except that
the option Enable Transparent HTTP proxy in the Forward Proxy tab of
Services → Web Proxy → Administration must be selected.

Now the proxy server is working in transparent mode, which means OPN-
sense does not have its fingers directly in the data stream. Without direct
contact, the proxy can not influence web access and filter anything.
It is not enough for the data to flow through the firewall. The web requests
must reach the proxy service inside the firewall. OPNsense solves this
problem by redirecting the traffic with an address translation (see Chap-
ter 8). Incoming HTTP packets from the clients on the LAN adapter are
manipulated using a NAT rule so that their destination address is the local
firewall (127.0.0.1) and their destination port is the TCP port of the proxy
(3128). The same trick works for HTTPS, where the destination port is
different and uses 3129.
Technically it is a Port Forward on the LAN adapter. Figure 14.10 shows
the two NAT rules to connect the proxy to the data stream.
After that, the further configuration behaves exactly like the standard web
proxy. It can prohibit, allow, filter, block, and categorize.

Exceptions work somewhat differently with transparent proxies because the

client cannot skip the proxy. In most cases, a single internal client should
benefit from an unfiltered Internet. To create this, another NAT rule must
prevent the data packets of the “special” client from arriving at the proxy.
An exemplary rule is already present in Figure 14.10 and works before the
general proxy NAT rules. Client 10.1.1.45 is excluded from the transparent
proxy and is not subject to its filtering rules. This client receives access to
the Internet via the usual firewall rules.

197
Chapter 14. Web Proxy

Figure 14.10: The transparent proxy requires two Port Forward rules

Technical background
OPNsense uses the number one free proxy in the open-source world: Squid.
Squid started in 1996 as a simple web cache and has worked his way up to
a full HTTP/1.1 proxy. In addition to the functions OPNsense uses, Squid
also supports Reverse Proxy and WCCP.
For content control, OPNsense uses its own methods for obtaining and
organizing categories. Based on the configured blacklist, OPNsense loads
all selected categories from the provider’s website. OPNsense then fills
these data sets of web pages, domains, and IP addresses into a simple file.
Squid then uses that file as a source for blocking its clients’ requests.

Limitations
The OPNsense web proxy is rich in features but is obviously designed for
smaller environments. This is shown by the fact that all settings, categories
and filter lists apply equally to all clients. There is no option to apply
different rules to one group of clients as to another group. This distinction
is essential in a business environment if the web proxy is implemented in a
company, hotel, or hospital.

198
 Outlook

Outlook
OPNsense has some plug-ins available for the web proxy. Solid virus
protection is provided by ClamAV, which is listed as os-clamav on the
firewall. After installation, ClamAV joins the web UI under Services →
ClamAV.
The web proxy can interact with the virus scanner via the Internet Content
Adaptation Protocol (ICAP). Both services communicate through a locally
installed ICAP service, which needs a further plug-in os-c-icap installed in
the local system.
The interaction with the web proxy starts with Services → Web Proxy
→ Administration in the ICAP Settings section. The mandatory fields of
Request Modify URL and Response Modify URL refer to the local ICAP
service:
icap://[::1]:1344/avscan

As soon as all services are working and ClamAV has updated its virus
database with the latest signatures, it is time for a small security audit.
That’s basically a web download of the EICAR virus [15]. Its purpose is to
test an Antivirus installation without harming your computer.
If everything works as expected, the browser should issue a Virus warning
and the ICAP service logs:
Thu Feb 11 20:24:35 2021, 99094/4247910144, VIRUS DETECTED: \
Win.Test.EICAR_HDB-1 , http client ip: 10.1.1.25, http user: -, \
http url: https://secure.eicar.org/eicar.com

If the web proxy also inspects HTTPS connections (see section TLS Inspec-
tion on page 192), then ClamAV will even analyze the encrypted content.

There is more: the additional plug-in os-web-proxy-useracl provides access

control for users and groups. For this to work, users must authenticate
themselves to the web proxy (see Chapter 15). The plug-in then checks for
each requested website whether this particular user is authorized to access
it.
The plug-in is available at Services → Web Proxy → Groups and Users. The
configuration just requires the name of a user or group, the list of domains,
and the filtering action.

199
Chapter 14. Web Proxy

Unfortunately, the plug-in does not use the category lists from section Filter
by category on page 186. It is limited to manually crafted lists. However, it
can create exceptions for individual users or groups.

Summary
OPNsense shines with the functional diversity of its proxy, which is a
fantastic product free of charge. In addition to the pure web proxy with
caching function, OPNsense also provides URL and category filters. The
latter blocks (or allows) a website based on its genre.
The transparent proxy demonstrates its strengths when the clients do not
allow or want proxy configuration. In networks without administrative
access to the clients (e.g. anonymous Wi-Fi networks and unknown devices)
this proxy type is the best choice.
OPNsense even ventures into the premier class and authorizes the proxy
service for a careful look into the encrypted web communication. Now,
even HTTPS data streams are subject to category filters and blacklists.

200
Chapter 15

Central Authentication

Some services require users to identify themselves to the OPNsense firewall

to utilize the web proxy or VPN dial-in. The firewall authenticates the
users against local accounts or validates the credentials with an external
authentication server.
In this chapter, OPNsense includes Microsoft’s directory service Active
Directory to check login attempts of its users. The well-known protocols
LDAP and RADIUS are available for the secure exchange of passwords.
The Active Directory is a directory service installed and operated in the
local infrastructure. Between the server and its authentication clients is a
local high-speed network.
However, directory services are also used in cloud environments. Several
providers offer an authentication service from the cloud: a Directory-as-a-
Service (DaaS). The service JumpCloud is a good starter for getting a taste
of the authentication cloud.

Protocols
There must be a common language between the authentication server and
the client. A proper protocol protects the transmitted information from
prying eyes, interconnects systems of different vendors, and checks the user
password (authentication) as well as its authority (authorization). OPN-
sense supports the proven and widely used protocols LDAP and RADIUS.

201
Chapter 15. Central Authentication

Table 15.1 shows the compatibility of various firewall services and authenti-
cation sources.

Service Local LDAP RADIUS

database
OpenVPN 2
 2
 2

IPsec VPN 2
 2
 2

Web Proxy 2
 2
 2

Web GUI 2
 1
2 1
2
Captive Portal 2
 2
 2

SSH 2
 1
2 1
2
Console 2
 1
2 1
2

Table 15.1: Authentication with external servers

Warning
If SSH, web GUI, or console authentication uses a remote server, a
local fallback login should be allowed as the secondary method. Thus,
a login is possible even if the LDAP/RADIUS server is not reachable.

LDAP
The Lightweight Directory Access Protocol (LDAP) is a language for query-
ing and manipulating records in a directory service. Since usernames and
passwords are stored in a central directory, LDAP is suitable for authentica-
tion and authorization.
Microsoft’s Active Directory is compatible with LDAP. Therefore, a do-
main controller also acts as an LDAP server. Access to the service and its
information is only password-protected and encrypted if required.

RADIUS
A genuine authentication protocol is the Remote Authentication Dial-In User
Service (RADIUS) because it is primarily concerned with authentication
1
There must be a local user account that contains the authorization details. The password
is then checked against the authentication server via LDAP or RADIUS.

202
 Lab setup

and authorization. RADIUS does not care about the management of user
accounts, so it expects some kind of directory service in the backend.
The possibilities of RADIUS are more versatile than with LDAP, because the
decision for authentication can also include the date and time as well as
the IPv4/IPv6 address of the requestor.
A Windows Server includes support for RADIUS with Network Policy Server
(NPS), which is installed as an additional service.

Lab setup

The RT-1 firewall becomes the central hub for authentication. Clients can
log on to this firewall via all available services that require authentication.
RT-1 provides OpenVPN, IPsec VPN, a web proxy, and the mandatory web
GUI for administration. The user accounts for accessing these services are
stored in an additional server, which the OPNsense firewall contacts via
LDAP and RADIUS.

winsrv RT-1 CL-2

Windows Server 2019 OPNsense Windows 10
RADIUS
oder LDAP C l i e n t-S e rv e r V P N -Tu n n e l

Ethernet1 em1 em4

10.1.1.16 10.1.1.1 192.0.2.1 RT-3

Standort-1 WAN-2 Standort-2

Figure 15.1: Lab setup for authentication with Active Directory

Figure 15.1 illustrates the network diagram. The authentication server is

a Windows Server 2019 with an Active Directory and the Network Policy
Server.
The querying client is CL-2 in site 2, which is connected via an additional
firewall RT-3.

203
Chapter 15. Central Authentication

Microsoft Server
Microsoft’s directory server represents an authentication system that man-
ages user data and accepts logins via LDAP and RADIUS. Alternative
products are OpenLDAP or Novell eDirectory. The examples and tutorials
refer to the Microsoft Windows Server, as this solution is widely used in
enterprise networks.
The Windows Server stores user accounts, passwords, and group member-
ships in its directory service Active Directory (AD). The server acts as a
Domain Controller and automatically provides an LDAP service. For the
lab environment, the root domain is opnsense.lab and the path within the
LDAP structure is derived from it.

Note
The Microsoft Windows Server will never disclose a user password via
LDAP.

How can the firewall verify a user password against the LDAP server? The
firewall does not check the authenticity of the password itself but logs on
to the LDAP server on behalf of the user. A successful login shows that the
user’s password is valid.
Usually, LDAP is secured with TLS to protect the transmitted data from
interceptions and manipulation. LDAP becomes LDAPS – the functionality
remains unchanged.

Note
By default, the Windows Server only offers unencrypted LDAP. The
server requires a certificate for encrypted LDAP communication.

To use encryption, install the Active Directory Certificate Services. Next,

make the LDAP service request a certificate. When complete, the LDAP
server listens on TCP port 636 and replies to encrypted requests.

A newly installed server has not yet activated any functions and requires
the Active Directory Domain Services role to get promoted to a domain
controller. For demonstration purposes, populate the initially blank Active

204
 Microsoft Server

Directory with several organizational units (OU) to provide storage for the
user objects.
Figure 15.2 shows the administrative view of the constructed Active Direc-
tory opnsense.lab with the organizational units Admins, User, Services and
Groups. The OU Services contains AD objects from non-personal accounts,
such as a firewall that wants to connect via LDAP.

Figure 15.2: Active Directory with users and groups

LDAP
The directory structure visible via LDAP depends on the structure of the
Windows domain and the organizational units. For that reason, config-
ure the LDAP authentication of OPNsense to match the Active Directory
structure. The examples in this chapter work only in combination with the
installed Windows Server.
On the client side, OPNsense expects an LDAP server to be configured. This
can be done by navigating to: System → Access → Servers.

205
Chapter 15. Central Authentication

Table 15.2 contains settings to match the constructed AD. Any valid AD
account is suitable to bind to the LDAP server. However, a dedicated service
account (here rt1 ) is recommended, which can be deactivated and moni-
tored in case of emergency.

Attribute Value
Descriptive name winsrv_ldap
Type LDAP
Hostname or IP address 10.1.1.16
Port value 389
Transport TCP - Standard
Protocol version 3
Bind credentials
User DN rt1
Password Password
Search scope Entire Subtree
Base DN dc=opnsense,dc=lab
Authentication containers Click Select button, tick all
container/OUs that contain user
objects
Extended Query blank
User naming attribute sAMAccountName

Table 15.2: OPNsense authenticates against the Active Directory

OPNsense provides a little helper tool at System → Access → Tester. This

tool validates the LDAP configuration by authenticating the given username
and password against the Windows Server. More troubleshooting tips follow
in section Troubleshooting on page 219.

With this configuration, every Active Directory user is authorized to use the
services of the OPNsense firewall. Typically, a firewall service is limited to a
group of users, which OPNsense implements via the Extended query. Their
content is an LDAP filter, which is applied during authentication. The login
only succeeds if the authenticating user passes the LDAP filter and provides
a valid password.

206
 Microsoft Server

LDAP filters have the operator in front, with the conditions following in
round brackets. The following practical examples provide possibilities to
limit the services of a firewall to selected users or groups.

&(ObjectClass=user)(memberOf=CN=VPN,OU=Groups,DC=OPNsense,DC=lab)

The condition of the filter is fulfilled if both checks within the brackets are
true. The authenticating user must be a user object, which is a member
(memberOf) of the VPN group. The leading symbol & represents the logic
operator AND.

|(memberOf=CN=VPN,OU=Groups,DC=OPNsense,DC=lab)(memberOf= \
CN=Webproxy,OU=Groups,DC=OPNsense,DC=lab)

The filter allows access if the applicant is a member of one of the two groups:
VPN or Webproxy. The leading symbol | represents the logical operator OR.

msNPAllowDialin=TRUE

The LDAP filter returns success if the AD user object has the attribute
msNPAllowDialin and the Boolean value true. This strange attribute is
Microsoft’s way of saying that a user may dial in. The AD user object shows
this property in the Dial-in tab. The LDAP filter evaluates to true when
Network Access Permission is set to “Allow Access”.

RADIUS
Windows Server provides a RADIUS service, which is not installed by de-
fault. Install the additional server role Network Policy and Access Services
and wait for the setup to complete. After installation, the Network Policy
Server (NPS) is available, which accepts RADIUS authentication requests
and decides the replies.

Microsoft gives its RADIUS service more decision power than LDAP. The
conditions, whether a login will be successful or not, can include a user-
name, computer, or Windows group. Besides, the registration can depend
on the time of day (e.g. only during lunch break from 12 p.m. to 1 p.m.) or

207
Chapter 15. Central Authentication

on certain days of the week (e.g. Monday to Friday). Finally, even the IPv4
or IPv6 address can be part of the decision.

Figure 15.3: Network Policy Server is an add-on role of the Windows Server

When authenticating via RADIUS, the server takes over the decision logic.
The client does not care about internal structures such as groups or or-
ganizational units. The client only sends a username with password and
expects a reply. As a result, setting up the network policy server is more
complicated than running an LDAP query.
The configuration starts in the Windows Server at Start → Windows Admin-
istrative Tools → Network Policy Server. In the RADIUS Clients section, the
NPS expects an IPv4/IPv6 address from each RADIUS client that is allowed
to use the service.
The shared secret key is a pre-shared short text that is used to encrypt parts
of the RADIUS package during communication. Figure 15.3 shows the
RADIUS client for firewall RT-1.
The Network Policies section defines the conditions under which a user may
dial in through any of the RADIUS clients. The settings in Figure 15.4 verify
whether the authenticating user is a member of the VPN group. It is even

208
 Microsoft Server

possible to check against multiple Windows groups, which are combined

with a logical OR.

Figure 15.4: The Network Policy Server grants access if the user
matches the policy

If the condition is met, the NPS behaves as specified in the Settings. In this
case, access is granted.
The network policies can consist of several individual policies that are
processed from top to bottom. If the condition of a policy matches the
authenticating user, the policy action is applied. If no policy is found, the
RADIUS server always rejects its clients. The processing logic corresponds
to the packet filter of a firewall, not for IP packets, but rather for RADIUS
clients.

Note
The network policy server logs its decisions by default to a log file in
the directory
C:\Windows\system32\LogFiles\

209
Chapter 15. Central Authentication

OPNsense places the authentication settings in the web UI at System →

Access → Servers. The RADIUS server contains the policy logic, so the
configuration on the client side is straightforward (see Table 15.3).

Attribute Value
Descriptive name winsrv_radius
Type Radius
Hostname or IP address 10.1.1.16
Shared Secret OPNsense18 (example)
Services offered Authentication
Authentication port value 1812
Authentication Timeout 5

Table 15.3: OPNsense uses RADIUS to authenticate users

against an Active Directory

Use the Tester to perform an authentication with a valid user account. The
NPS will reply to this request and print a message in its log file (abbrevi-
ated):
"WINSRV","IAS",02/11/2021,20:50:07, 2 ,,"opnsense.lab/User/ \
James Bond",,,,,,,,0,"10.1.1.1","RT-1",,,,,1,2,1,"VPN",0, \
"311 1 10.1.1.16 02/11/2021 19:39:49 1", [...]

A failed attempt is also logged. The log file then contains the (shortened)
entry:
"WINSRV","IAS",02/11/2021,20:52:28, 3 ,,"OPNSENSE\jbond",,,,,,,,0, \
"10.1.1.1","RT-1",,,,,,,1,,16, \
"311 1 10.1.1.16 02/11/2021 19:39:49 2", [...]

The main difference between the two lines is the marked response from the
RADIUS server: packet type 2 stands for a successful login (Access-Accept)
and type 3 represents a rejection (Access-Reject).

210
 Directory-as-a-Service

Directory-as-a-Service
In the as-a-Service concept, the prefix “Directory” represents an authentica-
tion system, which is obtained from a provider in the cloud. The directory
server is located somewhere and is installed, maintained, secured, and
updated by the service provider. The critical issues of availability, quality of
service, and licensing also fall within the provider’s responsibility.
The provider offers a configuration interface and authentication service;
this lab logs in with the LDAP and RADIUS directory service.

Site-1
RT-1
RADIUS OPNsense We b p ro x y

V P N tu n n e l

Internet
LDAP
We b GU I

Figure 15.5: The lab firewall authenticates with JumpCloud services

The Directory-as-a-Service is a young product and time will tell if it com-

petes with an on-premise directory service. To explore the possibilities, this
chapter uses the cloud provider JumpCloud [16] which offers both RADIUS
and LDAP and is free for small environments.
The firewall configuration only needs a slight modification. RT-1 no longer
communicates with the local Windows Server for user login, but with an IP
address on the Internet that leads to a JumpCloud server (Figure 15.5).
First, register with JumpCloud and use the GUI to create users, passwords,
and groups. For larger environments, there are migration paths from an
existing Active Directory. For the small lab environment, it is acceptable to
create a small number of user accounts manually.
After registering with JumpCloud and logging in as an administrator, the
web page for configuration appears, which JumpCloud refers to as Console.
Create a new user in the Users area. The user needs a first and last name,
an email address, and an unchangeable username. The Specify Initial
Password option accepts a pre-selected password; otherwise JumpCloud

211
Chapter 15. Central Authentication

generates a password and sends it to the selected email address. Finally,

the Enable as LDAP Bind DN option gives the user account permission to
log on to the directory server via an LDAP connection. The button save
user creates the new account and generates the LDAP Distinguished Name
(DN), which is later needed for the LDAP authentication of the OPNsense
firewall.

Figure 15.6: JumpCloud offers user management via a web page

Figure 15.6 shows a created user. The firewall needs an additional account
to connect to the LDAP directory and search for users. From this service
account, only the DN and the password are used later on. The unique LDAP
name of the newly created account for the RT-1 firewall is:
uid=rt1,ou=Users,o=59fd73a440c56b1234567890,dc=jumpcloud,dc=com

Now the authentication service from the cloud is ready for requests.

OPNsense as LDAP client

The OPNsense firewall RT-1 contacts the directory server via its DNS name
ldap.jumpcloud.com to check the user’s credentials. The setup for OPNsense
requires multiple steps:

212
 Directory-as-a-Service

1. Add the certification authority that JumpCloud uses for its servers.
This step is optional but recommended.
2. Next, create a trust relationship to the LDAP server.
3. Finally, configure the firewall services to use the LDAP authentication
for user login.
If the LDAP communication passes insecure networks, it is best to encrypt
the connection. JumpClouds authentication servers are somewhere on
the Internet, so TLS must protect the connection. Ironically, JumpCloud
allows unencrypted communication, which should not be used – not even
for testing purposes.
The LDAP server will identify itself with a certificate. The OPNsense firewall
can verify this certificate only if it knows the content or if it trusts the issuing
certificate authority. The Internet service provider GoDaddy currently
generates JumpClouds certificates. The Support Center of JumpCloud [17]
explains where to obtain the corresponding CA certificates. Download the
files and add them to the firewall at System → Trust → Authorities.
If there is any doubt as to which of the numerous GoDaddy certificates is
the right one, use the following command to reveal which GoDaddy CA is
responsible for JumpCloud:
echo -n | openssl s_client -connect ldap.jumpcloud.com:636 | \
sed -ne ’/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’

Now the firewall should have the correct certificate installed, so the con-
figuration of the LDAP client can begin. Navigate to System → Access →
Servers and click the button Add to set up a new LDAP server. Table 15.4 on
the following page shows the selected values for encrypted communication
with the JumpCloud LDAP server.
In the presented configuration, each user is authorized to use any service.
Use the Extended Query to apply any desired restrictions. That way, a user
must first match the conditions of an LDAP query before a service can be
used. Only if members of the VPN group receive a positive response from
the directory server, should the following filter be added as an Extended
query:
memberOf=cn=VPN,ou=Users,o=59fd73a440c56b1234567890, \
dc=jumpcloud,dc=com

213
Chapter 15. Central Authentication

Attribute Value
Descriptive name jumpcloud_ldap
Type LDAP
Hostname or IP address ldap.jumpcloud.com
Port value 636
Transport SSL – Encrypted
Protocol version 3
Bind credentials
User DN uid=rt1,ou=Users,o=59fd73a440c56 \
b1234567890,dc=jumpcloud,dc=com
Password Password
Search scope One Level
Base DN o=59fd73a440c56b12345,dc= \
jumpcloud,dc=com
Authentication containers ou=Users,o=59fd73a440c56b12345 \
67890,dc=jumpcloud,dc=com
User naming attribute uid

Table 15.4: Authenticate with JumpClouds LDAP server

Note
The LDAP directory of JumpCloud stores the created groups in the
organizational unit Users.

The query may span several groups, as described on page 207.

Warning
Each object in JumpCloud can have individual attributes in the section
Custom User Attributes. These attributes remain hidden from an LDAP
query and cannot be used for authentication.

Use OPNsense’s web GUI at System → Access → Tester to validate the

success of the logon process. See section Troubleshooting on page 219 for
further description.

214
 Directory-as-a-Service

OPNsense as RADIUS client

JumpCloud also offers RADIUS-as-a-Service for authentication. The goal
is the same as LDAP: validate a username and password. In addition,
JumpCloud’s RADIUS service always requires a group membership.
In JumpCloud’s web console, the RADIUS section in the left menu leads to
the configuration of the RADIUS clients. Such a client essentially has only
four properties: name, IP address, shared secret, and one or more groups
whose members can pass RADIUS authentication.

Note
RADIUS is a client-server model. The OPNsense firewall acts as a
RADIUS client and sends user information to the RADIUS server in the
JumpCloud data center. The web console of JumpCloud incorrectly
describes the RADIUS objects as RADIUS servers, which can lead to
confusion, as explained in the upcoming illustrations.

The expected IP address is the public address of the OPNsense firewall.

Here you must specify the exact IP address – a network range such as
198.51.100.0/24 gets rejected by the web console. A dynamic hostname
is also unacceptable. For Internet connections with dynamic addresses,
JumpCloud has a solution that works much like DynDNS.

Figure 15.7: Configuration settings for RADIUS client RT-1

215
Chapter 15. Central Authentication

Figure 15.7 on the previous page shows the configuration settings of the
RADIUS client. JumpCloud does not deliver any predefined groups, so
create them manually in the web menu at Groups. The new group type
must be Group of Users. Unlike the Active Directory, the group type is
already defined during creation.
The setup starts the remote side in OPNsense at System → Access →
Servers. The Add button immediately requests the login values shown in
Figure 15.8.

Figure 15.8: RADIUS configuration on firewall RT-1

JumpCloud reveals the IP addresses of the RADIUS servers in its Support

Center. Table 15.5 lists all available servers. The examples use a European
server.

IPv4 address DNS name Location

18.204.0.31 us1.radius.jumpcloud.com US East
54.203.27.225 us2.radius.jumpcloud.com US West
18.194.159.20 eu1.radius.jumpcloud.com Europe
18.182.131.248 ap1.radius.jumpcloud.com APAC

Table 15.5: JumpCloud offers RADIUS servers worldwide

216
 Directory-as-a-Service

The Shared Secret in OPNsense must match the long string specified as
Shared Secret in the JumpCloud web console. JumpCloud only offers
the Authentication service. The additional service accounting refers to
statistical data, which is intended for billing and monitoring. Each record
includes a timestamp of when users logged on and off, and the amount of
data transferred. JumpCloud silently ignores such Accounting Requests.

Dynamic IP address
JumpCloud expects a static IP address for the RADIUS client. This expec-
tation works well for customers with a fixed IP range on their Internet
connection. A RADIUS client behind a DSL connection, with a dynamic
IP address must inform JumpCloud of the new address after each change.
JumpCloud is kind enough to provide a programming interface (API) so
that the exchange of information can take place by a script.
Every JumpCloud customer has access to the API. The key for the API is
provided in the web console at API Settings. A key looks like this:
7e2563b0ca4c7988cf8ce0ec644da128351a3b62

and is required for every API access. The JumpCloud website provides a
simple example that uses the API to list all configured RADIUS clients:
1 curl --silent -X GET -H "Content-Type: application/json" \
2 -H "Accept: application/json" \
3 -H "x-api-key: 7e2563b0ca4c7988cf8ce0ec644da128351a3b62" \
4 "https://console.jumpcloud.com/api/radiusservers/" \
5 | json_pp -json_opt pretty

The addition in line 5 is optional and formats the output for better readabil-
ity. If the API key is correct, the command returns a JSON-formatted list of
RADIUS clients.
{
"totalCount" : 1,
"results" : [
{
"userPasswordExpirationAction" : "REMOVE",
"_id" : "59fe206fb63f4b3053ce0e55",
"networkSourceIp" : "203.0.113.226",
"organization" : "59fd73a440c56b127e6b7753",

217
Chapter 15. Central Authentication

"name" : "rt1.opnsense.lab",
"sharedSecret" : "hVXm5nxjZQzj5UtkKB9J@im4",
"id" : "59fe206fb63f4b3053ce0e55",
"userLockoutAction" : "REMOVE",
"mfa" : "DISABLED"
}
]
}

The address of the RADIUS object is present as variable networkSourceIp.

The _id attribute is JumpCloud’s internal identifier, required in the script
that follows.

JumpCloud offers a complimentary script via GitHub [18]. However, the

script is limited to Bourne-Again Shell (Bash), which is missing in OPN-
sense. Even though it is possible to install Bash, the JumpCloud script
contains special characters that will cause unexpected errors.
Appendix C links to a modified version of the script, which uses the C-Shell
and runs under OPNsense.
The script radiusUpdate.csh determines the public IP address of the local
firewall and compares it to the stored address from JumpCloud. If the
addresses differ, the script contacts the API to save the new address at
JumpCloud.
As soon as the WAN interface announces a change, the IP address and
JumpCloud’s server are compared. When this happens, the firewall calls
the script rc.newwanip to inform packet filters, services, routing table, and
gateway groups about the change. This script is the ideal place to tell
JumpCloud about the new IP address.
Only a few commands are required to download the script and place it in
the local file system of the OPNsense firewall:
curl --output /usr/local/bin/radiusUpdate.csh \
https://raw.githubusercontent.com/practical.opnsense/ \
practical.opnsense.github.io/master/chapter/15/ \
radiusUpdate.csh
chmod a+x /usr/local/bin/radiusUpdate.csh
cat <<EOF >> /usr/local/etc/rc.newwanip
# update JumpCloud about the modified IP address
system("/usr/local/bin/radiusUpdate.csh | logger -t jumpcloud");
EOF

218
 Troubleshooting

Before starting, insert API’s own key and the ID of the RADIUS client into
the script text. Appendix A explains how to modify files from the command
line.

Troubleshooting
The OPNsense web GUI comes with a quick authentication test at System
→ Access → Tester. This feature performs a login with real user credentials.
The user password is also required, which the administrator should not
know. For this reason, it is best to create a “demonstration” user, which is
not assigned to any person and is only used for troubleshooting.

Note
The methods and commands for troubleshooting are equally suitable
for the Active Directory and the services of JumpCloud.

Figure 15.9 shows a successful login to the JumpCloud authentication

server. If the result is negative, it is up to the client to solve the error
because JumpCloud does not provide insight into the log files.

Figure 15.9: OPNsense has a built-in authentication tester

219
Chapter 15. Central Authentication

If the authentication with the Tester fails for no apparent reason, continue
troubleshooting on the command line. The authentication process is the
same but the command line tool provides a much better error message.
Use the search command ldapsearch to run an authentication via LDAP.
When using RADIUS to debug, the os-freeradius plug-in provides a RADIUS
server as well as a command line version.

LDAP
The LDAP login from the command line expects – just like the web page – a
username and password. When the service user rt1 applies the ldapsearch
command, the user should connect to the JumpCloud server and issue the
following output:
ldapsearch -H ldaps://ldap.jumpcloud.com -n \
-b "ou=Users,o=59fd73a440c56b1234567890,dc=jumpcloud,dc=com" \
-D "uid=rt1,ou=Users,o=59fd73a440c56b1234567890,dc=jumpcloud,dc=com" \
-w my_secret_password

The command is successful if it does not issue an error message. The option
-n does nothing except for an LDAP-BIND. In case of a failure, the command
line displays an error message:

ldap_bind: Invalid credentials (49) The user account is unknown, or

the password is incorrect. This error also occurs if the Distinguished Name
(DN) of the user is incorrect.

ldap_sasl_bind(SIMPLE): Can’t contact LDAP server (-1) The LDAP

server is unreachable due to an intermediate firewall, a lack of (Internet)
connectivity, or an incorrect IP address in the ldapsearch command.

ldap_bind: Invalid DN syntax (34) The spelling of the user account is

incorrect. The path within the directory structure must be specified in the
LDAP syntax. If you are not sure about the path, JumpCloud allows you to
copy the correct DN from the user’s properties in the web console. When
using an Active Directory, the graphical LDAP client LDP visualizes the
directory structure.

220
 Troubleshooting

RADIUS
OPNsense does not provide a separated RADIUS client. The os-freeradius
plug-in is suitable for troubleshooting on the command line. It is installed
either via the web GUI or by the command:

pkg install os-freeradius

When complete, the command radtest is available, which can perform a

RADIUS authentication.

radtest USER PASSWORD \

eu1.radius.jumpcloud.com 100 hVXm5nxjZQzj5UtkKB9J@im4

The username and password appear in clear text in both the command and
in the output that follows. The puzzling string in line 2 is the Shared Secret
used by the RADIUS server.
A successful login with a valid user and correct group membership reports
this output:

Sent Access-Request Id 47 from 0.0.0.0:33069 to 18.194.159.20[...]

User-Name = "jbond"
User-Password = "Blofeld12#"
NAS-IP-Address = 10.5.1.1
NAS-Port = 100
Message-Authenticator = 0x00
Cleartext-Password = "Blofeld12#"
Received Access-Accept Id 47 from 18.194.159.20:1812 to [...]

If the RADIUS server refuses the login, use the command line tool radtest
to get the error message from the server. Ignore the Id field because it
changes in every response and is not relevant to the investigation:

Received Access-Reject Id 194 from 18.194.159.20:1812 The server

has actively rejected the login request. In the simplest case, only the
username is unknown, or the password is incorrect. It is also possible that
the user account is not a member of the group that the RADIUS server
requires for a successful login.

221
Chapter 15. Central Authentication

No reply from server for ID 157 The RADIUS server fails to send a
response. Use the ping command to briefly validate the end-to-end com-
munication at the network level. Incidentally, a RADIUS server also refuses
to reply if the Shared Secret is different, so double-check the settings on
both sides.
Note
Use the command pkg remove os-freeradius to remove the RADIUS
plug-in from the firewall after successfully troubleshooting.

Technical background
The possibility of an external authentication differs between the OPNsense
firewall services.
The IPsec VPN service strongSwan (see Chapter 10) and the web proxy
Squid (see Chapter 14) utilize a Pluggable Authentication Module (PAM)
for the user login. The PAM module pam_opnsense.so calls the method
authenticate() from the back-end framework AuthenticationFactory.
The authentication of OpenVPN users passes through a script to validate
a user account independent of their certificate. The final step during
validation ends in the AuthenticationFactory.
The services web GUI, SSH, and Console use the integrated authentication
of OPNsense. To authenticate the user, the services use the library:
/usr/local/etc/inc/authgui.inc
The Captive Portal asks the user to enter their credentials on a web page.
The service performs the authentication in the background and validates by
script if the user has the appropriate permissions:
/usr/local/opnsense/mvc/app/controllers/OPNsense/CaptivePortal/ \
Api/AccessController.php
In the background, the AuthenticationFactory will once again decide on the
validity of usernames and passwords.
The AuthenticationFactory mentioned several times is a PHP script of the
MVC framework of OPNsense (see Chapter 24). OPNsense uses this soft-
ware to implement the configuration backup, the API, and create various
firewall functions.

222
 Summary

Summary
The central authentication of OPNsense is a smart method to standardize
the login of users and administrators. Thus, the time of local accounts is a
thing of the past, since every authorized person can access every firewall
depending on his or her roles. Accounts can also be quickly disabled in case
of a security breach.
OPNsense limits protocol support to LDAP and RADIUS, which is compatible
with all popular authentication servers. For unsecured connections, LDAP
cooperates with TLS and encrypts the packet contents. RADIUS already has
a built-in crypto function, but it only protects the password.
Only the firewall troubleshooting is limited. The only way to track down a
serious problem is to use the command line.

OPNsense has more tricks that are not mentioned in this chapter. For
example, an additional one-time password can strengthen the local login
(see Chapter 9). OPNsense regulates Internet access for guests, as offered
in hotels and restaurants via a Voucher. With these generated vouch-
ers, a guest is authenticated without having to create a new user account
beforehand.

223
Chapter 15. Central Authentication

224
 Part IV

For Hackers

225
Chapter 16

Multi-WAN

When the performance limit of a computer system is reached, undesired

effects usually occur: servers refuse to authenticate users, memory is moved
to the hard disk, or firewalls do not allow new connections.
When network links are saturated, routers begin to discard IP packets. A
single lost packet is not too bad, and the TCP protocol can handle it very
well. However, if the network is overloaded, the loss rate increases, causing
all TCP connections to lack a few packets in their data stream. This results
in stalling applications, slow file transfers, and aborted applications. An
excessive load in the network affects all end devices.

The obvious solution is vertical scaling, where limited equipment is up-

graded. The server gets more powerful components, the firewall gets a
larger session table, or the network connection gets a technology update
with more bandwidth.

In most cases, it is not possible to improve the performance of a net-

work device. A Gigabit Ethernet switch does not become a TenGigabit
switch overnight. The VPN throughput of a router has an upper limit (see
Chapter 20) and the next generation Internet link is often over budget.

The alternative approach is horizontal scalability. It provides an increase in

performance through multiple parallel devices or links. A group of servers
answers the queries, or several bundled Gigabit links interconnect switches

227
Chapter 16. Multi-WAN

and routers with a second (low-cost) Internet connection to increase the

available bandwidth. The load is distributed among several components.
In the world of network devices, horizontal scaling is the most common
form of performance improvement because the financial investment is lower.
Besides, there are open standards (RFCs) for a parallel operation that are
accepted and implemented by many manufacturers: gateway redundancy
using CARP (see Chapter 12), bundling Ethernet interfaces as Link Aggre-
gation (according to IEEE 802.3ad), or with a dynamic routing protocol.

This chapter examines the parallel use of multiple uplink connections that
do not belong administratively to the local network. Typically, these are
Internet connections, but they could also be MPLS or leased lines. Generally,
it is expected that the network traffic must follow the rules of the ISP –
especially concerning public IP addresses.

Requirements

What does load distribution over several links achieve? The exact require-
ments depend on the surrounding infrastructure but share the following
key points:

• Outbound load distribution: the clients in the LAN reach the web
servers on the Internet via several links.

• Failure protection: a faulty line must be quickly detected and excluded

from the logic of load distribution.

• Configuration: the selection of the link must not be arbitrary, but

must be configurable.

• Weighting: an Internet link must be weighted depending on its band-

width.

This wish list only concerns the area of load distribution. The rules for
security on the Internet with a firewall policy still apply!

228
 Load distribution in the WAN

Load distribution in the WAN

What looks so simple in theory becomes a challenge in practice. Where

should the default route point when two Internet links are present? Do the
uplink bandwidths have to be the same? And how can critical applications
always use the stronger link?
The WAN load distribution distinguishes between outbound network traffic
(from LAN to the Internet) and inbound connections (from the Internet to
LAN or to DMZ). For outgoing packets, the last hop in the local network
decides which Internet Service Provider (ISP) will receive the data stream.
For this decision, the load distributing router uses its configured policy.
This policy consists of firewall rules (see Chapter 6). A rule may not only
reject or allow traffic, it may also decide the routing and ignore the routing
table. This concept allows a very fine-grained policy to determine which
application, which client or which target network accesses the Internet,
and via which service provider. A final default rule sends the traffic over a
random link or blocks it completely.
Before the packet leaves the firewall, network address translation (NAT, see
Chapter 8) must pick an IP address of the selected Internet link and insert
it into the IP header of the outgoing packet. This step is very important
because it ensures that the response packets return over the same line.

Lab environment

This chapter includes almost all firewalls, clients, and networks of the demo
lab (Figure 16.1 on the next page). Site 1 represents an internal network
that communicates with the simulated Internet via the load-sharing service
of RT-1. Firewall RT-1 has two Internet connections: the first ISP offers
an imaginary bandwidth of 34 Mbit/s via the WAN-1 network and RT-core
via an ISP router. The second Internet link uses the WAN-2 network with
10 Mbit/s to the ISP router RT-3.
ISP routers RT-3 and RT-core have a connection to site 2. In site 2, the
host CL-2 is used as a web server to offer a few demo pages. These web
pages should only check whether a TCP connection between the sites is
established and indicate from which ISP the request originates.

229
Chapter 16. Multi-WAN

ISP-1 / WAN-1
CL-1 198.51.100.0/24
RT-core
OPNsense
Debian 10 10.2.1.25
RT-1 198.51.100.6
em1 lan0
OPNsense
198.51.100.1 em2
192.0.2.6
10.1.1.1 em3
lan0
em1 em4
10.1.1.25
192.0.2.1 CL-2
Windows 10
em2 em1
192.0.2.3 10.2.1.3

Site-1 ISP-2 / WAN-2 Site-2

10.1.1.0/24 192.0.2.0/24 RT-3 10.2.1.0/24
OPNsense

Figure 16.1: Site-1 reaches Site-2 over two different links

Web server
For better traceability, the Site-2 network needs an HTTP server to display
a small web page with status information. The existing CL-2 machine is
suitable for this purpose, but any other server with web service does a good
job here.
An Apache HTTP server is quickly installed under Windows. The offered
CGI script ip.cgi (see Appendix C) provides a status page and belongs to
the folder

c:\apache2\cgi-bin\

Encryption and certificates improve security on the web but do not help
build the lab, so they are not used here. TCP port 80 is sufficient for
checking the load distribution.

Operation
OPNsense couples the gateway decision with the ruleset of the firewall.
This combination allows each firewall rule to make its own decision as to
which gateway or gateway group will forward the outgoing packets.
The firewall also needs to compare its getaway decision with the availability
of that gateway. Forwarding can only be used with a fully functional
gateway and a working Internet link.

230
 Configuration

Configuration
First, create a gateway and assign availability checks. Then use that gateway
in a firewall rule.

Gateways
The OPNsense firewall sends the regular network traffic via the configured
default gateway to the Internet. If there are multiple Internet links, OPN-
sense requires multiple gateways – one for each uplink and IP protocol. For
example, if two ISPs provide both IPv4 and IPv6 access, OPNsense needs
four gateways.
Use the web menu at System → Gateways → Single to create the new
gateways. The mandatory specifications for a gateway are an interface,
address family, gateway (IP address), a monitor IP, and a name that cannot
be changed later. Figure 16.2 on the following page shows the uplinks of
RT-1 and the configured gateways.
The Advanced button hides sophisticated options that allow fine-tuning of
the gateway monitoring. The weighting of a gateway is vital for a smooth
operation. It decides how much data traffic the configured gateways can
handle as soon as they are used in a gateway group – a higher weighting
number results in more network traffic.
The next section explains the selection of a good monitor IP and further
fine-tuning of the checks.

Health check
Checking the status of neighboring devices is usually performed by a routing
protocol. Routers exchange packets in regular intervals to show life and
to validate that the network adapter and the cable connection work. On
the Internet, this works via the Border Gateway Protocol (BGP), which
is intended for large scenarios. For modest environments, the OPNsense
firewall uses its own methods to check its uplinks for functionality.
In the simplest case, RT-1 regularly sends a ping to the gateway and waits
for a response. Whether or not the uplink works cannot be determined with
this check. It is best to pick a monitor IP located behind the ISP gateway.
The monitoring address must respond reliably because missing responses

231
Chapter 16. Multi-WAN

deactivate the gateway. Google’s DNS servers have achieved a high degree
of reliability, with addresses that are easy to remember: 8.8.8.8, 8.8.4.4,
2001:4860:4860::8888, and 2001:4860:4860::8844.

The frequency of the pings and the tolerance for unsuccessful tests have a
significant influence on the duration during which a broken link remains
undetected. Smaller values detect a failed Internet connection faster, but
increase the base load of the line and it may lead to false alarms.

Note
For comparison: routing protocols use a ping interval of 10 seconds
and alert after 40 seconds.

The destination address 203.0.113.6 is an additional address in the lab

environment on the ISP router RT-core, which is provided as a virtual IP.
This address allows simulating a failure by temporarily deactivating the
virtual IP.
The health check looks similar to the other link, except for the interface and
monitor IP, the latter answered by firewall RT-3. The view under System →
Gateways → Single and Figure 16.2 shows whether both Internet lines are
healthy and usable.

Figure 16.2: All monitored gateways are alive and kicking

232
 Configuration

Gateway Groups
A gateway group combines several gateways. Within the group, each
gateway gets a priority, which OPNsense calls Tier. It certifies to the
gateway how crucial it is within the group.
These groups and priorities can be used to configure multiple scenarios:

• Failure protection. Data communication takes place via the preferred

gateway (Tier 1). The backup gateway (Tier 2) is only used in the
event of an error.
• Load sharing. All Internet links are used, whereby the specific distri-
bution is less important than the availability. The gateways are all on
Tier 1 and have a weighting of 1.
• Load balancing. The outgoing connections are distributed in such a
way that the Internet lines are evenly utilized. All gateways are Tier 1,
but the weighting must match the bandwidth of the uplink.

It is even possible to combine these scenarios. OPNsense may distribute the

outgoing load over two gateways (Tier 1) and at the same time has another
backup link (Tier 2) in reserve.
With load balancing, the firewall ideally distributes sessions at the ratio of
34 to 10 over its Internet providers.

When is a gateway usable and when does it take a break? OPNsense

manages this via the Triggering Level of a gateway group.

• Member Down. The gateway goes out of service if it is unreachable;

thus, generating 100% packet loss.

• Packet Loss. OPNsense considers the gateway to be inactive if the

packet loss is greater than the specified upper threshold. As soon as
the loss rate decreases and falls below the lower threshold value, the
gateway is active again. The default values are 10% and 20%.

• High Latency. OPNsense detects the defective gateway if the latency

is greater than the upper threshold. If the situation improves and
the delay falls below the lower threshold, the gateway is considered
healthy again. The defaults are 200 ms and 500 ms.

233
Chapter 16. Multi-WAN

• Packet Loss or High Latency. OPNsense combines the two conditions

and ignores a gateway as soon as one of the two values climbs above
its threshold.

The mentioned thresholds for packet loss and latency are a property of the
gateway and not of its group. The configuration is located at System →
Gateways → Single.

Figure 16.3: Gateway groups decide on the gateway’s use

There are several grouping options for the two gateways in the lab network.
The example gateway group GOLD prefers a higher bandwidth gateway
RT-core and only switches to the reserve gateway RT-3 in case of an error
(Figure 16.3).
Group SILVER does it the other way around and sends traffic through the
smaller pipe to Gateway RT-3. In case of a failure, the traffic switches to
RT-core and then benefits from the higher bandwidth. Applications without
bandwidth requirements fit the DEFAULT group, which randomly passes IP
connections to a gateway.

It is not possible to mix gateways of different IP versions within a gateway

group, so each group must exist twice: once with the IPv4 addresses of the
gateways and once with the IPv6 counterpart.

234
 Configuration

Firewall
The OPNsense firewall controls how the gateway groups are used. The
firewall rules select an individual gateway or a gateway group. The default
gateway is preselected for each new rule. For policy-based routing, simply
replace the default gateway by one of the configured gateway groups. It
is possible for every single firewall rule to use a different routing path by
modifying the Gateway field. The available options in the dropdown menu
on the left contain the gateway groups (Figure 16.4).

Figure 16.4: The firewall rule decides the routing

It is even possible to globally replace the default gateway by a gateway

group. Take advantage of this option at System → Settings → General.

Address translation
The two WAN networks represent the Internet and, like the real Internet,
do not tolerate private IPv4 addresses. In addition to its new function as
load-balancer, the firewall RT-1 must replace the sender address of the
outgoing IPv4 packets with a public address (see Chapter 8). That way, the
packets of the internal clients in site 1 are also routable in the lab Internet.
When the ruleset (or randomness) has chosen the outbound interface,
OPNsense masquerade the outgoing IP packets with the IPv4 address of
that interface.

235
Chapter 16. Multi-WAN

Scenario
The load-balancer RT-1 is now ready for action but does not yet have any
rules for special routing. In this first scenario, the web access from client
CL-1 from site 1 to the remote server CL-2 in site 2 will be treated specially:
traffic to port 81 is sent over WAN-1 and RT-core, while traffic to port 82
travels over WAN-2 and RT-3.
The load-balancer will distribute access to port 80 evenly over both lines.
The web server on CL-2 provides information on the load balancing decision
based on the query’s source address.

A single firewall rule would be enough to permit the traffic. But since
there are three different requirements, three similar rules are needed. The
rules always have Allow as an action but differ in the destination port
number and their gateway decision. Figure 16.5 shows the required rules
for load balancing based on TCP ports.

Figure 16.5: Firewall rules now decide the next-hop gateway

Failure
The time has come: the Internet link via RT-core fails to work. The health
check detects this condition because there is no response to a ping. It is
easy to simulate the failure by using a filter rule on RT-core that does not
return pings or by deleting the virtual IP address from RT-core.
Shortly afterwards, the gateway status of load balancer RT-1 changes be-
cause the gateway RT-core is no longer available. In Figure 16.6 all gateway
groups are affected by this failure.

236
 Monitoring

Figure 16.6: The unreachable gateway is marked as Offline

The outgoing connections switch to the alternative link. Existing TCP con-
nections will terminate because the outgoing packets arrive at the target
system with a different source address. The TCP protocol does not like
changing IP addresses at all. Robust applications immediately establish a
new connection so that the user does not receive an error message.

Monitoring
Without monitoring, the failure of a single link remains unnoticed until the
second Internet link also dies and makes the web inaccessible. It is therefore
important that at least a responsible administrator or a monitoring team is
informed of the status change to begin troubleshooting.
OPNsense monitors the Internet links through the dpinger process and
shows availability at System → Gateways → Single or Group.
If an uplink fails, OPNsense can contact the admin by email. The settings
for email address and mail server are located at Services → Monit. The
pre-installed check gateway_alert interacts with dpinger, learns about the
faulty gateways, and alerts via email.

237
Chapter 16. Multi-WAN

Warning
Monit sends an email for each gateway’s state change and at regular
intervals. That way, OPNsense can become very chatty on flapping
Internet links.

IPv6

The described load distribution of IPv4 packets is based on address transla-

tion (NAT). The need for NAT results from the low number of remaining
IPv4 addresses. The IPv6 world has enough addresses and does not need
NAT anymore.
However, load balancing is also desirable in IPv6 networks, so NAT has
finally sneaked into the IPv6 landscape. About 20 years after the begin-
ning of IPv6, RFC 6296 introduced Network Prefix Translation (NPTv6).
That way, internal IPv6 networks are translated into external IPv6 networks.

When using this concept in the lab network, firewall RT-1 changes packets
from clients in their local network fd00:1::/64 to a global address from the
prefix 2001:db8:11::/64 using NPTv6. The client fd00:1::25 is then visible
on the Internet as 2001:db8:11::25. Section IPv6 of Chapter 8 on page 101
describes the configuration of NPTv6.

Warning
NPTv6 requires a static prefix, which is a feature of an Internet link for
business customers. Multi-WAN with OPNsense does not work with a
dynamic prefix on a private customer connection.

OPNsense requires two prefix translations for two gateways. The config-
uration in Figure 16.7 translates outgoing traffic into an IPv6 address of
the selected provider. If the corresponding interface or gateway has failed,
OPNsense ignores that prefix so that the IPv6 address always matches the
provider’s address range.

238
 Technical background

Figure 16.7: IPv6 prefix translation for multiple Internet uplinks

Technical background
When dealing with numerous Internet links, OPNsense combines several
techniques that work together hand in hand: the service dpinger checks
the gateways and the packet filter pf makes a decision about routing.
OPNsense uses the software dpinger to check whether the neighboring
gateways are accessible. The name indicates its duties, dpinger is a small
service in the background that sends ICMP echos to its configured target at
regular intervals. The results are stored in a UNIX socket and provided to
the web GUI for graphical representation. A separate dpinger instance runs
for each gateway, so that four gateways require four sockets. If a connection
fails, the socket provides the following exemplary content:
root@RT-1:˜ # cat /var/run/dpinger_*.sock
RT_3_ip4 383 79 0
RT_3_ip6 375 84 0
RT_core_ip4 779 206 36
RT_core_ip6 803 221 36

The first number shows the round-trip time in milliseconds, followed by the
deviation. The last number corresponds to the packet loss in percent.
When a firewall rule references a gateway group, the generated pf rule
uses the route-to keyword to bypass the main routing table. The routing
decision then returns the value of route-to, which consists of the outgoing
interface and the IP address of the neighboring gateway.
The syntax of an example rule at the LAN adapter for incoming TCP traf-
fic to port 3389 uses route-to to send the IP packets to the gateway
198.51.100.6.

239
Chapter 16. Multi-WAN

pass in quick on em1 route-to {( em3 198.51.100.6 )} inet \

proto tcp from {any} to {any} port {3389} keep state \
label "c489369db93b41c6be02ac140822f415"

In the case of load sharing and load distribution, i.e. with multiple routing
candidates, route-to may decide between several gateways.
pass in quick on em1 \
route-to {( em3 198.51.100.6 ) ( em4 192.0.2.6 )} \
round-robin inet proto tcp from {any} to {any} port {3389} \
keep state label "6b4c56b73616a0602b3f7830bc89d1e2"

If the gateway status changes, dpinger will notice this and adjust the pf
rules accordingly.

Summary
OPNsense is fully prepared for multiple Internet links to distribute the
data streams and satisfy the bandwidth needs of modern applications.
The firewall rules and gateway group can control very granularly which
traffic reaches the Internet via which ISP. And if an Internet connection
is interrupted, the health checks detect this issue, and OPNsense uses a
different uplink within seconds.
The load-sharing even works with IPv6, but only when the ISPs offer a fixed
prefix.

240
Chapter 17

DSL router

A regular DSL connection includes a DSL router that takes care of dialing
in, address translation, and securing the internal network. The home-use
routers are even equipped with wireless access point, telephony system,
and mini NAS. In the business environment, the router provides VPN and
manageability.
This chapter describes the use of an OPNsense firewall as a DSL router for a
VDSL2 connection with the German provider Deutsche Telekom using IPv4
and IPv6.

DSL types
The usable bandwidth of a DSL connection has increased over the last
decade. Initially, an IDSL link was capable of 144 kbit/s, which equals
two bundled ISDN channels. Nowadays, the throughput matches Ethernet
speeds at 100 Mbit/s. The available bandwidth allows Internet Service
Providers to offer more than just simple Internet access. Depending on
the bandwidth, customers can subscribe to telephony, television, or video
streaming. All provided services are wrapped in IP packets and sent through
the DSL link.
The marketing department of each ISP has picked great product names for
its services. However, the DSL technique below is similar and complies with
the rules of the International Telecommunication Union (ITU-T). Some
popular standards are briefly summarized:

241
Chapter 17. DSL router

• ITU-T G.992.5 Annex J. The transmission uses the technology of

ADSL2+ with modified frequencies described in Annex J. It offers
speeds of 24 Mbit/s downstream and 3.3 Mbit/s upstream.

• ITU-T G.993.1. Very-high-bit-rate DSL (VDSL) offers customers down-

stream rates of 52 Mbit/s and upstream rates of 16 Mbit/s.

• ITU-T G.993.2. VDSL2 is an enhancement of VDSL and reaches speeds

of 100 Mbit/s in both directions.

• ITU-T G.993.5. When the VDSL2 access link uses vectoring, it can
improve the data rates by mitigating negative physical effects of the
line signals. The speeds reach 100 Mbit/s downstream and 40 Mbit/s
upstream.
The offered bandwidth and implemented technology depend on the ISP
and regulatory situation in each country.

Lab setup
In this chapter, OPNsense runs on the physical hardware board APU 1D4
[7], manufactured by PC-Engines. Chapter 2 on page 42 describes the
equipment and installation.
The APU board is not equipped with an internal VDSL modem, so an
additional modem device must terminate the VDSL line. The APU is directly
connected to this modem.
Figure 17.1 shows the connectivity of a typical DSL router.

Note
The hardware platform uses RealTek network adapters, which FreeBSD
names re0 to re2. The naming convention is described in Chapter 4
on page 53.

The adapter re2 is connected to the modem and will negotiate the dial-in.
When the dial-in process is complete, it provides access to the Internet. The
re0 interface allows SSH and HTTPS access to the OPNsense management
interface. The remaining interface re1 leads to the internal network and
provides Internet communication to the client.

242
 PPPoE Dial-in

Site-2 RT-3
10.2.1.0/24 APU 1D4
OPNsense
fd00:2::/64 Internet
re1 pppoe
10.2.1.3 re2
Modem DTAG
re0 80.128.0.0/12
10.5.1.3 2003::/19

Management
10.5.1.0/24
fd00:5::/64
Figure 17.1: PC-Engines APU 1D4 as VDSL router

The separation of management access and internal LAN port only increases
security if there is a separate management network in the surrounding
infrastructure. If this is not the case, the interface re0 can also be used as a
regular LAN adapter to provide another physical port.

PPPoE Dial-in
Even though DSL is a leased line technology, most Internet providers expect
a prior dial-up with session negotiation. That way, the ISP can force a user
authentication, address assignment, and billing.
European ISPs mainly use PPPoE, which combines the point-to-point proto-
col (PPP) for dial-in lines with the fast Ethernet protocol.
The provider may even go one step further and use a VLAN tag for the
PPPoE packets. The VLAN concept divides one vast shared network into
several smaller subnets to isolate services, departments, or data centers
from each other.
For example, the ISP Deutsche Telekom implements this approach to sepa-
rate its IP-based television offering (IPTV) from regular Internet operations.
The IPTV platform initially transported packets parallel to normal Internet

243
Chapter 17. DSL router

traffic, which caused problems as the number of customers increased. For

this reason, VLANs separate the multicast IPTV traffic from regular Internet
traffic. Other service providers use the same approach and even separate
their VoIP technology with a VLAN.
The VLAN tag includes a number from 1 to 4094, which is used as an
identifier. The aforementioned German ISP picked VLAN number 7. As a
consequence, RT-3 must tag the outgoing PPPoE packets with the ID 7 so
that the provider recognizes them.
The OPNsense web GUI can assign a VLAN to an existing network adapter
at Interfaces → Other Types → VLAN. In this setup, the Parent interface is
re2, and the VLAN tag is 7.
Then a new interface is available at Interfaces → Assignments with the
name Vlan 7 on re2. A click on the plus symbol makes the interface usable.
Table 17.1 and Figure 17.2 provide the appropriate values for dialing into
Deutsche Telekom’s DSL network.

Attribute Value
Enable 2

Description VDSL
Block private networks 2

Block bogon networks 2

IPv4 Configuration Type PPPoE
IPv6 Configuration Type None
MTU 1500
Username 001234567890123456789012#0001 \
@t-online.de
Password 12345678

Table 17.1: Internet access via PPPoE

Right now, the focus of dial-in is limited to IPv4. The specifications for IPv6
are relevant in section IPv6 with prefix delegation on page 249 and can be
ignored for now.

Due to the additional PPPoE header with a fixed length of 8 bytes, the
Ethernet frame loses some space. Of the maximum 1500 bytes, the PPPoE
has already reserved eight bytes so that the maximum size is reduced to

244
 PPPoE Dial-in

Figure 17.2: OPNsense performs the PPPoE dial-in

1492 bytes (see Chapter 20). OPNsense does the math: for the dial-up
process to use the appropriate MTU of 1492 bytes, the web UI must show
the real value of the Ethernet interface of 1500 bytes.
The strange format of the login credentials is unique to this ISP and has its
origin in the ISDN era.

Note
If the modem takes over the VLAN marking or if the ISP does not use
VLANs at all, the firewall no longer needs a VLAN identifier and can
execute a regular PPPoE dial-in. In this case, skip the VLAN number
configuration on the firewall.

245
Chapter 17. DSL router

If the cabling is correct, the firewall should have already started the PPPoE
negotiations. If the DSL modem reports a synchronized connection, nothing
stands in the way of a dial-in.
After a successful PPPoE connection, the firewall receives IP addresses,
and the interface pppoe0 or VDSL appears in the list of network adapters
(Figure 17.3). In case of failure, take a look in the log file at Interfaces →
Point-to-Point → Log File.

Figure 17.3: Dashboard shows all local IP addresses and gateways

The following sections upgrade the OPNsense firewall step-by-step to a

fully-fledged DSL router. It is easier to locate and resolve errors within a
configuration step. The security aspect is of secondary importance.

LAN adapters
The network adapter re1 connects to the local network. Its IPv4 address
is a static address from the local IP range, as shown in Figure 17.1 on
page 243. The IPv6 address is unknown because the firewall receives it in
the following section IPv6 with prefix delegation from the Internet provider.

246
 DNS and DHCP

Network Bridge
If the employed hardware has multiple LAN ports, they can be used to
connect additional devices that require Internet access. These ports behave
like a small switch and are configured as a network bridge.
At Interfaces → Other Types → Bridge the available physical adapters
join the internal switch bridge0. The client’s devices now attached to the
network can communicate with each other through the bridge without
further setup. Promote the network bridge to a real interface at Interfaces
→ Assignments. The IPv4 address of the bridge is also the default gateway
for the connected computers.

DNS and DHCP

The two infrastructure protocols DNS and DHCP are usually explained
together, although they perform different tasks. The Domain Name Sys-
tem (DNS) handles the name resolution. It converts server names like
www.example.net into their IPv4 or IPv6 address used for communication.
A client computer will not be able to access a single web page if it cannot
ask a DNS server for the stored address.
The Dynamic Host Configuration Protocol (DHCP) is the plug-and-play
technology for IP networks. The DHCP server assigns an IP address to
each client and reveals more facts about the local network. In this way, the
requesting client receives all the necessary information to get along in the
network.

When it comes to DNS, OPNsense plays the role of mediator. The fire-
wall accepts all requests from the clients and relays them to an external
DNS server to get a qualified response.
OPNsense offers two competing services: Unbound DNS and Dnsmasq DNS,
the first being the acceptable default.
OPNsense learns the address of a DNS server during PPPoE negotiation. If
the ISP does not provide this information, the firewall gets its DNS servers
during the initial setup (see Chapter 4). If you do not know the IP addresses
of your provider’s DNS servers, use the easily remembered Google Public
DNS address (see Figure 17.4 on the next page).

247
Chapter 17. DSL router

Figure 17.4: OPNsense uses Google’s DNS resolvers

OPNsense configures the DHCP service at Services → DHCPv4. In the LAN

tab, the GUI expects information about the environment to be communi-
cated to the clients. In the configuration, specify at least the Range, DNS
servers, and Gateway – and make sure the checkbox Enable is activated.
Figure 17.5 shows the settings for RT-3.

Figure 17.5: OPNsense offers IPv4 addresses via DHCP

The gateway for the clients is the local firewall since it knows the path
to the Internet. For name resolution, the clients can choose between the

248
 IPv4 with Address Translation

firewall and Google DNS. The assignment of an IPv4 address is limited in

time and defaults to two hours (7,200 seconds).
Once a client computer is connected to a LAN port of the firewall and is set
up as a DHCP client, it will query for an IP address over the network. RT-3
will answer this request with the next free IPv4 address.

IPv4 with Address Translation

The previous configuration allows clients to reach the firewall, the firewall
then reaches the Internet. However, private IPv4 addresses are not allowed
on the Internet and must be replaced by a public address before entering the
global network. OPNsense performs this task with an address translation
(NAT, see Chapter 8) of all outgoing packets.
OPNsense replaces the address in the IPv4 header of outgoing packets
with its own public IP address (Figure 17.6). This replacement affects all
connections that originate from the IP network 10.2.1.0 and head toward
the Internet via the interface pppoe0.
Clients can now communicate with other servers on the Internet, visit
websites, and send an email – but only over IP version 4.

Figure 17.6: All IPv4 clients share the firewall’s public address

IPv6 with prefix delegation

Internet service providers have been offering IPv6 access for a few years.
For example, the German ISP Deutsche Telekom has been using dual-stack
technology on newer DSL lines since 2012. The customer router gets a
regular IPv4 address and – if it asks for it – also an IPv6 prefix (see Chap-
ter 5). This query allows the OPNsense firewall to communicate with the
IPv6 world.

249
Chapter 17. DSL router

Address translation is not a good option with IPv6, so the firewall must ap-
ply for another IPv6 prefix and share it with its clients by prefix delegation.
Deutsche Telekom generously provides its customers with /56 prefixes.
This prefix length allows the OPNsense machine to address 256 additional
internal IPv6 networks and to have control over more IPv6 addresses than
the entire IPv4 address space can provide.

First, the firewall has to request a modest /64 IPv6 prefix when dialing in.
With this inch, OPNsense takes a mile and immediately demands a big /56
prefix.
To get started with IPv6, OPNsense uses the same network adapter that was
used in the previous section PPPoE Dial-in. The method is different only
because IPv6 goes for DHCPv6. Table 17.2 lists the new adapter settings at
Interfaces → [VDSL] that comply with the German ISP. The configurations
for IPv4 and PPPoE remain unchanged.

Attribute Value
IPv6 Configuration Type DHCPv6
Configuration Mode Basic
Request only an IPv6 prefix 2
Prefix delegation size 56
Send IPv6 prefix hint 2

Use IPv4 connectivity 2


Table 17.2: Internet access via DHCPv6

This additional IPv6 network is intended for the client’s devices. The LAN
adapter of the firewall also provides itself with an IPv6 address from the
received prefix. The IPv6 configuration type Track Interface is best suitable
for this. That way, the firewall monitors the IPv6 address of the VDSL
interface (Figure 17.7). If this interface receives a new address, the LAN
adapter reserves a subnet of it. The firewall picks the address based on the
number selected at IPv6 Prefix ID.
The OPNsense firewall provides the IPv6 prefix in the LAN via Stateless Ad-
dress Auto-Configuration (SLAAC). With this method of address allocation,
OPNsense gives just enough information via Router Advertisement that the
client functions in the network.

250
 IPv6 with prefix delegation

Figure 17.7: OPNsense assigns a single IPv6 address to its LAN adapter

For a full-featured DHCPv6 server, OPNsense expects a static IPv6 address

that is not available in this prefix delegation scenario. Due to this limitation,
the client and the firewall agree on a stateless DHCPv6. This lightweight
DHCPv6 service only provides the hosts with the IPv6 prefix, DNS, and NTP
server of its environment – an IP address is missing! The client must take
care of the IPv6 address itself and build the address from the learned prefix
and its MAC address.

The prefix delegation process is dynamic. When the firewall disconnects

the DSL link and dials in again, it receives a different IPv6 address and IPv6
prefix from the provider. A DSL connection interruption causes all IPv6
devices to be re-addressed. After dialing in again, the firewall receives the
exemplary prefix 2003:e1:722:e600::/56 and attaches a /64 network to its
adapter bridge0 or re1.

Figure 17.8: The dashboard shows the local IPv4 and IPv6 addresses

251
Chapter 17. DSL router

The web UI at Interfaces → Overview or the dashboard (Figure 17.8 on the

preceding page) show which IPv4 and IPv6 addresses are assigned, and at
which adapter.

Firewall
A dual-stack firewall has its feet in both IP worlds and is vulnerable to attack
via both Internet protocols. Therefore, it needs a firewall policy for both
protocols to protect itself and its clients. OPNsense always distinguishes
between IPv4, IPv6, or acts for both protocols in the firewall rules (see
Chapter 6).

Note
The following firewall guidelines are minimal and should be adapted to
your own requirements before being used productively. The examples
do not claim to be complete.

IPv4/IPv6
The IPv4 traffic filtering policy limits access from the Internet to internal
resources or the DSL router. For data communication over IPv6, the same
rules apply as IPv4, but with a few extensions.
The Internet Protocol version 6 relies more on the Internet Control Message
Protocol (ICMP). The functionality of the Address Resolution Protocol
(ARP) in IPv4 belongs to IPv6, in the ICMP department. The SLAAC also
uses ICMPv6. For this reason, it is not possible to block ICMP completely.
The firewall policy must permit at least some features of ICMP. OPNsense
creates the necessary rules to permit an address assignment, and to not
accidentally block it. The same applies to rules for Bogon networks, i.e.
attempts to access IP addresses from unassigned prefixes.
Without an existing firewall rule on the VDSL adapter, OPNsense only
allows the packets required for Internet dial-up with PPPoE or DHCPv6.
Also, the firewall engine permits reply packets from existing connections.
The only custom rules for connections from the Internet apply to services
provided behind the DSL line.

252
 Technical background

Custom rules for connections from the Internet are only required if the DSL
line is used to offer services to the public (see Chapter 6).
The ruleset for the internal network of a DSL router looks rather tolerant.
OPNsense even has two default rules preconfigured. Rule one: allow any
IPv4 traffic to the Internet. Rule two: permit the same freedom for IPv6.

Management access
In environments with increased security requirements, it is best to separate
the administrative access to the firewall from the user traffic (see Chapter 9).
The SSH and web services are allowed only on the management adapter
and blocked on the other interfaces.
Also, the firewall rules can restrict access to its services by specific IP
networks. It also prevents the management interface from being misused
for transit traffic.
The firewall policy of this chapter allows incoming sessions via HTTPS,
SSH, and ping from the IP network 10.5.1.0/24. If additional IP networks
need access to the configuration interface, the firewall needs static routes
that send the traffic back to the correct gateway. Otherwise, the response
packets leak to the Internet via the default route.

Technical background
OPNsense realizes the PPPoE dial-up to the Internet with the netgraph multi-
link PPP daemon. (MPD5). This service works mostly in user space and
uses only a few methods from kernel space. MPD5 justifies the separation
between user and kernel space with security and speed. For the negotiation
and encapsulation of PPP packets, MPD5 uses the netgraph subsystem.
OPNsense writes the settings for PPPoE to the file mpd_opt2.conf and then
starts the MPD5 process. This process begins the dial-up, negotiation, and
monitoring of the Internet connection.
For IPv6, Internet Service Providers use the Router Advertisements from
ICMP to assign the IPv6 address, DNS server, and MTU to the requesting
clients. The DSL routers need to request a further prefix for their attached
clients. The ISP also uses DHCPv6 to offer an additional IPv6 network.
On the customer side, this setup expects a DHCPv6 client process that

253
Chapter 17. DSL router

requests the IPv6 network for delegation. This process is dhcp6c, and it
runs permanently because the ISP also announces changes in the network
via DHCPv6. The configuration is in the file:
/var/etc/dhcp6c_opt2.conf

The same service determines to which network adapter this prefix should
belong.

Who informs all the clients in the network about the IPv6 environment?
OPNsense provides this information via Router Advertisements and the
service radvd. This service floods its information every 3-10 seconds via a
multicast into the local network. The announcements contain the learned
DNS server and the IPv6 prefix. Further settings are present in radvd’s
configuration file:
/var/etc/radvd.conf

Summary
With the described configuration and a suitable modem, OPNsense becomes
a fully-fledged DSL router on any hardware, allowing clients to access the
Internet via IPv4 and IPv6.
OPNsense has other tricks that go beyond regular router operation:

• Dynamic DNS. With the plug-in DynDNS , the firewall deposits its
public IP address with a DNS provider and is thereby accessible under
its name on the Internet.

• Traffic analysis. OPNsense examines the data stream and keeps

statistics of the applications and web services used. The evaluation
provides insight into Internet use (see Chapter 13).

• Traffic shaping. The firewall can purposefully slow down data streams
by the IP address or application. For example, administrators can
restrict the bandwidth for the next Windows update to 6 Mbit/s in
order to free up bandwidth for other uses.

254
Chapter 18

Intrusion Detection

Firewalls do their best to enforce a security policy. However, the software

implementation might have bugs, and the configuration of the ruleset might
have errors. In the worst case, an attacker with enough knowledge and
time can get past a firewall into the heart of the network.

It is not possible to secure a network totally, no matter how good or expen-

sive the firewall is. And even an Intrusion Detection does not guarantee
100% security. However, with correct placement and careful adjustments,
an Intrusion Detection System can mitigate or prevent most attacks.

IPS and IDS

An Intrusion Detection System (IDS) takes an in-depth look into the trans-
mitted IP packets. If the analyzed content looks suspicious, the system will
trigger an alarm. When a packet contains a possible threat, the IDS may
detect the threat using known attack patterns stored in its database. This
process makes the IDS similar to a virus scanner: examine the data and
compare it with known patterns.

The IDS is limited to recognition, so the break-in is noticed but not pre-
vented. If the detected break-in must be stopped directly, the system
changes its role to an Intrusion Prevention System (IPS). The difference
between IPS and IDS lies only in the last step: while the IDS squeals on

255
Chapter 18. Intrusion Detection

any suspicious activity, the IPS will directly prevent the action by halting
packets or actively terminating connections.
In theory, the Intrusion Prevention System is a miracle drug: place the IPS
in the network and break-ins stop. Unfortunately, many IPS environments
suffer from a high rate of false positives. If legitimate connections break off
during regular communication because the IPS has discovered a seemingly
malicious packet, then this has nothing to do with protection against break-
ins. The IDS, on the other hand, does not modify the network connections
and becomes a pain in the neck for the monitoring team receiving the
inaccurate reports.

The art of intrusion detection is in the fine-tuning! After the initial setup,
the alarm rate is high because the IDS must first become familiar with the
new network. It is essential to minimize the number of false alarms by
exceptions or individual deactivated checks. The IDS get promoted to an
IPS only if the system classifies the environment as normal and no longer
produces a false alarm.

Network integration

For its investigation, the IDS requires insight into each network packet. The
system does not necessarily have to be in the data path for this, but can also
complete the analysis using packet copies. For real-time intrusion detection,
a mirror port at the network switch (see Chapter 23) is suitable, which
sends a copy of each transmitted packet to the IDS. Alternatively, the IDS
can analyze a recording of network traffic to find suspicious activity.
The IPS needs to be close to the network packets because it must be able to
discard packets, if necessary, to prevent worse things from happening. An
IPS usually sits in the data path and examines the packet contents in real
time.

The intrusion analysis of OPNsense is an additional service on the fire-

wall device and therefore is automatically in the data path.

256
 Lab setup

Lab setup

The firewall RT-1 becomes an Intrusion Detection System and examines

the network traffic destined for the server segment DMZ. The vulnerability
scanner OpenVAS [19] generates simulated attacks from the Internet via
WAN-1 to the server network. It explicitly checks the server labsrv for
security vulnerabilities. Between the attacker and the target, the IDS on
the firewall RT-1 can report and prevent any suspicious activity.

DMZ 10.4.1.0/24 fd00:4::/64

10.4.1.1 10.4.1.7

em2 eth1
RT-1 labsrv
OPNsense CentOS 8
em0
em0– 10.5.1.1 eth0
eth0– 10.5.1.7
em3
198.51.100.1

WAN-1
198.51.100.22 198.51.100.0/24
2001:db8:1::/64
eth1
OpenVAS
Greenbone OS
eth0
eth0– 10.5.1.22

Figure 18.1: OPNsense discovers trespassers in the network

The firewall RT-1 works on the level of a router (see Chapter 6) and is
visible by the attackers. Figure 18.1 shows the lab setup.
Other vulnerability scanners are also suitable for testing the IDS, as long as
they have good intentions and report the vulnerability instead of exploiting
it.
The vendor offers excellent documentation and a ready-to-use virtual appli-
ance making it straightforward to set up a vulnerability scanner in a short
time.

257
Chapter 18. Intrusion Detection

Attack
The vulnerability scanner audits the lab server and reports detected weak-
nesses. The firewall RT-1 implements only a few loose filter rules that allow
unrestricted access to the target server.

Note
This ruleset is unsuitable for practical use since the firewall is supposed
to be the first line of defense and the IDS is watching its back.

The vulnerability scanner begins its first investigation with the IDS still
turned off. This basic measurement shows whether the connection between
the systems is working and determines the target host’s security level.

Activate IDS
Just like a virus scanner, the IDS first needs a current status of attack signa-
tures and filters.
The web interface of OPNsense integrates the IDS at Services → Intrusion
Detection. The Download tab offers almost 60 pre-built rulesets that can
be included in the analysis (Figure 18.2). By default, all rules are inactive
and do not exist locally.

The title of each ruleset gives a good hint about their content: the strength
of ET open/emerging-imap obviously lies in the IMAP protocol and the
ruleset ET open/emerging-voip deals with Internet telephony. If the name
is meaningless, the ruleset details link to a web page that provides helpful
information.
After selecting multiple – or all – rulesets, hit the Download & Update Rules
button at the bottom of the page. The firewall will then download the latest
signatures and store them in the local system. The Enabled column informs
which rulesets are actively used for the intrusion analysis.
Every single rule is now listed in the Rules tab and can be switched on and
off individually – which is essential for suppressing false alarms!
With the current rules in an active state, the IDS is armed for the first
deployment and should carefully inspect the network packets. The small

258
 Activate IDS

Figure 18.2: The pre-built rulesets make it easier to get started with the IDS

checkbox Enabled at the tab Settings starts the background processes, and
monitoring begins. The emphasis is on monitoring, because in this mode
OPNsense only watches a break-in and reports any anomaly in the Alerts
tab.

Next attack

Another pseudo attack by the vulnerability scanner will provide an identical

security report. But now the IDS has been watching and can report many
(if not all) security incidents.
In Figure 18.3 on the next page OPNsense detected an exploit in the Bash,
which is known as Shellshock vulnerability. In addition to the source
and destination IPv4 addresses, the alarm message also contains the CVE
number under which this security exploit is filed. Further information is
available in the public vulnerability database [20].

259
Chapter 18. Intrusion Detection

Figure 18.3: OPNsense detects break-ins in the network

Fine tuning
In recognition mode, the IDS does not affect the excellent reputation of the
network because it does not actively prevent communication. Now it is
time to fine tune the IDS. It is essential to minimize the number of false
alarms because they are incredibly annoying once in IPS mode.
A good approach is to manually check the alert messages over several days
or weeks after the start of the IDS. If the IDS has triggered an alarm without
any real threat, set the rule action to Alert instead of Drop. If the target of
the attack does not exist in the network at all, it is acceptable to deactivate
the rule involved (register Rules). In both cases, the connection is not
interrupted in IPS mode, but an inactive rule will never look for this attack
again.

Activate IPS
The next step against break-ins is detection with immediate prevention: the
IDS becomes an IPS.

260
 Transparent IDS

Enable the checkbox IPS mode in the area Intrusion Detection → Ad-
ministration of the OPNsense web interface (Figure 18.4) to change the
well-tuned IDS to an IPS.

Figure 18.4: OPNsense prevents break-ins in the network

The IPS should now prevent the next planned attack on the lab server’s
services. The vulnerability scanner security report is now shorter and no
longer contains the exploits that the IPS has successfully prevented.
Even a good IPS cannot completely prevent any attack since many attacks
are masked by encrypted protocols such as HTTPS or SSH.

Transparent IDS
An Intrusion Detection System can work transparently on the Ethernet level,
just like a firewall (see Chapter 7). Thus, the IDS is mostly invisible to

261
Chapter 18. Intrusion Detection

neighboring hosts. In this way, it is straightforward to integrate an IDS

into an existing network with minimal modifications. In transparent mode,
neighboring devices do not need to change their IP addresses or routes.
The IDS is bridged into an existing physical connection. If problems occur,
the IDS can be removed just as quickly by switching the cables. Figure 18.5
shows the lab setup with a transparent IDS between the vulnerability
scanner and the target computer.

DMZ 10.4.1.0/24 fd00:4::/64

10.4.1.7
eth1
RT-1
bridge0 em2 OPNsense
10.4.1.1 em3 em0
em0– 10.5.1.1

labsrv
CentOS 8
eth0
eth0– 10.5.1.7
WAN-1
10.4.1.22

eth1
OpenVAS
Greenbone OS
eth0
eth0– 10.5.1.22

Figure 18.5: OPNsense detects intruders without annoying the neighbors

The implemented lab environment is minimal and therefore somewhat

naive. The structure shows an attacker and its target host in the same
network segment, which is an unusual scenario. This circumstance will not
affect the IDS operation since this chapter shows the possibilities of the IDS
and does not focus on firewall design.

262
 Transparent IDS

Network bridge
OPNsense needs a network bridge for transparent operation. This bridge
connects the DMZ and WAN adaptors. Data traffic can pass through the
network bridge, just like an Ethernet switch. There is no need for the hosts
to address the OPNsense firewall directly by its IP or MAC address.

Note
In a virtual environment, the two adapters of the network bridge
require extended access to the host system’s network card. In the
configuration interface of the virtual machine, this permission is called
Promiscuous mode.

Figure 18.6: A network bridge connects DMZ and WAN segments

The procedure is similar to the transparent firewall from Chapter 7 and is

only described here briefly. Refer to page 85 for the full description.

The setup of the bridge starts at Interfaces → Other Types. Figure 18.6
shows the newly created network bridge bridge0, which has two members
and presents itself as a regular network adapter in the system after setup.
Add the new bridge interface at Interfaces → Assignments. It will show
up in the list of interfaces as OPT3, whereby the number can vary. Go to
Interfaces → [OPT3] and give the new adapter a better name like br0.

263
Chapter 18. Intrusion Detection

Before the network bridge br0 gets an IPv4 address, the members DMZ
and WAN1 must lose their addresses. The former IPv4 address of the DMZ
adapter becomes the new address of the bridge interface.

Warning
If the firewall is managed by the IP address of the DMZ or WAN1
adapter, the web connection will fail once the addresses are removed.
In this case, access the firewall via a separate management adapter,
which is exclusively used for configuration.

Next, move the packet filter from the physical interfaces DMZ and WAN1
to the logical adapter br0. Unfortunately, OPNsense has no convenient
switch built into the web GUI but uses various kernel options. At least,
it is possible to change these options from the web interface: the kernel
parameters wait for their adjustment at System → Settings → Tunables.
• net.link.bridge.pfil_member = 0
prevents the kernel from filtering on the incoming and outgoing
member interfaces.
• net.link.bridge.pfil_bridge = 1
enables the kernel to filter on the bridge interface.
Finally, the network bridge is operational and receives a basic set of rules
similar to the one initially used for the physical WAN interface in section
Attack on page 258.

The IDS does not need any changes. For the intrusion detection soft-
ware, it makes no difference whether the packets have entered the local
system through an IP interface or a network bridge.

Technical background
The system for intrusion detection of OPNsense is Suricata [21] and is
licensed under the free GNU license. Suricata has been developed since
2010 and is an open-source IDS and IPS for all major operating systems.
FreeBSD provides the Suricata software as a single process that is configured
and controlled by the web interface.

264
 Technical background

/usr/local/bin/suricata -D --netmap \
--pidfile /var/run/suricata.pid \
-c /usr/local/etc/suricata/suricata.yaml

This process receives a copy of all network packets and compares the
content with its known rules describing attack patterns. For Suricata, it
does not matter how the packets reach the analysis engine. The IDS will
investigate and alert as long as the data stream can be examined locally.

Rules
The inspection rules are selected in advance from the web page, as described
in section Activate IDS on page 258. In the background, the script
/usr/local/opnsense/scripts/suricata/rule-updater.py

downloads the pattern files and stores them below

/usr/local/etc/suricata/rules/

Finally, the Suricata process receives the new rule material and loads it into
its memory. The update procedure does not interrupt the IDS.

OPNsense uses the rules of the community project Emerging Threats [22].
All offered rules are under the BSD license and can be used free of charge.
The provider gives direct insight into the rules on its public download page
https://rules.emergingthreats.net/open/suricata/rules/

The content of these files consists of individual rules that describe attacks
or unusual IP packets. A Suricata rule is similar to the rule of a packet filter.
It has a fixed syntax against which all fields of each IP packet are checked.
action protocol source port -> destination port (options)

If the conditions from the protocol, source, destination, ports, and options
apply, the action is executed.

For example, the file dns-events.rules contains a rule with the content:
alert dns any any -> any any (msg:"SURICATA DNS Z flag set"; \
app-layer-event:dns.z_flag_set; \
classtype:protocol-command-decode; sid:2240006; rev:2;)

265
Chapter 18. Intrusion Detection

The specified rule checks whether a DNS packet has set the Z flag. RFC
1035 specifies that the Z flag is reserved for future use and must always be
zero. If the Z flag is not zero, the packet is suspicious.

If the condition of the rule applies, the selected action alert will report
an alarm and forward the packet. The Z flag was later used in the DNS
extension of RFC 2535 for authentication. DNS packets with this security
extension can only pass the intrusion system if the rule action is not set to
drop. The choice between alert and drop therefore depends on the DNS
implementation of the network that the IDS is supposed to protect.

Summary
A system for intrusion detection is only as good as the quality and accuracy
of its rules. Suricata does an excellent job as a free and open-source IDS. It
runs in the heart of the firewall, which is a perfect location for analyzing
traffic streams both as IDS and IPS.
The programming language of the rules for detecting suspicious packets
remains in the background because Suricata uses pre-built rulesets from
different providers. The Schedule function always keeps the rules up to
date.
It is mandatory to fine-tune every Intrusion Detection System. Only when
the false alarm rate reaches an acceptable minimum, does the IDS become
a valuable member of the security family and possibly get promoted to IPS.

266
Chapter 19

Command Line

For everyday configuration work, OPNsense presents itself in a modern

web interface. The Responsive Design makes it look similar in all web
browsers. The command line is only required for initial setup or when
critical configuration changes have failed.
OPNsense does not have a fully functional command line interface (CLI) as
offered by many commercial routers. The command line is limited to use
as a first aid kit.

This chapter shows the text console possibilities. While the functions
of OPNsense are menu-based, the underlying UNIX operating system ex-
pects written commands. For convenient access to the file system via a file
browser, see the section Access from Windows on page 316.

configd
The central point of contact for services is configd, which also runs as a
service. It monitors, starts, and stops other services when the configuration
has changed or when the web UI queries for statistics. It is best to use the
wrapper script configd instead of messing directly with a system service.

configd does not restrict access but implements new features. For example,
in addition to the usual tasks start, stop, and status, configd enhances the
proxy service to understand the tasks fetchacls and downloadacls.

267
Chapter 19. Command Line

The configd service is accessible from the command line using the com-
mand configctl. The desired service and its task are appended to the
command:
configctl service task

For example, restart the scheduler cron with the command

root@RT-1:˜ # configctl cron restart
OK

The fascination of configd is its extensibility. The service is just a file in

the directory
/usr/local/opnsense/service/conf/actions.d/

and the task is a section within that file. Following the example of cron,
the content of actions_cron.conf is the simple code used to restart the
cron daemon.
[restart]
command:/etc/rc.d/cron restart
parameters:
type:script
message:restarting cron

Which tasks are available by a specific service? configctl prints a list of

all services and tasks on demand:
root@RT-1:˜ # configctl configd actions
filter kill table [ ]
filter find_table_references [ ]
dns reload [ ]
dhcpd list leases [ ]

Behind the scenes, the process configd.py is instructed by its master,

configctl, and consults the corresponding description file of the service
for each action. The file extension of configd.py already reveals that the
daemon is coded in Python. Python is a scripting language preferred by
OPNsense developers and has almost replaced the antiquated PHP code
from the pfSense era.
Python scripts can be opened with a text editor (see Appendix A) and
understood without detailed knowledge of the language.

268
 Configuration changes

Configuration changes
The development of a full command line for configuration changes is not
on the roadmap of OPNsense. The strategy is heading towards a RESTful
API, which in turn could be used by the CLI.
For this reason, modifications on the command line are not a good idea
and should be limited to emergencies. OPNsense wants to be admired as a
beautiful website and not in a black-and-white console window.
If you are fearless (but with a backup copy), you are welcome to open
the main configuration file /conf/config.xml and modify the respective
settings. The file uses the eXtensible Markup Language (XML) and is there-
fore an open text format. Direct changes are possible, but require a basic
understanding of the format.

A less risky approach is through the protected environment of the text-

based XML browser xmllint. When xmllint is used for a configuration
change, it guarantees at least a syntactical correct file.
xmllint represents the structure of the XML file, similar to a file system with
directories. Each hierarchy level of XML corresponds to a subdirectory that
can be entered or left with the command cd.
An example makes it much easier to understand: imagine you want to
change the display language of the web UI. Fire up xmllint and use the cd
command to descend to the appropriate level.
root@RT-1:˜ # xmllint --shell /conf/config.xml
/ > cd opnsense
opnsense > cd system
system > cd language
language > set de_DE
language > save
language > quit

The developers use the usual combination of language code and country
code to define the language of the web GUI. Finally, the save command
writes the changes back to the XML file.
The configuration file config.xml is modified accordingly, but the OPN-
sense services do not know about this new feature yet. Also, the configd
or the XML file is not queried, because this is an unexpected type of system
change.

269
Chapter 19. Command Line

Two unattractive solutions force the services to take a look at the main
configuration file:

• Restart the services. This is quick and painless, but does not work
with the IPS and the web proxy. The command

/usr/local/etc/rc.reload_all

completes this task and triggers each service individually. Depending

on the performance of the firewall hardware, the interruption is less
than one minute. This method fails to work for changes to interfaces,
routing, and anything that is not a service.

• Restart the firewall. This task works 100% of the time but interrupts
all services for several minutes.

Both approaches clearly show that OPNsense is not yet ready for daily
configuration work from the command line.
In case of an unusual security policy or a stubborn admin, it is best to
implement a high availability setup (see Chapter 12) to minimize the
outage duration. Administrative overhead increases, but users do not
notice configuration changes and downtime.

Undo changes
Critical configuration changes from the web interface can have devastating
consequences. Not to mention that a restart of the firewall usually does
not solve the problem, since OPNsense has already stored the error in the
configuration file.
OPNsense has integrated the Undo function into its text console. When
management access is out-of-band, access via SSH should still be possible
so that the text menu is still accessible (see Chapter 9).
Menu item 13 Restore a backup is responsible for managing recent config-
uration states. From the list of configuration files, an older, but working,
configuration is quickly identified by its timestamp. When selected, the
entry becomes the current configuration, and OPNsense recommends a
restart to finish the restore.

270
 Updates

Enter an option: 13

1. Tue Feb 2 11:52:11 CET 2021

2. Tue Feb 2 11:49:00 CET 2021
3. Tue Feb 2 11:33:24 CET 2021
4. Tue Feb 2 11:32:17 CET 2021
5. Tue Feb 2 11:31:51 CET 2021

Select backup to restore or leave blank to exit: 3

cp /conf/backup/./config-1612262004.2927.xml /conf/config.xml
Do you want to reboot to apply the backup cleanly? [y/N]:

Depending on the urgency of the recovery, postpone the restart to a later

point in time, which is less annoying for the users.

Updates
Manually repeating similar tasks for a large number of OPNsense firewalls
is time-consuming. It is best to automate regular tasks, like updates. The
command line is much better suited for this than the web interface.
The Check for updates button runs a command in the background:
configctl firmware check
/usr/local/opnsense/scripts/firmware/check.sh

This script takes a look at the public package repository. As soon as a newer
version appears, the web UI offers an update to the administrator.
The OPNsense developers use the FreeBSD package system to update their
software. The update process is essentially based on the command
/usr/local/sbin/opnsense-update

If the process requires a restart, the update dialog will inform and perform
the reboot when the software update is complete. It is best practice to run
the update task outside working hours.
A fully automated update with an optional subsequent reboot uses the
command
/usr/local/etc/rc.firmware

or

271
Chapter 19. Command Line

configctl firmware auto-update

Using the FreeBSD package structure does not reflect a lack of imagination
but is instead an excellent integration with the underlying operating system.

Packages
All available plug-ins at System → Firmware → Plugins are also software
packages managed by the package system.
In a virtual environment based on VMware, the installation of VMware
tools is mandatory for best performance and drivers. The CLI command
pkg install os-vmware

fetches the necessary packages and their dependencies from the public
OPNsense repository. It then installs the VMware tools and reports its
success to the packaging system. The installed package is now usable and
visible from the website.
In environments with minimal resources, it is recommended to install the
plug-in via the command line. The pkg command displays interaction
possibilities and also discloses the required memory space.

Summary
The functionality of the OPNsense command line is limited to the initial
setup of the network adapters and various connectivity tests. The web
interface provides a much better way to change configuration and gather
information. Troubleshooting is mainly done using the commands of the
FreeBSD operating system.

272
Chapter 20

Performance Tuning

OPNsense is based on UNIX. As a universal operating system, UNIX runs on

almost every hardware and its basic idea is grounded in versatility rather
than quick data packet transportation. Compared to highly optimized sys-
tems that have only throughput in mind, UNIX offers weak performance.
Even though the local system runs on a powerful hardware, the UNIX net-
work stack is more all-purpose, handling any environment without focusing
and specializing in high-performance networks.
Nevertheless, UNIX cuts a fine figure on network devices. The UNIX ker-
nel and its application programs offer several buttons and selections to
improve the packet processing. Custom improvements in the TCP/IP stack
are mostly out of scope, but even small steps can increase efficiency.

How much performance and bandwidth can be expected from a UNIX-

based firewall? The following sections show commands to measure the
system and possibly increase it.

Lab setup
This chapter examines the OPNsense firewall RT-1 for performance in pro-
cessing packets with and without encryption. The measurement endpoints
are two adjacent firewalls: RT-2 sends packets with maximum bandwidth
through RT-1 to RT-3, which then receives, measures, and discards the data.
Benchmarking the crypto performance uses a VPN tunnel between RT-1

273
Chapter 20. Performance Tuning

and RT-3, where RT-3 is also the measurement host. Figure 20.1 shows the
experimental assembly.

RT-2
OPNsense
em1
10.1.1.2 Site-1 Site-2
10.1.1.0/24 10.2.1.0/24

10.1.1.1 10.2.1.3
RT-1
em1 OPNsense
RT-3 em1
OPNsense
em4 em2
192.0.2.1 WAN-2 192.0.2.3
192.0.2.0/24

Figure 20.1: Experimental assembly for measuring throughput

Baseline

Before the firewall modification begins, it is best to determine the current

performance data. While a neighboring device pushes packets through the
firewall at the maximum rate, various commands provide detailed insight
into resource utilization.
These monitoring tools belong to the usual UNIX commands used for
troubleshooting, and should not be absent in any distribution. If one of
the following commands is missing on the local firewall, install it with the
command pkg install.

top In addition to a current list of UNIX processes, top also provides

real-time utilization of the processor, memory, SWAP, and system load.

274
 Virtual network adapter

netstat Which TCP/UDP connections are established through OPNsense?

And on which ports do applications listen for requests? netstat provides
the answers and statistics about buffers, timers, and queues. netstat also
lists the routing table when called with the -r option.

ifconfig At a quick glance, ifconfig lists loads of information about the

network interface card: MAC, IPv4, and IPv6 address, flags, and the status
of the negotiated speed and duplex mode.

ifinfo The ifinfo command lists information about the network adapt-
ers that refer to the physical layer. Also, it presents the number of bytes,
multicasts, collisions, discarded, and transferred packets.

iftop What does the firewall currently process for its clients? iftop shows
this information via the interface and lists the session as well as the transfer
rate of the last 2, 10, and 40 seconds.

pfctl controls the packet filter and returns the contents of the firewall
status table, statistics, and timeouts. With the -s all option, pfctl reveals
everything it knows.

pftop is a real-time display for the packet filter session table. For each
connection, pftop shows the duration and amount of transferred data.

systat reveals the processes that are currently creating the highest CPU
loads. Alternatively, systat also searches for processes with many disk I/O,
network accesses, or large memory requirements.

Virtual network adapter

A virtual environment offers multiple network adapter models. With a few
simple steps, it is even possible to change the adapter of the guest system
later.
VirtualBox includes the adapter type selection in the GUI. The dropdown
box is filled with network cards from AMD and Intel, which are presented

275
Chapter 20. Performance Tuning

to the guest system as a hardware adapter. However, it also works with-

out emulation: the paravirtualized network adapter virtio-net provides
the guest system with a software-based NIC. The guest operating system
needs prior awareness to this concept and should already have the software
drivers. Since virtio is an open standard, the developers of FreeBSD (and
OPNsense) included the appropriate code into the kernel back in 2012.

200

150
Throughput (Mbit/s)

e1000
pcnet2
100
virtio

50

0
128 256 512 1024 1280 1420

Packet size (byte)

Figure 20.2: Throughput rates of various packet sizes (VirtualBox)

The VMware Workstation automatically selects the Intel e1000 adapter.

ESXi chooses from e1000 and a proprietary adapter, but hardware from
AMD is also available. To increase performance, VMware also uses the para-
virtualization approach for network adapters, but with its own software
under the term VMXNET . The driver for the VMXNET adapter is included
in the VMware tools that must be installed separately on the guest system.

With VMware, it is required to change the adapter type by modifying the

guest system’s configuration file. Open a text console on the VM host and
navigate to the file with the extension .vmx in each guest’s directory. This
text file contains the configuration of the guest system, including the speci-
fication of the network adapter, e.g. ethernet0.virtualDev = "e1000"

276
 Virtual network adapter

400 e1000e
vmxnet3
vlance

300
Throughput (Mbit/s)

200

100

0
128 256 512 1024 1280 1420

Packet size (byte)

Figure 20.3: Throughput rates of various packet sizes (ESXi 7)

Modifications to the vmx-file are only effective when the virtual machine is
turned off. Possible adapter types are vlance, vmxnet3, e1000, and e1000e.
Which adapter is the best one? Figures 20.2, 20.3 and 20.4 show the
throughput rates of various adapters with different packet sizes and hyper-
visors. The exact measured bandwidth is less important than the difference
in adapter’s performance.
These measurements show the throughput rates of a virtual OPNsense
firewall without the enabled features IPsec, NetFlow, or NAT. Only the
packet filter is active and allows all traffic with a single rule.
Different operating systems and different drivers may lead to different
results. Therefore, the statements in the diagrams only apply to OPN-
sense 21.1.r_44 on the used host systems.
The measurement of the different adapter types is biased and only considers
the pure throughput rate. Capabilities such as jumbo frames, TCP checksum
offload, TCP segmentation offload, and the CPU load caused during the
measurement are not taken into account.

277
Chapter 20. Performance Tuning

400

300
Throughput (Mbit/s)

200

100

vlance
vmxnet3
e1000
0
128 256 512 1024 1280 1420

Packet size (byte)

Figure 20.4: Throughput rates of various packet sizes (VMware Player)

Routing throughput
If the firewall hardware is expected to deliver high data rates, it is worth
taking a look at the assignment of the network adapters to processor cores.
By default, OPNsense leaves this decision to the UNIX kernel.
In a worst-case scenario or a weak setup, the most active network adapters
end up on the same processor and fight for CPU cycles, while other cores
are underutilized.
Under those circumstances, it is time for intervention, and FreeBSD will
provide commands that can send the interrupts of a network card to a
specific CPU kernel. Check the list of interrupts to determine if this uneven
load distribution exists on the local system.
root@RT-3:˜ # vmstat -i | egrep "em|total"
interrupt total rate
irq16: em2:irq0 55810 157
irq17: em3:irq0+ 108623 305
irq18: em0:irq0+ 180 1
irq19: em1:irq0 7 0

278
 Routing throughput

In this example, the network adapter em3 drags the most cycles from the
CPU. During this load situation the two CPU cores are indeed unevenly
loaded, as the command top -P CC confirms:
CPU 0: 0.0% user, 0.0% nice, 3.5% system, 6.2% interrupt, 90.3% idle
CPU 1: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle

The next step is to distribute the work of the network adapters among the
CPU cores. The workhorse em3 (IRQ 17) could be assigned on the first core
while the adapters em0 (IRQ 18) and em2 (IRQ 16) operate on the second
core. FreeBSD provides the cpuset command for this purpose, which will
display or set an assignment:
root@RT-3:˜ # cpuset -l 0 -x 17
root@RT-3:˜ # cpuset -l 1 -x 16
root@RT-3:˜ # cpuset -l 1 -x 18
root@RT-3:˜ # cpuset -g -x 16 ; cpuset -g -x 17 ; cpuset -g -x 18
irq 16 mask: 1
irq 17 mask: 0
irq 18 mask: 1

Note
The addressing of the CPUs, or cores, is based on a bitmask shown
with examples in Table 20.1.

CPU Bitmask Hexadecimal

CPU 0 0001 1
CPU 1 0010 2
CPU 2 0100 4
CPU 3 1000 8
CPUs 0 or 1 0011 3
CPUs 0 or 2 0101 5
CPUs 0 or 3 1001 9
CPUs 0, 1 or 2 0111 7
CPUs 2 or 3 1100 C
All CPUs 1111 F

Table 20.1: Binary and hexadecimal numbering of CPU cores

279
Chapter 20. Performance Tuning

With this configuration, the high-speed NIC gets its own CPU core, and
the total load is distributed better. Subsequent performance measurement
should either show more throughput or cause less CPU load on the firewall
hardware.
CPU 0: 0.0% user, 0.0% nice, 73.5% system, 0.0% interrupt, 26.5% idle
CPU 1: 0.0% user, 0.0% nice, 88.1% system, 11.9% interrupt, 0.0% idle

IPsec throughput
If high throughput rates are required for an IPsec tunnel, it is best to
investigate the hardware specs first. Unfortunately, the IPsec performance
is not a fixed value printed in a datasheet. The encryption performance
depends on several factors:

• The processor. The CPU might have hardware support for crypto-
graphic instructions. This command set extension accelerates encryp-
tion and decryption, but the VPN software must also use it to take
full advantage of it.

• The crypto processor, hardware acceleration, or IPsec Offload. A

dedicated chip for the encryption relieves the main CPU and provides
a constant throughput rate, even at a high load.

• The bus. How fast does the internal data bus bring the packet contents
from the network card to the CPU and back?

• The crypto library. How efficiently have the programmers worked

and implemented the algorithms in their source code?

• The algorithm and key length. AES is mathematically simpler than

3DES. As a result, encryption with AES is faster and more packets can
be sent within the same time interval. The result is higher throughput.
The same applies to the key length: the longer the key, the greater
the effort involved when encrypting the content.

The measurement results in Figure 20.5 show an example of the available

bandwidth when packets are encrypted. A virtual machine with a vmxnet3
adapter running on VMware ESXi performed the benchmark. The packet

280
 IPsec throughput

175

150
Throughput (Mbit/s)

125

100

75

50
no encryption
AES-128 SHA-1
AES-192 SHA-256
25 AES-256 SHA-512
3DES MD5
128 256 512 1024 1280 1420

Packet size (byte)

Figure 20.5: Throughput rate with various crypto algorithms

size and algorithm vary to illustrate the influence on the measured through-
put rate.

Finally, the results also depend on the duration of the test. A short ex-
amination of fewer than 10 seconds hardly generates any packet loss,
because the firewall does not have to discard the overload packets, but can
cache them in buffers. This caching approach is basically an advantageous
behavior, but it tampers with the result. A meaningful result is achieved
after 60 to 120 seconds.

Measuring method
Several RFCs and Internet drafts propose differing ways to measure the
throughput. To determine the IPsec throughput rate, use a neutral mea-
suring computer to send packets to the first IPsec gateway. This device
encrypts the packets and forwards them to the second gateway for decryp-

281
Chapter 20. Performance Tuning

tion. Finally, the packages reach their destination. Figure 20.1 shows the
unmodified experimental setup.
The framework conditions of a measurement are:
• Test duration of 60 seconds,
• IPsec in tunnel mode,
• Repeat each test three times to find abnormalities and determine an
average value,
• Modify the packet size in fixed steps, as RFC 2544 recommends.
OPNsense has an excellent tool to run the measurement: iperf3 [23]. iperf3
transmits packets with maximum throughput between two devices and then
displays the achieved transfer rate. On the first host, start iperf as a server
that receives the measurement packets. The second host starts iperf as a
client with the IP address of the specified server. The client immediately
starts generating packets and sends them to the server.
The iperf client learns about the different package sizes via script:
#!/bin/csh
set INTERFACE=vmx1
foreach MTU ( 1420 1280 1024 512 256 128 )
echo "*** Benchmark with MTU $MTU Bytes"
ifconfig $INTERFACE mtu $MTU
sleep 2
iperf3 --client 192.0.2.3 --window 128K --time 60 \
--interval 60 | grep bits
end

ifconfig $INTERFACE mtu 1500

/usr/local/etc/rc.newwanip # enable new MTU

The result is a reasonable estimate of the possible firewall throughput rate

with different packet sizes.

Warning
iperf is available in versions 2 and 3, which are incompatible. The
examples in this book use version 3 because it is easier to install on
OPNsense.

282
 Increasing performance

Note
Performance and throughput of virtual machines vary depending on
the load situation of the host system and other guests.

The measurement results are then less meaningful and can have a certain
deviation.

Increasing performance
The performance improvement success depends on the hardware equipment
and network components. The following recommendations may, but do not
promise, improvements.

AES-NI
Recent CPUs from Intel and AMD support the AES-NI command set ex-
tension to handle encryption better (and faster). The name suggests its
purpose: the instruction set deals only with the AES algorithm. To take
advantage of AES-NI, the processor, software, and configuration must use
the new commands.
However, does the processor understand AES-NI? To do so, the CPU info
must include the keyword aes.
root@RT-1:˜ # dmesg | grep aes
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard

The hardware support of AES-NI is only half the battle because the kernel
must also support the command extension. OPNsense has outsourced its
knowledge of the crypto language to a kernel module, which is easy to
verify:
root@RT-1:˜ # kldstat | grep aes
19 1 0xffffffff82a31000 8d50 aesni.ko

OPNsense offers two algorithms, 3DES and AES, with different key lengths
for VPN tunnels. Today, AES is the preferred crypto algorithm because:

• Most x86 processors offer the optimized command set for AES.

283
Chapter 20. Performance Tuning

• The AES processing is typically faster because 3DES encrypts three

times in a row.
• AES is more secure than 3DES. This strength is due to the stronger
key, the longer block length, the immunity to the known attacks on
the algorithm, and the significantly longer period it takes for a brute
force attack to succeed.

3DES was developed for hardware use. When comparing the crypto perfor-
mance of a router with additional hardware acceleration, AES might lose
to 3DES regarding throughput.

IPsec

An IPsec VPN is negotiated through the charon process, and during opera-
tion is maintained by the kernel. The software code for AES encryption is
located in the kernel module aesni and offers all available key lengths.
OPNsense uses strongSwan [10] version 5.9.1 as an IPsec engine, which has
used AES-NI since mid-2015. The kernel module accelerates the encryption
of the VPN tunnels. StrongSwan does not use an additional SSL library
so AES-NI is used automatically if the VPN tunnel uses AES for encryption
instead of 3DES.

The only way to improve the IPsec VPN performance is to use the AES
encryption algorithm.

OpenVPN

The OpenVPN software developers use a library for all crypto operations.
OPNsense picks the two best libraries on the market and lets the administra-
tor choose which library to use. By default, the programs utilize OpenSSL,
but LibreSSL is an appealing alternative.
OPNsense automatically uses AES-NI, even if it is not selected in the web
GUI. And that is a good thing, because the VPN tunnel’s throughput is
30-50% more when AES-NI is activated. The pure crypto performance of
the CPU is many times higher; however, the path of the packet through
the firewall is not just encryption. The firewall also spends some time on
packet filtering, routing, address translation, and queueing of the packets.

284
 Increasing performance

100
Throughput (Mbit/s)

75

50

AES-128 SHA-1
25 Blowfish SHA-1
Camellia-256 SHA-1
AES-256 SHA-256
DES MD5
3DES MD5
128 256 512 1024 1280 1420

Packet size (byte)

Figure 20.6: OpenVPN throughput with OpenSSL and various ciphers

The results in Figure 20.6 show the throughput of an OpenVPN tunnel with
different crypto algorithms compiled against OpenSSL. The diverging rates
show the advantages of AES.
If desired, change the crypto library to LibreSSL from the web UI at Sys-
tem → Firmware → Settings. The next step is a firmware update because
OPNsense says goodbye to OpenSSL and installs the predefined software
packages with LibreSSL support.
Why make the effort to change the crypto library? OpenSSL and LibreSSL
both have their strengths and weaknesses. OpenSSL has been around for
almost 20 years and has had a few security incidents.
After the last major security breach in 2014, the OpenBSD developers
grabbed the source code from OpenSSL, removed a lot of useless lines,
and released LibreSSL. Since then, LibreSSL has scored when it comes to
security – so far no critical vulnerability has been noticed.

285
 Chapter 20. Performance Tuning

LibreSSL’s VPN tunnel throughput rate is lower than OpenSSL because

LibreSSL’s library needs a few more CPU cycles for its operation. The
difference is less than 10%, as Figure 20.7 shows. This chapter deals
with increasing throughput, and as such, OpenSSL is the best choice for
performance improvement.

125

100
Throughput (Mbit/s)

75

50

25 no tunnel
IPsec
OpenVPN with OpenSSL
OpenVPN with LibreSSL
128 256 512 1024 1280 1420

Packet size (byte)

Figure 20.7: Throughput of OpenVPN with OpenSSL and

LibreSSL library

Multiple CPU cores

A multi-core processor can perform parallel tasks, making them faster to
complete. However, parallel processing may not help when encrypting and
decrypting packets. The higher layer protocols expect the packets in the
correct order. If crypto work is distributed across multiple CPU cores, the
decrypted packets must first be sorted correctly, which takes time. When
encrypting and sending packets, the recipient must check the sequence
and sort them before the application receives the data. Depending on the
“disorder” of the packets received, sorting can take more time and reduce
the overall throughput.

286
 Increasing performance

In a virtual environment, it is best to measure the throughput rate of the

VPN gateway with a different number of CPU cores.

MTU and MSS

The transporting network specifies a maximum size for a single data packet.
The Ethernet uses a maximum length of 1500 byte. This limit is the Maxi-
mum Transmission Unit (MTU). The TCP protocol has a similar limitation,
which is called Maximum Segment Size (MSS) and determines the maxi-
mum number of byte that fit into a TCP segment. The principle of both
sizes is the same.
If Ethernet switches connect both client and server, the MTU is not an issue.
The challenge comes with topologies whose protocol adds some bytes to
the packet header. That way, the packet grows larger. Since the MTU of the
packet cannot grow, the amount of data must shrink.
A VPN tunnel has multiple headers (see Chapter 10), which all want a cut
from the header budget. Unfortunately, neither the client nor the server are
aware of the reduced amount of user data and continue sending with their
known MTU or MSS.
The performance problem starts at the VPN gateway, which receives full
1500-byte packets from the client. The gateway adds additional VPN head-
ers and forwards the packets on to the Internet. If the packets already
have the maximum size, the firewall has to split the oversized packet into
two packets and send them. The fragmented packets reach the destination
gateway, which decrypts them and delivers them to the server.
With full packets, both firewalls must encrypt and decrypt twice as many
packets. The throughput rate of the VPN connection degrades when the
packet size is larger than the MSS of the tunnel.
It is best if the client and server only send packets that do not have to be
fragmented by any firewall or router. However, the computers on the LAN
must know the tunnel’s MSS.
There are two methods to determine the MSS: by calculation or by trial and
error. In the calculation, the size of all headers involved is added up and
subtracted from the MTU. The result is the MSS.
Indeed, some web-based helpers graphically reproduce a tunnel packet and
display the calculated MSS. Check out [24] or search using the keywords
ipsec overhead calculator.

287
Chapter 20. Performance Tuning

The less academic way is to guess and try the right MSS. For this, ping is a
good partner: the packet size is slowly increased with a sweeping ping until
it reaches maximum size. As soon as the packets have to be fragmented,
the MTU is found.
root@RT-2:˜ # ping -D -g 1280 -G 1500 -i 0.2 10.2.1.3
PING 10.2.1.3 (10.2.1.3): (1280 ... 1500) data bytes
1288 bytes from 10.2.1.3: icmp_seq=0 ttl=64 time=2.578 ms
1289 bytes from 10.2.1.3: icmp_seq=1 ttl=64 time=0.383 ms
[...]
1419 bytes from 10.2.1.3: icmp_seq=131 ttl=64 time=0.199 ms
1420 bytes from 10.2.1.3: icmp_seq=132 ttl=64 time=0.638 ms
36 bytes from localhost: frag needed and DF set ( MTU 1440 )
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 05a1 0000 0 0000 40 01 ffbc 192.0.2.1 10.2.1.3

If both methods are too complex, or do not provide meaningful values, there
is still the tried-and-true value of 1300 byte. This MSS takes into account
the different tunnel modes, encryption types, and NAT traversal. However, a
few bytes remain unused in each packet, which slightly reduces throughput.

Figure 20.8 shows how the firewalls apply the determined MSS to the
communication path. When establishing a TCP connection, both partici-
pants announce their preferred MSS (steps 1 and 3), which will be used for
the duration of the connection.

RT-1 RT-3
10.1.1.1 VPN/WAN 10.2.1.3
em1 MTU 1361 em1
MTU 1500 MTU 1500

MSS=1460 MSS=1321
1 2
MSS=1321
4 MSS=1460
3
Figure 20.8: The OPNsense firewall manipulates the MSS values

OPNsense must now interfere with the TCP negotiation and overwrite the
MSS values (steps 2 and 4). From the point of view of the participants, the
respective partner proposed the lower MSS. In fact, the OPNsense gateway
has automatically changed the MSS to prevent fragmentation.

288
 Increasing performance

The firewalls have to manipulate the MSS at both ends of the VPN tunnel.
OPNsense does this by using Normalization rules, which is a bit outside
the web GUI at Firewall → Settings → Normalization. The normalization
rules are similar to the regular firewall rules, i.e. they do not allow or deny
data connections, but manipulate with packet properties. The intentional
manipulation is the MSS reduction. The required rule has the following
parameters:
• Interface: LAN
• Direction: Any
• Protocol: TCP
• Source: 10.1.1.0/24
• Destination: 10.2.1.0/24
• Max mss: 1321
This rule is required twice in total, whereby the second rule has swapped
the Source and Destination values. The rules ensure that both sides of the
connection use the lower MSS.
Figure 20.9 shows both rules with inverted IP networks for source and
destination.

Figure 20.9: OPNsense uses normalization rules to manipulate the MSS

The normalization rules reduce the MSS for all types of VPN tunnels. Also,
OpenVPN has the built-in parameter mssfix, which influences the MSS
without a normalization rule. The option is not visible in the web UI, but in
the Advanced field at VPN → OpenVPN → Servers. This location accepts
non-default configuration, so add the line

289
 Chapter 20. Performance Tuning

mssfix 1321

to the Advanced options and OpenVPN will reduce the MSS to 1321 byte.

Populate ARP cache

If a router does not know the MAC address of the next router or the target
host, it queries that host using the Address Resolution Protocol (ARP).
While the routers wait for a reply, the affected clients have to pause a few
seconds before communication starts.
The clients’ connection setup is somewhat faster if the ARP cache of the
routers already contains the MAC addresses of all neighbors.
The script in Listing 20.1 determines the next-hop routers from the routing
table and checks if there is an ARP entry for each neighbor. If this entry is
missing, ping triggers an ARP request whose response keeps the ARP cache
up-to-date.

1 #!/bin/sh
2
3 NEIGHBORS4=$(/usr/bin/netstat -4rn | grep UGS | awk ’{ print $2 }’)
4 for NHR in $NEIGHBORS4 ; do
5 arp -n $NHR >/dev/null || ping -t1 -c1 -W1 $NHR >/dev/null
6 done
7
8 NEIGHBORS6=$(/usr/bin/netstat -6rn | grep UGS | awk ’{ print $2 }’)
9 for NHR in $NEIGHBORS6 ; do
10 ndp $NHR >/dev/null || ping6 -X1 -c1 -x1 $NHR >/dev/null
11 done

Listing 20.1: Refresh the neighbor caches by script

The script only sends packets if the ARP cache lacks the neighbor record.
When called repeatedly, the script will not generate a high network load.
Starting in line 8, the script also populates the IPv6 neighbor list, which is
basically the same as the ARP cache, but for IPv6.
Copy the script from Listing 20.1 to the directory /usr/local/sbin/. Since
the OPNsense developers use the same directory for many other scripts, it
is an excellent choice for the new script.

290
 Summary

Next, use the task scheduler to execute the script with the exemplary name
fill_arp_cache.sh at regular intervals (see Chapter 22).
( crontab -l ; \
echo "*/5 * * * * /usr/local/sbin/fill_arp_cache.sh" ) \
| crontab -

Summary
An increase in performance is feasible if the settings and configuration
are inappropriately applied. The increase begins by measuring the current
performance. Next, the virtual network adapters show their performance
values. If the CPU has more than a single core, the scheduler can share
the load over multiple cores. Encryption is subject to further obstacles
because IPsec and OpenVPN depend on very different methods to increase
performance. And if the maximum amount of data per packet is exceeded,
there is an unexpected loss of performance.
If the firewall is already working with applied optimal settings, the next
step for more throughput is to use more powerful hardware.

291
Chapter 20. Performance Tuning

292
 Part V

For Admins

293
Chapter 21

Best Practice

Once everything is running smoothly, it is now time to improve details

and simplify workflows. The following Best Practices apply equally to
other manufacturers’ firewalls: however, the practical implementation is
different.

Factory reset
Any professional network device has the ability to discard all changes and
return to the default state. These factory settings are required when the
firewall is sold, or the demo equipment must be returned to the manufac-
turer. In the simplest case, the device only changes its intended use and
should not have any existing configuration. Basically, when a firewall is set
to Factory Default, it will delete all traces.
There are three ways to bring an OPNsense firewall to its factory status:

• Use the web UI and go to System → Configuration → Defaults

• Use the console menu and choose item 4 Reset to factory defaults
• Use the text console or dial-in via SSH. Then, run the delete com-
mand:
/usr/local/opnsense/scripts/shell/defaults.php

The firewall then shuts down and powers off. When booted again, the
device assigns the initial IPv4 address 192.168.1.1 to the LAN network

295
Chapter 21. Best Practice

adapter. The WAN adapter queries for its address via DHCP. The password
for the admin account is opnsense.

This method is sufficient if the firewall has a new purpose.

Thorough
If the firewall changes hands, it is mandatory to really remove all traces. It
is best to double check the factory settings and inspect the remaining file
system.
The safest and most thorough way to remove all traces is to reinstall the
operating system. This procedure is advisable if the firewall was enhanced
with custom modifications (see Chapter 15, 19, and 23). Also, not every
service or plug-in sticks to the folder structure and stores configuration data
in non-default paths. Finally, perform a new installation of OPNsense to
have a clear conscience that no sensitive data leaked.

Benchmark throughput
The available throughput between two firewalls is, in theory, the bandwidth
of the network interfaces. In reality, there are usually limiting factors that
reduce the performance.
It is possible to measure the available bandwidth between two OPNsense
firewalls quickly with iperf3 tool. Install iperf3 from the web UI as a plug-in
or via the command line. The tool is easy to use, delivers meaningful results,
and runs on Linux and Windows, with a Java-based GUI.
With iperf3 the sender sends data at maximum speed to the receiver, and
both determine the throughput.

One OPNsense firewall is the receiver, and the other is the sender. These
roles determine the direction of the measurement because the calculated
bandwidth applies from the transmitter to receiver. For an analysis in the
opposite direction, repeat the benchmark, but with interchanged roles.
Keep in mind that a firewall rule must allow the measurement packets from
iperf3 to port 5201. For this purpose, a floating firewall rule is a good
choice because this type of rule applies to multiple network adapters. Con-

296
 Benchmark throughput

figure that rule at the web UI at Firewall → Rules → Floating. Table 21.1
shows the details of the new rule.
Attribute Value
Action Pass
Interface LAN,DMZ,MGMT
TCP/IP Version IPv4+IPv6
Protocol TCP/UDP
Source any
Destination This Firewall
Destination port range 5201
Description iperf3

Table 21.1: A firewall rule allows measurement packets from iperf

The receiver starts in server mode with iperf3 -s and the sender executes:
iperf3 -c <IP_of_the_remote_station>

If an additional firewall is located in the path between the measuring hosts,

allow port 5201 on that firewall, too. iperf3 is capable of sending over both
IPv4 and IPv6 connections.

root@RT-2:˜ # iperf3 --server --interval 10

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.4.1.1, port 54922
[ 5] local 10.4.1.2 port 5201 connected to 10.4.1.1 port 23767
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 441 MBytes 370 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 441 MBytes 370 Mbits/sec

Listing 21.1: iperf3 server on RT-2

Listings 21.1 and 21.2 show an example in which RT-1 sends data to RT-2
and both display the calculated bandwidth. The measurement duration of
10 seconds is the default setting. Pick a higher value with the command
line argument --time. A comprehensive list of all options is available with
iperf3 --help.

297
Chapter 21. Best Practice

root@RT-1:˜ # iperf3 --interval 10 --client 10.4.1.2

Connecting to host 10.4.1.2, port 5201
[ 5] local 10.4.1.1 port 23767 connected to 10.4.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-10.00 sec 443 MBytes 372 Mbits/sec 0 1.16 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 443 MBytes 372 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 441 MBytes 370 Mbits/sec receiver

Listing 21.2: iperf3 client on RT-1

SSH login without password

OPNsense requires a username and password for SSH login. A good pass-
word consists of many letters, numbers, and special characters. However, it
is cumbersome to enter it repeatedly.
Behind the SSH login lies the versatile OpenSSH server, which not only
offers password authentication but an option to extend or replace the pass-
word by cryptographic keys.
If the security policy of the environment allows, the administrator can au-
thenticate themselves with their private key against the OPNsense firewall
(Figure 21.1). The firewall validates the offered key and starts a login shell,
which presents the CLI to the admin – without a password. And as long as
the admin does not lose their private key, this login method is even more
secure than the usual password.

SSH authentication
with

uses validates
Private key Public key
Figure 21.1: SSH login with cryptographic keys

298
 SSH login without password

To use this login method, first generate and distribute the key material:
1. Generate a key pair. Each administrator needs a separate pair of keys.

2. Store the private key in a secure place.

3. Enter the public key on all OPNsense firewalls that the admin is
expected to manage.
A Linux distribution often comes preinstalled with OpenSSH as an SSH
client. On Windows, the software PuTTY [25] has gained wide acceptance,
probably because it is versatile and easy to use. Furthermore, PuTTY does
not have to be installed and is free of charge.

Generate key pair

A Windows computer needs the graphical tool puttygen.exe to create a key
pair. The following examples are based on a 2048-bit RSA key. From today’s
perspective, this is a sufficiently strong key. For high-security requirements
the key can also be 4096 bit long.

Note
OPNsense version 17.7.7 and above does not accept SSH keys with a
length of fewer than 1024 bit.

The Generate button starts collecting random numbers and then displays
the generated public key. In the next steps, both parts of the key are
required so they must be stored with the save buttons. The file names do
not matter. With this in mind, an excellent choice to name the key files is
id_rsa.pub for the public key and id_rsa for the private key.
Under Linux, the tool for creating keys is pre-installed with the OpenSSH
package. The following command generates a key of the same type:
ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa -N ’’

The resulting files are stored in the .ssh/ folder of the home directory.
The private part of the key must be secret, hidden, secure, and should be
password-protected. If the private key is compromised, the security is gone!
When that happens, it is mandatory to quickly remove the corresponding
public keys from all firewalls to prevent misuse.

299
Chapter 21. Best Practice

Display public key

For checking purposes, take a quick look into the public key file. The key
shows up quite ordinary as a long word of ASCII characters. There is also
an identifier highlighted in grey in the following example. Under Windows,
the public key that has just been created is:
---- BEGIN SSH2 PUBLIC KEY ----
Comment: " rsa-key-20201015 "
AAAAB3NzaC1yc2EAAAABJQAAAQEAkYsyCWjp/SUbLvUx0JllPbF5eLL7Aw66hQHe
UjUKwF1haD4o0616PUr7bNEqwoFuajXIyk+WqKv6Pfaot5tKHMlvLrOyf9iwjXQr
cZ9IFoeFFlVdETxI6MwWcSO7yswuo9X2h6yHKRGq+okmyL8Y1XzyRnhiBunG8yH1
YYKs2xG4JlLS6z8LhhNUU7mwFXkGpfnpldVxn/STdj+qFJq3uGHodmC0NLsHeReq
udH/Tjpr6lQpXt2jFODk1AewFmtq9Zbl9myu2Pewp1vwCtvcuywgCdQFPjc5Esv/
fgkBV8JefQHo3iiJUPwl5NcnW4MmIWC/DdZ74zzWcG+fH8zEEQ==
---- END SSH2 PUBLIC KEY ----

Under Linux, the key is displayed in a different syntax:

ssh-rsa AAAAB3NzaC1yc2EAAQABAAABAQCxEQhUQB3EfYWLH[...] root@labsrv

Link public key with a firewall

Now the long text string of the public key must find its way into the web
GUI of the firewall. That way, the key is linked to a user account so that
a passwordless login is possible. The web interface manages the assigned
keys at System → Access → Users. Every user can view, delete, or add their
keys under Authorized Keys. The format matches the syntax under Linux:
ssh-rsa public_key_as_single_line root@labsrv

Note
Each user can have multiple keys, which helps when exchanging keys
or when multiple admins share an account.

Warning
The public key is a one-line string. Even if puttygen stores the key in
multiple lines, OPNsense only accepts the key without line breaks.

300
 Password reset

In the background, the web GUI stores the key string in the local text file
authorized_keys in the subfolder .ssh in the user’s home directory.

Login with the private key

The SSH client in Linux automatically uses the new key file from the
subdirectory .ssh/, so that a login to the firewall should already be pass-
wordless:

ssh root@10.5.1.1

PuTTY’s path to the private key file is at Connection → SSH → Auth. Then
the SSH login uses the crypto keys instead of the password.

Password reset
At some point in time there will be a situation where access to the firewall
fails due to an incorrect password or key. If it is not possible to log in to
the web GUI and SSH console, OPNsense provides a procedure to reset the
password.

Note
This method overwrites the lost password. It is not displayed later or
stored in plain text.

This procedure requires access to the text console. Choose menu item 3)
Reset the root password to set the password of root to a known value.

There is a nasty option in the web UI at System → Settings → Admin-

istration, which is called Password protect the console menu. If this option
is set, it is still possible to reset the password. For this purpose, OPNsense
uses methods of the operating system to grant its administrator root access
again.

1. Reboot. Restarting the firewall is required to boot the system in Single

User mode.

301
Chapter 21. Best Practice

Figure 21.2: The OPNsense boot menu

2. Wait for the boot menu (see Figure 21.2) and choose item 2. Boot
Single User to skip the password authentication.

3. The boot process is complete when the text console displays:

Enter full pathname of shell or RETURN for /bin/sh:

Hit the Enter key to confirm a default shell.

4. OPNsense and FreeBSD, respectively, store the passwords in a file

on the hard disk. Mount this disk in editing mode with the mount
command.

mount -o rw /

5. The OPNsense password program is then available and is started

with:

/usr/local/opnsense/scripts/shell/password.php

The following questions refer to a new password. The well-known

passwd command does not work here.

302
 Password reset

Figure 21.3: Password reset via command line in OPNsense

6. Reboot. The final restart brings the system back to normal mode. A
login with the new password should be possible via the text console,
SSH console, or web interface.

Figure 21.3 shows the commands within the text console.

303
Chapter 21. Best Practice

304
Chapter 22

Configuration

The unique thing about an OPNsense firewall is the content of the configu-
ration file. Everything else comes from the installation medium. For this
reason, it is best to keep a copy of the configuration file in a safe place. This
chapter enhances a configuration backup with remote file storage.
OPNsense saves the configuration to the local hard disk after each change
and remembers the last sixty changes by default.

In this chapter, OPNsense saves its configuration data to Dropbox and

Google Drive. That way, the configuration of all firewalls is stored in a
central location without the need for a separate server. The connection to
the provider’s cloud server is secure. However, it is not 100% clear where
and how the providers store the data.

Warning
The following steps make changes to the operating system and may
affect the stability of the OPNsense firewall.

Dropbox
Dropbox is the top dog for inexpensive storage space on the Internet. A
few gigabyte are free, and Dropbox accepts all kinds of files – including
configuration files.

305
 Chapter 22. Configuration

Dropbox has a comfortable web page for file management. However,

OPNsense cannot rely on the ease of use that has made the provider famous:
the file transfer runs on the command line, and it needs a Dropbox client.
OPNsense does not provide any compilers, and the choice of interpreters is
minimal. A Dropbox client, which only uses curl and the Bash commands,
is available at GitHub [26]. The installation runs from the command line,
and starts in the text console with menu entry 8) Shell.
1 pkg install bash
2 curl --user-agent opnsense --output /usr/local/bin/dropbox \
3 https://raw.githubusercontent.com/andreafabrizi/ \
4 Dropbox-Uploader/master/dropbox_uploader.sh
5 chmod a+x /usr/local/bin/dropbox

Unlike a Linux system, OPNsense lacks the Bash by default. A precompiled

package is available in the OPNsense repository and gets installed in the
local filesystem in line 1.
GitHub expects the git command to access project data, but curl does an
excellent job downloading the files. The GitHub project is full of files, but
only the Bash script is of interest. Line 2 fetches the script and places it
in a UNIX-typical path. The Dropbox client is now ready to start with the
command name dropbox.

At the start, dropbox welcomes you with a lot of information on how

to activate the API in your personal Dropbox account and how to gain
access. These steps are necessary for the dropbox command to securely
access the correct cloud storage.
Follow the steps as instructed by the command output. As soon as the
dropbox script has gained the app key, app secret, and access code, it is
possible to access Dropbox storage. Perform a test to validate that the
command sends the local configuration to Dropbox.
dropbox upload /conf/config.xml ‘hostname -s‘.config.xml
> Uploading "/conf/config.xml" to "/RT-1.config.xml"... DONE

The command exits successfully, but only the web interface of Dropbox can
validate an uploaded file, as Figure 22.1 confirms.

306
 Dropbox

Figure 22.1: Dropbox stores the configuration file from OPNsense’s firewalls

Note
If the upload was successful but Dropbox does not display the new file,
then the uploaded file does not contain any changes.

Automatic backup
The new backup method is a bit cumbersome because OPNsense only saves
to the local hard drive by default and ignores Dropbox.
OPNsense needs the help of its scheduler cron to save a copy of the con-
figuration file into Dropbox automatically. The scheduler can execute
commands regularly at fixed times and run the backup to the Dropbox.
The following command causes a daily upload to keep the dropbox archive
synchronized. If an error occurs, the process reports the failure to the logfile
(line 3).
1 cat <<EOF >> /etc/periodic/daily/900.backup-config
2 dropbox -q upload /conf/config.xml ‘hostname -s‘.config.xml || \
3 logger -t dropbox -p local0.error \
4 "Failed to upload the configuration file to Dropbox"
5 EOF
6 configctl cron restart

307
 Chapter 22. Configuration

Note
Upload the configuration file as often as required, but specify the
interval and commands for the upload process in /etc/crontab. The
man page of cron provides information and examples about the syntax:
man 5 crontab

With this extension, OPNsense regularly sends a copy of the configuration to

Dropbox. The configuration file storage requirement is only a few kilobytes
so that many thousands of configurations fit into the free basic account.
Also, Dropbox does versioning of the files (Figure 22.2), providing access
to previous configuration states.

Figure 22.2: Dropbox keeps older versions of configuration files

The configuration might contain sensitive information, so it is best to

encrypt the data before uploading it. That way, the contents of the configu-
ration are only accessible to third parties with password access.
When it comes to a firewall’s hardware failure, OPNsense uses the stored
XML file for system recovery. The recovery process will only accept an
encrypted file if it uses the same method for encryption as the built-in
configuration backup:
1 /usr/bin/openssl enc -e -aes-256-cbc -md md5 -in /conf/config.xml \
2 -pass pass:"password" | b64encode -m -r -o /tmp/config.enc .
3 /usr/local/bin/dropbox upload /tmp/config.enc $HOST.config.enc

308
 Google Drive

OPNsense calls openssl in the function encrypt() of its OPNsense\Backup

library to do the encryption stuff. Unfortunately, the script requires the
password in plain text (line 2) for encryption.

Google Drive
The configuration backup with Google Drive differs from Dropbox in that
OPNsense has integrated Configuration Cloud Backup in its web GUI since
March 2015. The setup on the client side does not require any unofficial
modifications to the operating system.
OPNsense uses the Google API to communicate with Google Drive. The
result is the same as with Dropbox: the configuration file is stored – after
some setup effort – in the cloud.

Access the API

Google has secured its Application Programming Interface (API): the parti-
cipants’ communication takes place over a secure HTTPS connection, and
for authentication Google recommends certificates. Not surprisingly, use a
Google account to access the API. It is required to activate the API before it
is used for the first time. Access the URL in a regular web browser:
https://console.developers.google.com/apis/api/drive.googleapis.com/

Hit the button Create Project to start a new project. The project name is
insignificant, but it does not hurt if it describes the subject, such as “OPN-
sense config backup.”
Next, go to the APIs & Services section of the Dashboard and click the
button Enable APIs and Services. After that, Google wants to know which
API because there is a long list of applications. Choose the Google Drive
API, which is enabled a few seconds later.

Navigate to APIs & Services → Credentials. Hit the button Create cre-
dentials to start the authorization procedure. Choose Service account from
the provided list (Figure 22.3). Pick something from “Service Accounts” for
the access role. Next, create a key for this new service account. OPNsense
requires the key type P12.

309
Chapter 22. Configuration

Figure 22.3: Login credentials for the Google API

Now Google creates a P12 certificate and offers it for download. This file is
imported into OPNsense later.
Also, take note of the service account ID. The ID looks like an email address
and matches the private key of the P12 file. For example:
rt-989@opnsense-config-backup-187541.iam.gserviceaccount.com

Both the service account ID and certificate file are required to access the
Google Drive API.

Set up Drive
Google Drive has an easy-to-remember URL https://drive.google.com.
Access that web page with a registered Google account. To keep Drive
organized, create a new folder for the configuration files. Push the New
button in the upper left corner to create this folder and choose a descriptive
name like “RT-1”. Next, share the newly created folder by clicking on the
folder icon and choose Share from the context menu.
Who may access this folder? Here is where the service account from the
previous section comes into play. Insert the service account ID into the field
people and groups. The last piece of the puzzle is the folder ID that Google
hides behind the Advanced button. Locate the Get link button and extract
the folder identifier. The following link shows an example URL for folder
sharing with the folder ID highlighted.

310
 Google Drive

https://drive.google.com/drive/ \
folders/ 0B65iPldCRuWEeF0hc1KpapnqkRK ?usp=sharing

The Done button completes the setup and sharing of Drive.

Upload
Unlike Dropbox, Google Drive does not automatically collect a version
history of files with the same name. After ten uploads from ten firewalls,
there are already one hundred files with similar names in the drive, which
only differ in timestamps. Now it’s OPNsense’s turn to connect its web UI
at System → Configuration → Backups to Google Drive.

Table 22.1 summarizes the sample values from the previous sections. Fur-
thermore, OPNsense needs a password to encrypt the configuration file so
that no plain text files reside in Drive.

Attribute Example
Enable 2

Email Address rt1-669@opnsense-config-backup-178 \
541.iam.gserviceaccount.com
P12 key Certificates file from section
Access the API (page 309)
Folder ID 19ZO8tCjGpFs-jS0WP_xnem1Zl3_R3PFL
Prefix hostname to
backupfile 2

Backup Count 10

Table 22.1: OPNsense gets access to Google Drive

Finally, it gets exciting, because the button Setup/Test Google Drive val-
idates the access to the Google API and uploads the first configuration
directly into Drive.

If successful, Drive contains the encrypted configuration file, of which

Google does not understand anything, as Figure 22.4 on the next page
demonstrates.

311
Chapter 22. Configuration

Figure 22.4: Google Drive only gets the encrypted firewall configuration

Automatic backup

A single backup to Drive is good, but a regular backup is better. And this is
where OPNsense automatically loads the configuration to Drive once a day.
Manual intervention is not necessary. After several days the drive fills up
and may resemble the folder in Figure 22.5.

Figure 22.5: Google Drive fills up with firewall configuration files

312
 Summary

Summary
OPNsense can upload its configuration to remote servers and cloud pro-
viders. This form of outsourcing stores the configuration file in an off-site
location so that the OPNsense firewall can be rebuilt in the event of a
disaster.
The default cloud provider is Google Drive, but it is also possible to use
other providers with an open API. This chapter presented the integration of
Dropbox. OPNsense can even talk to the private cloud provider Nextcloud
or run a simple backup via SCP.

313
Chapter 22. Configuration

314
Chapter 23

Life Hacks

What is the best way to work with OPNsense? As an open-source product,

it is possible to customize the firewall software, adjust unusual commands,
or add new features. This chapter describes methods to effectively handle
and troubleshoot OPNsense.

Warning
The sections in this chapter expect to modify files, specifically those
not intended to be modified during normal firewall operation. A copy
of the data should be made before the first edit so that the original
state can be quickly restored in case of problems.

Those familiar with Linux or UNIX can edit the files directly on the com-
mand line using available editors vim, nano, joe, emacs or Easy Editor.
Find a quickstart guide for working with UNIX files in Appendix A.
There is good news: Windows users will not have to mess with the UNIX
command line. There are some excellent graphical file explorers for Win-
dows. A well-known representative is introduced in the following section
Access from Windows.

The described modifications to the local files refer to OPNsense version

21.1.

315
Chapter 23. Life Hacks

Access from Windows

Windows supporters do not have to get their fingers dirty accessing the
UNIX directory structure. Under Windows, a graphical SFTP client presents
the directory layout in an Explorer-like view. The well-known representative
WinSCP [27] connects to the OPNsense firewall after entering the hostname
or IP address, username, and password. It then starts browsing in the user’s
home directory. That way, it is easy to navigate to a specific directory to
view or edit files.

Span port
Copying frames from one port to another has many names: Port mirroring,
Port monitoring, Switched Port Analyzer (SPAN), or Span port. It all means
the same, a Span port copies all incoming and outgoing network packets
from one interface to another interface. The traffic is not modified or
blocked in any way.
The target port receives a copy of every frame. It is typically connected to
some kind of network analyzer or Intrusion Detection system.

OPNsense requires a network bridge to mirror network traffic. First, pick

an unused network adapter to receive copies of the packets. The firewall
RT-1 uses the new adapter em5 for this. It gets the name MIRROR under
Interfaces → Assignments and works without an IP address.

In the following example, all packets between RT-1 and the DMZ should be
sent as a copy to interface MIRROR.
Next, set up a new network bridge in OPNsense at Interfaces → Other
Types → Bridge. The new bridge has only a single member interface –
the network adapter to be monitored (e.g. DMZ ). The mirror options are
hidden in the advanced options. Find the option Span port and choose
DMZ from the list, as shown in Figure 23.1.
Save the configuration by hitting the button Save at the bottom. OPNsense
will create the bridge and begin mirroring traffic. The network bridge
monitors traffic on its member interface and sends a copy of each packet
to the selected Span port. It is even possible to mirror multiple source

316
 Telegram

Figure 23.1: OPNsense requires a network bridge to mirror traffic

adapters to a single span port – just add more member interfaces to the
bridge. Keep in mind that the span port must not be a member of the bridge!

Nothing terrible happens if the management interface is chosen as a span

port by mistake because OPNsense sends the packet copies in addition to
the average network traffic. The SSH connection will not get disconnected.
The Switched Port Analyzer on Cisco routers and switches is not so benign
in this situation.

Telegram
Critical messages must trigger an alert on the administrator’s smartphone –
in real time.
So far, all log entries remain in the file system of the firewall or at the
central log host. If this logging server does not have a method for sending
critical messages, OPNsense’s syslog service can perform this task.
Strictly speaking, Syslog passes the copy of a critical message to a script for
review. If Syslog classifies the message as reportable, it will send an alarm
message to a smartphone.
Syslog-NG expects the configuration file telegram.conf from Listing 23.1
in the directory /usr/local/etc/syslog-ng.conf.d/. Restarting the ser-
vice will apply the change:

configctl syslog restart

The strings are just examples and do not represent a complete monitoring
solution. The new configuration file will relay critical syslog messages
(lines 7–10) to the telegram.sh script (line 2) in addition to its regular

317
 Chapter 23. Life Hacks

1 destination d_telegram {
2 program("/usr/local/bin/telegram.sh"
3 template("${DATE} ${HOST} ${MESSAGE}\n") );
4 };
5
6 filter f_telegram {
7 message(".*telegram-test.*")
8 or message(".*error for illegal user.*")
9 or message(".*GATEWAY ALARM.*")
10 or message(".*failed waiting for configd.*")
11 };
12

13 log {
14 source(s_all);
15 filter(f_telegram);
16 destination(d_telegram);
17 };

Listing 23.1: Syslog-NG reports important messages via Telegram

processing. Syslog sends the messages through STDIN. Also, make sure the
telegram.sh script from Listing 23.2 has the executable flag set:
chmod +x /usr/local/bin/telegram.sh

Note
This policy does not inspect old syslog messages for alerts. It will
process only new messages, which are currently logged in the system.

The smartphone must have a messenger application which is capable of

receiving messages directly from the command line. This chapter uses Tele-
gram [28]. The example in Listing 23.2 can send messages to a Telegram
client and requires API key and chat ID first. Telegram will reveal these
values after setting up a chatbot. The script receives input messages from
STDIN (line 4) and contacts the API of Telegram using a text-based web
client in line 5.

318
 Telegram

1 #!/bin/csh
2 set API_KEY=419840419:AAGPcJDOIe5YCS_ep3yfgPJM-f7E7m8MMMY
3 set CHAT_ID=273171581
4 set LINE = "$<"
5 /usr/local/bin/curl --insecure \
6 --data "chat_id=${CHAT_ID}&text=${LINE}" \
7 https://api.telegram.org/bot${API_KEY}/sendMessage

Listing 23.2: OPNsense sends log messages to Telegram as chatbot

Note
The regular logging process of an OPNsense firewall is not interrupted
if the telegram script is malfunctioning or unexpectedly stops.

Until now, the syslog process did not know anything about a Telegram
script. Therefore, OPNsense must reload the configuration file. This action
is done at System → Diagnostics → Services by pressing the button restart
next to syslog-ng or by rebooting the system. Next, generate a harmless
message from the command line to validate the new syslog alerter:
root@RT-1:˜ # logger telegram-test

This message is considered as an error and will trigger an alert. The script
will attach the syslog text in the message to the smartphone. Figure 23.2
shows the smartphone app, which has received various alarms from the lab
firewall.

Figure 23.2: OPNsense reports significant events with Telegram messenger

319
Chapter 23. Life Hacks

Firewall rules with category

The list of firewall rules can become long and confusing. Before the ruleset
fades into chaos, OPNsense provides a classifier for each rule. The classifier
is a free-text field in every firewall rule that assigns the rule to a category
(Figure 23.3).

Figure 23.3: Each firewall rule can be assigned to a category

Next, it is time to choose a category text for each rule. There are major
benefits in creating firewall filters using the categories rule, and it is well
worth the effort. The selection of rules in Figure 23.4 shows only the rules
categorized with the word dns. Multiple choices are possible.

Figure 23.4: Sort or filter the firewall ruleset by category

320
 Quick search

Note
The categories aim is to improve the organization of the ruleset. It
does not affect the operation of the packet filter.

Quick search
Users switching from pfSense will miss some topics because OPNsense has
placed the menu item elsewhere in the web interface. Manually searching
the web UI is quite frustrating and complicates the transition to OPNsense.
The web GUI offers a quick search in the title bar, which shows all available
topics as soon as the first letters are typed in. In Figure 23.5 the admin
searches for the magic word dns and immediately sees all areas dealing
with name resolution. A mouse click on the desired records leads to the
selected topic.

Figure 23.5: OPNsense will find the missing item with the quick search

321
Chapter 23. Life Hacks

322
Chapter 24

Application Programming
Interface

Access to an OPNsense system is not only possible via a web interface, but
also a programming interface. While web access is designed for the human
user, the programming interface is accessible from machines, scripts, and
monitoring systems.
OPNsense considers itself an open platform and presents its methods via
Application Programming Interface (API) so that the community can inte-
grate additional commands and processes.
Using the API is independent of any programming language, but requires
basic knowledge in Python to create new commands or modify existing
ones.
The open programming interface is a new concept of OPNsense, and did
not exist under the predecessor pfSense.

How does the API work?

The OPNsense programming interface follows the principle of Representa-
tional State Transfer (REST). The way REST works is based on client-server
architecture: the client asks the questions, and the server gives the answers.
The connection between the partners is stateless, i.e. the request always
contains all the necessary information that the server needs for its response.
Figure 24.1 shows the architecture of OPNsense. Access to the system is

323
Chapter 24. Application Programming Interface

granted on an equal basis via a web UI and API. Both methods have their
own view (View; see section Model View Controller on page 325) of the
OPNsense firewall. The requests end up in the background at the Controller,
which takes care of the internal processing.

Figure 24.1: OPNsense components interact with each other

The communication between client and server use the well-known HTTP
protocol. The client rephrases its question as a web address, e.g.

http://opnsense.example.net/api/core/firmware/running

The client wants to know whether a firmware update is in progress. The

server recognizes the question and checks what information it can provide.
The server formats the response as JavaScript Object Notation (JSON).

324
 How does the API work?

Despite its name, this format is known in most programming languages

and is easy to parse for scripts and programs. Even for the untrained eye,
looking at a JSON expression is not too mysterious:

{"status":"busy"}

An essential approach of REST is the uniform interface. The API syn-

tax should not change between OPNsense versions; this ensures that the
client-side programs do not have to be reprogrammed. Also, to maintain
uniformity, the API keywords are not translated into other languages.

The client can also change the OPNsense firewall configuration with the
API. How does the server know if the client wants to change something
or just gather information? This is where HTTP commands come in handy.
These commands are hidden in the background when using a regular web
browser.
The client can request information with the API using either the GET or
POST method. The GET method simply requests information while the
POST method sends additional data to the server, i.e. information not in-
cluded in the web address but required by the server. The GET request has
the reputation of an informational request without changes to the firewall.
The POST request, on the other hand, is generally used to change the state
in the server; in OPNsense this leads to a configuration change.
The HTTP protocol offers additional methods that are not used by the
OPNsense API.

Model View Controller

OPNsense divides software according to the principle of the Model View
Controller (MVC). This pattern supports code reuse and allows multiple
developers to work in parallel.
The three components model, view and controller, operate independently
of each other, as shown in Figure 24.2 on the following page.

• The model contains the data of the system, which it gets from the
controller and represents in the view.

• The view presents the data and expects user actions.

325
Chapter 24. Application Programming Interface

• The controller receives commands from the view and thus from the
user. The controller evaluates the commands, manipulates the model,
and updates the view, as a confirmation for the user.

View

Notify Write

Manipulate
Model Controller
Update
Figure 24.2: The interaction of Model, View, and Controller

Figure 24.1 on page 324 shows the usage at OPNsense. HTTP clients
observe the firewall through the view. One way to access the view, is from a
web browser. The replies are formatted as colorful web pages. Another view
for the API answers in pure JSON style and does not care about graphical
representation.
The view interacts with the controller, which is the central element of
data processing. The controller understands the requests and uses them
to create commands for the background services (see Chapter 19). For
change requests, the controller contacts the model that holds the data
inventory. With OPNsense, the data comprise of the current configuration,
which is primarily located in the central file config.xml. Simple or not, the
controller can not directly interact with the config.xml, but rather direct
all change requests to the model. This division occurs because the model is
the data expert and knows how to restart services after specific changes.

Documentation
The official documentation contains exact details about the usage and an
overview of the available options for each module and plug-in. Unfortu-
nately, the explanation is a bit too short – the meaning of an API resource is

326
 Read Access

only revealed by the function’s name. For example, it might be obvious that
the function upgrade from firmware starts a software update. However, it
is less obvious what the function dirty from monitoring tool Monit might
trigger.
The error messages are also of little help. If a call fails, the API returns only
brief responses such as

{"result":"failed"}

Extending the API with custom calls is well documented on the homepage
of OPNsense. In addition to examples, naming conventions, and guidelines,
there are templates to quick start the module programming.

Read Access
Before the client starts, it needs permission to call the API functions. This
access is comparable to a user login on a web interface because without
authentication the API refuses to answer anything.
The API key is available in the web menu under System → Access → Users.
The default account root of a test firewall is suitable for experiments. Locate
the API key section of the root user and click on the plus icon, which will
create an API key and offer to download it. The API access consists of a key
and a secret password. Together they allow access to the API.

key=j8i0QGGOq/vLJBq4RJpjZAheOvKYDFJYMuktnb2wo7EFwx/hxZo9nAdU[...]
secret=YbDe5YoLv7Cep8ATduQRr8StPO+YL3NE6l9EDiAEZS9OqJ4H0H1vh[...]

Each user account can create any number of API keys. The official OPN-
sense documentation recommends a separate key for every purpose.

Note
The API key is shortened in the following examples to improve the
readability of the commands.

curl --silent --user "j8i0QGGOq":"YbDe5YoLv" \

http://opnsense.example.net/api/core/firmware/status

327
 Chapter 24. Application Programming Interface

The command curl uses the API key as a combination of username and
password to authenticate a web page. Each request requires a repeated
authorization.
If the key is correct and the remote firewall is reachable, it will respond
with a JSON message:
{"connection":"ok","downgrade_packages":[],"download_size":"","l \
ast_check":"Tue Feb 2 09:29:51 CET 2021","new_packages":[],"os_ \
version":"FreeBSD 12.1-RELEASE-p12-HBSD","product_name":"opnsens \
e","product_version":"21.1","reinstall_packages":[],"remove_pack \
ages":[],"repository":"ok","updates":"0","upgrade_major_message" \
:"","upgrade_major_version":"","upgrade_needs_reboot":"0","upgra \
de_packages":[],"all_packages":[],"status_msg":"There are no upd \
ates available on the selected mirror.","status":"none"}

The API key is required in every curl command. It is recommended to store

the key string into variables to improve readability and scriptability.
1 key=j8i0QGGOq/vLJBq4RJpjZAheOvKYDFJYMuktnb2wo7EFwx/hxZo9v[...]
2 secret=YbDe5YoLv7Cep8ATduQRr8StPO+YL3NE6l9EDiAEZS9OqJ4H0H[...]
3 fw=10.5.1.1
4 curl --silent --user "${key}":"${secret}" \
5 http://${fw}/api/core/firmware/status | python -m json.tool
The command chaining with Python in line 5 formats the JSON code with
indented text and with line breaks. That way, it is easier to read the text
answer from the server. The actual content is unchanged, and the answer
is printed here in abbreviated form:
{
"connection": "ok",
"all_packages": [],
"downgrade_packages": [],
"download_size": "",
"last_check": "Tue Feb 2 09:31:51 CET 2021",
"new_packages": [],
"os_version": "FreeBSD 12.1-RELEASE-p12-HBSD",
"product_name": "opnsense",
"product_version": "21.1",
[...]
}

The command line is the ideal location for scripting. However, to explore
the API, there are graphical tools, one of which is introduced in the section
API browser on page 331.

328
 Write Access

Write Access
Accessing the API can also bring changes to the configuration. This write
access requires the HTTP method POST.
Using the intrusion detection system example (see Chapter 18), the API
shows the details of an IDS rule and allows changes to it. The information
for rule number 2013861 is provided by the subsequent call to getRuleInfo,
where the API key is expected in the environment variables.

curl --silent --user "${key}":"${secret}" \

http://10.5.1.1/api/ids/settings/ getRuleInfo /2013861 \
| python -m json.tool

The API lists all information and states of the requested IDS rule.

{
"action": {
"alert": {
"value": "Alert",
"selected": 1
},
"drop": {
"value": "Drop",
"selected": 0
}
},
"action_default": "alert",
"classtype": "bad-unknown",
"created_at": "2011_11_07",
"enabled": 1,
"enabled_default": 1,
"former_category": "HUNTING",
"gid": null,
"matched_policy": null,
"msg": "ET DNS Query for Suspicious .nl.ai Domain",
"reference": null,
"rev": 3,
"sid": 2013861,
"source": "emerging-dns.rules",
"status": "enabled",
"updated_at": "2020_09_17"
}

329
 Chapter 24. Application Programming Interface

The python script json.tool modifies the answer optically to make it more
readable to the human eye. The action of this rule is Alert.
Use the API command setRule to change the rule action to Drop.
1 curl --silent --user "${key}":"${secret}" \
2 --request POST \
3 --data ’{"action":"drop"}’ \
4 --header "Content-Type: application/json" \
5 http://10.5.1.1/api/ids/settings/ setRule /2013861

The API command requires the HTTP method POST (line 2), the intended
action (line 3) and the JSON format (line 4). The rule number in question
remains in the web address (line 5).

Note
Most of curl ’s options have an abbreviated form (e.g. -d) and a long
form (e.g. --data). The examples in this chapter print the longer
arguments for better readability.

Moving the result through json.tool is not necessary because the answer is
also clearly recognizable in its raw state:
{"result":"saved"}

Call getRuleInfo again to confirm that the rule action changed to Drop.

What does the API cover?

In theory, the API can do everything the web interface can do. However;
in practice, not all areas of the source code are API-enabled yet. So far
(OPNsense version 21.1) the API covers the following sections of OPNsense:

• Captive Portal
• Cron
• Diagnostics
• Firewall
• Firmware
• Interfaces

330
 API browser

• Intrusion Detection
• IPsec
• Monit
• NetFlow and Insight
• OpenVPN (only Client Export)
• Routes
• Shutdown and Reboot
• Syslog
• Traffic Shaper
• Web Proxy

The web interface displays whether a specific configuration section is al-

ready supported with the API. If the path of the web address begins with
/ui, the API is usable:
http://rt-1.opnsense.lab /ui /cron/

Otherwise, the configuration is still in the original code from the predecessor
pfSense, which was not intended to work with an API:
http://rt-1.opnsense.lab/system_general.php

A complete list of available API commands is provided by a sophisticated

look into the file system of an OPNsense firewall. The following command
reveals all the API calls, where the path is the only description of the
purpose.
egrep --no-filename --recursive "ajax[Get|Call]" \
/usr/local/opnsense/mvc/app/views/OPNsense/

API browser
Easy access to the programming interface comes with a graphical API
browser. This software generates the same syntax for a request, but the
parameters and headers section is crafted from an intuitive interface.
API browsers are available as a stand-alone software or as a plug-in for a
web browser. The Chrome extension RESTED [30] does an excellent job
and provides all necessary functions for the web API of OPNsense.

331
Chapter 24. Application Programming Interface

Figure 24.3: The API browser RESTED simplifies access

Figure 24.3 shows how to use RESTED to read the IDS rule from section
Write Access. First, insert the API key material into the browser; the API
key is stored as a username and the API secret as a password.
Do not get confused: the return code 200–OK means that the server has
understood and processed the request. Nevertheless, the answer can return
a negative result, which is not OK depending on the expected output.

Security
Nowadays, unencrypted access with HTTP to the API is only tolerated in
an isolated demo environment. It is best to use encrypted communication
with HTTPS in the local network and on the Internet.
However, the improved security of HTTPS only works if the client can
validate the server certificate. For this reason, the OPNsense firewall must
own and present a correctly issued certificate.
For environments with a weak security level, the OPNsense firewall can
act as a Certificate Authority (CA) and issue a certificate for its website

332
 Technical background

(see Chapter 14). If the API client trusts this CA, subsequent attacks on the
connection can be detected, and HTTPS saves the day.

First, convince the client to trust the CA. OPNsense hosts the certificate
management under System → Trust → Authorities. The export CA cert
button offers the certificate for download. If this button is missing, the
web server does not have a properly signed CA certificate, but still uses the
self-signed certificate from the initial installation. Give the downloaded CA
certificate the suitable filename ca-opnsense-lab.crt and copy it to the
client, which will be used later during negotiation with the API.
Finally, switch the protocol of the web UI from HTTP to HTTPS at System
→ Settings → Administration. From now on, the web interface of OPNsense
is only accessible through encrypted channels.

Armed with the CA certificate, the client can now validate the server’s
authenticity. The curl command either raises an alert while negotiating the
HTTPS connection or is silent when everything is fine.
1 curl --silent --user "${key}":"${secret}" \
2 --cacert ca-opnsense-lab.crt \
3 https ://rt-1.opnsense.lab/api/core/firmware/running

curl learns about the certificate by the command option in line 2. Once the
protocol is changed to https (line 3), the communication is encrypted and
authenticated between API client and server

Technical background
The HTTP server lighttpd is the service responsible for the configuration
interface and the web API. The service distinguishes the requests by the
path of a URL: a configuration web page always begins with /ui, while the
API uses the unique prefix /api.
All requests to the API end up in the script api.php on the server side. The
front-end of the API is the PHP file:
/usr/local/opnsense/www/api.php

The path of the web address is passed as an argument when calling api.php
so that the script knows what to do.

333
Chapter 24. Application Programming Interface

The API only works for the parts of the source code that have already been
rewritten from the pfSense code to the new OPNsense style. The developers
of OPNsense divide their software into components (see section Model
View Controller on page 325) and expect better code quality and shorter
development times.

Outlook
The OPNsense programming interface is still under construction, but the
basic structure is already in place. When the API is complete, it will be
possible to completely control an OPNsense firewall by scripts or a command
line. A uniform management software would also be imaginable, which
centrally manages all OPNsense firewalls in the corporate network and
configures them via the API.
Until now, the API has been used by the web interface to populate the page
with additional content, such as the current utilization of network cards and
system states. In this area, too, the developers are committed to dynamic
content, which is loaded via API calls.

Summary
The OPNsense programming interface is a way to query and configure the
firewall using a text-based web client. The syntax of the commands and
formats resemble common techniques so that a software developer can
quickly learn how to use the API.
Unfortunately, the API does not yet cover all areas offered in the web
interface. Virtually every new version of OPNsense enables more categories
to be used with the API.

334
Bibliography

[1] Deciso B.V.: Securing Networks. 2021. https://www.deciso.com/

[2] Deciso B.V.: Hardware sizing & setup. 2019.

https://docs.opnsense.org/manual/hardware.html

[3] The FreeBSD Project: FreeBSD 12.1-RELEASE Hardware Notes. 2021.

https://www.freebsd.org/releases/12.1R/hardware/

[4] OPNsense project team: OPNsense is a true open source firewall. 2021.
https://opnsense.org/

[5] Icons8 LLC: Icons by Icons8. 2021.

https://icons8.com/license/

[6] Ian Moore, Smart Guide Pty Ltd: phpVirtualBox.

2021. https://phpvirtualbox.github.io/

[7] PC Engines: apu1d4. 2018. http://www.pcengines.ch/apu1d4.htm

[8] Manuel Kasper: physdiskwrite. 2014.

http://m0n0.ch/wall/physdiskwrite.php

[9] MaxMind, Inc: GeoLite2 Free Geolocation Data. 2021.

https://dev.maxmind.com/geoip/geoip2/geolite2/

[10] Andreas Steffen: OpenSource IPsec-based VPN Solution. 2021.

https://www.strongswan.org/

[11] OpenVPN: Community Downloads. 2021.

https://openvpn.net/community-downloads/

335
Bibliography

[12] Peter Haag: nfdump. 2020. https://github.com/phaag/nfdump

[13] Simon Leinen: Samplicator. 2019.

https://github.com/sleinen/samplicator

[14] Shalla Secure Services KG: Shalla’s Blacklists. 2018.

http://www.shallalist.de/

[15] European Institute for Computer Anti-Virus Research e.V.:

Anti-Malware Testfile. 2019.
https://www.eicar.org/?page_id=3950

[16] JumpCloud: JumpCloud Directory Platform. 2019.

https://jumpcloud.com/

[17] GoDaddy: Certificate Repository. 2021.

https://certs.godaddy.com/repository

[18] Jason McFarland: TheJumpCloud. 2021.

https://github.com/TheJumpCloud

[19] Greenbone Networks GmbH: OpenVAS – Open Vulnerability

Assessment System. 2020. https://www.openvas.org

[20] National Institute of Standards and Technology: National

Vulnerability Database. 2021. https://nvd.nist.gov/vuln/search

[21] Open Information Security Foundation:

Suricata | Open Source IDS / IPS / NSM engine.
2021. https://suricata-ids.org/

[22] Emerging Threats: Emerging Threats Rule Documentation. 2019.

https://doc.emergingthreats.net

[23] ESnet: iperf3. 2020. http://software.es.net/iperf/

[24] Cisco Systems: IPsec Overhead Calculator Tool. 2021.

https://cway.cisco.com/tools/ipsec-overhead-calc/

[25] Simon Tatham: PuTTY: a free SSH and Telnet client. 2020.
https://www.chiark.greenend.org.uk/~sgtatham/putty/

336
 Bibliography

[26] Andrea Fabrizi: Dropbox Uploader. 2020.

https://github.com/andreafabrizi/Dropbox-Uploader

[27] Martin Prikryl: WinSCP. 2021. https://winscp.net/de/

[28] Telegram Messenger LLP: Telegram Messenger. 2021.

https://telegram.org/

[29] Daniel Stenberg et al.: curl: command line tool and library. 2021.
https://curl.se/

[30] Espen Henriksen: RESTED – A REST client for the rest of us. 2019.
https://github.com/RESTEDClient/RESTED

337
Bibliography

338
Index

2FA, see authentication AES-NI, 139, 283

32-bit, 32 AH, see IPsec
3DES, 280, 283 alert, 266
64-bit, 28 algorithm, 121, 143, 151, 280
Annex J, 242
access code, see Dropbox anti spoofing, 70, 78
Access-Accept, 210 antivirus, 182
Access-Reject, 221 API, 217, 309, 323
account, 205 APU, 43, 242
accounting, 217 ARP, 63, 252, 290
Active Directory, 204 attack, 255
address authentication, 182, 201, 301
assign, 54 two-factor, 111
dynamic, 217 VPN, 120, 138, 152
fake, 79 AuthenticationFactory, 222
global unicast, 63 authorization, 201
interface, 99
IP, 28 backup, 158, 307
IPv4, 57, 70 bandwidth limitation, 109
IPv6, 61, 101, 127 base, 166
link-local, 63 Bash, 218, 306
MAC, 33 beep, 49
multicast, 163, 170 benchmark, 296
private, 117 best practice, 295
translation, see NAT binat, see NAT
unique local, 63 bind, 220
virtual, 157, 158 bitmask, 279
advertising frequency, 158, 166, 168 block, 73
AES, 143, 280 Blowfish, 143

339
Index

bridge, see network bridge default route, 57

browser, 331 DHCP, 247
buffer, 281 DHCPv6, 251
direction, 70
CA, see certificate authority Directory as a Service, 211
cache, 182, 198, 290 disk, 24
captive portal, 222 distinguished name, 125, 212
CARP, 155, 189, 228 DNS, 56, 247
category, see proxy, 320 domain, 56, 204
certificate, 56, 139, 152, 193, 204, DoS, 109
213, 309, 332 dpinger, 239
certificate authority, 194, 213, 332 Drive, see Google Drive
charon, 131 drop, 266
Chatbot, see Telegram Dropbox, 305
check, 231 DSL, 241
Cisco, 317 dual stack, 249, 252
ClamAV, 199 DynDNS, 254
cluster, 156, 189
collector, 176, 177 e1000, 53, 276
command line, 267 Easy Editor, 348
compression, 151 editor, 315, 348
condition, 266 EICAR, 199
configd, 267 election, 166
configuration, 51, 305, 357 Emerging Threats, 265
connection setup, 120 encryption, 120
console, 51, 56 errata, 357
controller, see MVC ESP, see IPsec
core, 286 ESXi, see VMware
CPU core, 286
cpuset, 279 factory reset, 295
cron, 268, 307 failover, 160, 167, 236
crypto key, 152 failure protection, 228, 233
curl, 90, 97, 217, 306 filter, 183, 185
CVE, 259 LDAP, 206
operation, 89
Dashboard, 251 rule, see firewall rule
dead peer detection, 120, 126 firewall, 28, 69, 235, 252

340
 Index

backup, 156 transparent, 261

rule, 71, 96, 108, 122, 144 ifconfig, 275
transparent, 85 ifinfo, 275
troubleshooting, 83 iftop, 275
uncover, 91 IKE, 120, 130
first match, 70 in-band, see management
flash memory, 47 Inspect, 76
floating rule, 70, 83 Inspection, 192
flow, see NetFlow interface, see network card
FreeBSD, 19, 53, 84, 103, 167 interface identifier, see IPv6
FreeOTP, 112 Internet, 244
Internet Key Exchange, see IKE
gateway, 57, 231
interrupt, 278
group, 233
intrusion detection, see IDS
GeoIP, 80
IP
GEOM mirror, 46
address, see address
GET, 325
monitor, 231
GitHub, 306, 357
virtual, 95, 158
global unicast, see address
iperf, 282, 296
Google, 247
IPS, 260
Google Authenticator, 112
IPsec, 117, 139, 202, 222, 284, 287
Google Drive, 309
throughput, 280
group, see firewall
troubleshooting, 129
hard disk mirror, 46 IPTV, 243
HardenedBSD, 18 IPv4, 57, 249
hardware, 41 IPv6, 61, 101, 169, 178, 238, 249
header, 117 ISO file, 32
health check, 231 ITU-T, 242
heartbeat, 156
HMAC, 152 JSON, 217, 324
hostname, 56 JumpCloud, 211
hypervisor, 34
keepalive, 156
ICAP, 199 kernel, 56, 264
ICMP, 252 key, 298, 327
IDS, 255, 329 pre-shared, 138
rule, 266 private, 299

341
Index

public, 300 MTU, 55, 244, 282, 287

keyboard, 48 Multi-WAN, 227
multicast, 244, 254
lab MVC, 325
network, 23, 357
server, 29 NAT, 93, 124, 160, 235, 249
latency, 233 reflection, 103
LDAP, 202, 212 rule, 84, 87
filter, 206 traversal, 125
troubleshooting, 220 NDP, 63
LibreSSL, 153, 284 NetFlow, 173
License, 80 netstat, 145, 275
lighttpd, 333 network
link-local, see address bridge, 86, 89, 247, 263, 316
live image, 32 card, 55
load balancing, 167 policy, 209
load distribution, 227, 229 network adapter
load sharing, 233 assign, 53
log, 56, 73, 183, 210 name, 53
virtual, 275
management, 105, 253 Network Policy Server, 208
masquerading, 98 nfdump, 176
Master, see CARP normalization, see rule
MaxMind, 80 NPS, 207
memory, 24 NPTv6, 101, 238
mesh, 133
Microsoft, 201 offload, 277
mirror port, 256, 316 open source, 17
mode, see OpenVPN OpenBSD, 167
Model View Controller, 325 OpenSSL, 134, 137, 153, 284
modem, 242 openssl, 309
monitoring, 56, 237 OpenVAS, 257
monowall, 19 OpenVPN, 137, 202, 284
more, 347 GUI, 149
MPD5, 253 troubleshooting, 150
MSS, 287 operating system, 45
mssfix, 289 Oracle, 37

342
 Index

organizational unit, 205 PPPoE, 243

OTP seed, 112 pre-shared key, 130, 138
out-of-band, see management prefix, 62, 102, 238
delegation, 249
P12, 309 programming, 323
packet filter, 69, 85 promiscuous mode, 263
packet loss, 233 protocol, 201
PAM, 222 Proxy
password, 55, 201, 301 HTTPS, 192
time-based, 111 proxy, 140, 181
PAT, 98 ARP, 99
pattern, see regular expression cluster, 189
PC-Engines, 43, 242 explicit, 184
PCnet, 53 transparent, 197
PCRE, see regular expression troubleshooting, 188
peer identifier, 130 web, 202
peer-to-peer, 135 PuTTY, 299
performance, 273 puttygen, 299
encryption, 139 Python, 268, 323, 328
pf, 81, 104, 239
pfctl, 275 QR code, 112
pfInfo, 76 quick, 83
pfSense, 19, 334
pfsync, 162, 170 RADIUS, 202, 207, 215
pftop, 275 troubleshooting, 221
PHP, 222, 333 radvd, 254
phpVirtualBox, 38 rdr, 104
physdiskwrite, 43 regular expression, 186, 351
PIN, 111 reject, 73
ping, 58, 90, 288 REST, 323
pipe, see bandwidth limitation RFC2544, 282
PKI, 152 RFC5737, 29
plug-in, 48, 134, 199 RFC6296, 238
policy, 187, 209 route, 58, 64
port forward, 95, 100, 197 router
port group, 35 advertisement, 253
POST, 325 default, 57

343
Index

DSL, 241 synchronization, 162, 166

routing Syslog, 317
asymmetric, 165 syslog, 317
prefix, 62 systat, 275
static, 57 system
table, 57 embedded, 42
throughput, 278 sound, 49, 56
rule
filter, see firewall rule tap, 147
floating, 83 Telegram, 317
normalization, 289 throughput, 273, 296
order, 82 tier, 233
time-based, 77 time, 56
ruleset, 70, 75, 89, 259 timestamp, 217
TLS, 192, 204, 213
sAMAccountName, 206 TLS Inspection, 192
secure shell, 56 token generator, 111
SecureCopy, 313 top, 274
SFTP, 316 traceroute, 66, 128, 145, 159
SHA, 143 traffic analysis, 254
shaper, 109 traffic shaper, 109, 254
signature, 258 transparent, 85
skew, 166 tun, 147
SLAAC, 250 tuning, 273
SOCKS, 182 tunnel, 117
source NAT, see NAT client-server, 146
SPAN, 316 IPsec, 117
span port, 316 OpenVPN, 137
spoofing, see anti spoofing site-to-site, 122, 141
Squid, 185, 198 two-factor authentication, 111
SSH, 90, 202, 298 unique local, see address
SSL, see TLS UNIX, 273
SSL Inspection, 192 update, 56
state table, 162 URL filter, see filter
strongSwan, 131, 284 user space, 139
Suricata, 264
switch, 27 vboxmanage, 38

344
 Index

VDSL, 241 web proxy, see proxy

vectoring, 242 Windows, 65, 149, 204, 316
versioning, 308 WinSCP, 316
view, see Model View Controller
Vim, 348 XMLRPC Sync, 164, 190
virtio-net, 53, 276
ZeroTier, 135
virtual
IP, see IP
network adapter, 275
switch, 34
VirtualBox, 37
virtualization, 26
virus, 182, 199
VLAN, 28, 42, 243
vlance, 277
VMware
ESXi, 34, 276
Player, 33
tools, 48
Workstation, 32
VMXNET, 276
voucher, 223
VPN
compression, 151
dial-in, 133, 147
error pattern, 129
IPsec, 117
OpenVPN, 137
performance, 143
throughput, 128
Tinc, see Tinc VPN
VRRP, 156
vulnerability scanner, 258

WAN, 227
web cache, see cache
web GUI, 56

345
Index

346
Appendix A

Editing Files in FreeBSD

OPNsense has forked the web interface of pfSense, but unfortunately not
all commands of the operating system. Several chapters require access to
the underlying operating system FreeBSD because the intended task is not
available from the web interface. OPNsense does not put any barriers in the
way and allows access to the operating system level with the menu item 8
Shell in the text console.

From this point on, utmost caution is required, as OPNsense no longer

protects you from any disastrous changes.

Show content of a file

The contents of a text file are displayed page by page with the more com-
mand. Jump to the next page with the spacebar, while using the Enter key
to move to the next line.
For example, have a look at the configuration file of PHP with:

more /usr/local/etc/php.ini

While browsing within the file, the v key switches to the text editor if
changes to the content are desired. Quit using more with the q key.

347
Appendix A. Editing Files in FreeBSD

Edit a file
Do not change anything without backing up! Make a copy of the original
file before your fingers modify its content. The effort is minimal and keeps
emergencies caused by fatal changes from turning into disastrous results.
The command to copy files under UNIX is cp. Append the source file and
destination file as arguments. Create a backup copy of the above text file
with the following command:
cp /usr/local/etc/php.ini /usr/local/etc/php.ini .orig

The target file can be abbreviated as follows to save typing effort:

cp /usr/local/etc/php.ini{,.orig}

OPNsense provides two editors which differ in their operation and complex-
ity. For users with little previous knowledge of UNIX, the Easy Editor is a
good choice. If you want to learn more about UNIX, take a look at the vim
editor.

Easy Editor
The Easy Editor is a lightweight editor, which implements essential func-
tions for editing files. The starter command ee expects the file name as an
argument. It will then open the editor window and present the file content.
ee /usr/local/etc/php.ini

The top lines of the ee window are filled with commonly used commands.
The abbreviations ^v stands for the keyboard shortcut Ctrl-v and will jump
to the next page. The editor will save the file to disk when exited with
Escape+Enter. The editor offers more information in the integrated help
section at Escape+b and inside the man-page.

Vi IMproved
The vim text editor is an enhanced version of the older vi. It is more
user-friendly and has more features. Actually, it is overqualified for the task
of editing plain text files.

348
The vim has two modes of operation: normal mode and insert mode. During
normal mode all user input is recognized as commands. These commands
may delete lines, copy words, search-and-replace, and move around inside
the file. The i key (for insert) switches the vim to insert mode. Within this
mode, all user input is recognized as text. It is inserted in the text at the
cursor position. The Esc key switches the editor back to normal mode.
It is common to switch modes frequently while working with vim. Using
vim for the first time is somewhat counterintuitive, but with knowledge of
the most important commands, files can be edited very efficiently.
The editor is called by the command vi and expects a file name as argument:

vi /usr/local/etc/php.ini

Table A.1 contains frequently used command for the vim editor.

Command Effect
:w write. Save the file.
:q quit. Quit the editor.
:q! Quit the editor without saving the file.
:wq Quit the editor and save the file.
i insert text before the cursor.
I Insert text at the beginning of the current line.
a append text after the cursor.
A Append text at the end of the current line.
o Create a new line below the current line.
O Create a new line above the current line.
x Delete the character at the position of the cursor.
D delete from the cursor to the end of the current line.
dd Delete the current line.
yy yank. Copy the current line to the clipboard.
p paste. Insert the content of the clipboard after the cursor.
u undo. Revert the last action.

Table A.1: Important commands of the vim text editor

It is even possible to execute commands several times by prefixing them

with a number. For example, the 5dd command deletes five lines at once.
After entering 10x, the next ten characters disappear from the screen.

349
Appendix A. Editing Files in FreeBSD

If the vim has changed or deleted too much, pressing the u key several
times will undo the changes until the file is back to a known state. And if
the file content is hopelessly destroyed, hit :q! to exit the editor without
saving.
Complete books have been written about the vim, and the developer’s web-
site http://www.vim.org/ offers a good introduction.

This book was written with the vim editor.

350
Appendix B

Pattern Matching

Perl Compatible Regular Expression (PCRE) or Regex for short, is a method

to find a specific pattern within a text.
Simply speaking, Regex is an enhanced word processor search function. It
is more powerful but also more complicated. The search for patterns go
far beyond the recognition of simple words. You can search for anything:
regular expressions scan texts for numbers, strings, URLs, email addresses,
date, and time.
Many software developers and programming languages now use the PCRE
library. As such, developers do not have to design their own pattern recog-
nition logic for each program, and even end-users might get involved.

OPNsense uses the regular expressions for access control lists in the web
proxy. The help function of the website gives a few brief examples, which
are described in more detail here.
In the simplest case, the search pattern consists of only one word without
special characters. The string example.net finds all websites that have this
text somewhere in the URL. The letters in front and behind the string are
irrelevant.
The following text would match the pattern example.net:
mail.example.net
www.example.net/wp-content/uploads
http://docs.opnsense.org/search.html?q=example.net
This text does not contain example.net

351
Appendix B. Pattern Matching

Note
If the regular expression contains a space, the system searches for a
space in the text. The space is therefore not suitable for making the
Regex more readable.

Selections
The strength of the regular expressions lies in the wildcard characters,
which do not search for a specific letter, but a selection of characters. The
expression a-z finds all lowercase letters. With E-H, Regex will match the
capital letters E, F, G, and H. And 0-9 will match a single digit. When using
these tricks, the expression must be enclosed in square brackets.
It is even possible to combine multiple selections:
[a-zA-Z0-9]

will match a single letter that is either lowercase, uppercase, or a digit.

However, it will not match a question mark, forward slash, or @-key.

The square brackets may also appear several times or be mixed with normal
text. The following expression searches for the digits 1 to 3, followed by a
fixed text, the digits 6-9, any digit, and finally the word Meters written in
lowercase letters, beginning with either an upper- or lowercase letter.
[123]times [6-9][0-9] [Mm]eters

Quantifiers
If the pattern should match several characters of the same type, the plus
sign comes into play because plus searches “one or more” of the previous
expressions. The following pattern recognizes a multi-digit hexadecimal
number:
[a-fA-F0-9]+

This expression searches for the lower- and uppercase letter a to f or a digit.
And the string may appear multiple times, but at least once.

352
The asterisk has a similar effect as the plus, but it also accepts when
the desired pattern does not occur.
The following example shows a pattern that is optional by specifying *
behind the selection. That way, it may appear after the word July, but could
also be missing.

Today is the first of July [0-9]*

Between zero and any number of times, the regular expressions allow
precise specifications or a range.

• {5} expects the specified character exactly five times. [a-z]{5}

matches words consisting of five lowercase letters.

• {4,7} indicates a range from 4 to 7. The pattern [01]{4,7} will

match all binary numbers that have a length between 4 and 7 digits.

• The range {3,} has no upper limit. It will match all patterns with at
least three characters.

• Conversely, the specification of {,6} searches for patterns that have a

maximum of six characters. The Regex [0-9]{,6} will find numbers
from 0 to 999999.

The regular expression

[0-9]{4,5}0{3} [A-Z]{3}

searches for large amounts of money in any currency.

Characters
The regular expression of [A-Za-z0-9_] may be abbreviated as \w. Like-
wise, [0-9] can disguise itself as \d and \s stands for space bar, line feed,
and tabulator – basically anything that creates distance.
The pattern

\d{5}\s\w{4,}

matches a five-digit zip code for a city whose name has four or more letters.

353
Appendix B. Pattern Matching

Special characters
The period is a wildcard character and represents every single character
except the line break. The pattern fire.all matches firewall and fireball
but also senseless words like firenall and fire1all.
Table B.1 contains the most essential special characters.

Character Match
. Matches any single character (except a line break).
[ ] Matches any character within the square brackets.
[^] Matches anything except the characters in the square
brackets.
^ Matches any string that starts with the pattern.
$ Matches any string that ends with the pattern.
(|) Alternative. (gif|png|jpg)
* The pattern may appear once, multiple times, or not at all.
+ The previous pattern must appear once or more.
? The previous pattern may appear once or not at all.
\ Converts the following special character back to a regular
character.

Table B.1: Important special characters used in regular expressions

The following example illustrates the characters *, + and ?. In the pattern

fl?ight the letter l is optional due to the following question mark, so it
matches the words fight and flight. With fl+ight the l may occur several
times; It fits flight, fllight, flllight etc. but not fight. This is only the case
with the pattern fl*ight, because the asterisk allows the letter l to be
missing.

Examples
^https?:\/\/[a-zA-Z]+\.example\.(net|org|com)

The leading ^ character forces the string to begin with http. The http may
have an additional s, but that is not required. Next is a colon and two
forward slashes. The forward slash is also a special character, though it
needs a backslash to be interpreted as a regular character.

354
Next, the pattern expects one or more lower-/uppercase letters, followed
by a period and the fixed string example and another period. The period
also requires a backslash. Otherwise, it acts as wildcard and matches any
character. The pattern continues with either net or org or com.
The Regex stops after the top-level domain, but the matching string may
contain more characters. If that is not intended, complete the pattern with
the $ character to indicate that no characters may follow.

^(https?|ftp):\/\/\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\/

This Regex catches all http, https, and ftp requests that contain one to three
digit figures (\d{1,3}). The figures are separated by periods. When applied
to the blacklist of a proxy server, it will filter all requests that contain an
IPv4 address.
This method obviously only works for IPv4 addresses. The variable spelling
of IPv6 addresses (see Chapter 5) requires a more complex regular expres-
sion.

Testing
Regular expressions are an excellent source of errors and typos because a
complex Regex looks similar to encrypted text. It is best to develop and
validate the Regex in a Regex Tester. These web services expect the regular
expression and a text. The text then highlights what matches the selected
expression.
The websites listed below are a good starting point. Also, a web search with
the keywords pcre regex tester will provide a large selection.
https://regex101.com/
https://www.regextester.com/

355
Appendix B. Pattern Matching

356
Appendix C

Bonus Material

The printed examples in all chapters represent only a fraction of the infor-
mation relevant to the respective topic. The full configuration files of all
devices are available online at
https://practical-opnsense.github.io
https://github.com/practical-opnsense
or at
https://practical-opnsense.sourceforge.io
https://sourceforge.net/projects/practical-opnsense/

The URL contains all material that did not fit in this book, namely:

• The configuration of the firewalls from all chapters,

• Network diagram of the complete lab environment,

• Errata (list of corrections),

• Benchmark results of routing and crypto performance, and

• All scripts that are just mentioned in this book or printed in part.

357