SF-TAP: Scalable and Flexible Traffic Analysis Platform

Injecting pcap Files to SF-TAP Flow Abstractor

Sun, 05 Jun 2016 20:35:48 +0000

pcap Interface

SF-TAP flow abstractor provides an interface for injecting pcap files. The injection can be done by writing pcap files to the interface as follows.

$ ls /tmp/sf-tap/pcap
/tmp/sf-tap/pcap
$ cat dump01.pcap dump02.pcap | sudo nc -U /tmp/sf-tap/pcap

Limitation

Multiple pcap files beeing injected must be same endian, version numbers and timezone. If you want to inject pcap files captured different environments, inject the files one by one as follows.

$ cat dump01.pcap | sudo nc -U /tmp/sf-tap/pcap && \
cat dump02.pcap | sudo nc -U /tmp/sf-tap/pcap

Generally, traffic capturing is performed on singile environment, so, you may not need care about this.

Configuration of SF-TAP Flow Abstractor

Sun, 24 Apr 2016 03:51:42 +0000

Configuration File

SF-TAP flow abstractor reads a configuration file written in YAML when waking up. In this document, we explain how to write the configuration file for it.

Example Configuration

An example configuration is as follows. We explain each section of the configuration here.

# global configuration
global:
  home:    /tmp/sf-tap
  timeout: 30  # close long-lived (over 30[s]) but do-nothing connections
  lru:     yes # bring the least recently used pattern to front of list
  cache:   yes # use cache for regex
  tcp_threads:   2
  regex_threads: 2

pcap:
  if: pcap

loopback7:
  if:     loopback7
  format: text

tcp_default:
  if:     default # for every flow that wasn't matched by any rules
  proto:  TCP
  format: text
  body:   yes

udp_default:
  if:     default # for every flow that wasn't matched by any rules
  proto:  UDP
  format: text
  body:   yes

http:
  up:     '^[-a-zA-Z]+ .+ HTTP/1\.(0\r?\n|1\r?\n([-a-zA-Z]+: .+\r?\n)+)'
  down:   '^HTTP/1\.[01] [1-9][0-9]{2} .+\r?\n'
  proto:  TCP  # TCP or UDP
  if:     http
  format: text # text or binary
  body:   yes  # if specified 'no', only header is output
  nice:   100  # the smaller a value is, the higher a priority is
  utf8:   no   # treat data as UTF8 or latin1 (binary). used for regex
  balance: 4   # flows are balanced by 4 interfaces

http_proxy:
  up:     '^(CONNECT|connect) .+ HTTP/1\.(0\r?\n|1\r?\n([-a-zA-Z]+: .+\r?\n)+)'
  down:   '^HTTP/1\.[01] 200 .+\r?\n'
  proto:  TCP  # TCP or UDP
  if:     http_proxy
  format: text # text or binary
  body:   yes  # if specified 'no', only header is output
  nice:   90   # the smaller a value is, the higher a priority is
  utf8:   no   # treat data as UTF8 or latin1 (binary)

syslog_udp:
  up:     '^<([0-9]|[1-9][0-9]|1[0-8][0-9]|19[01])>'
  proto:  UDP
  if:     syslog
  format: text
  port:   514
  nice:   100
  utf8:   no

dns_udp:
  proto:  UDP
  if:     dns
  port:   53,5353,5355 # 53: UDP DNS, 5353: multicast DNS, 5355: LLMNR
  format: text
  nice:   200
  utf8:   no

global section

The global section can contain 6 subsections, which are “home”, “timeout”, “lru”, “cache”, “tcp_threads”, “regex_threads”, to control the behavior of SF-TAP flow abstractor.

home	used to indicate the root directory to which abstraction iterfaces are located
timeout	discard TCP sessions which do nothing for “timeout” seconds
lru	regular expressions are internally managed by the least-recently-used manner
cache	regular expressions are cached when matching
tcp_threads	the number of threads for handling TCP sessions
regex_threads	the number of threads for classifing flows by regular expressions

TCP section

TCP sections can contain following subsections.

up	the regular expression for up stream
down	the regular expression for down stream
proto	this subsection must be “TCP” for TCP
if	file name of the flow abstraction interface
format	spefify header format of the flow abstraction interface. this must be “text” or “binary”
body	the flow abstraction interface outputs TCP payload or only headers
nice	preference for classifing by regular expressions
utf8	palyload is treated as binary or UTF-8 string. this subsection affects behavior of regular expressions
port	classify by TCP port numbers. you can specify multiple port numbers such as 100,200,300-310
balance	the number of interfaces for load-balancing

The nice subsection indicate the preference for regular expressions. For example, if nice values of http and http_proxy are 100 and 90, flows are distinguished by using http_proxy’s regular expressions, and if the flows are classified as not http_proxy, then the flows are distinguished by using http’s regular expressions.

tcp_default section

This section is for the default interface of TCP flows. TCP flows are classified by regular expressions and port numbers indicated in TCP sections. The default TCP interfaces is the exit of flows which are not classified by the TCP sections. This section can contain following subsections.

if	interface name
proto	must be “TCP”
format	must be “text” or “binary”
body	output TCP payload or not
balance	the number for load-balancing

UDP section

UDP sections are for UDP packets. The subsections are quite similar to TCP section’s, but UDP can contain only 1 regular expression for classification, which are indicated by “up” subsection.

up	the regular expression for up stream
proto	this subsection must be “TCP” for TCP
if	file name of the flow abstraction interface
format	spefify header format of the flow abstraction interface. this must be “text” or “binary”
body	the flow abstraction interface outputs TCP payload or only headers
nice	preference for classifing by regular expressions
utf8	palyload is treated as binary or UTF-8 string. this subsection affects behavior of regular expressions
port	classify by TCP port numbers. you can specify multiple port numbers such as 100,200,300-310
balance	the number of interfaces for load-balancing

udp_default section

This section is for the default interface of UDP. This section can contains following subsections.

if	interface name
proto	must be “UDP”
format	must be “text” or “binary”
body	output TCP payload or not
balance	the number for load-balancing

loopback7 section

This section is used for the loopback7 interface. This section can contain “if” and “format” subsections as follows.

if	interface name
format	must be “text” or “binary”

pcap section

This section is for the pcap interface. This section can contain only the “if” subsection.

Re-injecting Flows via Loopback7 Interface

Wed, 02 Dec 2015 09:58:00 +0000

Loopback7 Interface

The loopback7 interfaces provided by SF-TAP flow abstractor is a special interface for injecting application flows into SF-TAP flow abstractor. This interfaces is helpful for handling encapsulated flows. For example, HTTP proxies allow HTTP clients to use HTTP tunneling by CONNECT method. Thus, actual flows, such as SSL/TLS, are encapsulated by HTTP protocol when using an HTTP proxy. In this case, SF-TAP flow abstractor does not properly work because of the encapsulation.

In order to handle encapsulated flows properly, the encapsulated flows must be decapsulated and re-injected to SF-TAP flow abstractor. The re-injecting is achieved by using the loopback7 interface. In this document, we show how to use the loopback7 intefrcace for HTTP proxy.

Re-injecting Strategy

Following figure shows the strategy for re-injecting flows of HTTP proxy. In this figure, the HTTP proxy handler connects to the HTTP proxy interface and the loopback7 interface for receiving and re-injecting flows of HTTP proxy. This handler strips data of HTTP proxy from encapsulated flows, and re-injects the internal flows to the SF-TAP flow abstractor via the loopback7 interface.

SF-TAP flow abstractor receives flows from the loopback7 interface, and classifies and outputs the flows to the corresponding interface; the SSL interface in this case. Accordingly, analyzers do not need to take care of encapsulations by HTTP proxy.

Pseudo Code for Re-injecting

We then show a pseudo code for re-injecting encapsulated flows as follows. This code is quite similar with the one shown before (Write Your Own Analyzers), but this code connects to 2 flow abstraction interfaces, which are for flows of HTTP proxy and re-injecting flows. As mentioned above, this code reads data and strips HTTP proxy’s header, and re-injects flows to the loopback7 interface.

// connect to socket
s1 = socket();
connect(s1, "/tmp/sf-tap/tcp/http_proxy");
s2 = socket();
connect(s2, "/tmp/sf-tap/loopback7");

function reinject(h, sid, buf, state) {
    if (state[sid][h["match"]] == HTTP_HEADER) {
        // return true, if read all HTTP header
        if (strip_http_header(buf[sid][h["match"]])) {
            state[sid][h["match"]] == HTTP_BODY;
        }
        
        if (state[sid][h["match"]] == HTTP_BODY) {
            header = gen_header(h, buf[sid][h["match"]].length);
            
            write(s2, header);
            write(s2, buf[sid][h["match"]]);
        }
    }
}

loop {
    // read header
    line = readline(s1);
    h = parse_header(line);
    sid = session_id(h);
    
    // read data
    if (h["event"] == CREATED) {
        state[sid]["up"] = HTTP_HEADER;
        state[sid]["from"] = HTTP_HEADER;
    } else if (h["event"] == DESTROYED) {
        state.remove(sid);
        buf.remove(sid);
    } else { // DATA
        read(s1, data, h["len"]);
        
        buf[sid][h["from"]].append(data);
        reinject(h, sid, buf, state);
    }
}

We are providing an example re-injector for HTTP proxy in Julia language. Please see it as a reference. (re-injector for HTTP proxy in Julia)

Header of Flow Abstraction Interface

The loopback7 interface accept headers as follows. Note that all headers must be ended with ‘\n’ (carriage return). The “hop” field is used for preventing infinite re-injection, and the value is internally incremented. Thus, you just do copy the original value to the re-injecting header.

CREATED event

The header of CREATED event must contain fields of “ip1”, “ip2”, “port1”, “port2”, “hop”, “l3”, “l4”,and “event”, where the value of “event” must be CREATED.

ip1=192.168.24.54,ip2=216.58.221.196,port1=59547,port2=80,hop=0,l3=ipv4,l4=tcp,event=CREATED\n

DESTROYED event

The header of DESTROYED event must contain fields of “ip1”, “ip2”, “port1”, “port2”, “hop”, “l3”, “l4”,and “event”, where the value of “event” must be DESTROYED.

ip1=192.168.24.54,ip2=216.58.221.196,port1=59547,port2=80,hop=0,l3=ipv4,l4=tcp,event=DESTROYED\n

DATA event

The header of CREATED event must also contain fields of “ip1”, “ip2”, “port1”, “port2”, “hop”, “l3”, “l4”,and “event”, where the value of “event” must be DATA. Furthermore, it must contain “from”, and “len” fileds. The “from” field indicates source of the data, and “len” field indicates the data length. The actual data follows after the DATA event header.

ip1=192.168.24.54,ip2=216.58.221.196,port1=59547,port2=80,hop=0,l3=ipv4,l4=tcp,event=DATA,from=2,len=494\n
494 bytes data is here

Load Balancing using Flow Abstraction Interface

Mon, 23 Nov 2015 05:13:00 +0000

Configuration for Load Balancing

SF-TAP flow abstractor provides a load balancing mechanism for application-level analyzers by using the flow abstraction interfaces. The flow abstraction interface can be divided into multiple interfaces by specifying the ‘balance’ field in the configuration file as follows.

http:
  up:      '^[-a-zA-Z]+ .+ HTTP/1\.(0\r?\n|1\r?\n([-a-zA-Z]+: .+\r?\n)+)'
  down:    '^HTTP/1\.[01] [1-9][0-9]{2} .+\r?\n'
  proto:   TCP
  if:      http
  format:  text
  body:    yes
  nice:    100
  balance: 4 # flows are balanced by 4 interfaces, must be power of 2

In this case, 4 HTTP interfaces are provided for HTTP protocol as follows.

$ ls /tmp/sf-tap/tcp
/tmp/sf-tap/tcp/http0=      /tmp/sf-tap/tcp/http2=
/tmp/sf-tap/tcp/http1=      /tmp/sf-tap/tcp/http3=

Effect of Multiple Processes

Thus, an HTTP analyzer can take advantage of multiple CPU cores by connecting load balancing interfaces as follows.

$ python3 http/sftap_http.py /tmp/sf-tap/tcp/http0

$ python3 http/sftap_http.py /tmp/sf-tap/tcp/http1

$ python3 http/sftap_http.py /tmp/sf-tap/tcp/http2

$ python3 http/sftap_http.py /tmp/sf-tap/tcp/http3

Following graphs show CPU loads of the HTTP analyzer with 1, 2, and 4 processes when generating 2500 HTTP requests per second. The analyzer could not handle the requests with 1 single process, but it could handle the requests with 2 or 4 processes.

Use Example HTTP and DNS Analyzers

Sat, 21 Nov 2015 10:01:00 +0000

Flow Abstraction Interfaces

SF-TAP flow abstractor provides flow abstraction interfaces by using UNIX domain socket. If you run it, there are files in a directory as follows.

$ cd /tmp/sf-tap
$ ls -R
loopback7=   tcp/         udp/

./tcp:
default=         http=            smtp=            torrent_tracker=
dns=             http_proxy=      ssh=             websocket=
ftp=             irc=             ssl=

./udp:
default=     dns=         torrent_dht=

Analyzers of SF-TAP connect to these interfaces and read flows to analyze. In this document, we show how to use example HTTP and DNS analyers provided by us.

Get Source

First of all, you must get source codes of the analyzers form GitHub.

$ git clone https://github.com/SF-TAP/protocol-parser.git
$ cd protocol-parser

HTTP Analyzer

An example HTTP analyzer written in Python3 connects a flow abstraction interface of HTTP, parses HTTP protocol, and outputs JSON to the standard output. You can execute it as follows.

$ python3 http/sftap_http.py /tmp/sf-tap/tcp/http nobody

Here, /tmp/sf-tap/tcp/http is a path to a flow abstraction interface of HTTP, and nobody is a flag specifying not to include HTTP body in outputs.

If you access to any HTTP server when running the HTTP analyzer, you can get information of HTTP access in JSON.

For example, if you connect to http://sf-tap.github.io/ in another prompt,

$ curl http://sf-tap.github.io/

, then you will get JSON data from the standard output as follows.

{
  "server":{
    "response":{
      "msg":"OK",
      "code":"200",
      "ver":"HTTP/1.1"
    },
    "ip":"103.245.222.133",
    "header":{
      "cache-control":"max-age=600",
      "content-length":"8418",
      "age":"0",
      "expires":"Sat, 21 Nov 2015 10:41:48 GMT",
      "date":"Sat, 21 Nov 2015 10:42:19 GMT",
      "x-fastly-request-id":"2fa8bb058315c7d938574efee8e564f03e4d5d08",
      "access-control-allow-origin":"*",
      "x-timer":"S1448102539.814963,VS0,VE180",
      "vary":"Accept-Encoding",
      "server":"GitHub.com",
      "x-github-request-id":"67F5E016:17E1:1286355:56504813",
      "x-cache":"HIT",
      "content-type":"text/html; charset=utf-8",
      "connection":"keep-alive",
      "via":"1.1 varnish",
      "accept-ranges":"bytes",
      "x-cache-hits":"1",
      "last-modified":"Sat, 21 Nov 2015 09:56:46 GMT",
      "x-served-by":"cache-itm7421-ITM"
    },
    "port":80
  },
  "client":{
    "method":{
      "method":"GET",
      "ver":"HTTP/1.1",
      "uri":"/"
    },
    "header":{
      "host":"sf-tap.github.io",
      "user-agent":"curl/7.43.0",
      "accept":"*/*"
    },
    "port":64755,
    "ip":"192.168.24.52"
  }
}

DNS Analyzer

An example DNS analyzer written in C++11 also analyzes DNS protocol and outputs JSON to the standard output. Note that this is only for UDP.

Build and Run DNS Analyzer

Build the DNS analyzer by using cmake and make.

$ cd protocol-parser/dns
$ cmake -DCMAKE_BUILD_TYPE=Release CMakeLists.txt
$ make

Then run it.

$ ./sftap_dns /tmp/sf-tap/udp/dns

If you send any DNS queries, you will get information of DNS access in JSON from the standard output.

For example, if you look up sf-tap.github.io in another prompt,

$ dig sf-tap.github.io

you will see DNS queries in JSON as follows.

{
  "src":{"ip":"192.168.24.52", "port":54459},
  "dst":{"ip":"192.168.24.1", "port":53},
  "id":60923, "qr":0, "op":0, "aa":0, "tc":0,
  "rd":1, "ra":0, "z":0, "ad":0, "cd":0, "rc":0,
  "query_count":1,
  "answer_count":0,
  "authority_count":0,
  "additional_count":0,
  "query":[{"name":"sf-tap.github.io", "type":"A", "class":1}],
  "answer":[],
  "authority":[],
  "additional":[]
}

{
  "src":{"ip":"192.168.24.1", "port":53},
  "dst":{"ip":"192.168.24.52", "port":54459},
  "id":60923, "qr":1, "op":0, "aa":0, "tc":0,
  "rd":1, "ra":1, "z":0, "ad":0, "cd":0, "rc":0,
  "query_count":1,
  "answer_count":2,
  "authority_count":4,
  "additional_count":4,
  "query":[{"name":"sf-tap.github.io", "type":"A", "class":1}],
  "answer":[
    {
      "name":"sf-tap.github.io", "type":"CNAME", "class":1,
      "ttl":3600, "cname":"github.map.fastly.net"
    },
    {
      "name":"github.map.fastly.net", "type":"A", "class":1,
      "ttl":30, "a":"103.245.222.133"
    }
  ],
  "authority":[
    {
      "name":"fastly.net", "type":"NS", "class":1,
      "ttl":71200, "ns":"ns1.p04.dynect.net"
    },
    {
      "name":"fastly.net", "type":"NS", "class":1,
      "ttl":71200, "ns":"ns4.p04.dynect.net"
    },
    {
      "name":"fastly.net", "type":"NS", "class":1,
      "ttl":71200, "ns":"ns2.p04.dynect.net"
    },
    {
      "name":"fastly.net", "type":"NS", "class":1,
      "ttl":71200, "ns":"ns3.p04.dynect.net"
    }
  ],
  "additional":[
    {
      "name":"ns1.p04.dynect.net", "type":"A", "class":1,
      "ttl":81318, "a":"208.78.70.4"
    },
    {
      "name":"ns2.p04.dynect.net", "type":"A", "class":1,
      "ttl":78428, "a":"204.13.250.4"
    },
    {
      "name":"ns3.p04.dynect.net", "type":"A", "class":1,
      "ttl":72003, "a":"208.78.71.4"
    },
    {
      "name":"ns4.p04.dynect.net", "type":"A", "class":1,
      "ttl":71749, "a":"204.13.251.4"
    }
  ]
}

Write Your Own Analyzers

Sat, 21 Nov 2015 09:39:00 +0000

How to Write Application Level Analyzers

In this document, we show how to implement analyzers for SF-TAP flow abstractor.

This tutorial is also available on SlideShare.

Tutorial of SF-TAP Flow Abstractor from Yuuki Takano

Flow Abstraction Interfaces

SF-TAP flow abstractor provdes flow abstraction interfaces by using UNIX domain socket. Thus, in order to capture flows (a.k.a. TCP streams), you must access to the interfaces. The directory structure of the interfaces is as follows.

$ cd /tmp/sf-tap
$ ls -R
loopback7=   tcp/         udp/

./tcp:
default=         http=            smtp=            torrent_tracker=
dns=             http_proxy=      ssh=             websocket=
ftp=             irc=             ssl=

./udp:
default=     dns=         torrent_dht=

Configuration File

Before explaining how to implement analyzers, we show a snippet of the configuration. For example, a configuration of HTTP is given as follows.

1 http:
2   up:     '^[-a-zA-Z]+ .+ HTTP/1\.(0\r?\n|1\r?\n([-a-zA-Z]+: .+\r?\n)+)'
3   down:   '^HTTP/1\.[01] [1-9][0-9]{2} .+\r?\n'
4   proto:  TCP  # TCP or UDP
5   if:     http # file name of UNIX domain socket
6   format: text # format of header. text or binary
7   body:   yes  # if specified 'no', only header is output
8   nice:   100  # the smaller a value is, the higher a priority is

Here, regular expressions for HTTP protocol are defined by ‘up’ and ‘down’ on line 2 and 3. Note that there are 2 regular expressions here because TCP is a connection oriented protocol. On the other hand, UDP protocols have only 1 regular expression as follows.

1 torrent_dht: # BitTorrent DHT
2   up:     '^d.*1:y1:[eqr].*e$'
3   proto:  UDP
4   if:     torrent_dht
5   format: text
6   nice:   100

Note that ‘down’ of UDP is meaningless.

TCP Event Abstraction

Actually, TCP handles many events (and many states), but SF-TAP flow abstractor abstracts these events as 3 events for simplicity. SF-TAP flow abstractor defines 3 events, which are CREATED, DATA, and DESTROYED events. CREATED event is invoked when TCP session is established. More precisely, it is invoked when finished a 3-way handshake and determined the application protocol by the regular expressions. DATA event is invoked when arriving data. DESTROYED event is invoked when closed TCP session because of some reasons, which are timeout, reset, and fin-close.

Protocol of Flow Abstraction Interface

The protocol on flow abstraction interfaces consists of chunks of header and data. Note that data is involved only if chunk’s event is DATA.

The header is ended \n (line feed), and denoted as follows.

ip1=192.168.24.54,ip2=216.58.221.196,port1=59547,port2=80,hop=0,l3=ipv4,l4=tcp,event=CREATED,time=1449325525.8732619\n

This is equivalent for

{
  "ip1":   "192.168.24.54",
  "ip2":   "216.58.221.196",
  "port1": 59547,
  "port2": 80,
  "hop":   0,
  "l3":    "ipv4",
  "l4":    "tcp",
  "event": "CREATED",
  "time":  1449325525.8732619
}

in JSON. When the event is CREATED or DESTROYED, the chunk does not involve data. Otherwise, when the event is DATA, the chunk involves data. The header of DATA event is denoted as follows.

ip1=192.168.24.54,ip2=216.58.221.196,port1=59547,port2=80,hop=0,l3=ipv4,l4=tcp,event=DATA,from=2,match=down,len=494,time=1449325525.8356969\n

In this case, the header indicates that data is coming from (ip2=216.58.221.196, port2=80) because the header denotes from=2. The entry of match=down indicates that the which regular expressions, provided in the configuration file shown above, is used for matching. Accordingly, you can determine whether the data is from server or client by the ‘match’ filed. The data length is indicated by the ‘len’ field. In this case, 494 bytes data follows after the header. The ‘time’ field indicates the rough timestamp, which is obtained by gettimeofday(), of the event. Thus, it is denoted in UNIX epoch time.

Since multiple flows are outputted to a single abstraction interface, it is required to identify chunks by flow identifiers. The SF-TAP flow abstractor identifies each flow by using 5-tuple as follows.

(ip1, port1, ip2, port2, hop)

The ‘hop’ filed is used for the loopback7 interface, and SF-TAP flow abstractor increments it internally. The loopback7 is used for re-injecting flows for encapsulated flows, such as HTTP proxy. We do not show more details of the loopback7 interface here.

Note that CREATED and DESTROYED events are not invoked when UDP protocols since UDP is not connection oriented. Thus, you need handle only DATA event for UDP protocols.

Pseudo Code for TCP Protocols

We give a pseudo code to analyze TCP protocols as follows.

// connect to socket
s = socket();
connect(s, "/tmp/sf-tap/tcp/http");

loop {
    // read header
    line = readline(s);
    h = parse_header(line);

    // generate session ID
    sid = new sessionID(h["ip1"], h["ip2"], h["port1"], h["port2"], h["hop"]);

    if (h["event"] == "DATA") {
        // data is arriving
        // h["from"] and h["match"] help to determine data origin
        read(s, buf, h["len"]);
    } else if (h["event"] == "CREATED") {
        // created TCP session
    } else if (h["event"] == "DESTROYED") {
        // destroyed TCP session
    }
}

In this code, first of all, we connect to the file of UNIX domain socket. Then we read a line, which is a header, from the socket. We parse the line to interpret the header, and generate the session ID from it. After that, we read data if the event is DATA where the data length is specified by ‘len’ field in the header.

The skeleton written in Python is available on GitHub Gist.

Pseudo Code for UDP Protocols

Then, we give a pseudo code to analyze UDP protocols as follows.

// connect to socket
s = socket();
connect(s, "/tmp/sf-tap/udp/dns");

loop {
    // read header
    line = readline(s);
    h = parse_header(line);

    if (h["event"] == "DATA") {
        // data is arriving
        // h["from"] help to determine data origin
        read(s, buf, h["len"]);
    }
}

Flow Separating, and L2 Mirroring and Bridging

Sat, 21 Nov 2015 09:22:00 +0000

Tutorial of SF-TAP cell incubator

Here, suppose that you have a following FreeBSD box, which has two 10 GbE (ix0 and ix1) and four 1 GbE (igb0, igb1, igb2 and igb3) interfaces.

Flow Based Separating

You can use qb-separator for traffic separating as follows, where -l and -t mean a prefix of “LEFT” and “TAP”, respectively. qb-separator must be executed with root privilege.

# ./qb-separator -l ix0 -t igb0,igb1,igb2,igb3

Here, qb-separator captures traffic from ix0, then separates and forwards it to igb[0-3]. Note that it separates traffic by using the hash values of IP addresses and port numbers of captured packets.

Mirroring

You can also use qb-tap for traffic mirroring as follows. qb-tap must be also executed with root privilege.

# ./qb-tap -l ix0 -t igb0,igb1,igb2,igb3

Here, qb-tap forwards all captured traffic from ix0 to igb[0-3].

L2 Bridging

You can use qb-separator or qb-tap as a simple software L2 bridge as follows, where -r means “RIGHT”.

# ./qb-separator -l ix0 -r ix1

L2 Bridging + Flow Based Separating

You can use qb-separator as a L2 bridge and a traffic separator as follows.

# ./qb-separator -l ix0 -r ix1 -t igb0,igb1,igb2,igb3

Here, qb-separator separates all traffic from ix0 and ix1, and forwards it to igb[0-3].

L2 Bridging + Mirroring

Similarly, qb-tap can work as an L2 bridge and a traffic mirroring box as follows.

# ./qb-tap -l ix0 -r ix1 -t igb0,igb1,igb2,igb3

Here, all traffic caputured from ix0 and ix1 is forwarded to igb[0-3].

Install SF-TAP Cell Incubator on FreeBSD

Sat, 21 Nov 2015 08:22:00 +0000

FreeBSD Version

This document is for FreeBSD 10.1, 10.2 and 10.3.

Build netmap Enabled Kernel

First of all, you need build netmap enabled kernel to use SF-TAP cell incubator. Edit the kernel config as follows to build the kernel.

# cd /usr/src/sys/amd64/conf
# cp GENERIC GENERIC.netmap
# vi GENERIC.netmap

device netmap # add device of netmap

Then, compile and install the kernel.

# cd /usr/src
# make -j 30 buildkernel KERNCONF=GENERIC.netmap
# make installkernel KERNCONF=GENERIC.netmap

And reboot.

# reboot

Finally, confirm that netmap is enabled.

# ls /dev/netmap
/dev/netmap

Install Dependencies

Install git to get the source code.

# pkg install git-x.x.x

Build Cell Incubator

Clone the source code form GitHub.

$ git clone https://github.com/SF-TAP/sf-incubator.git

Then, build it.

$ cd sf-incubator/src
$ make

Run

Before running SF-TAP cell incubator, disable offload engines of NICs. The shell script for disabling the engines for all NICs is included in the repository. Thus, you need just execute the script as follows.

# ./misc/ifcap_disable_freebsd.sh

Finally, you can run SF-TAP cell incubator as follows.

# ./qb-separator -r ix0 -t igb0,igb1,igb2,igb3

In this case, ix0 is a NIC to capture network traffic, and igb[0-3] are NICs to which separated flows are forwarded.

Increase Buffer Size for netmap (optional)

If you want to use many interfaces, increase the buffer size for netmap by sysctl.

# sysctl -w dev.netmap.if_size=2048
# sysctl -w dev.netmap.if_num=200
# sysctl -w dev.netmap.ring_size=73728
# sysctl -w dev.netmap.ring_num=400
# sysctl -w dev.netmap.buf_size=2048
# sysctl -w dev.netmap.buf_num=300000

Install SF-TAP Flow Abstractor on Ubuntu Linux

Sat, 21 Nov 2015 08:00:00 +0000

Ubuntu Version

This document is for Ubuntu 14.10, 15.04, 15.10, 16.04, and 16.10.

Install Dependencies

First of all, install build-essential, cmake, git, libevent-dev, libboost-all-dev, libpcap-dev, libre2-dev, libyaml-cpp-dev.

$ sudo apt-get install build-essential cmake git libevent-dev \
libboost-all-dev libpcap-dev libre2-dev libyaml-cpp-dev

Build and Run SF-TAP Flow Abstractor

Get Source

Then, clone the source of SF-TAP flow abstractor from GitHub.

$ git clone https://github.com/SF-TAP/flow-abstractor.git

Environment Variables

Before compiling, set environment variables for cmake as follows.

$ export CMAKE_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib
$ export CMAKE_INCLUDE_PATH=/usr/include:/usr/local/include

Build

Build by cmake and make.

$ cd flow-abstractor
$ cmake -DCMAKE_BUILD_TYPE=Release CMakeLists.txt
$ make

If you get an eorror regarding language locale, install suitable launguage pack.

$ sudo apt-get install language-pack-ja

Use netmap

If you want to use netmap, set an option of USE_NETMAP=1 to cmake.

$ cmake -DUSE_NETMAP=1 CMakeLists.txt
$ make

, and pass -n option when executing as follows.

$ ./src/sftap_fabs -i eth0 -c ./examples/fabs.yaml -n

Run

Run SF-TAP flow abstractor specifying a network interface and a config file.

$ sudo ./src/sftap_fabs -i eth0 -c ./examples/fabs.yaml

Welcome to Jekyll!

Fri, 20 Nov 2015 20:58:33 +0000

You’ll find this post in your _posts directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run jekyll serve, which launches a web server and auto-regenerates your site when a file is updated.

To add new posts, simply add a file in the _posts directory that follows the convention YYYY-MM-DD-name-of-post.ext and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.

Jekyll also offers powerful support for code snippets:

def print_hi(name)
  puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.

Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo. If you have questions, you can ask them on Jekyll Talk.