One month ago I bought a cubieboard to have a small, powerfull, unexpensive, low noise, low power consumption linux server at home, for my nerd projects.
The first project I started ? Run a NetFlow collector at home for my Ninux nodes.
The Cubieboard
The cubieboard is a small ARM pc. With 1Ghz processor and 1Gb of RAM is very flexible for any application. I bought it from miniand.com with the developer kit and I also got from Amazon a USB power adapter and a SD memory card. A few weeks after the new Cubieboard A20 came out ! So I suggest you get the new one even if it cost some dollars more, because it has a dual core CPU.
Cubieboard di ZioProto
For this project I installed the ARCH Linux distribution on the Cubieboard. To install it on the ARM Architecture you should go to this specific page: Arch Linux ARM. I did not need any graphical interface, and I found Arch Linux the best distribution for the CubieBoard, after I tried many different ones.
Netflow
What is Netflow ? Netflow is a network protocol spoken among the probes and a collector. The probes are running on your routers, and sniff all the traffic flows traversing the router. When the probes collected enough bits of information, this is sent via the netflow protocol to the collector, that collects the data from all the probes. You can learn more about netflow on Wikipedia.
The Nfsen Server
Nfsen is the software we are going to use as NetFlow collector. The official web page gives you detailed information. You can use any Linux box and with Apache and PHP. Keep in mind the nfsen server must be online 24h per day, this is why I choose the cubieboard. The Arch Wiki is really well done and installing apache and php was painless.
pacman -Sy apache php php-apache
Make sure you have a ntp client running that gives to the system the proper date and time, and your timezone is set correctly both on you system and on your /etc/php/php.ini file. This is very important or nfsen will not work correctly later.
Make sure you also uncomment the extension=sockets.so statement in the php.ini file because this is required by nfsen.
You can check your php settings creating a info.php file somewhere in the webserver root folder /srv/http/info.php with the following content:
<?php phpinfo(); ?>
And then just visit http://youripaddress/info.php
Now that you have Linux with Apache and PHP running, you need to compile and install nfdump and nfsen. First install a package called base-devel that gives you all the necessary tools to compile directly on the cubieboard. You will also need perl and rrdtool:
pacman -Sy base-devel perl rrdtool
First compile the nfdump suite, I downloaded the tarball nfdump-1.6.10.tar.gz from the nfdump web site.
I compiled from sources configuring as following:
./configure --enable-nfprofile --enable-nftrack --enable-readpcap --enable-nfpcapd make make install
Then you have to download the nfsen tarball nfsen-1.3.6p1.tar.gz After unpacking the tarball you will find in the “etc” folder a file nfsen-dist.conf. Copy this file to a new file called nfsen.conf in the same folder and make your changes. I did very little change from the original file:
[root@alarm nfsen-1.3.6p1]# diff -Naur etc/nfsen-dist.conf /data/nfsen/etc/nfsen.conf
--- etc/nfsen-dist.conf 2012-01-14 11:13:53.000000000 +0100
+++ /data/nfsen/etc/nfsen.conf 2013-06-26 14:47:32.567806501 +0200
@@ -36,7 +36,7 @@
# NfSen html pages directory:
# All php scripts will be installed here.
# URL: Entry point for nfsen: http://<webserver>/nfsen/nfsen.php
-$HTMLDIR = "/var/www/nfsen/";
+$HTMLDIR = "/srv/http/nfsen/";
#
# Where to install the docs
@@ -88,12 +88,12 @@
# This may be a different or the same uid than your web server.
# Note: This user must be in group $WWWGROUP, otherwise nfcapd
# is not able to write data files!
-$USER = "netflow";
+$USER = "http";
# user and group of the web server process
# All netflow processing will be done with this user
-$WWWUSER = "www";
-$WWWGROUP = "www";
+$WWWUSER = "http";
+$WWWGROUP = "http";
# Receive buffer size for nfcapd - see man page nfcapd(1)
$BUFFLEN = 200000;
@@ -160,9 +160,14 @@
# Ident strings must be 1 to 19 characters long only, containing characters [a-zA-Z0-9_].
%sources = (
- 'upstream1' => { 'port' => '9995', 'col' => '#0000ff', 'type' => 'netflow' },
- 'peer1' => { 'port' => '9996', 'IP' => '172.16.17.18' },
- 'peer2' => { 'port' => '9996', 'IP' => '172.16.17.19' },
+ 'NBGalliaNazza' => { 'port' => '10000', 'col' => '#0000ff', 'type' => 'netflow' , 'IP' => '10.183.1.10'},
+ 'NBGalliaBramante' => { 'port' => '10000', 'col' => '#ff0000', 'type' => 'netflow' , 'IP' => '10.183.1.11'},
+ 'Conbipel' => { 'port' => '10000', 'col' => '#ddeeee', 'type' => 'netflow' , 'IP' => '10.145.0.1'},
+ 'M5ConbipelGarib' => { 'port' => '10000', 'col' => '#bbbbcc', 'type' => 'netflow' , 'IP' => '172.16.145.5'},
+ 'M5ConbipelCruto' => { 'port' => '10000', 'col' => '#ffbbcc', 'type' => 'netflow' , 'IP' => '172.16.145.6'},
+ 'Gallia' => { 'port' => '10000', 'col' => '#ee0000', 'type' => 'netflow' , 'IP' => '10.183.1.1'},
+ 'TuscoloZioProto' => { 'port' => '10000', 'col' => '#7fff00', 'type' => 'netflow' , 'IP' => '192.168.3.214'},
);
#
[root@alarm nfsen-1.3.6p1]#
when you finished the configuration you can install nfsen using the installation perl script. The script will tell you about any missing dependency you still have to fix:
./install.pl etc/nfsen.conf
after installation the necessary web files are in /srv/http/nfsen/. Point the browser to nfsen.php to start. You will find everything else in the folder /data/ and you can start nfsen to start to collect data doing:
/data/nfsen/bin/nfsen start
if you add new netflow probes in your network at a later time, these are called sources in the nfsen configuration file. After changing the configuration you should use the reconfig command, start and stop of nfsen is not enough.
/data/nfsen/bin/nfsen reconfig
make sure you read all the documentation for a complete reference.
The NetFlow probes
The routers where I wanted to install the probes are running OpenWRT Attitude Adjustment. I tried three different netflow probes: fprobe-ulog, fprobe and softflowd.
fprobe-ulog is the most lightweight probe. It works together with the iptables ulog target to log packets from kernel space to a userspace application. Basically you match the packets you want to log with a iptables rule that sends the information to the userspace application, that speaks the netflow protocol with the collector:
iptables -I FORWARD -o wlan0 -j ULOG --ulog-cprange 48 --ulog-qthreshold 50 iptables -I FORWARD -i wlan0 -j ULOG --ulog-cprange 48 --ulog-qthreshold 50 iptables -I FORWARD -o eth0 -j ULOG --ulog-cprange 48 --ulog-qthreshold 50 iptables -I FORWARD -i eth0 -j ULOG --ulog-cprange 48 --ulog-qthreshold 50 fprobe-ulog -Xeth0:0,wlan0:1 10.183.1.5:10000
because there is not a ULOG target for ip6tables you will not be able to export IPv6 flows using fprobe-ulog, unless in the future a ULOG target will be available. Make sure you dont have ulogd running, if you do it might suck up all the available RAM matching the IP packets from the iptables ULOG target. On devices with 32MB of RAM you have to be carefull.
fprobe is the same as fprobe-ulog but it is libpcap based. There is no need for iptables rules. However also with the libpcap fprobe I was not able to export IPv6 flows.
The last probe I tried was softflowd. This probe is libpcap based and it is the only one which worked for me, exporting both IPv4 and IPv6 flows. The current version available on OpenWRT was outdated and not working, so I upgraded the OpenWRT package and compiled the latest version. I sent my patch upstream to the OpenWrt developers, (patch also available here), and the binary package is for target ar71xx is here. You can install the probe like this:
opkg install http://stud.netgroup.uniroma2.it/~saverio/softflowd_0.9.9-1_ar71xx.ipk
So I just started the softflowd probe like this:
softflowd -i wlan0 -n 10.183.1.5:10000 -v 9 -6
At this point wait some minutes and you will start to see data on your collector. I just put one graph here to give you an idea, but a complete set of screenshots is available on the nfsen website.
grafico_nfsen
as a final step I secured my nfsen installation with username and password using a htaccess file. To do this you can find a quick note on the Ninux Wiki.
I suggest if you are in the Ninux network to get yourself a Cubieboard, or some similar board, and play with network applications 🙂
Saverio
