IP Reputation: Noting the Obvious

I mentioned previously about using a local IP reputation database for help when doing packet analysis.  It's handy to have to lookup the reputation of a source or destination IP address, especially if your database contains reputation data from various sources.  Last night I decided to use that data in a blacklist with snort's reputation preprocessor.  Today I'm going to remove that blacklist and I'll tell you why.

I was so ensnared by the idea of having a blacklist to block traffic to sites with bad reputations, that I forgot one fundamental aspect...the fact that many sites share the same IP address.  An example that I can think of would be shared hosting, where something like Apache virtualhosts are used.  In this scenario, each website might share a single public IP address. 

This means that if one of those sites is hosting malware files that you can download, then the reputation for the single public IP address will be bad, even if the other sites that are hosted on that IP address are clean and not hosting any malware or other baddies.

Because I didn't want to block traffic to legitimate sites, I turned on the preprocessor and enabled the requisite GID 136 rules to alert only and to not actually stop any of the traffic.

After a while, I had thousands of alerts.  Spot checking many of the alerts, I noticed that they did not appear to have any untoward content.  I looked them up on clean-mx.de and realized that it was other sites hosted on those IP addresses that contained malicious content, not the ones that were being visited from my network.

If I had chosen to drop or block that traffic, then I would have stopped communications with legitimate sites that just so happen to be hosted on the same IP as a malicious site. 

If you're extremely security conscious and do not want to have any affiliations with malicious sites, then maybe you would want to block traffic to any IP's having any type of malicious content.  However, I don't think it is fair to punish other websites because they chose the wrong hosting provider, or simply got stuck on the same IP address as a site hosting malware.

Snort Inline with AFPacket DAQ

I decided to put my snort instance into inline mode this past weekend.  It had been running for a while in passive mode, getting fed traffic from a SPAN port on my switch.  I had been running it this way so that I could do some initial tuning.  I disabled rules which don't apply to my environment, set thresholds for other rules, etc.

I placed it inline just inside my firewall.  So it is positioned on the network between my perimeter firewall and my internal network switch.  It will be able to inspect the traffic going out of my internal network to the Internet as well as the traffic coming into my internal network.

I thought for sure that when I switched to inline that I wouldn't have any performance issues.  I thought wrong.

I changed my startup script to start snort with these options:  "snort -Q -i eth1:eth2 --daq afpacket -c /etc/snort/snort.conf"

Everything seemed to work well.  I could create drop rules that were effective and I could even block access from my internal network to certain web addresses.  The problem came when I tried to use speedtest.net.  I noticed it was very slow.  I was getting less than a megabit per second, even when choosing a nearby test server.

I tried disabling all of my rules in snort, disabling inline normalization, and nothing seemed to fix the problem.  I stopped the snort instance and manually created a bridge for eth1 and eth2 using brctl.  Without snort running, this worked fine and gave me the expected results.  That told me that there wasn't a problem with the hardware or the way I have everything networked.  All signs were pointing to a problem with snort.

Last night I finally discovered the problem.  My snort instance is running as a KVM virtual machine on Proxmox.  When I set up the machine, I had chosen to use the virtio network drivers.  Changing this to the e1000 (I have Intel NICs in the physical host) driver fixed the bandwidth problem when using snort inline.  Now I'm getting the expected results when using speedtest.net.

I guess when snort automatically bridges your inline interfaces, it has some performance issues with the virtio drivers.  Either that or it is just treating them like some kind of generic driver, hence the performance problems even though it is technically able to bridge the ports and use them.

IDS/IPS User Annoyances

As you know, I use snort as my IDS product of choice at home.  Sometimes I see questions from other people who use snort that just strike me as being dumb questions.  Now I understand if you were suddenly told by your boss, "Guess what?  You're the new person in charge of our IDS".  You have my sympathy.  There will obviously be  a learning curve for you if you've never managed an IDS or IPS.  The people I don't have sympathy for are the ones who have fancy titles like Sr. Security Analyst or Senior Network Engineer, who ask the dumb questions.  These people should just know better. 

Below is a list of things I thought of off the top of my head that these people should know:

Place your snort sensor inside your firewall.  Put it as close to the same network segment that you want to monitor as possible.  Don't put it outside your perimeter firewall.  You don't need to care about everyone who is knocking at your door.  If you want to keep them out, that's why you have a firewall.  If you place the sensor outside the firewall, you will get so many alerts that you won't be able to manage them.

Which brings me to my next point...Snort is not a firewall.  Please don't treat it like one.  Before you ask yourself if snort can block certain traffic, the better question would be to ask if your firewall can block it before it even makes it inside your network.  So if you want to do something like rate-limit certain traffic coming from outside to your internal network, do it at your firewall. 

Tune your IDS/IPS for your environment.  An IDS isn't a set it and forget it type of device.  Yes, you should go through all the rules and turn off the ones that do not apply to your environment.  If you don't have time, make time.  It's that important to the performance of your sensor.  If you're not running any Windows machines, then turn off all the rules that apply to Windows hosts.

Don't write rules for every piece of malware under the sun.  Let your antivirus software do its job.  You do have antivirus software, right?

Do not enable the portscan preprocessor.  It will affect performance and gives you very little in terms of value.  So what if someone ran a portscan?  A portscan is not an exploit.  It is not even necessarily a precursor to an exploit.  And trust me, someone is always port scanning at your perimeter.  Know that and move on.

Using snort to block access to particular websites.  Again, this is a task that is better suited for your firewall or proxy.

Don't expect sympathy if you are concerned about running a rule for a 10 year old exploit that only affects certain older versions of software.  You had 10 years to move off of that software version.   Just because you chose not to does not make it my problem.  An IDS isn't designed to replace your patching procedure.  It's designed to buy you the time you need to get your software patched.  Once the software is patched you should turn off that specific rule because you don't need it any longer.

I'm sure that given enough time I could probably write a book with tips like the above.  Right now, however, I'm a little short on time and think that what I've said so far is a good start for anyone who manages an intrusion system.

Snort: Flow-IP Statistics Parser

I recently enabled flow-ip statistics in my snort.conf by editing the perfmon preprocessor line like so:

preprocessor perfmonitor: time 300 file /var/snort/snort.stats pktcnt 10000 flow-ip-file /var/snort/ipflow.csv flow-ip

As you can see, I am logging snort performance data to /var/snort/snort.stats every 5 minutes or 10000 packets.  In addition, I'm logging the flow-ip data to /var/snort/ipflow.csv.

The flow-ip data can be used to help identify the top talkers on your network and is useful for troubleshooting performance issues, such as CPU spikes, in snort.  But because the output is contained in a CSV file, it isn't very easy to read.

I wrote a small perl script that uses bash commands, such as awk, to make it a little easier to read.  The script adds together the total TCP bytes from Host A with the total TCP bytes from Host B and outputs the top 10 in descending order by the total TCP bytes sent between the two hosts.

It does not look for unique hosts, though.  And it does the calculations per line in the CSV.  So if your CSV file contains multiple lines with the same hosts, and they all happen to have more TCP traffic than the other lines in the CSV, then those hosts will be listed multiple times in the output, sorted only by differences in the total TCP bytes transferred between them.

Feel free to modify and use the code any way you see fit.  You can execute the code by saving it to a file, such as flowipparser.pl and then calling it with your CSV file as an argument from the command line:  ./flowipparser.pl ipflow.csv

The output to the console will contain three columns:  Host A IP Address, Host B IP Address, and the total TCP bytes transferred between the two hosts.

Here's the code:

#!/usr/bin/perl -w

$INPUTFILE="$ARGV[0]";

system qq(awk -F "," '{print \$1, \$2, \$4, \$6}' $INPUTFILE | sort -k3n,3 | sort -r -n -k3 | head | awk -F " " '{ print \$1, \$2, sum=\$3+\$4 }' | sort -r -n -k3 );