I still use Base as a web frontend to my snort installation. I know a lot of people are using things like Snorby now, but I think Base does everything I need it to do. Anyway, I was looking at Base this afternoon and I noticed over 200 new alerts.
All of the alerts were from my main router and they were of the type "ICMP Test". Closer examination showed that the router was trying to ping a machine that was unreachable. Since my router also acts as my DNS and DHCP server, I checked the syslog on that machine.
The syslog was full of DHCP offers to the same IP address that snort was showing as unreachable. I took the MAC address and ran it through an online MAC to vendor lookup and it showed me that it was a MAC from Motorola CHS. I went through the house restarting all of my Motorola cable boxes. Since doing that I noticed that the DHCP log shows that an acknowledgment was sent in response to the DHCP offer. Snort has also stopped alerting for that particular ICMP Test.
I guess one of the cable boxes just got hung up a bit. It happens from time to time. Usually I don't catch the problem until it is too late (e.g. my favorite TV shows aren't recording as scheduled in the DVR). Thanks to Snort and Base, that won't be a problem tonight.
Friday, May 4, 2012
NTFSclone
I installed ntfsprogs on my Debian desktop because I have a Windows partition that I'd like to create an image of on my NAS. I ran ntfsclone with the --save-image option and directed it to place the output in an NFS share to my NAS. I started it last night and it's almost 60% of the way finished.
My lessons learned are as follows:
- Software RAID sucks. I should probably spring for a decent hardware RAID controller.
- Consumer hard disks also suck for large file copies like this one. Those cheap Western Digital disks in my NAS may have seemed like a great deal, but they just don't compare to higher-end SCSI disks. The IO Wait is what's causing it to take so long. It was over 50% when I last looked at it on the NAS.
I should really invest in better equipment at home :)
Update: The ntfsclone imaging finally finished. It turns out that I may have tracked down another culprit relating to the slow file transfer and the high iowait. I have a 3-disk RAID 5 array in my Openfiler NAS. Running mdadm -D /dev/md0 showed that one of the disks was faulty. I rebooted the NAS and re-added that disk to the RAID array. Right now it is in the process of rebuilding, so I'll have to wait a while to see how that goes. Even if it comes back online okay, I'll still probably order an extra disk to add to the array as a spare.
My lessons learned are as follows:
- Software RAID sucks. I should probably spring for a decent hardware RAID controller.
- Consumer hard disks also suck for large file copies like this one. Those cheap Western Digital disks in my NAS may have seemed like a great deal, but they just don't compare to higher-end SCSI disks. The IO Wait is what's causing it to take so long. It was over 50% when I last looked at it on the NAS.
I should really invest in better equipment at home :)
Update: The ntfsclone imaging finally finished. It turns out that I may have tracked down another culprit relating to the slow file transfer and the high iowait. I have a 3-disk RAID 5 array in my Openfiler NAS. Running mdadm -D /dev/md0 showed that one of the disks was faulty. I rebooted the NAS and re-added that disk to the RAID array. Right now it is in the process of rebuilding, so I'll have to wait a while to see how that goes. Even if it comes back online okay, I'll still probably order an extra disk to add to the array as a spare.
Snort: Sensitive Data
Man, I'll tell you, the sensitive data processor in snort was not designed to be used with web traffic. If you've used it at all, you know it fires all the time when used in conjunction with regular web traffic. If seems to throw alerts for detecting email addresses if it so much as finds the '@' symbol in a packet. Any string of numbers in a packet makes it alert for finding supposed credit card numbers.
Since in my current setup snort is processing all packets sent from my router, I'm going to have to disable sensitive data processing. I guess if I was only monitoring traffic from my internal network, then there would be fewer of these alerts. And then, the alerts I do get would probably at least be worth taking a look at. Right now, though, I'm just getting flooded with false positives.
Since in my current setup snort is processing all packets sent from my router, I'm going to have to disable sensitive data processing. I guess if I was only monitoring traffic from my internal network, then there would be fewer of these alerts. And then, the alerts I do get would probably at least be worth taking a look at. Right now, though, I'm just getting flooded with false positives.
Saturday, April 28, 2012
Underscore in Hostname
I had to add this line to my Bind9 configuration for my home DNS server:
Had to do this mainly because I was getting crap like "bad owner name (check-names)" in my syslog everytime my Android phone tried to join the network. Apparently, DNS doesn't like special characters in hostnames. It seems Android phones tend to have an underscore character in their hostnames. The annoying thing about this is that you can't change the hostname on a non-rooted phone.
But anyway, using the line up above in your named.conf file will stop your syslog from adding these alerts. I think it's probably not the best thing to do, but until my phone uses a normal hostname, I have to do it to avoid sifting through tons of useless syslog alerts.
Interestingly enough, before adding that to clean up the log, Snort was also firing each time my phone joined the network, citing it as a possible Bind9 DoS attempt. So even if I wasn't checking my logs regularly, snort would have at least let me know that I should pay closer attention to my phone.
check-names master warn;
Had to do this mainly because I was getting crap like "bad owner name (check-names)" in my syslog everytime my Android phone tried to join the network. Apparently, DNS doesn't like special characters in hostnames. It seems Android phones tend to have an underscore character in their hostnames. The annoying thing about this is that you can't change the hostname on a non-rooted phone.
But anyway, using the line up above in your named.conf file will stop your syslog from adding these alerts. I think it's probably not the best thing to do, but until my phone uses a normal hostname, I have to do it to avoid sifting through tons of useless syslog alerts.
Interestingly enough, before adding that to clean up the log, Snort was also firing each time my phone joined the network, citing it as a possible Bind9 DoS attempt. So even if I wasn't checking my logs regularly, snort would have at least let me know that I should pay closer attention to my phone.
IPtables Port Spanning -- Sort of
I installed Snort on a Debian virtual machine in my home test lab. I thought it would be fun to learn more about it. What better way to learn than to just start using it?
I pretty much just followed one of the guides available at snort.org to get snort installed with PulledPork for downloading updated rulesets. I've actually installed snort before, but it was several years ago and on Slackware instead of Debian. I have noticed some things that are new to snort since the last time I installed it. Things like the sensitive data preprocessor. No, SDP, you're wrong. None of those web packets contain CCN's or SSN's. I actually disabled SDP in the end. If you just comment out the sensitive data preprocessor line in snort.conf and the sensitive-data-rules line, you'll probably find that snort will die often and complain about something with "sd_pattern". Just look in the snort.rules file and comment out the alert lines containing "sd_pattern". That should solve that problem. Unless of course, you really do need to look for sensitive data. In that case, don't turn it off. But do get ready for loads of false positives.
Having an IDS installed is not fun if you can't get packets for it to process, though. I don't really care about chatter inside my network between the different machines. I just wanted to process any packets sent or received through my router (machine running Debian and iptables).
I own a switch, not a hub, so plugging in the physical network connection used by the snort VM won't do any good. And my switch is a cheap, unmanaged switch. So I can't do a span port on it, either.
Then I learned about the TEE target in iptables. It will copy the packets passing through the firewall and forward them over the network to another machine. Yes, it's probably not the most secure method in the world, but it was exactly what I needed for my home network. Kind of like a simulated span port.
However, it was a pain to get going on Debian stable. In fact, you have to use xtables-addons along with iptables. But it's not easy to get working correctly because of the versions of xtables-addons and iptables in the stable repos. Instead I just ended up upgrading my router to Debian testing. It was the easy way out. That way everything is a new version and I got a nice shiny new 3.x kernel. Once you get your system to recognize the TEE target and the --gateway option, you can add something like this to your iptables rules:
Now that that's over, I can get back to configuring snort. Any traffic coming into or out of my network through my router will also get copied over to snort for processing. Yay!
I pretty much just followed one of the guides available at snort.org to get snort installed with PulledPork for downloading updated rulesets. I've actually installed snort before, but it was several years ago and on Slackware instead of Debian. I have noticed some things that are new to snort since the last time I installed it. Things like the sensitive data preprocessor. No, SDP, you're wrong. None of those web packets contain CCN's or SSN's. I actually disabled SDP in the end. If you just comment out the sensitive data preprocessor line in snort.conf and the sensitive-data-rules line, you'll probably find that snort will die often and complain about something with "sd_pattern". Just look in the snort.rules file and comment out the alert lines containing "sd_pattern". That should solve that problem. Unless of course, you really do need to look for sensitive data. In that case, don't turn it off. But do get ready for loads of false positives.
Having an IDS installed is not fun if you can't get packets for it to process, though. I don't really care about chatter inside my network between the different machines. I just wanted to process any packets sent or received through my router (machine running Debian and iptables).
I own a switch, not a hub, so plugging in the physical network connection used by the snort VM won't do any good. And my switch is a cheap, unmanaged switch. So I can't do a span port on it, either.
Then I learned about the TEE target in iptables. It will copy the packets passing through the firewall and forward them over the network to another machine. Yes, it's probably not the most secure method in the world, but it was exactly what I needed for my home network. Kind of like a simulated span port.
However, it was a pain to get going on Debian stable. In fact, you have to use xtables-addons along with iptables. But it's not easy to get working correctly because of the versions of xtables-addons and iptables in the stable repos. Instead I just ended up upgrading my router to Debian testing. It was the easy way out. That way everything is a new version and I got a nice shiny new 3.x kernel. Once you get your system to recognize the TEE target and the --gateway option, you can add something like this to your iptables rules:
iptables -t mangle -A PREROUTING -j TEE --gateway x.x.x.x(where x.x.x.x is the IP of the machine you want to receive the packets.)
Now that that's over, I can get back to configuring snort. Any traffic coming into or out of my network through my router will also get copied over to snort for processing. Yay!
Saturday, March 31, 2012
Optical Character Recognition with Linux
My company currently uses ABBYY products to perform OCR on PDF documents. I thought it would be fun to see what open source alternatives existed out there for this purpose. After a little bit of searching, I found that most people seemed to agree that Tesseract is one of the more accurate OCR programs available that is also open source. I decided to give it a try on a Debian virtual machine.
I first tried using Tesseract on a TIFF file with cursive handwriting. This failed miserably and just gave me a bunch of garbage as text output. Maybe if I put forth the time and effort to "train" Tesseract I could get this to work somewhat. But then again, everybody's cursive handwriting is different, so I can't ever see getting this to work reliably.
I then tried a PDF document with typed text. Tesseract wouldn't even read the PDF, as is. I first had to convert it into an image file (I chose TIFF, again). This worked much better. I'd say it was probably about 90 - 95% accurate. I then tried the same PDF in Windows using ABBYY Corporate Edition version 10. ABBYY had much better results. In fact, it was almost 100% accurate. I have to give the point to ABBYY on this one.
Finally, I tried scanning a gas station receipt and OCR-ing it. Using Tesseract produced more garbage. This time, though, ABBYY also produced garbage. Granted, though, this receipt was very crumpled and the text on the receipt was printed very lightly. I had to strain my eyes to read it myself, so I can't blame the two pieces of software for not being able to properly OCR it.
The fact is that I don't really have any personal uses for OCR software. That makes it difficult for me to think up more testing scenarios. My results seem to show that if I needed to OCR documents for a business, I'd probably put my trust in the commercial ABBYY product line. The Tesseract project does show a lot of promise, though. Tesseract did seem to perform much faster and use far less resources than ABBYY. With some improved out-of-the-box accuracy, my recommendation could certainly shift in favor of Tesseract.
I first tried using Tesseract on a TIFF file with cursive handwriting. This failed miserably and just gave me a bunch of garbage as text output. Maybe if I put forth the time and effort to "train" Tesseract I could get this to work somewhat. But then again, everybody's cursive handwriting is different, so I can't ever see getting this to work reliably.
I then tried a PDF document with typed text. Tesseract wouldn't even read the PDF, as is. I first had to convert it into an image file (I chose TIFF, again). This worked much better. I'd say it was probably about 90 - 95% accurate. I then tried the same PDF in Windows using ABBYY Corporate Edition version 10. ABBYY had much better results. In fact, it was almost 100% accurate. I have to give the point to ABBYY on this one.
Finally, I tried scanning a gas station receipt and OCR-ing it. Using Tesseract produced more garbage. This time, though, ABBYY also produced garbage. Granted, though, this receipt was very crumpled and the text on the receipt was printed very lightly. I had to strain my eyes to read it myself, so I can't blame the two pieces of software for not being able to properly OCR it.
The fact is that I don't really have any personal uses for OCR software. That makes it difficult for me to think up more testing scenarios. My results seem to show that if I needed to OCR documents for a business, I'd probably put my trust in the commercial ABBYY product line. The Tesseract project does show a lot of promise, though. Tesseract did seem to perform much faster and use far less resources than ABBYY. With some improved out-of-the-box accuracy, my recommendation could certainly shift in favor of Tesseract.
Subscribe to:
Posts (Atom)