Saturday, March 31, 2012
Optical Character Recognition with Linux
My company currently uses ABBYY products to perform OCR on PDF documents. I thought it would be fun to see what open source alternatives existed out there for this purpose. After a little bit of searching, I found that most people seemed to agree that Tesseract is one of the more accurate OCR programs available that is also open source. I decided to give it a try on a Debian virtual machine.
I first tried using Tesseract on a TIFF file with cursive handwriting. This failed miserably and just gave me a bunch of garbage as text output. Maybe if I put forth the time and effort to "train" Tesseract I could get this to work somewhat. But then again, everybody's cursive handwriting is different, so I can't ever see getting this to work reliably.
I then tried a PDF document with typed text. Tesseract wouldn't even read the PDF, as is. I first had to convert it into an image file (I chose TIFF, again). This worked much better. I'd say it was probably about 90 - 95% accurate. I then tried the same PDF in Windows using ABBYY Corporate Edition version 10. ABBYY had much better results. In fact, it was almost 100% accurate. I have to give the point to ABBYY on this one.
Finally, I tried scanning a gas station receipt and OCR-ing it. Using Tesseract produced more garbage. This time, though, ABBYY also produced garbage. Granted, though, this receipt was very crumpled and the text on the receipt was printed very lightly. I had to strain my eyes to read it myself, so I can't blame the two pieces of software for not being able to properly OCR it.
The fact is that I don't really have any personal uses for OCR software. That makes it difficult for me to think up more testing scenarios. My results seem to show that if I needed to OCR documents for a business, I'd probably put my trust in the commercial ABBYY product line. The Tesseract project does show a lot of promise, though. Tesseract did seem to perform much faster and use far less resources than ABBYY. With some improved out-of-the-box accuracy, my recommendation could certainly shift in favor of Tesseract.
I first tried using Tesseract on a TIFF file with cursive handwriting. This failed miserably and just gave me a bunch of garbage as text output. Maybe if I put forth the time and effort to "train" Tesseract I could get this to work somewhat. But then again, everybody's cursive handwriting is different, so I can't ever see getting this to work reliably.
I then tried a PDF document with typed text. Tesseract wouldn't even read the PDF, as is. I first had to convert it into an image file (I chose TIFF, again). This worked much better. I'd say it was probably about 90 - 95% accurate. I then tried the same PDF in Windows using ABBYY Corporate Edition version 10. ABBYY had much better results. In fact, it was almost 100% accurate. I have to give the point to ABBYY on this one.
Finally, I tried scanning a gas station receipt and OCR-ing it. Using Tesseract produced more garbage. This time, though, ABBYY also produced garbage. Granted, though, this receipt was very crumpled and the text on the receipt was printed very lightly. I had to strain my eyes to read it myself, so I can't blame the two pieces of software for not being able to properly OCR it.
The fact is that I don't really have any personal uses for OCR software. That makes it difficult for me to think up more testing scenarios. My results seem to show that if I needed to OCR documents for a business, I'd probably put my trust in the commercial ABBYY product line. The Tesseract project does show a lot of promise, though. Tesseract did seem to perform much faster and use far less resources than ABBYY. With some improved out-of-the-box accuracy, my recommendation could certainly shift in favor of Tesseract.
Saturday, March 17, 2012
Ethernet Auto-Negotiation
I've probably mentioned this before, but I'm surprised how many people I encounter that still don't get how auto-negotiation works. The easiest way to spot network administrators who just don't get it is to analyze their network for duplex mismatches. And you usually don't end up looking for those until you have users complaining about "slowness" over the network.
The problem is that most people think auto-negotiation will work if they just set the network interface in their computer to use it. This will not guarantee a successful negotiation of the correct duplex setting. That is because auto-negotiation requires each end of the connection to be set for auto-negotiation. That means that in addition to the NIC in the computer, the switch that it is connected to must also be set for auto-negotiation. This is defined as part of the IEEE 802.3 standard for Ethernet connections.
The standard says that if only one end of the connection is set for auto-negotiation, the speed (100Mbps, 1Gbps, etc..) will negotiate fine, but the duplex setting will become half-duplex. If you want to see how you are negotiated from your PC, then you would need a tool from your NIC manufacturer. Some manufacturers like Broadcom do offer such tools that you can install, but not all manufacturers will provide such a tool.
A more complex example would be using small, unmanaged switches with managed switches. Those cheap unmanaged switches usually do not allow you to specify the duplex setting for each port. Therefore, they use auto-negotiation by default. So if you connect that to a port on a managed switch that is explicitly set for, let's say, full-duplex, then you will have a duplex-mismatch resulting in a half-duplex connection. In this case, even if you checked the connection with a tool from a workstation connected to the unmanaged switch, you would not get correct results. The tool would most likely show a full-duplex connection. That is because the connection from the PC to the unmanaged switch may be fine. The issue would be with the backbone connection from the unmanaged switch to the managed switch.
The easiest remedy for these types of duplex mismatches is to simply always use auto-negotiation on all of your devices. Almost all modern devices are going to use auto-neg by default. So anything you purchase and connect to your network will always negotiate the proper duplex as long as all of your network devices are configured for auto-negotiation.
Remember, it takes two for auto-negotiation over Ethernet networks.
The problem is that most people think auto-negotiation will work if they just set the network interface in their computer to use it. This will not guarantee a successful negotiation of the correct duplex setting. That is because auto-negotiation requires each end of the connection to be set for auto-negotiation. That means that in addition to the NIC in the computer, the switch that it is connected to must also be set for auto-negotiation. This is defined as part of the IEEE 802.3 standard for Ethernet connections.
The standard says that if only one end of the connection is set for auto-negotiation, the speed (100Mbps, 1Gbps, etc..) will negotiate fine, but the duplex setting will become half-duplex. If you want to see how you are negotiated from your PC, then you would need a tool from your NIC manufacturer. Some manufacturers like Broadcom do offer such tools that you can install, but not all manufacturers will provide such a tool.
A more complex example would be using small, unmanaged switches with managed switches. Those cheap unmanaged switches usually do not allow you to specify the duplex setting for each port. Therefore, they use auto-negotiation by default. So if you connect that to a port on a managed switch that is explicitly set for, let's say, full-duplex, then you will have a duplex-mismatch resulting in a half-duplex connection. In this case, even if you checked the connection with a tool from a workstation connected to the unmanaged switch, you would not get correct results. The tool would most likely show a full-duplex connection. That is because the connection from the PC to the unmanaged switch may be fine. The issue would be with the backbone connection from the unmanaged switch to the managed switch.
The easiest remedy for these types of duplex mismatches is to simply always use auto-negotiation on all of your devices. Almost all modern devices are going to use auto-neg by default. So anything you purchase and connect to your network will always negotiate the proper duplex as long as all of your network devices are configured for auto-negotiation.
Remember, it takes two for auto-negotiation over Ethernet networks.
Wednesday, March 7, 2012
Overheard Conversation at Work
I just had to share this with the world. I was just sitting at my desk and caught the tail end of a conversation between two coworkers. They were talking about the disk configuration in a server.
Person #1 -- How are the disks configured?
Person #2 -- When data is written to one drive, it is also written to another drive.
Person #1 -- Okay, the disks are mirrored.
Person #2 -- No, it isn't mirrored. It's RAID-ed.
I wasn't a part of the conversation, so maybe Person #2 meant something else. But he just kept describing disk mirroring. While there are RAID levels for mirroring, it can also be called just that...mirroring. One isn't right and the other wrong.
You really had to be there. It was just how Person #2 was talking down to Person #1 acting like saying "mirroring" in this situation was completely wrong. Could you imagine if someone was telling you that, let's say, RAID1 does not equal "disk mirroring"?
In this case, I think the person had just learned the term "RAID" and was trying to shun any other words to describe what his pretty new word was describing.
Person #1 -- How are the disks configured?
Person #2 -- When data is written to one drive, it is also written to another drive.
Person #1 -- Okay, the disks are mirrored.
Person #2 -- No, it isn't mirrored. It's RAID-ed.
I wasn't a part of the conversation, so maybe Person #2 meant something else. But he just kept describing disk mirroring. While there are RAID levels for mirroring, it can also be called just that...mirroring. One isn't right and the other wrong.
You really had to be there. It was just how Person #2 was talking down to Person #1 acting like saying "mirroring" in this situation was completely wrong. Could you imagine if someone was telling you that, let's say, RAID1 does not equal "disk mirroring"?
In this case, I think the person had just learned the term "RAID" and was trying to shun any other words to describe what his pretty new word was describing.
Sunday, March 4, 2012
Debian Wheezy & GDM3 :: Disable User List
I wanted to disable the user list that shows on the GDM3 greeter. You know, where you can point and click on a username and then enter the password. I don't find this very secure, since knowing the username is half the battle for someone breaking into your computer. And Jesus F-ing Christ was it hard to disable! Hey, Gnome devs, please dear God, start working on some better system settings tools. Stuff like this should be easier. I mean, I can put up with this whole one desktop environment to rule them all (PCs, tablets, and phones) mentality that seems to be infecting the industry right now. I'd even say that things like Unity and Gnome 3 are growing on me the more I use them. But even Microsoft still gives me a bunch of Control Panel applets with the Windows 8 Consumer Preview. There are almost no important settings which can be modified in Gnome 3 or Unity by using a point and click method. If I can do it by editing a file or running a command on a server, then great. But at the end of the day, I'd like to simply point and click on my desktop system.
Sorry for the rant. Back to the problem at hand...
At first I had tried things like adding a line to /etc/gdm3/greeter.gconf-defaults to disable the user list, as is documented all over the Internet to do so. This did not work.
I finally found my solution on the Linux Mint forum. The command I had to run was sudo gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.mandatory --type Boolean --set /apps/gdm/simple-greeter/disable_user_list True.
Sorry for the rant. Back to the problem at hand...
At first I had tried things like adding a line to /etc/gdm3/greeter.gconf-defaults to disable the user list, as is documented all over the Internet to do so. This did not work.
I finally found my solution on the Linux Mint forum. The command I had to run was sudo gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.mandatory --type Boolean --set /apps/gdm/simple-greeter/disable_user_list True.
Saturday, March 3, 2012
Debian :: Upgrade Stable to Wheezy
I've been running Debian Stable on my laptop and desktop at home for a couple of months now. Even though I thought I could live with fairly out of date software in the repositories, I decided last night that I couldn't. So I set out to upgrade to Wheezy (Testing).
I simply changed every mention of "stable" to "testing" in my /etc/apt/sources.list. Then I ran sudo aptitude update and sudo aptitude dist-upgrade. The second command would exit every once in a while, so I'd have to run it again. I just accepted whatever it told me it needed to do. I kept running it until it finally didn't show any packages that needed to be upgraded. At first, I had tried using sudo apt-get dist-upgrad, but after a while I couldn't get it to run anymore -- it kept showing unresolvable dependency errors. Aptitude worked fine, though. After it was done, I crossed my fingers and rebooted.
On the desktop, after the reboot, I was greeted with a text login instead of with gdm3. I ran sudo apt-get install gdm3 gnome-core. Rebooted again, and still no graphical login.
I thought the issue might have been related to my NVIDIA drivers (I have a GTS 250 video card), so I uninstalled all the nvidia driver stuff that I had installed. After that I tried installing the binary driver from NVIDIA's website. It just SIGTERM-ed after accepting the license agreement.
I re-installed all the nvidia dkms stuff and then deleted any xorg.conf that I had in /etc/X11. Then I created a directory called /etc/X11/xorg.conf.d and (as root -- not with sudo) ran the command echo -e 'Section "Device"\n\tIdentifier "My GPU"\n\tDriver "nvidia"\nEndSection' > /etc/X11/xorg.conf.d/20-nvidia.conf.
This time after restarting, I got my gdm3 login screen that I was hoping for. I logged in and everything seemed to be in order. I brought up the dash and decided to type in the name of some software to search for. As soon as I hit a key, the dash would crash. This happened every time I tried to type something into the dash's search box. I was able to fix that by clearing the contents of a file and then making the file immutable with the chattr command: echo > ~/.local/share/recently-used.xbel && sudo chattr +i ~/.local/share/recently-used.xbel.
None of my NFS mounts were mounting as specified in /etc/fstab, and I discovered I needed to install nfs-common again using apt-get. I also installed gnome-tweak-tool so I could have some more control over the Gnome environment. One of the big things I used the tweak tool for was adding minimize and maximize buttons to windows.
Now the only problem I have is that the Software Center doesn't open for me. Well, the window opens and it looks like it's trying to do something for a minute, but then just leaves me with a blank window. I'm not too terribly concerned about it because I can still use aptitude, apt-get, and synaptic to install software. It would be nice if it still worked, but I'm betting it will be fixed in a future update. Either that or I'll end up fixing it myself or just re-installing at some point.
I simply changed every mention of "stable" to "testing" in my /etc/apt/sources.list. Then I ran sudo aptitude update and sudo aptitude dist-upgrade. The second command would exit every once in a while, so I'd have to run it again. I just accepted whatever it told me it needed to do. I kept running it until it finally didn't show any packages that needed to be upgraded. At first, I had tried using sudo apt-get dist-upgrad, but after a while I couldn't get it to run anymore -- it kept showing unresolvable dependency errors. Aptitude worked fine, though. After it was done, I crossed my fingers and rebooted.
On the desktop, after the reboot, I was greeted with a text login instead of with gdm3. I ran sudo apt-get install gdm3 gnome-core. Rebooted again, and still no graphical login.
I thought the issue might have been related to my NVIDIA drivers (I have a GTS 250 video card), so I uninstalled all the nvidia driver stuff that I had installed. After that I tried installing the binary driver from NVIDIA's website. It just SIGTERM-ed after accepting the license agreement.
I re-installed all the nvidia dkms stuff and then deleted any xorg.conf that I had in /etc/X11. Then I created a directory called /etc/X11/xorg.conf.d and (as root -- not with sudo) ran the command echo -e 'Section "Device"\n\tIdentifier "My GPU"\n\tDriver "nvidia"\nEndSection' > /etc/X11/xorg.conf.d/20-nvidia.conf.
This time after restarting, I got my gdm3 login screen that I was hoping for. I logged in and everything seemed to be in order. I brought up the dash and decided to type in the name of some software to search for. As soon as I hit a key, the dash would crash. This happened every time I tried to type something into the dash's search box. I was able to fix that by clearing the contents of a file and then making the file immutable with the chattr command: echo > ~/.local/share/recently-used.xbel && sudo chattr +i ~/.local/share/recently-used.xbel.
None of my NFS mounts were mounting as specified in /etc/fstab, and I discovered I needed to install nfs-common again using apt-get. I also installed gnome-tweak-tool so I could have some more control over the Gnome environment. One of the big things I used the tweak tool for was adding minimize and maximize buttons to windows.
Now the only problem I have is that the Software Center doesn't open for me. Well, the window opens and it looks like it's trying to do something for a minute, but then just leaves me with a blank window. I'm not too terribly concerned about it because I can still use aptitude, apt-get, and synaptic to install software. It would be nice if it still worked, but I'm betting it will be fixed in a future update. Either that or I'll end up fixing it myself or just re-installing at some point.
Subscribe to:
Posts (Atom)