The Best Firewall Software of 2008: Online Armor
The decision is in. After a year and a half of testing, and with the help of more than a thousand Scot’s Newsletter readers who’ve written detailed descriptions of their software firewall experiences, I’m happy to announce that Tall Emu’s Online Armor 2.1 is The Scot’s Newsletter Blog Best Firewall Software of 2008.
There are many reasons why I’ve selected Online Armor (OA) as the best software firewall for Windows users; the rest of this story delivers the details. But boiled down to a single thought, the most important reason is this: Online Armor offers the best blend of a high degree of protection with a high level of usability.
That may sound simplistic, but in this software category such a balance is the toughest thing for a software development company to achieve. It’s very easy to throw up a blizzard of pop-up user-prompts. You can make your system so secure that you’ll never want to use it again. It’s also easy to dumb down the security so much that you’ll rarely, if ever, see a pop up — and in the process, render the firewall ineffective. The trick is to offer solid protection with minimal user interruptions. OA 2.1 is the only firewall software I’ve tested that delivers a near-perfect balance.
Online Armor firewall comes in two editions: free and paid. Version number 188.8.131.52 was the latest one tested for both editions. [Editor’s Note: As of 4/19/2008, Online Armor’s latest version is 184.108.40.206.] Tall Emu updates the product frequently; to check on the latest versions of OA and read the release notes, see this Tall Emu support-forum post. In addition to the two different editions of Online Armor, Tall Emu also packages it with an antivirus module. For information about the differences among Online Armor Free, Online Armor (paid), and Online Armor AV+, see Tall Emu’s Online Armor Comparison page.
This review is specific to the paid version of Online Armor, which costs about $40. I’ve extensively tested both the free and the paid versions, and both work well. But it’s the paid version that I prefer and recommend (for reasons I’ll detail further along). Online Armor AV+ has not been tested for this evaluation. It contains the Kaspersky antivirus engine, which, while a good product, is not as good as Eset’s NOD32 2.7. Because I named NOD32 2.7 the Best Antivirus Product of 2007, I have tested Online Armor extensively with NOD32 running. I’ve experienced zero incompatibility issues between OA and NOD32 2.7. (Note: I’m still using and recommending NOD32 version 2.7, not the newer 3.0 version. Version 2.7 is still available from Eset.)
The second place Comodo Firewall Pro 3.0 software from the Comodo Group is also a very good product. The latest version tested for this review was 3.0.020.320. If your overriding concern is security, security, security, and you don’t mind a less-than-ideal user experience, Comodo is worthy of consideration. Its superb security ratings and great configurability make it well suited to more experienced users who prefer a belt-and-suspenders approach. This is not, though, the firewall to install on your mother’s PC. Comodo also comes in both 32-bit and 64-bit Vista versions. Comodo Firewall Pro is free. The Comodo Group is working on several features and functionalities that it believes will markedly improve Comodo usability, so this is also a product to keep an eye on.
Security Testing and Gating Criteria
This evaluation kicked off in September 2006 as a series review (long-term testing with progress reports). I have written many firewall articles during this period about my gating criteria, interim findings, products I tested, and reasons why specific products were eliminated from the running. To review that information, please visit SNB’s Firewall category archive page. By scrolling, you’ll find every installment I’ve written for the Best Firewall series over the past 19 months. Among other things, you’ll discover the reasons why I eliminated Outpost, ZoneAlarm, Sunbelt Personal Firewall (a.k.a. Kerio), and Kaspersky’s firewall (part of a suite). Each of these firewalls was a strong contender, but each had a fatal flaw that eliminated it from contention. The companies that make them could rectify those issues, but have not done so to date.
When it was first established, this evaluation used the results of FirewallLeakTester.com’s tests as a method of screening out lesser-performing firewalls. Later in the process, I switched to Matousec’s more in-depth and more regularly updated results. Matousec has recently updated its test results; Comodo gets the highest score, with Online Armor placing second. Corroborating my test results of past year, Matousec scores Eset Smart Security’s leak-protection level as “none.”
I have also performed a set of my own security tests on Online Armor 2.1, Comodo 3, and some of the other firewalls I considered along the way. The latest versions of Online Armor 2.1 and Comodo 3 offer superb protection when used properly. (Most importantly: In both products, the HIPS module must be enabled.) Both firewalls have received significant security improvements over the past six months, too. Earlier versions were not as secure.
Most of my research, however, has focused on usability, company support, stability, compatibility, and bug resolution. These are the areas that make the difference between a security product that you rely on and one you use until you find something better. Too many people are in limbo with products like this, just tolerating them at best. The goal of this research has from the start been selecting security products that you can live with, perhaps even love.
Why Programs Were or Weren’t Tested
The impetus for this review came after more than a decade of using and reviewing multifaceted, everything-but-the-kitchen-sink security suites such as Norton Internet Security. When I kicked that habit, I looked around for something better and realized that most mainstream computer publications were for the most part reviewing only the big-name, large-footprint products. It was clear to me that there was a better way that involved selecting a small set of best-of-breed security products that work well together. So my first determination was that fat security-suite products need not apply. Many of the other gating criteria spring from that decision.
This evaluation assumes that the software firewall is running behind a hardware router or broadband “modem” that offers network address translation (NAT) and stateful packet inspection (SPI), or in other words, a hardware firewall. For home use, consumer-class wired or wireless hardware firewall routers are available from D-Link, Linksys, and Netgear that are for security purposes comparable. Even if you do not have a network, I recommend that you purchase this low-cost hardware. If you have a wireless network, you should also be running password-enabled WPA encryption with a password that isn’t easy to guess.
Finally, over the long term of this evaluation, many new firewall products emerged. It was not possible to test all of them, and in some cases I relied on the input of Scot’s Newsletter readers to help me vet products. The review was also closed to new entrants late last year while I focused on the two finalists: Online Armor 2.1 and Comodo 3.0.
With those points in mind, these are the gating criteria used to determine the Best Firewall Software of 2008:
- Very low system overhead with a strong preference for stand-alone software — no full-blown security suites
- Full compatibility with effective third-party stand-alone security products from other software categories
- Excellent inbound and outbound security protection with an emphasis on solid leak protection, as prescreened by Matousec.com
- A simple, informative, configurable, and highly usable user interface
- Software that is reliable and as bug-free as possible
- Backed by a software development company that is stable, communicative, responsive to customer issues, and actively developing the product. As with any security product, the company behind it should have something to lose — its reputation — if it doesn’t properly stand behind and update the product. It also needs a strong, responsive development team whose development process emphasizes bug fixing and customer experience, not hurrying the product out the door to meet arbitrary deadlines.
- Quiet operation; alerts you when there are real problems. Excessive or repetitive warnings or pop-ups aren’t acceptable.
- Protects but doesn’t cause intermittent problems with Windows local-area network functionality
- A feature that lets users rapidly shut down all inbound and outbound activity
- Vista support, while not mandatory, is preferred. (Note: Online Armor does not yet have a Vista version, but it’s under development.)
Comodo 3: The Next-Best Thing
Comodo Firewall, from the Comodo Group, is a full-fledged software firewall that is free to download and use. Comodo has strong pluses and minuses. The 3.0 upgrade was highly ambitious and was not adequately beta tested. The result was a long series of incremental updates following the release of Comodo 3 — at least six updates over the past six months or so. For details about the releases, including what’s in them, check out Comodo’s Release Notes page. The good news is that Comodo is being actively updated.
The Comodo 3 software has a lot to offer. It comes with a server-based whitelist for its HIPS (host-intrusion-prevention system) module, called Defense+, whose purpose is to cut back on pop-ups. The product also offers an operational mode called Clean PC that, at your option, scans all your current applications and then registers them as safe. That means fewer pop-ups for you, especially in the early going. I also prefer the functionality of Comodo’s “install mode” to those of most other firewalls. It is capable of disabling several types of pop-ups for about 15 minutes in an attempt to let you complete a new program installation in peace. When the 15 minutes expire, it prompts you to turn off the install mode to reinstate full protection. The only problem with Comodo’s install mode is that figuring out how turn it on may not be immediately obvious to the average Comodo user.
At its core, Comodo 3 is a highly protective software firewall that takes itself seriously. Its primary design criterion appears to be that great security requires the program to ask the user to approve or deny any and all actions that might possibly be caused by something malicious. I can’t disagree with that thinking in principle — assuming the people running computers know enough to make the right decisions. Because many of them don’t, Comodo is trying very hard to minimize pop-ups with its whitelist, install mode, and initial hard drive scan. The company also has other features in the works (not evident in this build of its software) that aim to improve usability by reducing pop-ups and improving the software’s ability to detect threats.
Even so, Comodo 3’s Defense+ experience is not ideal. In the kind of usage scenario where several programs are downloaded each week, Comodo users are likely to experience a lot of pop-ups. If you don’t install new applications very often, my personal experience has been that Comodo settles in and the operation of the HIPS becomes less intrusive. It is, though, noticeably noisier than Online Armor’s HIPS protection. It also doesn’t appear to remember user inputs quite as well as the OA HIPS does.
The Main Difference
The primary reason why Comodo Firewall didn’t take top honors in this review is that it errs on the side of protection at the expense of usability. Comodo’s protection takes it a bit beyond the bounds of acceptable usability — a subjective determination on my part. In a nutshell, it has too many pop-ups in this release. And even though it is able to “learn” to have fewer pop-ups and can also be controlled by settings, both the initial and the long-term user experiences are diminished by this behavior.
For example, I was recently confronted with over a dozen pop-ups when I left Comodo running in memory while choosing to uninstall it from the Add or Remove Programs control panel. At least one user prompt is requisite in this scenario because otherwise, a malware routine could be written to uninstall or disable the firewall. You must approve anything that disables your firewall, even when you initiate that action yourself. From a security perspective, there’s a sound argument to be made for more than one pop-up, since most software products are made up of multiple modules that might be selectively turned off to create specific vulnerabilities. But a dozen pop-ups is well beyond the tolerable level in my book.
In another instance, when I directed Windows to install a single Windows Update patch, I was immediately faced with a pop-up — an acceptable experience. I did everything I could in that first prompt window to make Comodo trust the process that was running. But the software firewall nevertheless prompted me with 11 additional pop-ups before that one patch was installed. Windows Update (update.exe) should be a trusted app. I realize that the executable might be spoofed, but if a user validates it, Comodo should learn to be quiet after that trust is confirmed — without having to figure out Install Mode.
It may sound counterintuitive that I’m preferring a balance of usability and security over pedal-to-the-metal security. There’s an important reason for that: When pop-ups are too repetitive or too frequent, it’s only human nature for a large segment of the user base to start ignoring them. That behavior leads to a severe loss of security.
The build of Comodo I tested to wrap up this review, 220.127.116.110, has benefitted from the the long series of bug-fix updates since 3.0 was introduced. According to the company, most of the initial incremental updates were aimed at solving unexpected problems when running Comodo 3 on Vista, support for which was added for the first time in Comodo 3. But many Scot’s Newsletter Blog readers who use Windows XP also emailed me descriptions of problems with the first three incremental updates to Comodo 3.
Meanwhile, even though Comodo 2.4 was something of a cult favorite, it’s absolutely true that a wide range of people experienced significant trouble with that firewall too. So for a period of time, Comodo users were stuck between a rock and a hard place. Many of them tried version 3 and returned to version 2.4. Others wrote me that they left for other firewalls. But the period of disturbance settled down, and I’m no longer receiving email after email with tales of woe.
What that tells me is that Comodo 3 is a good firewall product, potentially a great one, that quite possibly was shipped to end users without adequate QA testing. As is always the case with free, publicly available software, some early adopters were ill-equipped to handle the problems they encountered. Most of those issues appear to have been fixed now. Comodo 3 was also an ambitious release, and bugs happen. But this kind of management of a development process does not inspire confidence — especially when it’s the type of product that can wreak havoc on your computer.
If the Comodo team can focus on software quality, and if it can add additional functionality that pares back on pop-ups, future updates of Comodo 3 could improve the overall usability of the firewall markedly. Solid protection plus good usability is a winning combination. For now, Comodo 3 misses on the usability front — the main reason it has come in second in this review. But because Vista compatibility is a Comodo 3 strength, for the time being at least, it’s the firewall I recommend to Vista users.
The Top Dog: Online Armor 2.1
Online Armor was the late entrant in this evaluation. A bevy of readers suggested it last fall after Matousec gave it a 100% security rating in an earlier version of its test suite. (Comodo received the same top score.) Since I began testing it and calling for input on it, the most common sentiment I’ve heard from people who try it is: “I like it.” Even people who’ve had issues with it have said that. And that’s been my reaction too.
Online Armor’s user experience is on par with ZoneAlarm Free and Sunbelt Personal Firewall — the two firewalls I’ve pointed to in the past as having the best user interfaces in this field. It’s also a relatively young product that is being intensively developed by its makers. OA’s basic UI is very solid, very easy to figure out without help. But the simple interface sometimes lures you away from finding some of the power that lies beneath. OA relies a little too heavily on context menus for access to power features. As you use this product, try right-clicking things. Somewhere down the road Tall Emu should add a column to many of its config screens with a link reading something like “options” or “configure” that opens the context menu. That would be more discoverable. Still, this is a minor issue. All in all, I’m very happy with OA 2.1’s usability.
Several new features debuted in the significant Online Armor 18.104.22.168 update released February 19, 2008, including a resizable main program window, improved on-demand system scan, install mode, and multiple network detection and management.
Version 22.214.171.124 also added a useful convenience feature to the Run Safer capability of OA’s Program Guard. Run Safer let’s you force Internet-connected programs — such as your Web browser, email, and IM package — to run with reduced Windows user-account rights, giving you added protection from malware. The new feature is a context-menu item that lets you temporarily run a Run-Safer-restricted program in a normal (or admin-level) mode.
The OA facility called Autoruns (Startup Items), which gives you a user interface for managing and controlling applications and services that launch automatically on Windows boot, has also been extended to watch additional aspects of the operating system.
The firewall’s Computers tab offers a network-access monitor that shows all the computers connected to your machine via your network. Available details include IP address, MAC address, computer name, and gateway IP address. You can right-click any of the other computers you see and direct the firewall not to trust it.
Probably the most improved aspect of Online Armor beginning with its 126.96.36.199 version is the online-accessible database of program information, which Tall Emu calls OASIS (Online Armor Software Information Service). The company has committed additional resources to keeping this database updated. As it has grown and become more fleshed out over the past several weeks, OASIS has become more useful. The main benefit of the online app database is evident on OA pop-up windows that display the “More…” link. By clicking this link, you’ll get useful information that identifies the program or process that initiated the pop-up — which can be a big help in deciding whether to block or allow the action. You can also get this information by working the context menus in the Programs area, which displays all the programs on your system. And Tall Emu expects to surface this data in other ways too.
The single most important point of failure with most firewalls is user error — usually involving the wrong decision on a pop-up dialog. It’s absolutely essential for firewalls to help educate users about programs running on their PCs. The time has long since past when firewall makers could reasonably expect users to already possess the knowledge to make these decisions. So it was an excellent decision by Tall Emu to make this change.
Tall Emu offers this list of product features on its website that will help you get up to speed on the program. This list doesn’t cover some of the recent improvements.
One of Online Armor’s very best attributes isn’t a feature or functionality; it’s the people behind the product. Tall Emu’s CEO, Mike Nash, is the most visible person behind OA. He posts frequently in the OA support forums. What’s especially impressive about the talk and actions emanating from Australia-based Tall Emu is a strong corporate culture that values communication, honesty, a willingness to talk openly about problems, a responsive attitude, open-mindedness, and respect. I’m not sure how to say this, but I trust Tall Emu to do the right thing. I can’t remember the last time I felt that way about a software company in the post-Microsoft-antitrust era.
Getting back to the tangible, for the last month or two I’ve been directly aware — from emails written to me by SNB readers, OA forum posts, and emails from Mike Nash — of two or three serious issues with the most recent major Online Armor release (initially 188.8.131.52). Most bugs happen to only a small percentage of the overall users of a software product. I didn’t experience any of these more notable issues — in fact, probably most people didn’t. The point I’m trying to get at is this: I’ve been impressed with the transparency and alacrity with which Tall Emu attacks and resolves such problems. This nastier class of bugs, the worst of which is an occasional but recurring crash of Windows Explorer, have all been identified and fixed. (The fix for the Windows Explorer bug is being tested and should be released shortly.)
No product is perfect, and that’s probably more true of software firewalls than many other types of software. Online Armor has bugs just like all of its competitors. It’s what happens when problems are identified that distinguishes development teams. What I’ve seen from Tall Emu is that they do it the right way.
What about the free version of Online Armor? It’s very good. The most important aspects of firewall and HIPS protection are in there. But the paid version offers several additional security layers that are easily worth the $39.95 price of admission.
There’s also a somewhat controversial limitation of the free version: It doesn’t automatically update with new versions of Online Armor. In other words, to install a new version of Online Armor Free you must uninstall the old version and then install the new version. No big deal you say? Not quite. That also means you should go through the initial setup wizard and then, to get through all the pain, launch and trust your most-often-used applications.
Online Armor (paid) can automatically download and install version updates. So, yes, this is something Tall Emu has done purposely to incent you to pay for the full version.
This decade has seen a dramatic rise of free software, but people don’t dedicate themselves full-time to a project like Online Armor without having to eat and do other expensive things. I urge all those of you who can afford the $40 to pay it — in fact, I urge you pay for all the “free” programs you use regularly.
Finally, for Vista users, a new version of Online Armor developed for Vista is very close to being released in an initial public beta test. It could take a couple of months, or longer, for Tall Emu to work through the bugs and deliver a final Vista version. As I wrote earlier in this story, use Comodo until then. When Online Armor for Vista ships, I will give it a look and post something about it.
Online Armor 2.1
.0.112 (the paid version) is the best firewall I’ve ever tested, offering a blend of usability and hard-wired security that’s near-ideal for maximizing protection and ensuring a good user experience. A great firewall doesn’t have to be, and shouldn’t be, a chore to use. Online Armor isn’t.
A year and a half after launching this quest, naming OA the Best Firewall Software of 2008 came naturally. The very best products have a way of standing out.