Digging In
This is the third time we've built out a logging environment for a SIM bake-off, and the experience allowed us to avoid some pitfalls (see our previous reviews: Security Information Management Tools: NetForensics Leads a Weary Fleet and Too Much Information ).
For starters, we knew that using a combination of syslog-ng with a UDP syslog relayer would reduce the number of steps required for troubleshooting (see "How We Tested SIM Products,"). For the uninitiated, syslog-ng is an open-source package that was created by Balazs Scheidler as a more functional replacement to the original Unix syslog daemon (see Balabit.com). Syslog-ng uses the syslog protocol, but it offers a healthy amount of additional functionality--for example, support for syslog over TCP--and has gained immense popularity, in part for its flexibility when routing and retransmitting syslog messages.
By implementing syslog-ng and sending all our log data to a relay host we ensured all the SIM products received the exact same data, and verified that device data was being received. We didn't want to get stuck in any finger-pointing contests with vendors claiming their devices weren't receiving logs.
On the transportation front, we faced an all too familiar dilemma: Either place agents on all our log sources to move data off them securely, or go the classic, unreliable route of using syslog over UDP (see "Stuck on Syslog"). In the case of appliances, such as Cisco PIX firewalls, we didn't have a choice--you can't run agents on a PIX. Grudgingly, we opted to go with syslog because we couldn't stomach the idea of trying to deploy agents everywhere. Had we gone the agent route, however, ArcSight ESM would have been our best option because its collectors not only encrypt traffic, they also perform bandwidth throttling and batch transfers. These features come in handy when dealing with remote sites and limited WAN connectivity. In really large organizations, we could see the advantage of going agentless in some areas and using agents in others. We're skeptical of vendors that claim one model is superior; there's a time and place for each.
On the storage front, our first order of business was assessing our log volume sizes (see "The Windows Logging Headache"). We doubted that we'd hit the multi-terabyte ceilings that plague many SIM deployments, but we wanted to be certain. Once we had all of our devices speaking syslog, we brought a syslog-ng Linux system online to serve as a temporary repository. After a few weeks of log collections we could estimate our average weekly log volume sizes. We had fewer than 40 devices generating log data (though some were quite chatty). They delivered between 40 and 60 events per second on average, about 3 GB of data a week.
Organizations should go through this exercise to understand how much data they'll have. We've spoken to organizations that generate more than 1 TB a month. Knowing how much data you'll have and how long to keep it, and understanding how much to keep online versus offline are critical to a successful SIM deployment. We knew going into this review that 500 GB to 1 TB of back-end storage would suit our needs just fine.
Also critical is an understanding of the limitations of back-end systems. Network Intelligence and SenSage have some advantages in this department, as they've moved away from conventional relational database technology to more proprietary warehousing mechanisms (see "Is RDBMS Bad in the SIM World?" ). The avoidance of expensive table reindexing and removal of unnecessary RDBMS features, such as record locking, could lead to much better performance when getting into multi-terabytes of data. Given our environment's size, we didn't have performance problems with any of the products we used.
Once we had the SIM products installed and data flowing freely, we turned our attention to operational needs. Our requirements revolved around two action areas: monitoring a select number of our critical assets and using investigation and query tools to determine if events ID'd by our monitoring efforts were real threats.
Monitoring capabilities are heavily dependent on correlation capabilities, especially when you're dealing with hundreds of events per second. If the SIM can't summarize and dismiss the vast majority of events coming into your console, you're fighting a losing battle. By analyzing information from firewalls, IDSs, authentication services and system hosts, good correlation rules can help identify what's of concern and what isn't. On the correlation front, ArcSight ESM's rule sets are the most powerful, but Q1 Labs QRadar's correlation logic is by far the easiest to use. Network Intelligence's enVision and Symantec's Security Information Manager 9500 also can perform a respectable amount of real-time correlation. By comparison, we found the prospect of authoring rules with SenSage's Enterprise Security Analytics falling somewhere between extremely painful and impossible; the product has a long way to go in the ease-of-use department.
For our tests we designed several custom correlation rules (see chart at left). Only a select number of systems and services in our lab are accessible from the outside world, for example, and those openings in our firewalls represent our primary attack surface; we keep a sharp eye on them at all times. One rule we devised was to cross-reference firewall-allow statements with successful authentication sessions. One potential sign of a successful service attack would be an inbound connection that didn't result in a normal user session. Building this rule took a little time, but we succeeded with products from ArcSight, Q1 Labs and Symantec. We struggled with all of the others, and eventually threw in the towel for this particular rule.
Classifying individual systems also helps the prioritization effort. A system responsible for financial transactions or housing R&D data, for example, is more critical than the file server that holds marketing literature. When we went to classify our own systems we found High Tower Security Event Manager's asset-classification process easiest and most useful, but enVision and ArcSight ESM also offer asset-weighting functions. After setting up classification weightings, we're not sure how we ever survived without them.
Finally, when it comes to reporting features and general interface usability, most of the products need significant work. If you're spending any serious amount of time behind a console, you want to easily drill down, perform ad hoc queries on user names and IP addresses, actually use the pretty graphs by being able to click on them, and avoid frustration while navigating the UI. The products from High Tower and ArcSight are the two easiest products to use, interface-wise, with ArcSight ESM's UI far more flexible and comprehensive than rivals'. EnVision has made advances in the UI department, but it still has a way to go. The field goes downhill from there. With the exception of LogLogic's ad hoc querying ability, which made some tasks ridiculously easy, the rest of the SIMs tested need serious work in the UI department.
In the end, we awarded ArcSight ESM our Editor's Choice, followed closely by Network Intelligence enVision, High Tower and Q1 Labs' QRadar. In the raw functionality department, ArcSight ESM is the most mature product on the market, but it gets expensive fast in large environments. High Tower SEM's simplicity is attractive--we'll watch it closely over the coming year. Symantec also has a unique offering--its integration with its DeepSight data feeds lets organizations use information and data from other global resources. No other product offers this cross-organization data-sharing capability.
One thing to note about as-tested pricing scores, which are based on the SIM collecting and analyzing log data from just fewer than 40 devices: Products with a per-device pricing model become exponentially more expensive in large organizations. Although our deployment was relatively small, quadrupling our device load with some of the appliances, like High Tower's or LogLogic's, wouldn't have cost us a dime more. However, the same level of expansion with the software from ArcSight, Sensage or Symantec would have cost a pretty penny. This is why we weighted price at 25 percent of the score--the issue isn't just upfront outlay, but scaling costs (as-tested pricing can be found in the features chart).
