Big Data Security Challenges

Collecting massive amounts of security data is easy. Data analysis and visualization? Not so much.

According to ESG Research, 47% of enterprise organizations collect 6TB of security data or more on a monthly basis to support their cybersecurity analysis requirements. Furthermore, 43% of enterprise organizations collect “substantially more” security data then they did 2 years ago while an additional 43% of enterprise organizations collect “somewhat more” security data then they did 2 years ago.

Just what types of data are they collecting? Everything. User activities, firewall logs, asset data, vulnerability scans, DNS logs, etc. Most enterprises aren’t collecting, storing, and analyzing large volumes of network packets (i.e. Full-packet capture or PCAP) today but they will increasingly do so in the future. Once this happens, security data volume collection will take another quantum leap.

If this activity doesn’t signal the need for big data security analytics than nothing does. Nevertheless, CISOs’ need go beyond dumping a bunch of unstructured data in a Hadoop cluster.

So what’s required? To find out, ESG recently surveyed 257 security professionals working at North American-based enterprise organizations (i.e. more than 1,000 employees) and asked them a series of questions about security data collection, processing, and analysis. As part of this project, security professionals were asked to identify specific difficulties around security data collection and analysis. The top 2 problems revealed were:

• 62% of enterprise organizations have “significant difficulties “ or “some difficulties” with security data visualization
• 53% of enterprise organizations have “significant difficulties “ or “some difficulties” with security data analysis

More..