This guest post comes from Andy Thurai. Andy is the Chief Architect & Group CTO for the Intel unit that is responsible for Cloud/ Application security, API, Big Data, SOA and Mobile middleware solutions. You can follow him @AndyThurai (Twitter) or at thurai.net.
As promised in my last blog “Big Data, API, and IoT …..Newer technologies protected by older security” here is a deep dive on Big Data security and how to effortlessly secure Big Data effectively.
Like many other open source models, Hadoop has followed a path that hasn’t focused much on security. In order to effectively use Big Data, it needs to be secured properly. However if you try to force fit into an older security model, you might end up compromising more than you think. But if you make it highly secure, it might interfere with performance.
In order to effectively secure Big Data, you must mitigate the following security risks that aren’t addressed by prior security models.
In a hosted environment, the provider holds the keys to your secure data. If a government agency legally demands access, the providers are obligated to provide access to your data. While it is necessary, the onus should be on you to control when, what, and how much you are giving others access to and also keep track of the information released to facilitate internal auditing processes.
Keep the keys to the kingdom with you. An encryption proxy can provide a tighter control.
If you encrypt the entire data, it could slow the performance down significantly. In order to avoid that, some of the Big Data, BI, and analytics programs choose to encrypt only portions of sensitive data. It is imperative to use a Big Data eco-system that is intelligent enough to encrypt data selectively.
A separate and more desirable option is to run faster encryption/ decryption. Solutions such as Intel Hadoop security Gateway use Intel chip based encryption acceleration (Intel AES-NI instruction set as well as SSE 4.2 instruction set) which is several orders of magnitude faster than software based encryption solutions. It is not only faster, but it is also more secure as the data never leaves the processor for an on or off-board crypto processor.
Sensitive data can be classified into two groups: Risk or Compliance. Safeguarding your data might include one of the following:
If you let applications/services access the raw data that could be disastrous. Instead, you might want to enforce the data access controls, as close to the data as possible. You need to distribute data, associated properties, classification levels, and enforce them where the data is. One way to enforce this would be to have an API expose data that can control the exposure based on data attributes locally.
Many of the Big Data components communicate via APIs (i.e. HDFS, HBase, and HCatalog). When you allow such powerful APIs to be exposed with very little, or no protection, it could lead to disastrous results. The most effective way to protect your Big Data goldmine would be to introduce a touchless API security Gateway in front of the Hadoop clusters. The clusters can be made to trust calls ONLY from the secure gateway. By choosing a hardened Big Data security gateway you can enforce all of the above by using very rich authentication and authorization schemes.
This issue is important enough for me to call this out as a separate issue. This arises from the architectural perspective that, if no proper resource protection is enforced, the NameNode can become the single point of failure making the entire Hadoop cluster useless. It is as easy as someone launching a DOS attack against webHDFS by producing excessive activity that can bring webHDFS down.
You need to have an effective Identity Management and Access control system in place to make this happen. You also need to identify the user base and effectively control access to the data consistently based on access control policies without relying on additional identity silos. Ideally, authentication and authorization for Hadoop should leverage existing identity management investments. The enforcement should also take into account time based restrictions as well (such as certain users having access to certain data only during specific periods, etc.).
Once you have implemented an effective data access controls based classification, you also need to monitor and log the usage patterns. You need to constantly analyze the usage patterns to make sure that there is no unusual activity. It is very crucial to catch any unusual activity and access-pattern early enough so you can avoid dumps of data making it out of your repository to a hacker.
As more and more organizations are rushing to implement and utilize the power of Big Data, care should be exercised to secure Big Data. Extending the existing security models to fit Big Data may not solve the problem; as a matter of fact it might introduce additional performance issues as discussed above. A solid security framework needs to be thought out before organizations can adopt enterprise grade Big Data.