Apache Kafka 0.9 Released with SSL Security, Cluster Usage Regulation, and More

By CIOReview | Thursday, November 26, 2015
849
1505
295

FREMONT, CA: Bringing an end to user’s wait for a new version, the availability of the 0.9 version of Apache Kafka is finally announced.

.Heightened Security for Streaming Data

Strengthening the security structure of Kafka, the 0.9 version incorporates three new security features. One of the features is user authentication that verifies the user either through using Kerberos or TLS client certificates to provide information on each request made to Kafka. Its support for authorization can be based on different available session attributes or context including user, IP, common name in certificate etc, says Neha Narkhede, Co-founder and CTO, Confluent.Secondly, addition of unix-like permissions system, version 0.9 keeps track and controls users access to data. To protect sensitive data across unknown network, the new version supports encryption on the wire which is available through Secure Sockets Layer (SSL), a standard security technology for establishing an encrypted link between a server and a client. Encryption has been kept optional and not in default mode.

At-rest encryption can be achieved either by encrypting individual fields in the message or via filesystem security features. To drive adoption of the new implementations and to mitigate risk, the new producer and consumer APIs and consumer implementations have been installed with the new security features.

Redesigned Consumer Client

Keeping the promise of redesigning the consumer client, the 0.9 version comprises beta support for the newly redesigned consumer client. The new consumer offers pluggable offset management support that allows user to choose between the default Kafka backed offset management or offset management through an external data store. Offering a unified consumer API, it is useful for managing distributed producing and consuming of data in Kafka.

Simplified Kafka Adoption

Kafka enables streaming of data flow between external systems to unlock data siloes and to exchange data in real-time. But for users every such integration tool is difficult to understand and operate as their data infrastructure and system increases. As a result adoption of Kafka for data integration purposes becomes complex. To make data streaming hassle-free, Kafka 0.9 extends support for a new feature called Kafka Connect that facilitates large-scale, real-time data import and export for Kafka. It encourages the growth of an ecosystem of open source connectors that share the same APIs, behavior that can be monitored, deployed, and administered in a consistent manner.

As earlier versions of Kafka , consumers could consume data fast and monopolize broker resources as well as cause network saturation. Even producers were able to drag large amount of data creating pressure on memory and large I/O on broker instances. In such a situation, collateral damage occurred within the clusters that impacted service-level agreement (SLA).

User-defined Quotas

To offer user the ability to prevent a rogue application from making the cluster unavailable for other clients, Kafka introduces support for user-defined quotas. It enables user to enforce quotas on a per-client basis in terms of bytes read per second per client id. Each client receives a default quota which is enforced on a per-broker basis. On violating their quota, Kafka slows down the client while increasing available network bandwidth and cluster resources of other clients of the cluster.