What are Kafka Streams and How are they implemented?

Apache Kafka Streams API is an Open-Source, Robust, Best-in-class, Horizontally scalable messaging system. In layman terms, it is an upgraded Kafka Messaging System built on top of Apache Kafka. In this article, we will learn what exactly it is through the following docket.

• What is Kafka?
• What is a Stream?
• What exactly is Kafka Stream?
• Apache Kafka Stream API Architecture
• Kafka Stream Features
• Kafka Streams Example
• Differences between Kafka and Kafka Streams
• Use cases of Apache Kafka Streams API

What is Kafka?

Apache Kafka is basically an Open-Source messaging tool developed by Linkedin to provide Low-Latency and High-Throughput platform for the real-time data feed. It is developed using Scala and Java programming Languages.

What is a Stream?

In general, a Stream can be defined as an unbounded and continuous flow of data packets in real-time. Data packets are generated in the form of key-value pairs and these are automatically transferred from the publisher, there is no need to place a request for the same.

What exactly is Kafka Stream?

Apache Kafka Stream can be defined as an open-source client library that is used for building applications and micro-services. Here, the input and the output data is stored in Kafka Clusters. It integrates the intelligibility of designing and deploying standard Scala and Java applications with the benefits of Kafka server-side cluster technology.

Apache Kafka Stream API Architecture

Apache KStreams internally use The producer and Consumer libraries. It is basically coupled with Kafka and the API allows you to leverage the abilities of Kafka by achieving Data Parallelism, Fault-tolerance, and many other powerful features.

The Different Components present in the KStream Architecture are as follows:

• Input Stream
• Output Stream
• Instance
• • Consumer
  • Local State
  • Stream Topology

• Input Stream and Output Streams are the Kafka Clusters that store the Input and Output data of the provided task.
• Inside every instance, we have Consumer, Stream Topology and Local State
• Stream Topology is actually the flow or DAG in which the given task is executed

• Local State is the memory location that stores the intermediate results of the given operations like Map, FlatMap etc.

To increase data parallelism, we can directly increase the number of Instances. Moving ahead, we will understand the features of Kafka Streams.

Kafka Stream Features

Now, let us discuss the important features of Kafka streams that give it an edge over other similar technologies.

• Elastic

Apache Kafka is an open-source project that was designed to be highly available and horizontally scalable. Hence, with the support of Kafka, Kafka streams API has achieved it’s highly elastic nature and can be easily expandable.

• Fault-tolerant

The Data logs are initially partitioned and these partitions are shared among all the servers in the cluster that are handling the data and the respective requests. Thus Kafka achieves fault tolerance by duplicating each partition over a number of servers.

• Highly viable

Since Kafka clusters are highly available, hence, they can be preferred any sort of use cases regardless of their size. They are capable of supporting small, medium and large scale use cases.

• Integrated Security

Kafka has three major security components that offer the best in class security for the data in its clusters. They are mentioned below as follows:

• • Encryption of data using SSL/TLS
  • Authentication of SSL/SASL
  • Authorization of ACLs

Followed by Security, we have its support for top-end programming languages.

• Support for Java and Scala

The best part about Kafka Streams API is that it gets integrated itself the most dominant programming languages like Java and Scala and makes designing and deploying Kafka Server-side applications with ease.

• Exactly-once processing semantics

Usually, stream processing is a continuous execution of the unbounded series of data or events. But in the case of Kafka, it is not. Exactly-Once means that the user-defined statement or logic is executed only once and the updates to state, managed by SPE(Stream Processing Element) are committed only once in a durable back-end store

Kafka Streams Example

This particular example can be executed using Java Programming Language. Yet, there are a few prerequisites for this. One needs to have Kafka and Zookeeper installed in the Local System.

The code is written is for wordcount which documented below as follows.

import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
import org.apache.kafka.streams.kstream.Produced;
import org.apache.kafka.streams.state.KeyValueStore;
import java.util.Arrays;
import java.util.Properties;

      public class WordCountApplication {
            public static void main(final String[] args) throws Exception {
                  Properties props = new Properties();
                  props.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-application");
                  props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092");
                  props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
                  props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
                  StreamsBuilder builder = new StreamsBuilder();
                  KStream<String, String> textLines = builder.stream("TextLinesTopic");
                  KTable<String, Long> wordCounts = textLines.flatMapValues(textLine -> Arrays.asList(textLine.toLowerCase().split("W+"))).groupBy((key, word) -> word).count(Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>as("counts-store"));
                  wordCounts.toStream().to("WordsWithCountsTopic", Produced.with(Serdes.String(), Serdes.Long()));
                  KafkaStreams streams = new KafkaStreams(builder.build(), props);
                  streams.start();
            }
      }

Now, we will learn about some of the important differences between Kafka and Kafka Streams.

//Text given

Welcome to Edureka Kafka Training.

This article is about Kafka Streams.

//Output:

Welcome(1)
to(1)
Edureka(1)
Kafka(2)
Training(1)
This(1)
article(1)
is(1)
about(1)
Streams(1)

Differences between Kafka and Kafka Streams

Use cases of Apache Kafka Streams API

Apache Streams API is used in multiple use cases. Some of the major Applications where Streams API is being used are mentioned below.

The New York Times

The New York Times id one of the powerful media in the United States of America. They use Apache Kafka and Apache Streams API to store and distribute the real-time news through various applications and systems to their readers.

Trivago

Trivago is the Global Hotel Search platform. They use Kafka, Kafka Connect and Kafka Streams to enable their developers to access details of various hotels and provide their users with the best in class service at the lowest prices

Pinterest

Pinterest uses Kafka at a larger scale to power the real-time predictive budgeting system of their advertising system. with Apache streams API backing them up, they have more accurate data than ever.

With this, we come to an end of this article. I hope I have thrown some light on to your knowledge on the Kafka Streams along with its implementation in real-time.

Now that you have understood Big data and its Technologies, check out the Hadoop training by Edureka, a trusted online learning company with a network of more than 250,000 satisfied learners spread across the globe. The Edureka Big Data Hadoop Certification Training course helps learners become expert in HDFS, Yarn, MapReduce, Pig, Hive, HBase, Oozie, Flume and Sqoop using real-time use cases on Retail, Social Media, Aviation, Tourism, Finance domain.

If you have any query related to this “Kafka Streams” article, then please write to us in the comment section below and we will respond to you as early as possible.