Real Time Business Statistics with Kafka and Elasticsearch

This technical report details the implementation of a real-time business statistics system using Kafka, KSQL, and Elasticsearch. The objective was to enable a client to view business event statistics, such as the number of orders placed for various products. The system processes business events from different actors, aggregates them, and presents the data through interactive Kibana dashboards. The report outlines the architecture, starting with the creation of a Kafka topic to receive business events, followed by the use of KSQL to perform continuous queries that aggregate statistics. It describes the integration of Kafka with Elasticsearch using a Sink Connector to index the results for visualization. The report also includes specific examples of KSQL queries for creating streams and tables, as well as the configuration of the Elasticsearch connector. The final implementation resulted in a system capable of processing a significant number of events per second, fulfilling the client's requirements for real-time data analysis.