Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures
Abstract & Details
Research Area
Computer Science
Keywords
- Big Data
stream analytics
Apache Kafka
Spark Structured Streaming
adaptive partitioning
exactly-once processing
Delta Lake
fault tolerance.
Abstract
Modern enterprise systems produce continuous, high-velocity event streams whose burstiness, skewed keys, and recovery requirements expose limitations in fixed-partition ingestion pipelines. This paper presents SPE-Engine, a distributed stream-processing architecture that unifies Kafka-based ingestion, adaptive partition rebalancing, zero-copy buffer handling, state-aware checkpointing, and transactional Delta Lake sinks. The paper makes four contributions: (i) a formalized architecture and dynamic rebalancing algorithm that migrates only overloaded key ranges within bounded cost budgets; (ii) an end-to-end processing and recovery protocol guaranteeing idempotent commits and exactly-once semantics without record duplication; (iii) an empirical evaluation, including ablation and scalability analysis, against legacy MapReduce, default Spark Structured Streaming, and optimized Flink pipelines; and (iv) an operational analysis detailing trade-offs across throughput, tail latency, horizontal scalability, and hardware efficiency. In controlled cluster benchmarks, SPE-Engine achieves 1.48 million events per second, a 0.32 s mean latency, a 0.75 s P99 tail latency, and a 1.85 s recovery time under worker failure, maintaining stable sub-second latency even under 80% key skew. The architecture, baseline comparisons, and statistical assumptions are reported explicitly to allow reproducible verification.
License
This work is licensed under a Creative
Commons
Attribution-ShareAlike 4.0 International License.
Author Information
| # | Name | Institute / Affiliation |
|---|---|---|
| 1 | Mohamed Ahmed Elmubarak Mohamed | Nuba College, Sudan Technological University, Sudan |
| 2 | Abulfathi Alhussaini | American Open University Nigeria |
| 3 | Salah Eldin Deng Aljack Shuol | University of Bahri,Khartoum-Sudan |
| 4 | Fathelrhman Mohammed Abker Mohammed | University of Hail |
How to Cite
Use the following formats to cite this article in your research.
APA Style
Mohamed, Mohamed Ahmed Elmubarak, Alhussaini, Abulfathi, Shuol, Salah Eldin Deng Aljack, & Mohammed, Fathelrhman Mohammed Abker (2026). Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures. International Journal of Advance Research and Innovative Ideas In Education, 12(5), 903-909.
MLA Style
Mohamed, Mohamed Ahmed Elmubarak, et al. "Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures." International Journal of Advance Research and Innovative Ideas In Education, vol. 12, no. 5, 2026, pp. 903-909.
IEEE Style
Mohamed Ahmed Elmubarak Mohamed, Abulfathi Alhussaini, Salah Eldin Deng Aljack Shuol, and Fathelrhman Mohammed Abker Mohammed, "Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures," International Journal of Advance Research and Innovative Ideas In Education, vol. 12, no. 5, pp. 903-909, 2026.
Vancouver Style
Mohamed Mohamed Ahmed Elmubarak, Alhussaini Abulfathi, Shuol Salah Eldin Deng Aljack, Mohammed Fathelrhman Mohammed Abker. Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures. International Journal of Advance Research and Innovative Ideas In Education. 2026;12(5):903-909.
Harvard Style
Mohamed, Mohamed Ahmed Elmubarak, Alhussaini, Abulfathi, Shuol, Salah Eldin Deng Aljack, & Mohammed, Fathelrhman Mohammed Abker (2026) 'Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures', International Journal of Advance Research and Innovative Ideas In Education, 12(5), pp. 903-909.
Chicago Style
Mohamed, Mohamed Ahmed Elmubarak, et al. "Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures." International Journal of Advance Research and Innovative Ideas In Education 12, no. 5 (2026): 903-909.
Turabian Style
Mohamed, Mohamed Ahmed Elmubarak, et al. "Distributed Real-Time Stream Analytics and Ingestion Engine for High-Velocity Big Data Architectures." International Journal of Advance Research and Innovative Ideas In Education 12, no. 5 (2026): 903-909.