A Scalable Real-Time Stock Market Prediction Framework Using LSTM Network and XGBoost Model

This paper introduces a real-time prediction system for stock prices that unifies machine learning (ML), deep learning, and data engineering methods for improving forecasting precision. The system integrates historical stock information and analysis of news sentiment to make accurate predictions. Historical data are accessed through Yahoo Finance, whereas news articles applicable to the stocks are accessed using the News API to incorporate the effects of the external market. The framework makes use of Long Short-Term Memory (LSTM) networks and XGBoost models that are supplemented using a stacked ensemble method to obtain higher accuracy and resilience. A FastAPI-backed backend supports API endpoints for prediction and history and is backed up by Redis to support real-time data streaming and storage. There is a special Redis processor to listen to streaming stock data, process it, and create predictions from trained models. Apart from this, a Streamlit dashboard provides an interactive, user-friendly representation of the predictions, historical patterns, model comparison, and performance measures. This paper provides an example of a scalable, real-time stock market prediction solution that can aid investors and analysts in making data-driven financial choices.