Leakage-aware sentiment and multimodal stock-index analysis

Abstract

Short-horizon stock-index forecasting remains difficult because returns are noisy, non-stationary, and only partly explained by historical prices and public news. This thesis investigates whether financial-news sentiment and multimodal market representations improve three-class direction classification, while asking when high accuracy reflects genuine forecasting rather than target construction. To address these questions, it develops a staged, leakage-aware experimental framework. A finance-domain language model first transforms dated headlines into lagged daily sentiment features. The primary forecasting experiment combines these features with historical market variables to classify the genuine three-trading-day log return from 𝑡 to 𝑡 + 3, using fixed ±0.5% thresholds. Candidate look-back windows of 5, 10, 15, and 20 trading days are compared using validation macro-F1 only; the 10-day window is then evaluated once on the locked test. The study also evaluates a smoother EMA-50 trend-state task and reconstructs a published ViT–TFT–HOG design that fuses numerical sequences, candlestick images, visual descriptors, and candle geometry across multiple stock indices. Chronological splits, simple baselines, feature ablations, per-market diagnostics, and matched target controls are used throughout. On 232 locked test examples, the Temporal Transformer reaches 37.93% accuracy, 0.3788 macro-F1, and 0.0393 MCC. The majority baseline has higher accuracy at 41.38% because of class imbalance, but its macro-F1 is only 0.1951. A matched ablation reaches 0.3808 macro-F1 without sentiment, so prior-day sentiment does not provide measurable incremental forecasting value in this experiment. Trading outcomes are mixed across markets and do not establish consistent economic value. The EMA-50 task is substantially easier because its labels are smoother and more persistent. The reconstructed multimodal model achieves stable accuracy above 93% for the reported market-state task and transfers well to additional indices. However, a deterministic audit reveals that the prices defining this target are already available in the input window; when the endpoint is moved into the genuine future, performance falls close to chance. These findings show that high classification accuracy can measure state reconstruction rather than forecasting skill. The main contribution is a reproducible, target-explicit evaluation that preserves strong reconstruction results without overstating future-return predictability.