System Overview

1. System Overview

This system answers one question for 15 actively-monitored US equities: “Will this stock go UP, DOWN, or stay FLAT in the next 5 minutes, 25 minutes, 75 minutes, or 5 hours?”

It does this through a 6-stage pipeline that runs every 10 minutes during market hours, fully automated via N8N workflow orchestration:

Symbol Coverage — Active Watchlist (15 symbols)

The system is not limited to these symbols — it processes any valid US equity on-demand.

Cloud Stack: PostgreSQL (all data), Azure Blob Storage (models), FastAPI (inference), N8N (orchestration), Docker (containerized deployment). Estimated cost: ~$20–40/month using free tiers.

2. Architecture

Foundation: 5-Minute Bar Resolution

The entire system is built on a 5-minute bar foundation — a deliberate choice that balances day trading liquidity (78 bars per trading day), API rate compliance (all providers support 5-minute data on free tiers), and model training quality (sufficient observations for statistical significance).

All time horizons are expressed as bar counts, not absolute time. This makes the architecture resolution-independent:

If the foundation changes to 1-minute bars in the future, the same configuration produces 1-minute, 5-minute, 15-minute, and 60-minute windows — zero code changes required.

Three-Container Deployment

Separating these provides fault isolation (a crash in trading doesn’t affect predictions), independent scaling (inference is CPU-bound, pipelines are I/O-bound), and deployment flexibility (update trading logic without reloading ML models).

Clean Architecture

• Interfaces — FastAPI endpoints, CLI entry points. Handles HTTP requests and responses.
• Application — Pipeline orchestrators, decision engines. Coordinates workflow and validates constraints.
• Domain — Trading strategies, position sizing, exit evaluation. Pure business logic with no external dependencies.
• Infrastructure — Broker API clients, database adapters, storage adapters. All external communication.

This separation means trading strategies can be unit tested without databases or broker APIs, and swapping Alpaca for Interactive Brokers requires zero changes to business logic.

System Architecture

3. Stage 1: Ticks — Market Data Ingestion

What It Does

Fetches real-time OHLCV (Open, High, Low, Close, Volume) data from multiple market data providers, deduplicates overlapping data using source priority, and stores it in PostgreSQL for downstream processing.

Data Sources

When multiple sources provide data for the same symbol and timestamp, the system deduplicates using the priority order above. All source data is preserved — only the highest-priority version is used for downstream processing.

Each provider implements a common interface, making it straightforward to add new data sources.

Intelligent Mode Selection

Mode selection happens per-symbol within each run. A single execution might use incremental for AAPL, hybrid for META (correction detected), and skip for PRM (off-market).

Revision Tracking

Market data providers occasionally correct historical values — a high price revised from 650.50 to 650.99, a volume figure adjusted after settlement. The system detects these corrections automatically, logs the revision (what changed, old value, new value, source), and propagates the correction through all downstream pipelines.

Resilience

• 97% network success rate across all production runs
• 3-attempt retry with exponential backoff per API call
• 60-second timeout prevents hanging on unresponsive providers
• Session-aware processing: Correctly handles pre-market, regular, after-hours, and extended sessions
• Zero data loss confirmed across all production runs since December 2025

4. Stage 2: Bars — Feature Engineering

What It Does

Transforms raw OHLCV data into a 23-column feature matrix by computing 18 technical features across 5 categories. This feature matrix is what ML models train on and what inference uses for predictions.

Technical Features

Base Features (5): Open, High, Low, Close, Volume — the raw OHLCV data carried forward from ticks.

Price-Based Features (5)

Momentum Features (4)

Volume Features (3)

Sentiment Features (4) — Reserved for Future Integration

Day Trading Features (2) — Computed During Labeling

Incremental Processing

The pipeline does not blindly recalculate all bars every run. It detects which data has changed:

1. New data — Any new market data since the last bar computation
2. Revised data — Any corrections from data providers

Only bars matching either condition are reprocessed. During market hours, this typically means 5–15 new bars per run. Off-market, it means zero bars processed — the pipeline exits in under 1 second.

Computation Approach

• Vectorized operations: All indicator calculations use vectorized functions — no row-by-row loops
• No look-ahead bias: Features use only data up to and including the current bar
• Idempotent writes: Results are written in batches with upsert logic
• Typical performance: ~10 bars in under 8 seconds during incremental runs

5. Stage 3: Labels — Movement Classification

What It Does

Looks into the future to classify price movements as UP, DOWN, or FLAT for each of the 4 time horizons. This creates the ground truth that models learn to predict.

Classification Logic

For each bar, the pipeline calculates the forward return at each horizon:

The 0.5% threshold was chosen to capture meaningful intraday moves while filtering out noise. Too low catches random fluctuations; too high misses actionable trading opportunities.

Multi-Horizon Processing

A single run across 15 symbols with ~540 new bars produces approximately 2,160 labeled samples (540 bars × 4 horizons).

Additional Computed Features

Efficient Processing

The pipeline checks whether any bars have changed since the last run. If no new or updated bars exist, the entire labels pipeline exits in under 1 second — no computation, no writes. During off-market hours, this is the case for every run.

During market hours, only labels for new or revised bars are processed. This scoped approach keeps labeling fast even as the database grows.

6. Stage 4: Training — Model Building

What It Does

Trains one RandomForest classifier per time horizon using the labeled data from Stage 3, evaluates performance, and uploads versioned model files to Azure Blob Storage for inference.

Feature Selection

Production Algorithm: RandomForest

Why Separate Models Per Horizon

Each horizon exhibits different feature importance patterns:

• Short-term (H1, H5): Momentum indicators dominate — RSI, MACD Histogram, and Bollinger %B have the highest importance.
• Medium-term (H15): Mix of momentum and price-based features — EMA Spread and VWAP Gap gain importance as trend sustainability matters more.
• Long-term (H60): Trend and volume indicators dominate — EMA-based features, Relative Volume, and VWAP become the strongest predictors.

Training separate models per horizon yields 60–70% accuracy versus ~55% with a single multi-horizon model — a significant improvement for a 3-class classification problem (random baseline: 33%).

Model Performance

Expandable Algorithm Architecture

The system supports multiple algorithms through configuration — switching requires changing a single config value:

Tabular Models (ready to use):

Sequence Models (planned — require sliding window feature engineering):

Model Storage and Versioning

Models are versioned and stored in Azure Blob Storage organized by version and horizon. All models can be hot-reloaded by the inference API without container restart.

Training is currently executed 2–3 times daily using a configurable lookback window. The flow is: new labels → retrain → upload → hot reload inference.

Model Lifecycle

7. Stage 5: Inference — Real-Time Predictions

What It Does

Loads all 4 trained models into memory and serves real-time predictions via a FastAPI service. For each symbol, the service produces a prediction per horizon — UP, DOWN, or FLAT — with probability scores for all three classes.

API Endpoints

Prediction Output

Performance

Models are cached in memory at startup. Hot reload refreshes the cache without restarting the container — critical for deploying newly trained models during market hours.

Predictions are stored in the database with full linkage to the input bar, and each prediction is tracked to ensure idempotent evaluation — the same prediction never triggers multiple trades across runs.

8. Stage 6: Trading — Signal Generation & Execution

What It Does

Reads unprocessed predictions from the database, generates trading signals through multi-horizon consensus, validates each signal against portfolio constraints, submits approved orders to the configured broker, and tracks every decision for audit compliance.

The Complete Trading Workflow

1. Symbol Resolution — Identify symbols with unprocessed predictions, plus symbols with open positions
2. Prediction Fetch — For each symbol, fetch the latest predictions across all 4 horizons
3. Exit Evaluation — Check all open positions for exit conditions and close positions before opening new ones
4. Signal Generation — Pass predictions to the configured strategy to determine if a BUY or SELL signal is warranted
5. Forecast Marking — Mark all evaluated predictions, regardless of signal outcome
6. Signal Validation — Decision Engine validates portfolio constraints for each signal
7. Order Submission — Approved signals are submitted to the broker
8. Trade Persistence — Every executed order is recorded with full linkage to the triggering prediction
9. Summary — Return a structured summary for both machine consumption and human notification (Telegram)

Trading Decision Flow

Signal Generation: Multi-Horizon Consensus

Rather than acting on any single prediction, the system requires agreement across multiple time horizons. This reduces noise and increases signal quality significantly.

Trading Strategies

Conservative Strategy:

Tiered Strategy (active):

Evaluation proceeds from strongest to weakest tier. First match wins.

Tiered Signal Evaluation

Decision Engine: Portfolio Constraints

Position Sizing

The system uses fixed-dollar position sizing: each trade targets a configurable dollar amount (default $5,000), and the quantity is calculated based on current price. The tiered strategy applies a multiplier: Tier 1 signals get 2.0× (i.e., $10,000 positions), while Tier 2 and Tier 3 get 1.0× ($5,000). Fractional shares are supported for long positions.

Exit Conditions

The exit framework is extensible — planned additions include smart EOD (hold overnight winners), profit-taking, and tier-based exits.

Manual Trading

The system supports manual trades through the same pipeline. A user provides symbol, quantity, and side — the request bypasses signal generation but passes through the identical Decision Engine constraint checks, position sizing validation, and audit logging. Manual trades appear alongside automated trades in all analytics and dashboards.

9. Multi-Broker Architecture

Provider-Agnostic Design

All broker interactions flow through a single abstract interface that provides rate limiting (per-broker configurable), retry logic with exponential backoff, run tracking integration, API call logging, and standardized response format regardless of broker.

Every consumer — the trading pipeline, CLI tools, REST API — works with this abstract interface. Switching brokers requires zero changes to any consumer code.

The interface defines 9 methods that every broker must implement:

Three-Broker Implementation

Broker Selection

1. Runtime override — Pass a provider parameter in the API request or CLI flag
2. Environment variable — Set for automated pipelines
3. Config default — Read from configuration file

Alpaca (Active)

The production broker for US equities, using Alpaca’s Paper Trading API. Stateless HTTP REST with separate URLs for trading and market data. Supports market orders, fractional shares (long positions), real-time position tracking, and P&L calculation. 200 requests/minute rate limit.

Interactive Brokers (Code Complete — Pending Testing)

All 9 interface methods implemented. Uses an event-driven TCP socket protocol where requests trigger asynchronous callbacks. A synchronous wrapper layer makes the async API compatible with the rest of the system. Pending end-to-end testing with IB Gateway.

Kotak Securities (In Progress)

India’s National Stock Exchange (NSE) for equities and Gold/Silver ETFs. Uses a synchronous Python SDK with automated TOTP-based daily login. Supports delivery-based (CNC) and intraday (MIS) product types. Single config switch between sandbox and production.

Multi-Market Database Support

• All trade records include currency, country, and exchange information
• The symbol registry supports symbols across multiple exchanges and regions (US, India, Japan, UK, Germany, Canada, Australia)
• Exchange codes are standardized across providers
• Trading hours, settlement rules, and timezone handling are metadata-driven, not hardcoded

When adding a new market, only the broker client needs implementation. All business logic works unchanged because metadata drives market-specific behavior.

10. End-to-End Performance

Azure Production Performance

The system runs every 10 minutes during market hours as Docker containers on Azure Web App, with all pipelines executing sequentially in a single cycle.

Intelligent mode selection per symbol — incremental, hybrid, skip, or full — keeps cycle times low:

Azure Web App (B1) → Azure PostgreSQL, February 2026.

Training Performance

Training runs 2–3× daily using a configurable lookback window. The current algorithm is scikit-learn RandomForest (CPU-only). The database holds 3 years of backfilled data available for training and backtesting.

15 symbols, RandomForest (100 estimators), CPU-only. February 2026.

All 4 models upload to Azure Blob Storage after training and are hot-reloaded by the inference API without restart.

Row Accounting

Every pipeline run tracks inserts, updates, and skips separately. Zero variance between expected and actual counts has been maintained across all production runs — if the pipeline says it processed 540 bars, exactly 540 rows were modified in the database.

11. Observability & Data Quality

Three-Tier Run Tracking

Validation Tools

12. Reporting API & Appsmith Dashboard

The Fourth Container

Alongside the three core containers (Pipeline, Inference, Trading), the system includes a dedicated Reporting API on port 8003 — a read-only sidecar that serves dashboard data, portfolio analytics, and broker account information.

Separating read-heavy dashboard queries from write-critical trading actions ensures a slow analytics query never blocks trade execution.

Single Generic Endpoint

The API follows a registry pattern — one universal endpoint handles all report types: GET /report/{report_type}. Adding a new report requires two steps: add a generator method and register it. No route changes needed.

Available Reports

All endpoints support optional query parameters: days (lookback), status (order filter), symbol (single-symbol filter), activity_type (FILL, DIV, etc.), and limit (max records).

The Dashboard: Appsmith on Azure

The Appsmith dashboard runs as an Azure Web App at denduluri.net, with the four API sidecars (ports 8000–8003) deployed as Azure sidecar containers alongside it. The dashboard reads pre-computed data directly from PostgreSQL via SQL views and calls sidecars over localhost for live broker data — zero middleware overhead.

Additional capabilities: dark trading theme, role-based access (Admin / Developer / Viewer), TradingView embedded charts, mobile-responsive layout, and CSV export on all data tables.

13. Experiment & Backtesting Framework

The Problem

The production system trains models and executes trades — but how do we know if a different algorithm, different features, or different strategy would perform better? Without systematic experimentation, we’re flying blind.

The Solution: YAML-Driven Experiment Framework

A dedicated backtesting framework that lets a researcher define an experiment in a YAML config, run it from a single CLI command, and get comprehensive classification + trading + stability metrics — all tracked for comparison.

What’s Configurable

Walk-Forward Validation

The most rigorous testing method. Instead of a single train/test split, the framework slides a window across historical data, producing not just accuracy numbers but stability metrics — does the model work consistently across different time periods, or was it just lucky in one window?

Comprehensive Metrics

Classification Metrics — How accurate are the predictions?

Trading Metrics — Would this make money?

Stability Metrics (walk-forward only) — Is performance consistent?

Experiment Tracking

The database schema uses a bt_ prefix (4 tables, 7 views, 28 indexes) — completely isolated from production data. Safe to drop and recreate without any impact on live trading.

Parameter Sweep

Beyond single experiments, the framework supports grid search: 4 algorithms × 3 tree counts × 3 thresholds = 36 experiments, each tracked and comparable. This systematically finds the best configuration rather than relying on intuition.

14. Developer Tooling

CLI commands for daily operations and automated tools for code quality and data maintenance — all following the same Clean Architecture patterns as the core system.

CLI Commands

All CLI commands share the same factories and services as the API containers — execute_trades uses the identical TradingPipelineFactory that powers the Trading API on port 8002.

Operational Tools

Code Quality & Maintenance

15. What’s Next

Immediate Roadmap

Future Phases

Design Philosophy

The project follows a disciplined approach: validate before risking capital. Every component is built to be auditable (complete decision logging), config-driven (no hardcoded values in business logic), and incrementally extensible (new algorithms, brokers, and data sources slot in through configuration rather than code changes).