ChronosDefrag

"Chronological adjacency ≠ Statistical similarity"

A trending regime at t = 1000 has more in common with a trending regime at t = 50 than with the mean-reverting regime at t = 999. Standard sequence models trained over chronological order waste capacity learning regime transitions that carry no predictive signal. ChronosDefrag reorders the timeline to exploit regime structure.

Why This Matters

Financial markets cycle through regimes — mean-reversion, momentum, high-volatility dispersion — with no fixed periodicity. Most sequence models treat non-stationarity as noise to smooth over.

ChronosDefrag treats it as structural signal to exploit.

❌ Standard Approach

Train over raw chronological order
Learn and forget regimes repeatedly as they recur
Computationally wasteful
Statistically fragile under regime shifts
No explicit regime awareness

✓ ChronosDefrag

Unsupervised regime discovery via contrastive learning
Reorder timeline so regimes are locally adjacent
Train predictor on pseudo-timeline (locally stationary)
Self-supervised throughout—no labels required
Interpretable regime routing at inference

The defragmentation hypothesis: If we reorder the training timeline so that same-regime windows are locally adjacent, the predictor's learning problem simplifies from non-stationary sequence modelling to locally stationary interpolation.

Architecture Overview

ChronosDefrag operates in two phases: offline training and online inference.

Training Phase

Raw Tick Stream (T, F)
         │
         ▼
   Block Slicer
   ├─ Causal · non-overlapping
   ├─ Timestamp-tracked
   └─ (N, block_len, F)
         │
         ▼
   ╔════════════════════════════════╗
   ║    RegimeEncoder               ║
   ║  ├─ Input projection           ║
   ║  ├─ N × ResidualBlock          ║
   ║  │  ├─ CausalConv1D            ║
   ║  │  ├─ Pre-norm + GELU         ║
   ║  ├─ Temporal mean pooling      ║
   ║  ├─ LayerNorm                  ║
   ║  ├─ Latent head (L2-norm)      ║
   ║  ├─ Projection head            ║
   ║  ├─ InfoNCE loss               ║
   ║  ├─ TER regularisation         ║
   ║  └─ LCR regularisation         ║
   ╚════════════════════════════════╝
         │  (N, latent_dim) ∈ S^(d-1)
         ▼
   ╔════════════════════════════════╗
   ║   VectorDefragRouter           ║
   ║  ├─ All-pairs cosine sim       ║
   ║  ├─ Greedy NN chain            ║
   ║  └─ Permutation π              ║
   ╚════════════════════════════════╝
         │  Pseudo-timeline z[π]
         ▼
   ╔════════════════════════════════╗
   ║   TemporalPredictor            ║
   ║  ├─ Stacked GLURecurrentBlocks ║
   ║  ├─ Operates on z[π]           ║
   ║  └─ Latent reconstruction      ║
   ╚════════════════════════════════╝

Inference Phase

Live Tick Window (block_len, F)
         │
         ▼
   RegimeEncoder
   └─ Query latent z_q
         │
         ▼
   VectorDefragRouter.reconstruct_context()
   ├─ Batched mm() → similarities
   ├─ topk() → neighbors
   ├─ SPD weighting (temporal decay)
   └─ Weighted aggregate
         │
         ▼
   TemporalPredictor.infer()
         │
         ▼
   {alpha_prediction, regime_confidence, system_latency_ms}

Key Components

Module	Responsibility
`RegimeEncoder`	Maps market windows to unit-sphere latent embeddings via InfoNCE + regularisers
`VectorDefragRouter`	Greedy nearest-neighbour chaining on latent manifold to construct pseudo-timeline
`TemporalPredictor`	Gated linear recurrence units trained on defragmented timeline
`ChronosDefragEngine`	End-to-end orchestration: fit(), predict_live(), manage registry

Novel Mechanisms

ChronosDefrag introduces two mechanisms not found in standard deep learning libraries:

1. Temporal Entropy Regularisation (TER)

Location: RegimeEncoder.temporal_entropy_reg()

Constructs a soft assignment matrix P ∈ ℝ^(B × B) via row-wise softmax over pairwise cosine similarities within a batch. Computes mean Shannon entropy and penalises it falling below a target:

L_{\text{TER}} = \max(0,\, H_{\text{target}} - \bar{H}(P)) \cdot \lambda_{\text{entropy}}

Why it works: Prevents latent collapse without requiring discrete cluster assignments or prototype banks. Operates directly on the geometry of the cosine similarity matrix.

2. Similarity Persistence Decay (SPD)

Location: VectorDefragRouter.query()

A temporal-distance-aware weighting function applied to cosine similarities at inference. Parameter-free, differentiable with respect to neither query nor registry, adds zero training overhead:

\hat{s}(i) = \cos(q, h_i) \cdot \delta^{(|t_q - t_i| / |H|)}

Key property: Monotonically down-weights registry entries temporally far from the query, respecting the implicit non-stationarity structure of financial data without explicit regime labelling.

Mathematical Foundations

Latent Manifold & InfoNCE Loss

Each market window $x \in \mathbb{R}^{T \times F}$ is mapped to a unit-sphere embedding $z \in \mathcal{S}^{d-1}$ via encoder $\varphi_\theta$. The contrastive objective (InfoNCE / NT-Xent) maximises agreement between augmented views of the same block while repelling embeddings of different blocks:

L_{\text{NCE}} = -\mathbb{E}\left[\log \frac{\exp(z_i \cdot z_j / \tau)}{\sum_k \exp(z_i \cdot z_k / \tau)}\right]

where $\tau$ is a learnable log-temperature parameter, clipped to $(0, 1)$ to prevent numerical instability.

Pseudo-Timeline Construction

The defragmentation permutation $\pi$ is found via greedy nearest-neighbour chaining on $\mathcal{S}^{d-1}$. Starting from block 0, each subsequent position is the unvisited block with highest cosine similarity to the current position:

\pi^* \approx \arg\min_{\pi \in \text{Sym}(N)} \sum_{i=1}^{N-1} \left\|z_{\pi(i+1)} - z_{\pi(i)}\right\|^2

This is the Euclidean TSP on the unit sphere. The greedy solution is $O(N^2)$ but executes inside a single torch.mm() call—no Python loops.

Timeline Entropy Diagnostic

After construction, ChronosDefrag reports a mixing ratio that indicates regime separation quality:

\eta = 1 - \frac{|\text{corr}(\pi, \text{arange}(N))|}{N}

$\eta \approx 1$ indicates high regime mixing (pseudo-timeline substantially reordered). $\eta \approx 0$ signals latent collapse — increase encoder depth or adjust regularisation.

Performance & Benchmarks

Measured on Apple M2 Pro, CPU-only, PyTorch 2.3, Python 3.11. No GPU required.

Training Performance

Stage	Blocks (N)	Time
Encoder (30 epochs, batch 32)	96	~40s
Pseudo-timeline construction	96	~0.4ms
Predictor (20 epochs, batch 32)	88 windows	~2s

Inference Latency

Mean Latency

1.14ms

p50 Latency

1.09ms

p95 Latency

1.89ms

p99 Latency

2.31ms

Throughput

~877

QPS

Peak Memory

~180MB

Includes: encoder forward pass + registry lookup + SPD weighting + predictor forward pass. First 5 calls excluded (warm-up).

Test Suite

19 passed in 4.1s

Coverage: tensor shapes · causal padding correctness · block non-overlap · pseudo-timeline validity · collapse detection · augmentation integrity · latency budget · determinism under fixed seed.

Installation & Quickstart

Installation

pip install chronosdefrag

Requirements: Python ≥ 3.10, PyTorch ≥ 2.1, NumPy ≥ 1.24

Minimal Example

import torch
from defrag.config import DefragConfig
from defrag.engine import ChronosDefragEngine

# Configuration
cfg = DefragConfig()
engine = ChronosDefragEngine(cfg)

# (T, F) chronological tick-level features
# F = 6 features by default (open, high, low, close, volume, returns)
data = torch.randn(4096, 6)

# Train: discover regimes, build pseudo-timeline, train predictor
engine.fit(data)

# Inference: route live window, predict regime
live_window = torch.randn(cfg.encoder.block_len, 6)
result = engine.predict_live(live_window)

print(f"alpha      : {result.alpha_prediction.item():+.5f}")
print(f"confidence : {result.regime_confidence * 100:.1f}%")
print(f"latency    : {result.system_latency_ms:.2f}ms")

Run Verification Suite

python main.py

Expected output with synthetic market data:

  [TICK 0001]  latency= 1.23ms  confidence= 24.3%  alpha=+0.31847
  [TICK 0002]  latency= 1.11ms  confidence= 26.1%  alpha=+0.28902
  [TICK 0003]  latency= 1.08ms  confidence= 27.6%  alpha=+0.19374
  ...
  mean : 1.14ms  p95 : 1.89ms  p99 : 2.31ms  QPS : 877

Configuration Reference

All hyperparameters are typed dataclasses. Override at instantiation:

from defrag.config import DefragConfig, DeploymentMode

cfg = DefragConfig(
    mode    = DeploymentMode.PRODUCTION,
    device  = "auto",  # cuda > mps > cpu
    seed    = 42,

    # Encoder configuration
    encoder = {
        "input_dim"        : 6,
        "block_len"        : 64,           # must be power of 2
        "hidden_dim"       : 128,
        "latent_dim"       : 64,
        "num_layers"       : 4,
        "entropy_weight"   : 0.01,         # TER coefficient
        "continuity_weight": 0.005,        # Latent continuity
        "temperature"      : 0.07,
    },

    # Routing configuration
    routing = {
        "registry_capacity": 8192,
        "top_k"            : 16,
        "similarity_decay" : 0.92,         # SPD δ coefficient
        "min_confidence"   : 0.05,
    },

    # Predictor configuration
    predictor = {
        "latent_dim"   : 64,
        "hidden_dim"   : 128,
        "num_layers"   : 3,
        "forecast_dim" : 1,
        "grad_clip"    : 1.0,
    },

    # Optimization configuration
    optim = {
        "encoder_epochs"  : 40,
        "predictor_epochs": 30,
        "batch_size"      : 64,
        "encoder_lr"      : 3e-4,
        "amp_enabled"     : False,
    },
)

Deployment Modes

Mode	Assertions	Logging	Use Case
`RESEARCH`	Enabled	Verbose	Development & debugging
`STAGING`	Enabled	Reduced	Pre-production validation
`PRODUCTION`	Disabled	Minimal	Low-latency inference

Design Philosophy

🔐 Causality by Construction

Every Conv1d uses left-only padding (CausalConv1d). Block slicer tracks timestamps explicitly. predict_live() asserts query timestamp exceeds training boundary.

🎯 No External Vector DBs

Registry is a fixed-size circular buffer of L2-normalised embeddings. Lookup is pure tensor algebra: mm() → topk(). Stays on-device with zero serialisation overhead.

🧠 Self-Supervised Throughout

No price labels, forward returns, or external annotations required. Encoder learns via contrastive pairs. Predictor reconstructs latent embeddings. Works on any multivariate tick series.

📦 Minimal Dependencies

torch, numpy, Python stdlib only. No pandas, scikit-learn, Lightning, Hydra, Ray, or vector DB frameworks. Full install graph inspectable in under 1 minute.

🔄 Deterministic by Default

Single cfg.seed_everything() covers Python random, NumPy, PyTorch CPU/CUDA. Every test is deterministic and passes on a cold run.

⚡ Pure PyTorch

Leverages PyTorch 2.0+ features: torch.compile support, native dtype handling, integrated autograd. No custom CUDA kernels or external compute libraries.

Limitations & Failure Modes

Known Limitations

Experimental research framework with no live-trading validation.
alpha_prediction is a latent-space quantity — a proxy for regime continuation, not an interpretable return forecast.
Minimum training size: $N \geq 50$ blocks ($T \geq 50 \times$ block_len ticks).
Greedy pseudo-timeline construction is $O(N^2)$. For $N > 5000$ blocks, chunked approximation required.
No uncertainty quantification over predictions. Point estimates during transitional regimes should be treated with scepticism.

Failure Modes & Mitigations

Latent Collapse: All embeddings converge to a small region. Symptoms: uniform confidence ~1.0, timeline entropy $\eta \approx 0$. Mitigation: increase entropy_weight, lower temperature, increase batch_size.

Similarity Drift: Encoder representation geometry shifts during training. Early registry entries become stale. Mitigation: reduce encoder_lr late in training; re-fit when regime_confidence degrades persistently.

Non-Stationarity Mismatch: Structural regime change (crisis, halving, regulation) has no historical analogue. Symptom: uniformly low regime_confidence. Mitigation: periodic re-fit on rolling window; monitor confidence distribution in production.

Pseudo-Timeline Pathology: Greedy chain produces near-chronological permutation when latent space doesn't separate regimes. Symptom: timeline entropy $\eta < 0.1$. Mitigation: increase encoder depth, verify feature normalisation.