Skip to content
OpenQuantOpenQuant
AuthoredWritten by hand against the code; every example on it is executed by the docs checks. No human has reviewed this page yet.

microstructural_features

Most price data carries no record of the things a trader most wants to know about a market: how wide the spread was, how far a given order would have moved the price, whether the flow was one-sided. Market microstructure theory offers ways to infer them from what is recorded, and AFML’s Chapter 19 collects those estimators as features. The ones that need only bars go back decades and work on any history; the ones that need signed trades are sharper and need tick data. Chapter 18’s entropy estimators, which measure how predictable a sequence of price moves is, live here too. The Python module is openquant.microstructural.

The estimators

Section titled “The estimators”

Spread. The Roll measure (1984) uses the fact that prices bouncing between bid and ask are negatively autocorrelated. If trades hit either side at random around an efficient price, the spread is 2−cov⁡(Δpt,Δpt−1)2\sqrt{-\operatorname{cov}(\Delta p_t,\Delta p_{t-1})}. Corwin–Schultz (2012) needs only highs and lows: the high is usually a buy and the low a sell, so the high–low range contains the spread once, while volatility grows with the horizon; comparing one-bar with two-bar ranges separates them. It returns a relative spread. Bekker–Parkinson volatility is the high–low volatility estimate with that spread taken out.

Price impact, the λ\lambda in “price moves λ\lambda per unit of flow”. Kyle’s lambda relates the price change to signed volume; Amihud’s relates the absolute return to dollar volume and needs no trade signs; Hasbrouck’s relates the return to the signed square root of dollar volume. Each comes in two forms. The bar_based functions take closes and volumes, infer the sign from the direction of the bar, and return a rolling mean of per-bar ratios. The trades_based functions take the trades of one bar, with their aggressor flags, and fit a regression through the origin.

Order flow. get_bvc_buy_volume is bulk volume classification: with no trade signs, attribute the share Φ(Δp/σΔp)\Phi(\Delta p/\sigma_{\Delta p}) of a bar’s volume to buyers. get_vpin is the volume-synchronised probability of informed trading of Easley, López de Prado and O’Hara (2012), the rolling mean absolute imbalance between buy and sell volume as a share of volume.

All rolling functions return a vector as long as the input, with NaN until a full window of valid values is available.

Checked against a market whose answers are known

Section titled “Checked against a market whose answers are known”

The simulation fixes a spread and a price impact, generates trades, and asks each estimator to recover them, first from the trades and then from 50-trade bars.

import random


from openquant import microstructural as ms


# 60,000 trades around an efficient price that follows a random walk. Every trade prints
# half a spread above or below it: the true spread is 0.10 on a price near 100, or 10 bp.
# Each trade also moves the efficient price by 0.002 per unit of signed volume.
rng = random.Random(1)
mid, spread, impact = 100.0, 0.10, 0.002
prices, volumes, sides = [], [], []
for _ in range(60_000):
    side, size = rng.choice((-1.0, 1.0)), rng.uniform(1, 10)
    mid += impact * side * size + rng.gauss(0, 0.01)
    prices.append(mid + side * spread / 2)
    volumes.append(size)
    sides.append(side)


changes = [0.0] + [b - a for a, b in zip(prices, prices[1:])]
print(f"Roll spread            {ms.get_roll_measure(prices[-10_000:], 5000)[-1]:.4f}   (true 0.1000)")
print(f"Kyle lambda, by trade  {ms.get_trades_based_kyle_lambda(changes, volumes, sides):.5f}  (true 0.00200 + bounce)")


# The same trades as 1,200 bars of 50: only high, low, close and volume survive.
bars = [prices[i:i + 50] for i in range(0, len(prices), 50)]
high, low = [max(b) for b in bars], [min(b) for b in bars]
cs = [v for v in ms.get_corwin_schultz_estimator(high, low, 20) if v == v]
print(f"Corwin-Schultz spread  {sum(cs) / len(cs):.5f}  (true 0.00100 relative); "
      f"{sum(v == 0 for v in cs) / len(cs):.0%} of bars floored at zero")


# Order-flow toxicity from bar data alone: bulk volume classification, then VPIN.
close = [b[-1] for b in bars]
bar_volume = [sum(volumes[i:i + 50]) for i in range(0, len(volumes), 50)]
true_buys = [sum(v for v, s in zip(volumes[i:i + 50], sides[i:i + 50]) if s > 0)
             for i in range(0, len(volumes), 50)]
bvc = ms.get_bvc_buy_volume(close, bar_volume, 20)
pairs = [(e / v, t / v) for e, t, v in zip(bvc, true_buys, bar_volume) if e == e]
mx, my = sum(a for a, _ in pairs) / len(pairs), sum(b for _, b in pairs) / len(pairs)
corr = (sum((a - mx) * (b - my) for a, b in pairs)
        / (sum((a - mx) ** 2 for a, _ in pairs) * sum((b - my) ** 2 for _, b in pairs)) ** 0.5)
print(f"BVC buy share vs true buy share: correlation {corr:.2f}")
Roll spread            0.1127   (true 0.1000)
Kyle lambda, by trade  0.00942  (true 0.00200 + bounce)
Corwin-Schultz spread  0.00128  (true 0.00100 relative); 25% of bars floored at zero
BVC buy share vs true buy share: correlation 0.67

Three of these are good news with a caveat each. Roll recovers the spread to within 13%; it runs high here because price impact adds negative autocorrelation of its own, which the model attributes to the spread. Corwin–Schultz gets the right order of magnitude from highs and lows alone, but a quarter of its values are exactly zero: the estimator goes negative whenever a window’s volatility swamps the spread and is floored, so its mean is usable and its individual values are not. Bulk volume classification, given nothing but closes and volume, correlates 0.67 with the true buy share.

Kyle’s lambda is the instructive failure: 0.0094 against a true 0.0020. The regression is of transaction price changes on signed volume, and a buy prints at the ask, so every trade carries half a spread in the direction of its sign. The estimate is impact plus bounce. Regress mid-price changes if you have quotes; otherwise read the trades-based lambda as a cost of trading, not as permanent impact.

Entropy of a price sequence

Section titled “Entropy of a price sequence”

Chapter 18’s idea is that a market in which prices are unpredictable produces incompressible sequences, and that a drop in entropy — patterns appearing — marks inefficiency worth investigating. The route has two steps.

Encode the series as a string. encode_tick_rule_array maps tick signs to letters (1 → a, −1 → b, 0 → c). For returns or volumes, quantile_mapping(values, n_letters) builds a codebook with equally populated bins and sigma_mapping(values, step) one with bins of fixed width; encode_array applies either, one letter per value. Both mappings reject an empty array and NaN values, sigma_mapping rejects a step that is not a positive finite number, and encode_array raises on a NaN value or an empty codebook rather than dropping anything, so the message is always as long as the input.

Estimate the entropy of the string, in bits per symbol:

FunctionEstimator
get_shannon_entropy(msg)−∑plog⁡2p-\sum p\log_2 p over single symbols; ignores order entirely
get_plug_in_entropy(msg, w)Shannon entropy of overlapping words of length w, divided by w
get_lempel_ziv_entropy(msg)size of the Lempel–Ziv dictionary divided by message length; lower means more repetition
get_konto_entropy(msg, window)Kontoyiannis’ estimator from longest-match lengths; window = 0 uses an expanding window, and a fixed window is clamped to half the message length (Snippet 18.4)

Only the last three can see structure. The string abababab… has a Shannon entropy of exactly 1 bit, the same as a fair coin, because it has as many as as bs; its plug-in entropy with three-letter words is 0.33 and its Kontoyiannis entropy 0.21, against 1.00 and 0.87 for a random string of the same length (400 symbols, a Kontoyiannis window of 20).

From Rust

Section titled “From Rust”
use openquant::microstructural_features::{
    encode_tick_rule_array, get_lempel_ziv_entropy, get_plug_in_entropy, get_shannon_entropy,
    get_trades_based_kyle_lambda, get_vpin, vwap,
};


// Regression through the origin of price change on signed volume: sum(xy) / sum(x^2).
let lambda = get_trades_based_kyle_lambda(&[0.2, -0.1, 0.4], &[10.0, 5.0, 20.0], &[1.0, -1.0, 1.0])?;
assert!((lambda - 0.02).abs() < 1e-12);


// VPIN over 3 bars of equal volume: mean |buys - sells| / volume. NaN until the window fills.
let vpin = get_vpin(&[100.0; 5], &[80.0, 20.0, 50.0, 90.0, 10.0], 3)?;
assert!(vpin[0].is_nan() && vpin[1].is_nan());
assert!((vpin[2] - 0.4).abs() < 1e-12);


assert!((vwap(&[1000.0, 2000.0], &[10.0, 10.0])? - 150.0).abs() < 1e-12);


// A perfectly periodic message: maximal Shannon entropy, low entropy by every other measure.
assert_eq!(encode_tick_rule_array(&[1, 1, -1, 0])?, "aabc");
assert_eq!(get_shannon_entropy("abababab"), 1.0);
assert_eq!(get_plug_in_entropy("abababab", 2)?, 0.5);
assert_eq!(get_lempel_ziv_entropy("abababab"), 0.5);

MicrostructuralFeaturesGenerator::new_from_csv(path, tick_num_series, volume_encoding, pct_encoding) streams a three-column trades file — timestamp, price, volume — and emits the trades-based features and entropies per bar. tick_num_series holds the cumulative trade numbers (1-based) at which bars close, such as the tick numbers of volume or dollar bars built from the same file — [100, 250, 400], not per-bar counts like [100, 150, 150]. It is Rust-only.

What to watch for

Section titled “What to watch for”
  • get_trades_based_hasbrouck_lambda changed in a fix. Earlier versions regressed the absolute return on signed volume, so buys and sells cancelled and the estimate was near zero under balanced flow. It now regresses the signed return, as in AFML: on 5,000 simulated trades with a true λ\lambda of 1e-5 it returns 1.0001e-5, where it returned −4.2e-8 (#105). Results computed before the fix are not comparable.
  • The Roll measure takes the absolute value of the covariance. Roll’s formula needs a negative autocovariance and is undefined otherwise. Following mlfinlab, this function uses 2∣cov⁡∣2\sqrt{\lvert\operatorname{cov}\rvert}, so a trending series with positive autocovariance also reports a “spread”. Check the sign of the autocovariance yourself on anything that is not a liquid, mean-reverting tick series.
  • VPIN divides by the current bar’s volume, not the window’s. It is the published measure only on volume bars, where every bar has the same volume. On time bars a quiet bar inflates it and a busy bar deflates it.
  • Bar-based lambdas are means of ratios, not regressions. One bar with tiny volume contributes an enormous Δp/V\Delta p/V and dominates the window. Winsorise, or use volume or dollar bars so that the denominators are comparable.
  • Codebooks start at character 0. quantile_mapping and sigma_mapping assign letters from the start of the 256-character table, so encoded strings begin with control characters including NUL. They are fine as input to the entropy functions and unsafe to print, log, or pass through anything that treats a string as C text. Alphabets are capped at 256 symbols.
  • Entropy estimates are biased on short messages, downward for plug-in with long words (most words are seen once) and erratically for Lempel–Ziv. Compare messages of equal length and encoding; the level means little, the change over time is the feature.
  • The Lempel–Ziv and Kontoyiannis estimators are quadratic or worse in message length. Keep messages to a bar’s worth of ticks, not a day’s.
Section titled “Related modules”
  • data-structures — the bars these features are computed on; volume and dollar bars suit most of them better than time bars.
  • streaming-hpc — incremental VPIN and concentration for live data.
  • util-volatility — range-based volatility estimators related to Bekker–Parkinson.
  • structural-breaks — the other family of regime features.

API reference

Section titled “API reference”

Python

Section titled “Python”

Signatures, parameters, return values and errors for every function: openquant.microstructural.

Rust

Section titled “Rust”

Rustdoc for this commit, with every public item of openquant::microstructural_features (after a crates.io release the same pages are on docs.rs):