openquant.data

Pure Python. Signatures are as written in the source; descriptions are the docstrings. All Python modules.

Classes

CacheMissError

class CacheMissError(LookupError)

Raised by fetch(..., offline=True) when a request is not in the cache.

CallableSource

class CallableSource

Adapt a function fn(symbol, start, end) -> frame into a DataSource.

This is the pattern for user-supplied, fetch-only adapters: fn calls the user's own vendor account, reading the user's own key from an environment variable inside fn (never pass or store the key here). Fetched data is cached on the user's machine and is never part of the repository.

CallableSource.__init__

__init__(
    fn: Callable[[str, date, date], Any],
    *,
    name: str,
    version: str = '',
    terms: str | None = None,
) -> None

CallableSource.fetch_symbol

fetch_symbol(symbol: str, start: date, end: date) -> Any

Call the wrapped function as fn(symbol, start, end) and return its result unchanged.

Parameters

symbol : str

Symbol to fetch.

start : datetime.date

First date requested (inclusive).

end : datetime.date

Last date requested (inclusive).

Returns

Any

Whatever fn returns; fetch converts it with polars.DataFrame(...) if needed.

DataSource

class DataSource(Protocol)

What fetch needs from a market data source.

name identifies the source in the cache path and the run manifest; keep it stable. fetch_symbol returns daily bars for one symbol with dates in [start, end] (inclusive), as a Polars DataFrame or anything polars.DataFrame(...) accepts, with any column names clean_ohlcv recognizes (date/ts, open, high, low, close, volume, optional adj_close and symbol).

Optional attributes: version (a string; changing it starts a new cache namespace for the source) and terms (a URL or statement of the data's terms, copied into fetch's metadata so it lands in the run manifest).

DataSource.fetch_symbol

fetch_symbol(symbol: str, start: date, end: date) -> Any

Return daily OHLCV bars for one symbol with dates in [start, end].

Parameters

symbol : str

Symbol to fetch.

start : datetime.date

First date requested (inclusive).

end : datetime.date

Last date requested (inclusive).

Returns

Any

A polars DataFrame, or anything polars.DataFrame(...) accepts, with column names clean_ohlcv recognizes. Rows for other symbols make fetch fail; rows outside [start, end] are filtered out by fetch.

LocalFileSource

class LocalFileSource

A source backed by one local CSV/Parquet OHLCV file with a symbol column.

version is derived from the file's bytes, so editing the file starts a new cache namespace instead of serving stale bars.

LocalFileSource.__init__

__init__(
    path: str | Path,
    *,
    name: str = 'local-file',
    terms: str | None = None,
) -> None

LocalFileSource.symbols

symbols() -> list[str]

Sorted list of the distinct symbols in the file (loads the file on first use).

LocalFileSource.fetch_symbol

fetch_symbol(symbol: str, start: date, end: date) -> pl.DataFrame

Return all rows of the file for symbol.

The file is loaded and cleaned with load_ohlcv once and cached on the instance. start and end are accepted for the DataSource interface but not applied here; fetch filters the returned rows to [start, end].

Parameters

symbol : str

Symbol to return.

start : datetime.date

Unused.

end : datetime.date

Unused.

Returns

polars.DataFrame

The canonical clean_ohlcv columns, restricted to symbol.

Raises

LookupError

If symbol is not in the file.

LocalSampleSource

class LocalSampleSource(LocalFileSource)

The SYNTHETIC daily OHLCV sample shipped with openquant.

Symbols SYN_A ... SYN_E (not real tickers), Monday-Friday from 2022-01-03 to 2023-12-29, generated by scripts/data/make_synthetic_sample.py from a fixed seed. It is not market data; it exists so tests, docs and runbooks run offline.

LocalSampleSource.__init__

__init__() -> None

Functions

align_calendar

align_calendar(
    df: pl.DataFrame,
    *,
    interval: str = '1d',
    return_report: bool = False,
) -> pl.DataFrame | tuple[pl.DataFrame, dict[str, Any]]

Reindex each symbol onto a regular time grid, marking the bars that are missing.

The frame is cleaned with clean_ohlcv, then for each symbol a grid is built from that symbol's first to last timestamp (inclusive) in steps of interval, anchored at its first timestamp. Grid points with no row get nulls in the price and volume columns and is_missing_bar=True. Nothing is forward-filled. A daily grid includes weekends and holidays.

A bar whose timestamp is not on its symbol's grid (for example a daily bar stamped at a different time of day) is **not** in the output. With return_report=True those bars are returned in the report; otherwise a UserWarning names how many were dropped.

Parameters

df : polars.DataFrame

Long OHLCV frame with ts, symbol, open, high, low, close and volume (or their aliases).

interval : str, default "1d"

Grid step: an integer followed by d, h, m or s (for example "1h").

return_report : bool, default False

Also return a report of what alignment removed, instead of warning about it.

Returns

polars.DataFrame or tuple of (polars.DataFrame, dict)

Columns ts, symbol, open, high, low, close, volume, adj_close, is_missing_bar, sorted by symbol then ts; is_missing_bar is True where the grid point had no input row. With return_report=True, (aligned, report), where report has off_grid_bars (the dropped bars, canonical columns), off_grid_bar_count and rows_removed_by_deduplication.

Raises

ValueError

If interval does not have a supported form or is not positive, or a required column is missing.

clean_ohlcv

clean_ohlcv(
    df: pl.DataFrame,
    *,
    dedupe_keep: Literal['first', 'last'] = 'last',
    return_report: bool = False,
) -> pl.DataFrame | tuple[pl.DataFrame, dict[str, Any]]

Normalize a long OHLCV frame to the canonical schema, sorted and de-duplicated.

Column names are matched case-insensitively against known aliases (for example date, timestamp or datetime for ts, ticker for symbol, adjusted_close for adj_close). ts is cast to string and parsed to Datetime (a "%Y-%m-%d %H:%M:%S[.f]" form exactly, anything else by polars' inference); symbol is cast to string and the price and volume columns to Float64. A missing adj_close is filled from close. Rows with a null in any canonical column, including timestamps that fail to parse, are dropped silently. Extra columns are discarded.

Parameters

df : polars.DataFrame

Long OHLCV frame with ts, symbol, open, high, low, close and volume (or their aliases), optionally adj_close.

dedupe_keep : {"first", "last"}, default "last"

Which row to keep among rows sharing a (symbol, ts) key, by position after the frame is sorted by symbol and ts.

return_report : bool, default False

Also return a data quality report of the cleaned frame.

Returns

polars.DataFrame or tuple[polars.DataFrame, dict[str, Any]]

Columns ts, symbol, open, high, low, close, volume, adj_close, sorted by symbol then ts. With return_report=True, a (frame, report) tuple; the report has the keys of data_quality_report, with rows_removed_by_deduplication set to the number of duplicate rows dropped, duplicate_key_count 0, and null_counts counting the nulls of the input (whose rows were dropped).

Raises

ValueError

If dedupe_keep is not "first" or "last", or a required column is missing.

data_quality_report

data_quality_report(df: pl.DataFrame) -> dict[str, Any]

Summarize row counts, duplicate keys, gaps and time range of an OHLCV frame.

The frame is normalized as in clean_ohlcv (aliases, casts) but not de-duplicated, so duplicate keys are counted rather than removed. Rows with a null in a canonical column are counted in null_counts and then left out of every other figure.

Parameters

df : polars.DataFrame

Long OHLCV frame with ts, symbol, open, high, low, close and volume (or their aliases).

Returns

dict[str, Any]

row_count (rows without nulls), symbol_count, duplicate_key_count (rows repeating the previous row's (symbol, ts)), inferred_interval_us (the most common bar spacing), gap_interval_count (consecutive bars of one symbol with a skipped weekday between them on daily data, or further apart than the inferred spacing otherwise), ts_min and ts_max (as "%Y-%m-%d %H:%M:%S" strings, with .ffffff when there are microseconds; None when empty), rows_removed_by_deduplication (always 0 here) and null_counts (nulls per canonical column of the input after aliasing and casting: a timestamp that does not parse counts as a null ts, and a missing adj_close has close's nulls).

Raises

ValueError

If a required column is missing.

dataset_hash

dataset_hash(df: pl.DataFrame) -> str

Deterministic content hash of a table, as "sha256:<64 hex>".

The hash describes the data, not its layout. Definition (oq-dataset-sha256-v1):

1. Columns are taken in sorted name order. 2. Each column is mapped to a kind: int, float, str, bool, date (days since the Unix epoch) or datetime[us,<tz>] (microseconds since the Unix epoch). Other types raise TypeError. 3. Rows are sorted by every column in that order (nulls last), so input row order does not matter; duplicate rows count. 4. The SHA-256 input is the line oq-dataset-sha256-v1, one line "<name>":<kind> per column, then one line per row of tab-separated fields: floats as Python repr (exact round-trip), strings as JSON, integers in decimal, booleans as true/false and nulls as \N.

It does not depend on Parquet/Arrow byte layout or the Polars version. Any change to a value, a column name, a column type or the set of rows changes it.

default_cache_dir

default_cache_dir() -> Path

$OPENQUANT_DATA_CACHE, else $XDG_CACHE_HOME/openquant/data, else ~/.cache/openquant/data.

fetch

fetch(
    symbols: str | Iterable[str],
    start: date | datetime | str,
    end: date | datetime | str,
    *,
    source: DataSource | None = None,
    cache_dir: str | Path | None = None,
    refresh: bool = False,
    offline: bool = False,
    return_meta: bool = False,
) -> pl.DataFrame | tuple[pl.DataFrame, dict[str, Any]]

Fetch daily OHLCV bars for symbols over [start, end] through a local cache.

Each (source, symbol, start, end) request is cached as one Parquet file (plus a JSON sidecar) under cache_dir, default :func:default_cache_dir. A repeated request is read from disk without calling the source, so it works offline. source defaults to :class:LocalSampleSource, which is SYNTHETIC data. refresh=True refetches; offline=True raises :class:CacheMissError instead of calling the source.

Returns the canonical frame of :func:clean_ohlcv (ts, symbol, open, high, low, close, volume, adj_close, sorted by symbol then ts), which has passed :func:data_quality_report (see :func:quality_failures). With return_meta=True it also returns provenance for a run manifest: the source's name, version and terms, the request, each symbol's cache status and the frame's :func:dataset_hash.

load_ohlcv

load_ohlcv(
    path: str | Path,
    *,
    symbol: str | None = None,
    return_report: bool = False,
) -> pl.DataFrame | tuple[pl.DataFrame, dict[str, Any]]

Read an OHLCV CSV or Parquet file and clean it with clean_ohlcv.

Parameters

path : str or pathlib.Path

File to read; the suffix selects the reader (.csv, .parquet or .pq, case-insensitive).

symbol : str or None, default None

Symbol to assign to every row when the file has no symbol column (or alias). Ignored when the file has one.

return_report : bool, default False

Also return the clean_ohlcv quality report.

Returns

polars.DataFrame or tuple[polars.DataFrame, dict[str, Any]]

The canonical frame of clean_ohlcv, or (frame, report) with return_report=True.

Raises

FileNotFoundError

If path does not exist.

ValueError

If the suffix is unsupported, the file has no symbol column and symbol is None, or a required OHLCV column is missing.

quality_failures

quality_failures(report: Mapping[str, Any]) -> list[str]

Reasons a :func:data_quality_report fails fetch's bar; empty when it passes.

The bar: at least one row, no duplicate (symbol, ts) keys, no nulls. Gaps are reported but not failed, since daily data has weekends and holidays.

record_dataset_hash

record_dataset_hash(
    manifest: dict[str, Any],
    frame: pl.DataFrame | None = None,
    *,
    digest: str | None = None,
    **provenance: Any,
) -> dict[str, Any]

Record a dataset's content hash, and its provenance, in a run manifest.

Sets manifest["dataset_hash"] and merges hash, hash_version, rows and any provenance keywords (for example the metadata returned by fetch(..., return_meta=True)) into manifest["dataset"]. Pass the frame, or a digest already computed by :func:dataset_hash. Returns manifest, modified in place.