Historical — API Reference

Historical([key]; gateway=DEFAULT_HIST_GATEWAY, timeout=100, user_agent=...)

Client for the Databento Historical (HTTP) API.

key falls back to ~/.databento/config.toml then ENV["DATABENTO_API_KEY"] (see load_api_key).

client = Historical()
store  = get_range(client; dataset="XNAS.ITCH", schema=Schema.TRADES,
                          symbols=["AAPL"], start=DateTime(2024,1,2,14,30),
                          end_=DateTime(2024,1,2,14,31), stype_in=SType.RAW_SYMBOL)

get_range(client; dataset, schema, symbols, start, end_=nothing,
          stype_in=SType.RAW_SYMBOL, stype_out=SType.INSTRUMENT_ID,
          limit=nothing, typed=true, size_hint=nothing)

Fetch a time range of records from the Databento Historical API. Returns a DBNStore.

The endpoint streams zstd-compressed DBN bytes; this function decompresses and decodes them in memory using DBN.jl. For very large ranges consider submit_job instead.

When typed=true (default) and the schema is type-pure (almost all schemas are, the exception is Schema.MIX), the records vector is Vector{T} for the concrete record type — roughly 10× faster decode and 60% less allocation than the generic path. Pass typed=false to force the legacy Vector{DBN.DBNRecord} return for callers that rely on the Union element type.

size_hint lets callers who know the record-count bound (e.g. from a prior get_record_count call) pre-size the records vector exactly. The default heuristic over-allocates by 10–20% to avoid realloc churn under growth; an exact hint avoids that waste.

foreach_record(f, client::Historical; dataset, schema, symbols, start, end_=nothing,
               stype_in=SType.RAW_SYMBOL, stype_out=SType.INSTRUMENT_ID,
               limit=nothing, record_type=nothing)

Streaming variant of get_range. Calls f(record) for each DBN record as it arrives off the wire — overlaps HTTP download with decompress + decode and never materialises the compressed payload or the records vector in memory. Use this for very large queries where get_range's in-memory buffering would be wasteful.

By default, the concrete record type is inferred from schema (e.g. Schema.TRADES → DBN.TradeMsg) and the type-specific zero-allocation decode path is used (~2× faster than generic dispatch). Pass record_type = DBN.DBNRecord to force the generic Union-typed path, or record_type = SomeT to override.

Returns the DBN.Metadata from the response header.

list_publishers(client::Historical)

List every Databento publisher (venue / data provider). Returns a JSON array of objects with at least publisher_id, dataset, and description. Wraps GET /v0/metadata.list_publishers.

list_datasets(client; start_date=nothing, end_date=nothing)

List Databento datasets, optionally filtered to those that have data in the [start_date, end_date] window. Returns a JSON array of dataset names ("XNAS.ITCH", "OPRA.PILLAR", …). Wraps GET /v0/metadata.list_datasets.

list_schemas(client; dataset)

List the schemas offered by dataset (e.g. ["trades", "tbbo", "mbp-1", ...]). Wraps GET /v0/metadata.list_schemas.

list_fields(client; schema=nothing, encoding=nothing)

List the fields available for a given (schema, encoding) combination. schema accepts a Schema value or its lowercase string; encoding accepts an Encoding value or its lowercase string. Either kwarg can be nothing to broaden the query. Wraps GET /v0/metadata.list_fields.

list_unit_prices(client; dataset)

Per-feed-mode and per-schema price list for dataset (USD per GB). Useful as an input to manual cost calculations; for a quote on a specific query, prefer get_cost. Wraps GET /v0/metadata.list_unit_prices.

get_dataset_range(client; dataset)

Earliest and latest available timestamps for dataset. Use the result to bound start/end_ on subsequent get_range or submit_job calls. Wraps GET /v0/metadata.get_dataset_range.

get_dataset_condition(client; dataset, start_date=nothing, end_date=nothing)

Per-date data quality / availability status for dataset in the [start_date, end_date] window. Helpful for skipping known-bad days before running a long backfill. Wraps GET /v0/metadata.get_dataset_condition.

get_record_count(client; dataset, symbols, schema, start, end_=nothing,
                 stype_in=SType.RAW_SYMBOL, limit=nothing)

Count of records the same query would return from get_range without actually downloading them. Free; use this before issuing an expensive get_range on an unknown date range. Wraps GET /v0/metadata.get_record_count.

get_billable_size(client; dataset, symbols, schema, start, end_=nothing,
                  stype_in=SType.RAW_SYMBOL, limit=nothing)

Billable size in bytes (uncompressed CSV-equivalent) the same query would return from get_range. Free; combine with list_unit_prices or use get_cost directly for a dollar estimate. Wraps GET /v0/metadata.get_billable_size.

get_cost(client; dataset, symbols, schema, start, end_=nothing,
         mode=nothing, stype_in=SType.RAW_SYMBOL, limit=nothing)

Dollar cost preview for the same query as get_range under the given FeedMode (default HISTORICAL). Returns a number in USD. Free; call before issuing large queries. Wraps GET /v0/metadata.get_cost.

submit_job(client; dataset, symbols, schema, start, end_=nothing, ...) -> Dict

Submit an asynchronous batch job. The Historical API queues the request and returns a job_id; poll list_jobs until the JobState is DONE, then call list_files and batch_download to fetch the results.

Useful when the synchronous get_range would be too large to stream back in one HTTP response. Cost is the same as the equivalent get_range query; preview it with get_cost before submitting.

Optional kwargs control output framing:

• encoding — "dbn" (default), "csv", "json".
• compression — "zstd" (default), "none", "zip".
• stype_out — output symbology (default SType.INSTRUMENT_ID).
• split_duration/split_size/split_symbols — slice the output across multiple files; see SplitDuration.
• packaging — bundle files for download; see Packaging.
• delivery — destination; see Delivery.
• limit — cap on records returned.

Wraps POST /v0/batch.submit_job.

list_jobs(client; states=nothing, since=nothing)

List batch jobs submitted under the calling API key, optionally filtered to one or more JobState values (states accepts a single state, a vector, or a comma-separated string) and/or to jobs created after since (DateTime, ISO-8601 string, or unix-ns integer). Returns a JSON array of job objects. Wraps GET /v0/batch.list_jobs.

list_files(client; job_id)

List the output files for a batch job identified by job_id. Each entry includes filename, size, hash (sha256), and a urls.https download URL. Used internally by batch_download but exposed for callers that need to inspect or selectively fetch files. Wraps GET /v0/batch.list_files.

batch_download(client; job_id, output_dir, verify_hash=true,
               filenames=nothing, overwrite=false)

Download files from a completed batch job into output_dir. Returns the list of written file paths.

Each file's download URL is taken from list_files' urls.https field. The response also includes the file's sha256 hash; when verify_hash=true the downloaded bytes are checked against it.

By default every file in the job is downloaded. Pass filenames to restrict to a subset (matched by exact filename). Pass overwrite=true to re-download a file that's already present and the right size.

Named batch_download to avoid clashing with Base.download.

resolve(client; dataset, symbols, stype_in, stype_out,
        start_date, end_date=nothing)

Resolve symbols from stype_in (e.g. SType.RAW_SYMBOL) to stype_out (e.g. SType.INSTRUMENT_ID) within the [start_date, end_date] date range. Returns a JSON object whose result field maps each input symbol to a list of (d0, d1, s) intervals where the mapping is valid (instruments expire, roll, get renamed) — see SymbologyResolution for the per-symbol status convention.

Example:

resolve(client; dataset = "GLBX.MDP3", symbols = ["ES.FUT"],
        stype_in = SType.PARENT, stype_out = SType.INSTRUMENT_ID,
        start_date = Date(2024,1,2), end_date = Date(2024,1,5))

Wraps POST /v0/symbology.resolve.