Skip to content

StratCraftsAI/histdata

BranchesTags

Repository files navigation

histdata

Forex Historical Data Downloader and Parquet Converter

Python 3.10+ | histdata.com | 66 Currency Pairs | Apache Parquet Output

CI Python 3.10+ Parquet zstd MIT License 66 Pairs Mentioned in Awesome

Features | Quick Start | Usage | Data Format | Pairs | Docs | Community

Why histdata?

histdata is a simple Python ETL tool that downloads free Forex historical data from histdata.com and converts it to Apache Parquet format with zstd compression. It is designed for quantitative researchers and trading system developers who need clean, columnar market data without manual download steps.

"One command to download 20+ years of 1-minute Forex data across 66 pairs, ready for pandas, Polars, or any Arrow-compatible tool."

Features

  • 66 Currency Pairs - Forex majors, crosses, commodities (WTI, Brent), and indices (S&P 500, Nikkei, DAX, etc.)
  • Automatic Download - Two-step token extraction (GET page + POST download) with polite 2-second throttle
  • Smart Year Handling - Past years download as single yearly ZIPs; current year downloads month-by-month
  • Parquet Output - Apache Parquet with zstd compression, ready for pandas/Polars/DuckDB/Spark
  • Typed Schema - timestamp[ms], open, high, low, close, volume (all float64)
  • CLI Tool - Install once, run from anywhere via the histdata command
  • Zero Heavy Dependencies - Only requests, beautifulsoup4, and pyarrow

Quick Start

Installation

pip install git+https://github.com/StratCraftsAI/histdata.git

Or install from source:

git clone https://github.com/StratCraftsAI/histdata.git
cd histdata
pip install -e .

Requirements

  • Python: 3.10+
  • Dependencies: requests, beautifulsoup4, pyarrow (installed automatically)

First Run

# Download EURUSD 2020-2025 and convert to Parquet
histdata --pairs EURUSD --year-start 2020 --year-end 2025

Usage

Download Specific Pairs

# Single pair
histdata --pairs EURUSD --year-start 2020 --year-end 2025

# Multiple pairs
histdata --pairs EURUSD GBPUSD USDJPY --year-start 2015 --year-end 2025

# All 23 major Forex pairs (default)
histdata --year-start 2020 --year-end 2025

# All 66 pairs (Forex + Commodities + Indices)
histdata --pairs all --year-start 2020 --year-end 2025

Convert Existing ZIPs

# Skip download, only convert previously downloaded ZIPs
histdata --skip-download --zip-dir ./downloads

Options

Option Default Description
--pairs Major Forex (23) Currency pairs to download. Use all for all 66
--year-start 2020 First year to download
--year-end 2025 Last year to download
--timeframe M1 Timeframe: M1 (1-minute bars) or tick
--output-dir ./output Directory for Parquet output
--zip-dir ./downloads Directory for downloaded ZIPs
--skip-download off Skip download, convert existing ZIPs only
-v, --verbose off Enable debug logging

Python API

from histdata_supplementary.downloader import download_pair
from histdata_supplementary.parser import parse_zip
from histdata_supplementary.converter import write_parquet

import requests
from pathlib import Path

session = requests.Session()

# Download EURUSD 2023
zips = download_pair("EURUSD", 2023, 2023, Path("./downloads"), session=session)

# Parse and convert
for zip_path in zips:
    rows = parse_zip(zip_path)
    write_parquet(rows, Path(f"./output/EURUSD_M1.parquet"))

Read Output with pandas

import pandas as pd

df = pd.read_parquet("output/EURUSD_M1.parquet")
print(df.head())
#              timestamp     open     high      low    close  volume
# 0 2023-01-02 17:00:00  1.07045  1.07048  1.07045  1.07048     0.0
# 1 2023-01-02 17:01:00  1.07045  1.07050  1.07044  1.07050     0.0

Data Format

Source

  • Provider: histdata.com (free registration, no API key)
  • Format: Generic ASCII CSV, semicolon-delimited
  • Layout: DateTime;OPEN;HIGH;LOW;CLOSE;Volume
  • Timestamps: EST (Eastern Standard Time, no DST adjustment)
  • Timestamp format: YYYYMMDD HHMMSS
  • Past years: One ZIP per year
  • Current year: One ZIP per month

Output

  • Format: Apache Parquet
  • Compression: zstd
  • Schema:
Column Type Description
timestamp timestamp[ms] Bar open time (EST)
open float64 Open price
high float64 High price
low float64 Low price
close float64 Close price
volume float64 Tick volume

Supported Pairs

Forex (23 pairs)

Majors Crosses JPY Crosses
EURUSD EURAUD AUDJPY
GBPUSD EURCAD CADJPY
USDJPY EURCHF CHFJPY
USDCHF EURGBP EURJPY
USDCAD EURNZD GBPJPY
AUDUSD GBPAUD NZDJPY
NZDUSD GBPCAD
GBPCHF
GBPNZD
AUDNZD

Commodities (2 pairs)

Pair Description
BCOUSD Brent Crude Oil
WTIUSD WTI Crude Oil

Indices (9 pairs)

Pair Description
AUXAUD ASX 200
FRXEUR CAC 40
GRXEUR DAX 30
HKXHKD Hang Seng
JPXJPY Nikkei 225
NSXUSD NASDAQ 100
SPXUSD S&P 500
UDXUSD US Dollar Index
UKXGBP FTSE 100

Project Structure

histdata/
├── histdata_supplementary/
│   ├── __init__.py
│   ├── cli.py            # CLI entry point
│   ├── converter.py      # OHLCVRow -> PyArrow Table -> Parquet (zstd)
│   ├── downloader.py     # GET token + POST download ZIP (2s throttle)
│   ├── pairs.py          # 66 currency pair constants
│   └── parser.py         # Unzip -> parse ASCII CSV -> OHLCVRow
├── pyproject.toml
├── LICENSE
├── README.md
├── CONTRIBUTING.md
├── CODE_OF_CONDUCT.md
├── SECURITY.md
└── SUPPORT.md

Pipeline

histdata.com  ──(GET page)──>  Extract Token
                                    │
                              (POST download)
                                    │
                                    v
                                  ZIP file
                                    │
                              (unzip + parse CSV)
                                    │
                                    v
                              list[OHLCVRow]
                                    │
                              (PyArrow conversion)
                                    │
                                    v
                            Parquet (zstd compressed)

FAQ

Do I need an API key?

No. histdata.com provides free data without registration or API keys.

How long does a full download take?

Approximately 2 seconds per ZIP file (polite throttle). Downloading all 66 pairs for 5 years takes roughly 10-15 minutes.

Can I use this data with StratCraft?

Yes. The Parquet output follows the standard OHLCV schema and can be loaded directly into StratCraft as a data source.

What timezone are the timestamps in?

All timestamps are in EST (Eastern Standard Time, UTC-5). histdata.com does not apply DST adjustment.

Does it support tick data?

Pass --timeframe tick to download tick-level data instead of 1-minute bars. Tick data parsing uses the same pipeline.

Documentation

Community

Contact

For questions or collaboration: nonagonal.portal@gmail.com

Contributing

This project is maintained by StratCraftsAI.

  • Issues & Discussions: Welcome for bug reports, data quality questions, and feature ideas
  • Pull Requests: Bug fixes welcome (see CONTRIBUTING.md)
  • Feature PRs require prior discussion in Issues

License

MIT License - See LICENSE file.

Built for quantitative researchers who need clean Forex data without the hassle

About

No description, website, or topics provided.

Resources

Readme

License

MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages