Data Engineering Skills for Claude

Expert knowledge skills for Claude Code and Claude.ai covering modern data engineering technologies: table formats, stream processing, streaming AI agents, streaming storage, ML-native formats, and local orchestration.

Each skill follows the Anthropic Agent Skills standard: a folder containing a SKILL.md with YAML frontmatter that Claude loads on demand when the trigger conditions match.

Available Skills

Skill	Domain	File
Apache Iceberg	Open table format / lakehouse	iceberg/SKILL.md
Apache Paimon	Streaming lake format	paimon/SKILL.md
Apache Fluss	Streaming storage for real-time analytics	fluss/SKILL.md
Apache Flink	Stream processing framework	flink/SKILL.md
Apache Flink Agents	Streaming AI agents, durable execution, and recovery	flink-agents/SKILL.md
Apache Iggy	Rust-native message streaming	iggy/SKILL.md
Lance	Columnar format for ML/AI + vector search	lance/SKILL.md
Firn	Object-storage-backed vector and full-text search	firn/SKILL.md
Docker Compose	Container orchestration (v2 and v5)	docker-compose/SKILL.md

Release and compatibility guidance was refreshed on 3 November 2026. Each skill includes dated guidance and upstream sources. These are verified snapshots, not automatic dependency upgrades; check the installed versions and current compatibility matrix before applying them.

The Flink Agents skill focuses on integration and operations. Apache also maintains flink-agents-dev for application scaffolding, authoring references, and YAML schemas. Flink Agents 0.4.0 is still a preview release.

How Skills Work

Skills use the Agent Skills progressive disclosure model:

1. YAML frontmatter (always loaded). name and description tell Claude when the skill is relevant.
2. SKILL.md body (loaded on trigger). Core instructions and guidance.
3. Bundled files (optional, loaded on demand). Deeper references, scripts, or templates referenced from SKILL.md.

The current skills intentionally keep only concise SKILL.md files. Add references/, scripts/, or assets/ only when a skill needs substantial offline detail, deterministic helpers, or reusable output assets.

Using These Skills

Claude Code

Place a skill folder (or symlink it) into ~/.claude/skills/ or a project .claude/skills/ directory. Claude Code discovers SKILL.md files automatically and loads them based on the frontmatter description.

Alternatively, reference a skill file directly in a conversation:

@iceberg/SKILL.md help me migrate an Iceberg v2 table to v3 deletion vectors

Claude.ai

Zip a skill folder and upload via Settings → Capabilities → Skills.

Claude API

Pass the skill via container.skills on the Messages API (requires the Code Execution Tool beta). See Using Agent Skills with the API.

Skill Structure

Each skill conforms to the Agent Skills standard:

skill-name/
└── SKILL.md     # Required: YAML frontmatter + Markdown instructions

Minimum frontmatter:

---
name: skill-name
description: What the skill does and when Claude should use it (trigger phrases).
---

Contributing

To add a new skill:

1. Create a kebab-case folder: your-skill-name/.
2. Add a SKILL.md with valid YAML frontmatter (name matches the folder; description includes both what it does and when to use it, with specific trigger phrases).
3. Cover the required sections: Scope, Inspect First, Safety, Verify, and Update Checklist. These keep each skill an operating guide rather than a reference article.
4. Keep SKILL.md focused; move deep reference material to references/ and executable helpers to scripts/ within the skill folder.
5. Update the table above.
6. Date release-specific guidance and link authoritative sources. Check engine, connector, and client compatibility independently; distinguish stable releases from previews and development branches.

Run the validator before opening a pull request:

pip install pyyaml
python3 .github/scripts/validate_skills.py

It checks frontmatter validity, name and directory agreement, description length, required sections, file size, directory layout, relative links, and README coverage. A skill folder may contain only SKILL.md plus references/, scripts/, and assets/. CI runs the same script on every pull request, alongside a weekly link check.

For full authoring guidance, see Anthropic's Complete Guide to Building Skills for Claude (PDF, ~33 pages) and the Skill authoring best practices documentation.