punycoder

High-performance Unicode and Punycode encoding/decoding for internationalized domain names (IDNs) in R.

Overview

The punycoder package provides fast, standards-based conversion between Unicode and ASCII representations of domain names, across two distinct surfaces:

• a low-level Punycode codec — puny_encode() / puny_decode() — the raw RFC 3492 transform with xn-- A-label framing (RFC 5890/5891) and letter-digit-hyphen checks, not an IDNA normalization API (no Unicode NFC, UTS #46 mapping, or case folding);
• an IDNA/UTS-46 host-normalization surface — host_normalize() — mapping a host name to its canonical lowercase ASCII comparison form under a pinned UTS #46 non-transitional profile.

host_normalize() is a UTS #46 profile, not IDNA2008 conformance — UTS #46 is compatibility processing and deliberately accepts labels IDNA2008 would reject (e.g. ☕.example → xn--53h.example). See ?host_normalize and normalization_profile_info() for the normative profile and full standards references (RFC 3492/5890/5891/5892/5893, UTS #46, UAX #15/#44, STD 3, RFC 8753).

Normalization runs against vendored Unicode data, and a build ships a set of Unicode versions with one pinned as the default (currently 17.0.0, with 16.0.0 also shipped). unicode_versions() reports what the installed build carries, and host_normalize(x, unicode_version = "16.0.0") selects another one for a single call — which normalization_profile_info() reflects by appending +unicode-<version> to the reported profile token.

Installation

Install the released version of punycoder from CRAN with:

install.packages("punycoder")

Or install the development version from r-universe with:

install.packages(
  "punycoder",
  repos = c("https://bart-turczynski.r-universe.dev", "https://cloud.r-project.org")
)

To build it from source on GitLab instead:

# install.packages("remotes")
remotes::install_gitlab("bart-turczynski/punycoder")

System requirements

punycoder needs R 4.1.0 or later and Rcpp, which install.packages() pulls in. A binary install needs nothing else. A source install needs a C++ compiler, which the standard R build tools for each platform provide. libidn2 2.3.5 or later and pkg-config are optional: configure uses them for a native backend when it finds both.

Optional native backend (libidn2)

punycoder works without extra system libraries. If libidn2 is available at build time, the package enables a native backend automatically; otherwise it uses the built-in C++ fallback backend.

To install the recommended optional dependency:

• macOS (Homebrew):
  • brew install libidn2 pkg-config
• Debian/Ubuntu:
  • sudo apt-get install libidn2-0-dev pkg-config
• Fedora/RHEL/CentOS:
  • sudo dnf install libidn2-devel pkgconf-pkg-config
• Arch Linux:
  • sudo pacman -S libidn2 pkgconf

Verify the library is visible before installing punycoder from source:

system("pkg-config --modversion libidn2")

Then install/reinstall punycoder:

remotes::install_gitlab("bart-turczynski/punycoder")

Example

library(punycoder)

# Basic encoding
puny_encode("café.com")
#> [1] "xn--caf-dma.com"

# Check if domain is punycode
is_punycode("xn--example")
#> [1] TRUE

# Validate domains
validate_domain("test.com")
#> Punycoder Domain Validation Results
#> ==================================
#> 
#> 1 domain: 1 valid, 0 invalid (strict = TRUE)
#> 
#> Domain: test.com 
#> Valid:  TRUE

Command-line use

punycoder is a library, not a CLI, but a one-line Rscript wrapper covers the usual shell need: turning an IDN into the ASCII form that dig, nslookup, or curl will actually accept. No extra install, no bin/ script. The snippets below are POSIX sh/bash — keep the -e argument in single quotes so the shell leaves the R code alone.

Unicode host → ASCII, via the UTS #46 canonical form:

Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de
#> xn--mnchen-3ya.de

Use puny_encode() instead when you want the raw RFC 3492 transform with no UTS #46 mapping, and puny_decode() to go back the other way:

Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE)), sep = "\n")' münchen.de
#> xn--mnchen-3ya.de

Rscript -e 'cat(punycoder::puny_decode(commandArgs(TRUE)), sep = "\n")' xn--mnchen-3ya.de
#> münchen.de

Feeding a resolver is then just command substitution (drop sep so nothing but the host is printed):

dig +short "$(Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)))' münchen.de)"
#> 194.246.166.100

nslookup "$(Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)))' münchen.de)"

All three functions are vectorized, so several hosts can go through one R startup and out to xargs:

Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de москва.рф |
  xargs -n 1 dig +short

Failed conversions

host_normalize() never aborts: an input it cannot normalize comes back as NA, which prints as a literal NA line rather than silently disappearing from the pipeline.

Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de 'bad..domain'
#> xn--mnchen-3ya.de
#> NA

puny_encode() and puny_decode() behave differently: they are strict by default, so a bad input stops the script with a non-zero exit status.

Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE)), sep = "\n")' 'bad..domain'
#> Error: Error encoding domain: Domain contains empty label
#> Execution halted

Pass strict = FALSE to get the per-element NA behavior instead, so one bad host does not take down a batch:

Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE), strict = FALSE), sep = "\n")' münchen.de 'bad..domain'
#> xn--mnchen-3ya.de
#> NA

Since NA is printed, not swallowed, drop it before piping to a resolver — otherwise dig dutifully looks up a host named NA:

Rscript -e 'h <- punycoder::host_normalize(commandArgs(TRUE)); cat(h[!is.na(h)], sep = "\n")' münchen.de 'bad..domain' |
  xargs -n 1 dig +short

Key Features

• Reliable Encoding/Decoding: RFC 3492 compliant punycode conversion
• High Performance: Vectorized operations for processing large datasets
• Comprehensive Validation: Robust error handling with informative messages
• Flexible Backend: Automatically uses libidn2 when available, with a built-in fallback backend

Use Cases

Data Analysis

Clean and standardize URL datasets:

# Identify international domains
is_idn(c("café.com", "example.com", "москва.рф"))

# Validate domain names
validate_domain(c("valid.com", "invalid..domain"))

Non-goals

punycoder is a standards primitive for Punycode and host normalization. It is deliberately agnostic about resolvability and safety; the following are not part of its acceptance criteria:

• No spoof / homograph / mixed-script / display-safety detection. host_normalize() is not a safety gate — a successful result says the host is valid and normalized under the pinned UTS #46 profile, nothing about whether it is visually safe or non-deceptive. Confusable and restriction-level checks (UTS #39 / UTR #36, which UTS #46 itself recommends only as application/UI-layer steps) belong upstack.
• No URL parsing or canonicalization. RFC 3986 / WHATWG URL parsing and canonicalization belong upstack; see the rurl package, which consumes punycoder’s host functions.
• No DNS resolvability or registrability / PSL classification.
• No address parsing. There is no email-to-ASCII helper; splitting an address and IDNA-encoding its domain part is an addressing concern for an upstack consumer, not a Punycode primitive.
• No per-TLD repertoire / allowed-character validation. host_normalize() validates against the pinned UTS #46 profile, not against registry-specific IDN tables (which evolve independently of Unicode). TLD policy belongs upstack.

These opinions belong in higher layers that consume punycoder’s host functions.

How it compares

punycoder descends from the libidn-based hrbrmstr/punycode and follows IDNA2008 / UTS #46 rather than IDNA2003. The Prior art and comparison article compares it with Punycode libraries in other ecosystems and with the comparable R packages, on the same inputs.

Acknowledgments

These packages build on data, libraries, and prior work from many others. See ACKNOWLEDGMENTS.md for the full list of thanks.

Related packages

punycoder is part of a small ecosystem of R packages by the same author:

• pslr — Public Suffix List engine that uses punycoder for IDNA canonicalization. Use it for eTLD and registrable-domain queries.
• rurl — Full URL parsing, normalization, and joining toolkit built on top of both punycoder and pslr.

Citation

If you use punycoder in your work, please cite it. Run citation("punycoder") for the current citation, or see CITATION.cff.

Each release is archived on Zenodo. Cite the concept DOI 10.5281/zenodo.20973629 to refer to the software in general (it always resolves to the latest version), or the version-specific DOI shown on the Zenodo record for a particular release.

Contributing

We welcome contributions. See CONTRIBUTING.md for the current development workflow, ARCHITECTURE.md for how the package is structured, and DECISIONS.md for the design-decision log.

Code of Conduct

Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

License

MIT