Skip to content

feat: add CategoricalEncoder (one-hot / ordinal, handle_unknown) - #12

Merged
akifkulll merged 4 commits into
octacity-org:mainfrom
mehmeterencakmak:feat/categorical-encoder
Nov 2, 2026
Merged

akifkulll merged 4 commits into
octacity-org:mainfrom
mehmeterencakmak:feat/categorical-encoder

Conversation

@mehmeterencakmak

Copy link
Copy Markdown
Contributor

Closes #3

Summary

Adds CategoricalEncoder, the block that turns categorical columns into numbers.

  • method="onehot" (default) replaces each categorical column, in place, with one 0/1 integer column per category, named <column>_<category>.
  • method="ordinal" replaces it with the category's position in categories_[column].
  • Only object, string and category columns are encoded; numeric (and other) columns pass through unchanged.
  • Categories are learned in fit only, sorted, and stored in categories_ as {column: [category, ...]}.
  • Unknown categories in transform: handle_unknown="error" raises a ValueError naming the column and values; "ignore" gives all zeros (one-hot) or -1 (ordinal).
  • Overrides get_feature_names_out; it reuses the base class checks for fitted state and input_features.

Acceptance criteria

  • Subclasses BasePreprocessor, implements only _fit / _transform; __init__ only stores its arguments.
  • Invalid method / handle_unknown raise ValueError in fit.
  • Categories come from training data only; test-only categories are unknown.
  • Missing values in a categorical column raise ValueError in fit and transform, asking to impute first.
  • A generated one-hot name that collides with an existing column raises ValueError in fit.
  • get_feature_names_out() matches the columns transform produces for both methods.
  • transform returns a new DataFrame and does not modify its input (index is kept).
  • Works in make_pipeline + cross_val_score; GridSearchCV tunes method.
  • Exported from preprocessing_toolkit.
  • ruff check, ruff format --check, pytest pass locally (60 tests, Python 3.13).

Notes

  • Codes are computed with pd.Index(categories).get_indexer(values); building a pd.Categorical from values outside its categories is deprecated in recent pandas.
  • __init__.py will need a trivial merge with feat: add missing value imputer #10 (imputer export) depending on merge order.

🤖 Generated with Claude Code

mehmeterencakmak and others added 2 commits October 21, 2026 12:16
Encodes object, string and category columns using sorted categories
learned in fit; numeric columns pass through. Overrides
get_feature_names_out for the generated one-hot columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cover both methods, unknown handling, validation, missing values,
name collisions, get_feature_names_out and scikit-learn compatibility.
Compute codes with Index.get_indexer, since building a Categorical from
values outside its categories is deprecated in pandas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@akifkulll

Copy link
Copy Markdown
Collaborator

Nice work. CI is green and all 60 tests pass. Two things before merge:

  1. Ordered categoricals: for pd.Categorical(['low','high','medium'], categories=['low','medium','high'], ordered=True), categories_ becomes ['high','low','medium'], so ordinal codes ignore the declared order. For category dtype columns, use list(X[col].cat.categories) instead of sorting (this also keeps unused levels).
  2. Mixed-type object columns (['a', 1, 'b']) crash in sorted() with a bare TypeError. Use sorted(..., key=str) or raise a clear ValueError.
    Note: feat: add FeatureScaler (standard / minmax) #11 also edits init.py, so whichever is merged second will need a quick rebase.

Eline sağlık Eren, CI yeşil ve 60 testin hepsi geçiyor. Merge'den önce iki küçük şey var:

  1. Sıralı kategoriler: pd.Categorical(['low','high','medium'], categories=['low','medium','high'], ordered=True) gibi bir sütunda categories_ ['high','low','medium'] oluyor. Yani ordinal kodlar tanımlanan sırayı (low < medium < high) dikkate almıyor. category dtype'lı sütunlarda sıralamak yerine list(X[col].cat.categories) kullanırsan sorun çözülür. Kullanılmayan kategoriler de bu sayede kaybolmaz.

  2. Karışık tipli sütunlar: ['a', 1, 'b'] gibi object sütunlarda sorted() açıklamasız bir TypeError veriyor. sorted(..., key=str) kullanabilirsin ya da anlaşılır bir ValueError fırlatabilirsin.

Not: #11 de init.py dosyasını değiştiriyor. Önce #11'i merge edeceğiz, sonra burada küçük bir rebase gerekecek: iki import satırı ve all içinde iki isim yan yana kalacak.

@akifkulll akifkulll added this to the v0.1 milestone Oct 31, 2026
mertcankaratass added a commit to mertcankaratass/PreToolkit that referenced this pull request Oct 31, 2026
mehmeterencakmak and others added 2 commits October 31, 2026 21:45
…coder

# Conflicts:
#	src/preprocessing_toolkit/__init__.py
Category dtype columns now use their declared categories, so ordered
categoricals keep their order and unused levels are kept. Mixed-type
object columns no longer crash in sorted(); they fall back to sorting
by string value, with the type name breaking ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mehmeterencakmak

Copy link
Copy Markdown
Contributor Author

Thanks! Both are fixed:

  • category columns now keep their declared order (and unused levels).
  • Mixed-type columns (like ['a', 1, 'b']) no longer crash.

I also merged main into the branch to resolve the __init__.py conflict. CI is green.


Teşekkürler! İki düzeltmeyi de yaptım:

  • category tipli sütunlar artık tanımlı sıralarını (ve kullanılmayan kategorileri) koruyor.
  • Karışık tipli sütunlar (['a', 1, 'b'] gibi) artık çökmüyor.

Ayrıca __init__.py çakışmasını çözmek için main'i branch'e merge ettim. CI yeşil.

@akifkulll

Copy link
Copy Markdown
Collaborator

Thanks Eren, both fixes verified. Teşekkürler!”

@akifkulll
akifkulll merged commit 5b5812b into octacity-org:main Nov 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CategoricalEncoder (one-hot / ordinal, handle_unknown)

2 participants