CatBoost – An Algorithm you need

Ankit Tomar,

Hi there!

In this post, we’ll explore CatBoost in depth — what it is, why it was created, how it works internally (including symmetric trees, ordered boosting, and ordered target statistics), and guidance on when to use or avoid it.

🐈 What is CatBoost?

CatBoost is a gradient boosting library developed by Yandex, uniquely built to handle categorical features natively. Instead of manually applying one-hot or target encoding, CatBoost automatically processes categorical variables in a way that avoids overfitting.

🔍 Why was CatBoost introduced?

Most boosting libraries struggle with categorical variables:

  • One-hot encoding increases dimensionality and sparsity.
  • Naive target encoding can introduce data leakage and overfitting.

CatBoost uses ordered target statistics to calculate target encodings in a way that avoids future data leakage.

🧠 How CatBoost builds models

  • Starts with an initial prediction.
  • Computes residuals (negative gradients).
  • Fits new trees on residuals and updates predictions.
  • Uses ordered boosting and ordered target statistics to reduce prediction shift and prevent target leakage.

🌳 Symmetric trees and ordered boosting

Symmetric trees: CatBoost grows balanced, symmetric trees where all leaves at the same depth use the same split. This:

  • Simplifies model structure.
  • Speeds up inference.
  • Improves parallelism.

Ordered boosting: Instead of training on the whole dataset directly, CatBoost permutes the dataset and ensures each data point’s encoding only depends on data before it. This combats prediction shift, where earlier trees might bias the model.

🔐 Preventing target leakage (ordered target statistics)

CatBoost computes target statistics (like mean target values for categories) in a way that each row’s encoding only depends on preceding rows in the permutation. This prevents the model from “peeking” into future data and overfitting.

⚙️ Important hyperparameters

  • depth: Tree depth.
  • iterations: Number of boosting rounds.
  • learning_rate: Step size.
  • l2_leaf_reg: Regularization.
  • random_strength: Adds randomness to prevent overfitting.

🧮 Key formulas and intuition

CatBoost, like other gradient boosting methods, fits trees to minimize a loss:

  • Regression:
  • Classification:

At each boosting step, a new tree predicts negative gradients to reduce the loss.

✅ When to use CatBoost

  • Datasets with many categorical features.
  • Projects needing good out-of-the-box accuracy.
  • Situations where prediction stability is key.

⚠️ When to consider alternatives

  • Purely numeric data, large-scale data: LightGBM or XGBoost may be faster.
  • When model size is critical.
  • When categorical handling isn’t important.

Catboost is very powerful algorithm. You can read more here https://catboost.ai/docs/en/concepts/tutorials

🐈 CatBoost Algorithm Example – How it Handles Features, Splits & Reuse

Imagine you’re working with a dataset containing 10 features, like:

  • purchase_count
  • membership_level
  • visit_days
  • age
  • … and so on.

You might wonder:

How does CatBoost decide which feature to split on? Does it use the same features in all trees? And can it reuse the same feature multiple times inside one tree?

Let’s walk through this step by step.

🌳 Step 1: Building symmetric trees

CatBoost builds symmetric, balanced trees:

  • At each depth (tree level), all nodes must split on the same feature and threshold.
  • This makes trees balanced and allows faster parallel prediction.

🔍 Step 2: How CatBoost picks splits

At each depth, CatBoost:

  1. Considers all available features (all 10 in our example).
  2. Computes which feature & threshold best reduce impurity (e.g., using Gini, entropy, or loss gradient).
  3. Chooses the split that offers the most gain.

Importantly, this process repeats at each depth:

  • The model checks again across all 10 features — including any it already used earlier in the tree.

♻️ Step 3: Reusing the same feature

Yes — the same feature can appear multiple times in the same tree.

For instance:

  • At depth 1, CatBoost might choose:
    purchase_count > 5
  • At depth 2, after data is split, it could again find:
    purchase_count > 12 is the best next split.

Each reuse can have a different threshold or grouping because:

  • The data subset in each node is different.
  • The optimal split point changes.

✅ Step 4: Using all features in all trees

  • CatBoost keeps the same set of features (the full 10) available at each depth.
  • It does not drop features permanently just because they’ve been used.
  • Across different trees, the splits can vary — some trees might reuse a feature heavily; others might not use it at all.

🧠 Why does this matter?

  • Reusing the same feature helps capture non-linear relationships and complex patterns.
  • It helps the model remain flexible and powerful, despite using symmetric trees.
  • Each split is chosen only if it truly improves the model on that branch of data.

🔄 Symmetric trees recap

  • All nodes at the same depth must split on the same feature & threshold.
  • But across depths (e.g., depth 1 vs depth 2), CatBoost is free to:
    • Choose different features
    • Reuse the same feature with new thresholds

📦 Putting it together (quick example)

Tree depth 1:

  • Split: purchase_count > 5

Tree depth 2:

  • Re-evaluates all 10 features.
  • Finds purchase_count > 12 again gives best gain → reuses purchase_count.

Tree depth 3:

  • Re-evaluates again.
  • Chooses a different feature this time, e.g., visit_days > 3.

✏️ Summary

  • CatBoost keeps the same feature set at each depth.
  • At each depth, it re-evaluates all features to find the best split.
  • The same feature can appear multiple times in one tree.
  • Symmetric trees enforce same split per level, but not across levels.

This design helps CatBoost remain fast, powerful, and very effective at modeling complex data — especially with categorical features.

Loading

Post Views: 272
Machine Learning

Related Posts