Boost Trees

Back to Index

---

Objective

Learn Boost Tree algorithms

Prerequisite Reading

• Brush up on Decision Trees
• Brush up on Random Forest

Essentials Reading

Boost Trees

• Gradient Boosting Classifiers in Python with Scikit-Learn - Good explanation of how boost trees work
• A Gentle Introduction to the Gradient Boosting Algorithm for Machine Learning - good overview of theory

Extra Reading

• Gradient Boosting Machine Learning - video of a talk by Professor Trevor Hastie
• Section. 8.2.3 “Boosting”, page 321 in Introduction to Statistical Learning

Implementing Boosting trees in Scikit-Learn

• Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost
• Scikit ensemple methods
• GradientBoostingClassifier in Scikit doc

Knowledge Check

• What is the basic principle for boost trees?
• What are some implementations of boost trees?

Exercises

We will be using BT in the same exercises we did in Decision Trees section

Difficulty Level

★☆☆ - Easy
★★☆ - Medium
★★★ - Challenging
★★★★ - Bonus

EX-1: Classification - Synthetic data (★☆☆)

Use Scikit’s make_blobs or make_classification to generate some sample data.

Try to separate them using BT

EX-2: Classification (★★☆)

• Here is Bank marketing dataset
• You may want to encode variables
• Use DT to predict yes/no binary decision
• Visualize the tree
• Create a confusion matrix
• What is the accuracy of the model
• Run Cross Validation to gauge the accuracy of this model

EX-3: Regression - Synthetic data (★☆☆)

Use Scikit’s make_regression to generate some sample data.

Use RandomForestRegressor to solve this

EX-4: Regression (★★☆)

• Use Bike sharing data
• Use RandomForestRegressor to predict bike demand
• Visualize the tree
• Use RMSE, R2 to evaluate the model
• Use Cross Validation to thoroughly test the model performance

More Exercises

---

Index