Skip to content
All tags

#bagging

2 posts
aiguideReading NTU Hsuan-Tien Lin Machine Learning Foundations & Techniques

Random Forests: Bagging plus Decision Trees and the Out-of-Bag Estimate | NTU Hsuan-Tien Lin Machine Learning Techniques Lecture 10

A random forest is bagging applied to fully-grown C&RT trees, with each branch looking only at a few randomly projected features. When bootstrap draws N times, each example has about a 1/e ≈ 36.8% chance of not being used by a given tree. These out-of-bag examples give E_oob directly, a validation that needs no held-out data and no re-training; add the permutation test and you get an importance for every feature. The price is that the result wobbles with the random seed, so you keep adding trees until G is stable.

#ntu-htlin-ml#ntu#ai-course#machine-learning#ensemble#decision-trees#bagging
aiguideReading NTU Hsuan-Tien Lin Machine Learning Foundations & Techniques

Blending and Bagging: Why Combining Hypotheses Works Better | NTU Hsuan-Tien Lin Machine Learning Techniques Lecture 7

With T hypotheses gₜ in hand, you can do more than pick one by validation: give each one vote (uniform blending), learn a set of ballot counts (linear blending), or learn another model that combines them (the form the handout calls any blending, also named stacking). This lecture proves in six lines of algebra that, under squared error, the averaged G is never worse than the average performance of the gₜ, and the gap is exactly how spread out the gₜ are, which is the variance. Finally, bootstrapping re-samples T data sets from the one data set you have, trains one gₜ on each, and lets them vote; that is bagging.

#ntu-htlin-ml#ntu#ai-course#machine-learning#ensemble#bagging