Visible to Intel only — GUID: GUID-1EABD4BE-5FFC-4E2F-8766-18926948294C
basic_statistics_dense_batch.cpp
basic_statistics_dense_online.cpp
column_accessor_homogen.cpp
cor_dense_batch.cpp
cor_dense_online.cpp
cov_dense_batch.cpp
cov_dense_biased_batch.cpp
cov_dense_biased_online.cpp
cov_dense_online.cpp
csr_accessor.cpp
csr_table.cpp
dbscan_brute_force_batch.cpp
df_cls_hist_batch.cpp
df_cls_hist_batch_random.cpp
df_cls_traverse_model.cpp
df_reg_hist_batch.cpp
df_reg_hist_batch_random.cpp
df_reg_traverse_model.cpp
heterogen_table.cpp
homogen_table.cpp
kmeans_init_dense.cpp
kmeans_lloyd_dense_batch.cpp
knn_cls_brute_force_dense_batch.cpp
knn_reg_brute_force_dense_batch.cpp
knn_search_brute_force_dense_batch.cpp
linear_kernel_dense_batch.cpp
linear_regression_dense_batch.cpp
linear_regression_dense_online.cpp
logistic_regression_dense_batch.cpp
pca_cor_dense_batch.cpp
pca_cor_dense_online.cpp
pca_cov_dense_batch.cpp
pca_cov_dense_online.cpp
pca_precomputed_cor_dense_batch.cpp
pca_precomputed_cov_dense_batch.cpp
pca_svd_dense_batch.cpp
rbf_kernel_dense_batch.cpp
svm_two_class_thunder_dense_batch.cpp
basic_statistics_dense_batch.cpp
basic_statistics_dense_online.cpp
column_accessor_homogen.cpp
connected_components_batch.cpp
cor_dense_batch.cpp
cor_dense_online.cpp
cov_dense_batch.cpp
cov_dense_biased_batch.cpp
cov_dense_biased_online.cpp
cov_dense_online.cpp
csr_accessor.cpp
csr_table.cpp
dbscan_brute_force_batch.cpp
df_cls_dense_batch.cpp
df_reg_dense_batch.cpp
directed_graph.cpp
graph_service_functions.cpp
heterogen_table.cpp
homogen_table.cpp
jaccard_batch.cpp
jaccard_batch_app.cpp
kmeans_init_dense.cpp
kmeans_lloyd_dense_batch.cpp
knn_cls_brute_force_dense_batch.cpp
knn_cls_kd_tree_dense_batch.cpp
knn_search_brute_force_dense_batch.cpp
linear_kernel_dense_batch.cpp
linear_regression_dense_batch.cpp
linear_regression_dense_online.cpp
logloss_dense_batch.cpp
louvain_batch.cpp
pca_cor_dense_batch.cpp
pca_cor_dense_online.cpp
pca_cov_dense_batch.cpp
pca_cov_dense_online.cpp
pca_precomputed_dense_batch.cpp
pca_svd_dense_batch.cpp
pca_svd_dense_online.cpp
polynomial_kernel_dense_batch.cpp
rbf_kernel_dense_batch.cpp
shortest_paths_batch.cpp
sigmoid_kernel_dense_batch.cpp
subgraph_isomorphism_batch.cpp
svm_multi_class_thunder_csr_batch.cpp
svm_multi_class_thunder_dense_batch.cpp
svm_nu_cls_thunder_csr_batch.cpp
svm_nu_cls_thunder_dense_batch.cpp
svm_nu_reg_thunder_csr_batch.cpp
svm_nu_reg_thunder_dense_batch.cpp
svm_reg_thunder_csr_batch.cpp
svm_reg_thunder_dense_batch.cpp
svm_two_class_smo_csr_batch.cpp
svm_two_class_smo_dense_batch.cpp
svm_two_class_thunder_csr_batch.cpp
svm_two_class_thunder_dense_batch.cpp
triangle_counting_batch.cpp
K-Means Clustering
Density-Based Spatial Clustering of Applications with Noise
Correlation and Variance-Covariance Matrices
Principal Component Analysis
Principal Components Analysis Transform
Singular Value Decomposition
Association Rules
Kernel Functions
Expectation-Maximization
Cholesky Decomposition
QR Decomposition
Outlier Detection
Distance Matrix
Distributions
Engines
Moments of Low Order
Quantile
Quality Metrics
Sorting
Normalization
Optimization Solvers
Decision Forest
Decision Trees
Gradient Boosted Trees
Stump
Linear and Ridge Regressions
LASSO and Elastic Net Regressions
k-Nearest Neighbors (kNN) Classifier
Implicit Alternating Least Squares
Logistic Regression
Naïve Bayes Classifier
Support Vector Machine Classifier
Multi-class Classifier
Boosting
Visible to Intel only — GUID: GUID-1EABD4BE-5FFC-4E2F-8766-18926948294C
Gradient Boosted Trees
Details
Given n feature vectors 
of np-dimensional feature vectors and n responses 
, the problem is to build a gradient boosted trees classification or regression model.
The tree ensemble model uses M additive functions to predict the output 
where 
is the space of regression trees, T is the number of leaves in the tree, w is a leaf weights vector, 
is a score on i-th leaf. q(x) represents the structure of each tree that maps an observation to the corresponding leaf index.
Training procedure is an iterative functional gradient descent algorithm which minimizes objective function over function space by iteratively choosing a function (regression tree) that points in the negative gradient direction. The objective function is
where l(f) is twice differentiable convex loss function and 
is a regularization term that penalizes the complexity of the model defined by the number of leaves T and the L2 norm of the weights 
for each tree, 
and 
are regularization parameters.
Training Stage
Library uses the second-order approximation of objective function
where 
, 
and following algorithmic framework for the training stage.
Let 
be the set of observations. Given the training parameters, such as the number of iterations M, loss function l(f), regression tree training parameters, regularization parameters and , shrinkage (learning rate) parameter 
, the algorithm does the following:
Find an initial guess
, 
For
:Update
and 
, Grow a regression tree
that minimizes the objective function 
, where 
, 
, 
, 
.Assign an optimal weight
to the leaf j, 
.Apply shrinkage parameter
to the tree leafs and add the tree to the modelUpdate
The algorithm for growing the tree:
Generate a bootstrap training set if required (stochastic gradient boosting) as follows: select randomly without replacement
observations, where f is a fraction of observations used for training of one tree.Start from the tree with depth 0.
For each leaf node in the tree:
Choose a subset of feature for split finding if required (stochastic gradient boosting).
Find the best split that maximizes the gain:
For more details, see [Chen2016].
The library supports the following termination criteria when growing the tree:
Minimal number of observations in a leaf node. Node t is not processed if the subset of observations is smaller than the predefined value. Splits that produce nodes with the number of observations smaller than that value are not allowed.
Maximal tree depth. Node t is not processed, if its depth in the tree reached the predefined value.
Minimal split loss. Node t is not processed, if the best possible split is smaller than parameter
.
Prediction Stage
Given a gradient boosted trees model and vectors 
, the problem is to calculate the responses for those vectors. To solve the problem for each given query vector 
, the algorithm finds the leaf node in a tree in the ensemble which gives the response by that tree. Resulting response is based on an aggregation of responses from all trees in the ensemble. For detailed definition, see description of a specific algorithm.
Split Calculation Mode
The library supports two split calculation modes:
exact - all possible split values are examined when searching for the best split for a feature.
inexact - continuous features are bucketed into discrete bins and the possible splits are restricted by the buckets borders only.
Batch Processing
Gradient boosted trees classification and regression follows the general workflow described in Classification Usage Model and Regression Usage Model.
Training
For description of the input and output, refer to .
At the training stage, the gradient boosted trees batch algorithm has the following parameters:
Parameter |
Default Value |
Description |
|---|---|---|
splitMethod |
inexact |
Split computation mode. Possible values:
|
maxIterations |
50 |
Maximal number of iterations when training the model, defines maximal number of trees in the model. |
maxTreeDepth |
6 |
Maximal tree depth. If the parameter is set to 0 then the depth is unlimited. |
shrinkage |
0.3 |
Learning rate of the boosting procedure. Scales the contribution of each tree by a factor |
minSplitLoss |
0 |
Loss regularization parameter. Minimal loss reduction required to make a further partition on a leaf node of the tree. Range: |
lambda |
1 |
L2 regularization parameter on weights. Range: |
observationsPerTreeFraction |
1 |
Fraction of the training set S used for a single tree training, |
featuresPerNode |
0 |
The number of features tried as the possible splits per node. If the parameter is set to 0, all features are used. |
minObservationsInLeafNode |
5 |
Minimal number of observations in the leaf node. |
memorySavingMode |
false |
If true then use memory saving (but slower) mode. |
engine |
SharePtr< engines:: mt19937:: Batch>() |
Pointer to the random number generator. |
maxBins |
256 |
Used with inexact split method only. Maximal number of discrete bins to bucket continuous features. Increasing the number results in higher computation costs |
minBinSize |
5 |
Used with inexact split method only. Minimal number of observations in a bin. |
. The observations are sampled randomly without replacement.