FastTreesRegressor Class
Machine Learning Fast Tree
- Inheritance
-
nimbusml.internal.core.ensemble._fasttreesregressor.FastTreesRegressorFastTreesRegressornimbusml.base_predictor.BasePredictorFastTreesRegressorsklearn.base.RegressorMixinFastTreesRegressor
Constructor
FastTreesRegressor(number_of_trees=100, number_of_leaves=20, minimum_example_count_per_leaf=10, learning_rate=0.2, normalize='Auto', caching='Auto', best_step_trees=False, use_line_search=False, maximum_number_of_line_search_steps=0, minimum_step_size=0.0, optimizer='GradientDescent', early_stopping_rule=None, early_stopping_metrics=1, enable_pruning=False, use_tolerant_pruning=False, pruning_threshold=0.004, pruning_window_size=5, shrinkage=1.0, dropout_rate=0.0, get_derivatives_sample_rate=1, write_last_ensemble=False, maximum_tree_output=100.0, random_start=False, filter_zero_lambdas=False, baseline_scores_formula=None, baseline_alpha_risk=None, position_discount_freeform=None, parallel_trainer=None, number_of_threads=None, random_state=123, feature_selection_seed=123, entropy_coefficient=0.0, histogram_pool_size=-1, disk_transpose=None, feature_flocks=True, categorical_split=False, maximum_categorical_group_count_per_node=64, maximum_categorical_split_point_count=64, minimum_example_fraction_for_categorical_split=0.001, minimum_examples_for_categorical_split=100, bias=0.0, bundling='None', maximum_bin_count_per_feature=255, sparsify_threshold=0.7, first_use_penalty=0.0, feature_reuse_penalty=0.0, gain_conf_level=0.0, softmax_temperature=0.0, execution_time=False, feature_fraction=1.0, bagging_size=0, bagging_example_fraction=0.7, feature_fraction_per_split=1.0, smoothing=0.0, allow_empty_trees=True, feature_compression_level=1, compress_ensemble=False, test_frequency=2147483647, feature=None, group_id=None, label=None, weight=None, **params)
Parameters
- feature
see Columns.
- group_id
see Columns.
- label
see Columns.
- weight
see Columns.
- number_of_trees
Specifies the total number of decision trees to create in the ensemble. By creating more decision trees, you can potentially get better coverage, but the training time increases.
- number_of_leaves
The maximum number of leaves (terminal nodes) that can be created in any tree. Higher values potentially increase the size of the tree and get better precision, but risk overfitting and requiring longer training times.
- minimum_example_count_per_leaf
Minimum number of training instances required to form a leaf. That is, the minimal number of documents allowed in a leaf of regression tree, out of the sub-sampled data. A 'split' means that features in each level of the tree (node) are randomly divided.
- learning_rate
Determines the size of the step taken in the direction of the gradient in each step of the learning process. This determines how fast or slow the learner converges on the optimal solution. If the step size is too big, you might overshoot the optimal solution. If the step size is too small, training takes longer to converge to the best solution.
- normalize
If Auto
, the choice to normalize depends on the
preference declared by the algorithm. This is the default choice. If
No
, no normalization is performed. If Yes
, normalization always
performed. If Warn
, if normalization is needed by the algorithm, a
warning message is displayed but normalization is not performed. If
normalization is performed, a MaxMin
normalizer is used. This
normalizer preserves sparsity by mapping zero to zero.
- caching
Whether trainer should cache input training data.
- best_step_trees
Option for using best regression step trees.
- use_line_search
Should we use line search for a step size.
- maximum_number_of_line_search_steps
Number of post-bracket line search steps.
- minimum_step_size
Minimum line search step size.
- optimizer
Default is sgd
.
- early_stopping_rule
Early stopping rule. (Validation set (/valid) is required.).
- early_stopping_metrics
Early stopping metrics. (For regression, 1: L1, 2:L2; for ranking, 1:NDCG@1, 3:NDCG@3).
- enable_pruning
Enable post-training pruning to avoid overfitting. (a validation set is required).
- use_tolerant_pruning
Use window and tolerance for pruning.
- pruning_threshold
The tolerance threshold for pruning.
- pruning_window_size
The moving window size for pruning.
- shrinkage
Shrinkage.
- dropout_rate
Dropout rate for tree regularization.
- get_derivatives_sample_rate
Sample each query 1 in k times in the GetDerivatives function.
- write_last_ensemble
Write the last ensemble instead of the one determined by early stopping.
- maximum_tree_output
Upper bound on absolute value of single tree output.
- random_start
Training starts from random ordering (determined by /r1).
- filter_zero_lambdas
Filter zero lambdas during training.
- baseline_scores_formula
Freeform defining the scores that should be used as the baseline ranker.
- baseline_alpha_risk
Baseline alpha for tradeoffs of risk (0 is normal training).
- position_discount_freeform
The discount freeform which specifies the per position discounts of examples in a query (uses a single variable P for position where P=0 is first position).
- parallel_trainer
Allows to choose Parallel FastTree Learning Algorithm.
- number_of_threads
The number of threads to use.
- random_state
The seed of the random number generator.
- feature_selection_seed
The seed of the active feature selection.
- entropy_coefficient
The entropy (regularization) coefficient between 0 and 1.
- histogram_pool_size
The number of histograms in the pool (between 2 and numLeaves).
- disk_transpose
Whether to utilize the disk or the data's native transposition facilities (where applicable) when performing the transpose.
- feature_flocks
Whether to collectivize features during dataset preparation to speed up training.
- categorical_split
Whether to do split based on multiple categorical feature values.
- maximum_categorical_group_count_per_node
Maximum categorical split groups to consider when splitting on a categorical feature. Split groups are a collection of split points. This is used to reduce overfitting when there many categorical features.
- maximum_categorical_split_point_count
Maximum categorical split points to consider when splitting on a categorical feature.
- minimum_example_fraction_for_categorical_split
Minimum categorical example percentage in a bin to consider for a split.
- minimum_examples_for_categorical_split
Minimum categorical example count in a bin to consider for a split.
- bias
Bias for calculating gradient for each feature bin for a categorical feature.
- bundling
Bundle low population bins. Bundle.None(0): no bundling, Bundle.AggregateLowPopulation(1): Bundle low population, Bundle.Adjacent(2): Neighbor low population bundle.
- maximum_bin_count_per_feature
Maximum number of distinct values (bins) per feature.
- sparsify_threshold
Sparsity level needed to use sparse feature representation.
- first_use_penalty
The feature first use penalty coefficient. This is a form of regularization that incurs a penalty for using a new feature when creating the tree. Increase this value to create trees that don't use many features.
- feature_reuse_penalty
The feature re-use penalty (regularization) coefficient.
- gain_conf_level
Tree fitting gain confidence requirement (should be in the range [0,1) ).
- softmax_temperature
The temperature of the randomized softmax distribution for choosing the feature.
- execution_time
Print execution time breakdown to stdout.
- feature_fraction
The fraction of features (chosen randomly) to use on each iteration.
- bagging_size
Number of trees in each bag (0 for disabling bagging).
- bagging_example_fraction
Percentage of training examples used in each bag.
- feature_fraction_per_split
The fraction of features (chosen randomly) to use on each split.
- smoothing
Smoothing paramter for tree regularization.
- allow_empty_trees
When a root split is impossible, allow training to proceed.
- feature_compression_level
The level of feature compression to use.
- compress_ensemble
Compress the tree Ensemble.
- test_frequency
Calculate metric values for train/valid/test every k rounds.
- params
Additional arguments sent to compute engine.
Examples
###############################################################################
# FastTreesRegressor
from nimbusml import Pipeline, FileDataStream
from nimbusml.datasets import get_dataset
from nimbusml.ensemble import FastTreesRegressor
from nimbusml.feature_extraction.categorical import OneHotVectorizer
# data input (as a FileDataStream)
path = get_dataset('infert').as_filepath()
data = FileDataStream.read_csv(path)
print(data.head())
# age case education induced parity ... row_num spontaneous ...
# 0 26 1 0-5yrs 1 6 ... 1 2 ...
# 1 42 1 0-5yrs 1 1 ... 2 0 ...
# 2 39 1 0-5yrs 2 6 ... 3 0 ...
# 3 34 1 0-5yrs 2 4 ... 4 0 ...
# 4 35 1 6-11yrs 1 3 ... 5 1 ...
# define the training pipeline
pipeline = Pipeline([
OneHotVectorizer(columns={'edu': 'education'}),
FastTreesRegressor(feature=['induced', 'edu'], label='age')
])
# train, predict, and evaluate
metrics, predictions = pipeline.fit(data).test(data, output_scores=True)
# print predictions
print(predictions.head())
# Score
# 0 35.171112
# 1 35.171112
# 2 34.118595
# 3 34.118595
# 4 32.484325
# print evaluation metrics
print(metrics)
# L1(avg) L2(avg) RMS(avg) Loss-fn(avg) R Squared
# 0 4.095952 24.049695 4.904049 24.049695 0.124438
Remarks
FastTreesRegressor is an implementation of FastRank. FastRank is an efficient implementation of the MART gradient boosting algorithm. Gradient boosting is a machine learning technique for regression problems. It builds each regression tree in a step-wise fashion, using a predefined loss function to measure the error for each step and corrects for it in the next. So this prediction model is actually an ensemble of weaker prediction models. In regression problems, boosting builds a series of of such trees in a step-wise fashion and then selects the optimal tree using an arbitrary differentiable loss function.
MART learns an ensemble of regression trees, which is a decision tree
with
scalar values in its leaves. A decision (or regression) tree is a
binary tree-like flow chart, where at each interior node one decides
which
of the two child nodes to continue to based on one of the feature
values
from the input. At each leaf node, a value is returned. In the
interior nodes, the decision is based on the test "x <= v"
, where
x
is the value of the feature in the input sample and v
is
one
of the possible values of this feature. The functions that can be
produced
by a regression tree are all the piece-wise constant functions.
The ensemble of trees is produced by computing, in each step, a regression tree that approximates the gradient of the loss function, and adding it to the previous tree with coefficients that minimize the loss of the new tree. The output of the ensemble produced by MART on a given instance is the sum of the tree outputs.
- In case of a binary classification problem, the output is converted
to a probability by using some form of calibration.
- In case of a regression problem, the output is the predicted value
of the function.
- In case of a ranking problem, the instances are ordered by the
output value of the ensemble.
Reference
Wikipedia: Gradient boosting (Gradient tree boosting)
Greedy function approximation: A gradient boosting machine.
Methods
get_params |
Get the parameters for this operator. |
get_params
Get the parameters for this operator.
get_params(deep=False)
Parameters
- deep