| Title: | Rapid Calculation of Model Metrics |
|---|---|
| Description: | Collection of metrics for evaluating models written in C++ using 'Rcpp'. Popular metrics include area under the curve, log loss, root mean square error, etc. |
| Authors: | Tyler Hunt [aut, cre] |
| Maintainer: | Tyler Hunt <[email protected]> |
| License: | GPL (>= 2) |
| Version: | 1.2.3 |
| Built: | 2024-11-08 05:13:27 UTC |
| Source: | https://github.com/jackstat/modelmetrics |
Calculates the area under the curve for a binary classifcation model
auc(...) ## Default S3 method: auc(actual, predicted, ...) ## S3 method for class 'glm' auc(modelObject, ...) ## S3 method for class 'randomForest' auc(modelObject, ...) ## S3 method for class 'glmerMod' auc(modelObject, ...) ## S3 method for class 'gbm' auc(modelObject, ...) ## S3 method for class 'rpart' auc(modelObject, ...)auc(...) ## Default S3 method: auc(actual, predicted, ...) ## S3 method for class 'glm' auc(modelObject, ...) ## S3 method for class 'randomForest' auc(modelObject, ...) ## S3 method for class 'glmerMod' auc(modelObject, ...) ## S3 method for class 'gbm' auc(modelObject, ...) ## S3 method for class 'rpart' auc(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels. Can be |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') auc(testDF$y, Preds) # using s3 method for glm auc(glmModel)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') auc(testDF$y, Preds) # using s3 method for glm auc(glmModel)
Calculates the Brier score
brier(...) ## Default S3 method: brier(actual, predicted, ...) ## S3 method for class 'glm' brier(modelObject, ...) ## S3 method for class 'randomForest' brier(modelObject, ...) ## S3 method for class 'glmerMod' brier(modelObject, ...) ## S3 method for class 'gbm' brier(modelObject, ...) ## S3 method for class 'rpart' brier(modelObject, ...)brier(...) ## Default S3 method: brier(actual, predicted, ...) ## S3 method for class 'glm' brier(modelObject, ...) ## S3 method for class 'randomForest' brier(modelObject, ...) ## S3 method for class 'glmerMod' brier(modelObject, ...) ## S3 method for class 'gbm' brier(modelObject, ...) ## S3 method for class 'rpart' brier(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
Calculates the classification error
ce(...) ## Default S3 method: ce(actual, predicted, ...) ## S3 method for class 'lm' ce(modelObject, ...) ## S3 method for class 'glm' ce(modelObject, ...) ## S3 method for class 'randomForest' ce(modelObject, ...) ## S3 method for class 'glmerMod' ce(modelObject, ...) ## S3 method for class 'gbm' ce(modelObject, ...) ## S3 method for class 'rpart' ce(modelObject, ...)ce(...) ## Default S3 method: ce(actual, predicted, ...) ## S3 method for class 'lm' ce(modelObject, ...) ## S3 method for class 'glm' ce(modelObject, ...) ## S3 method for class 'randomForest' ce(modelObject, ...) ## S3 method for class 'glmerMod' ce(modelObject, ...) ## S3 method for class 'gbm' ce(modelObject, ...) ## S3 method for class 'rpart' ce(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
Create a confusion matrix given a specific cutoff.
confusionMatrix(actual, predicted, cutoff = 0.5, use_names = FALSE)confusionMatrix(actual, predicted, cutoff = 0.5, use_names = FALSE)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
use_names |
If |
actual <- c(0, 0, 1) predicted <- c(1, 0, 1) confusionMatrix(actual, predicted) confusionMatrix(actual, predicted, use_names = TRUE)actual <- c(0, 0, 1) predicted <- c(1, 0, 1) confusionMatrix(actual, predicted) confusionMatrix(actual, predicted, use_names = TRUE)
Calculates the f1 score
f1Score(actual, predicted, cutoff = 0.5)f1Score(actual, predicted, cutoff = 0.5)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
Calculates the F score and allows different specifications of the beta value (F0.5)
fScore(actual, predicted, cutoff = 0.5, beta = 1)fScore(actual, predicted, cutoff = 0.5, beta = 1)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
beta |
the desired beta value (lower increases weight of precision over recall). Defaults to 1 |
Calculates the GINI coefficient for a binary classifcation model
gini(...) ## Default S3 method: gini(actual, predicted, ...) ## S3 method for class 'glm' gini(modelObject, ...) ## S3 method for class 'randomForest' gini(modelObject, ...) ## S3 method for class 'glmerMod' gini(modelObject, ...) ## S3 method for class 'gbm' gini(modelObject, ...) ## S3 method for class 'rpart' gini(modelObject, ...)gini(...) ## Default S3 method: gini(actual, predicted, ...) ## S3 method for class 'glm' gini(modelObject, ...) ## S3 method for class 'randomForest' gini(modelObject, ...) ## S3 method for class 'glmerMod' gini(modelObject, ...) ## S3 method for class 'gbm' gini(modelObject, ...) ## S3 method for class 'rpart' gini(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels. Can be |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') gini(testDF$y, Preds) # using s3 method for glm gini(glmModel)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') gini(testDF$y, Preds) # using s3 method for glm gini(glmModel)
Calculates kappa statistic. Currently build to handle binary values in actual vector.
kappa(actual, predicted, cutoff = 0.5)kappa(actual, predicted, cutoff = 0.5)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
Calculates the log loss or entropy loss for a binary outcome
logLoss(...) ## Default S3 method: logLoss(actual, predicted, distribution = "binomial", ...) ## S3 method for class 'glm' logLoss(modelObject, ...) ## S3 method for class 'randomForest' logLoss(modelObject, ...) ## S3 method for class 'glmerMod' logLoss(modelObject, ...) ## S3 method for class 'gbm' logLoss(modelObject, ...) ## S3 method for class 'rpart' logLoss(modelObject, ...)logLoss(...) ## Default S3 method: logLoss(actual, predicted, distribution = "binomial", ...) ## S3 method for class 'glm' logLoss(modelObject, ...) ## S3 method for class 'randomForest' logLoss(modelObject, ...) ## S3 method for class 'glmerMod' logLoss(modelObject, ...) ## S3 method for class 'gbm' logLoss(modelObject, ...) ## S3 method for class 'rpart' logLoss(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
a binary vector of the labels |
predicted |
a vector of predicted values |
distribution |
the distribution of the loss function needed |
modelObject |
the model object. Currently supported |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') logLoss(testDF$y, Preds) # using s3 method for glm logLoss(glmModel)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') logLoss(testDF$y, Preds) # using s3 method for glm logLoss(glmModel)
Calculates the mean absolute error
mae(...) ## Default S3 method: mae(actual, predicted, ...) ## S3 method for class 'glm' mae(modelObject, ...) ## S3 method for class 'randomForest' mae(modelObject, ...) ## S3 method for class 'glmerMod' mae(modelObject, ...) ## S3 method for class 'gbm' mae(modelObject, ...) ## S3 method for class 'rpart' mae(modelObject, ...)mae(...) ## Default S3 method: mae(actual, predicted, ...) ## S3 method for class 'glm' mae(modelObject, ...) ## S3 method for class 'randomForest' mae(modelObject, ...) ## S3 method for class 'glmerMod' mae(modelObject, ...) ## S3 method for class 'gbm' mae(modelObject, ...) ## S3 method for class 'rpart' mae(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
Calculates the area under the curve for a binary classifcation model
mauc(actual, predicted)mauc(actual, predicted)
actual |
A vector of the labels. Can be |
predicted |
A data.frame of predicted values. Can be |
setosa <- glm(I(Species == 'setosa') ~ Sepal.Length, data = iris, family = 'binomial') versicolor <- glm(I(Species == 'versicolor') ~ Sepal.Length, data = iris, family = 'binomial') virginica <- glm(I(Species == 'virginica') ~ Sepal.Length, data = iris, family = 'binomial') Pred <- data.frame( setosa = predict(setosa, type = 'response') ,versicolor = predict(versicolor, type = 'response') ,virginica = predict(virginica, type = 'response') ) Predicted = Pred/rowSums(Pred) Actual = iris$Species mauc(Actual, Predicted)setosa <- glm(I(Species == 'setosa') ~ Sepal.Length, data = iris, family = 'binomial') versicolor <- glm(I(Species == 'versicolor') ~ Sepal.Length, data = iris, family = 'binomial') virginica <- glm(I(Species == 'virginica') ~ Sepal.Length, data = iris, family = 'binomial') Pred <- data.frame( setosa = predict(setosa, type = 'response') ,versicolor = predict(versicolor, type = 'response') ,virginica = predict(virginica, type = 'response') ) Predicted = Pred/rowSums(Pred) Actual = iris$Species mauc(Actual, Predicted)
Calculates the Matthews Correlation Coefficient
mcc(actual, predicted, cutoff)mcc(actual, predicted, cutoff)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
Calculated the multi-class log loss
mlogLoss(actual, predicted)mlogLoss(actual, predicted)
actual |
A vector of the labels. Can be |
predicted |
matrix of predicted values. Can be |
Calculates the mean square error
mse(...) ## Default S3 method: mse(actual, predicted, ...) ## S3 method for class 'lm' mse(modelObject, ...) ## S3 method for class 'glm' mse(modelObject, ...)mse(...) ## Default S3 method: mse(actual, predicted, ...) ## S3 method for class 'lm' mse(modelObject, ...) ## S3 method for class 'glm' mse(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') mse(testDF$y, Preds)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') mse(testDF$y, Preds)
Calculates the mean square log error
msle(...) ## Default S3 method: msle(actual, predicted, ...) ## S3 method for class 'lm' msle(modelObject, ...) ## S3 method for class 'glm' msle(modelObject, ...) ## S3 method for class 'randomForest' msle(modelObject, ...) ## S3 method for class 'glmerMod' msle(modelObject, ...) ## S3 method for class 'gbm' msle(modelObject, ...) ## S3 method for class 'rpart' msle(modelObject, ...)msle(...) ## Default S3 method: msle(actual, predicted, ...) ## S3 method for class 'lm' msle(modelObject, ...) ## S3 method for class 'glm' msle(modelObject, ...) ## S3 method for class 'randomForest' msle(modelObject, ...) ## S3 method for class 'glmerMod' msle(modelObject, ...) ## S3 method for class 'gbm' msle(modelObject, ...) ## S3 method for class 'rpart' msle(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
True Negatives / (True Negatives + False Negatives)
npv(actual, predicted, cutoff = 0.5)npv(actual, predicted, cutoff = 0.5)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') npv(testDF$y, Preds, cutoff = 0)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') npv(testDF$y, Preds, cutoff = 0)
True Positives / (True Positives + False Positives)
ppv(actual, predicted, cutoff = 0.5)ppv(actual, predicted, cutoff = 0.5)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') ppv(testDF$y, Preds, cutoff = 0) precision(testDF$y, Preds, cutoff = 0)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') ppv(testDF$y, Preds, cutoff = 0) precision(testDF$y, Preds, cutoff = 0)
True Positives / (True Positives + False Negatives)
recall(actual, predicted, cutoff = 0.5)recall(actual, predicted, cutoff = 0.5)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') recall(testDF$y, Preds, cutoff = 0) sensitivity(testDF$y, Preds, cutoff = 0) tpr(testDF$y, Preds, cutoff = 0)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') recall(testDF$y, Preds, cutoff = 0) sensitivity(testDF$y, Preds, cutoff = 0) tpr(testDF$y, Preds, cutoff = 0)
Calculates the root mean square error
rmse(...) ## Default S3 method: rmse(actual, predicted, ...) ## S3 method for class 'lm' rmse(modelObject, ...) ## S3 method for class 'glm' rmse(modelObject, ...)rmse(...) ## Default S3 method: rmse(actual, predicted, ...) ## S3 method for class 'lm' rmse(modelObject, ...) ## S3 method for class 'glm' rmse(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') rmse(testDF$y, Preds)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') rmse(testDF$y, Preds)
Calculates the mean square log error
rmsle(...) ## Default S3 method: rmsle(actual, predicted, ...) ## S3 method for class 'lm' rmsle(modelObject, ...) ## S3 method for class 'glm' rmsle(modelObject, ...) ## S3 method for class 'randomForest' rmsle(modelObject, ...) ## S3 method for class 'glmerMod' rmsle(modelObject, ...) ## S3 method for class 'gbm' rmsle(modelObject, ...) ## S3 method for class 'rpart' rmsle(modelObject, ...)rmsle(...) ## Default S3 method: rmsle(actual, predicted, ...) ## S3 method for class 'lm' rmsle(modelObject, ...) ## S3 method for class 'glm' rmsle(modelObject, ...) ## S3 method for class 'randomForest' rmsle(modelObject, ...) ## S3 method for class 'glmerMod' rmsle(modelObject, ...) ## S3 method for class 'gbm' rmsle(modelObject, ...) ## S3 method for class 'rpart' rmsle(modelObject, ...)
... |
additional parameters to be passed the the s3 methods |
actual |
A vector of the labels |
predicted |
A vector of predicted values |
modelObject |
the model object. Currently supported |
True Negatives / (True Negatives + False Positives)
tnr(actual, predicted, cutoff = 0.5)tnr(actual, predicted, cutoff = 0.5)
actual |
A vector of the labels |
predicted |
A vector of predicted values |
cutoff |
A cutoff for the predicted values |
data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') tnr(testDF$y, Preds, cutoff = 0) specificity(testDF$y, Preds, cutoff = 0)data(testDF) glmModel <- glm(y ~ ., data = testDF, family="binomial") Preds <- predict(glmModel, type = 'response') tnr(testDF$y, Preds, cutoff = 0) specificity(testDF$y, Preds, cutoff = 0)