Overview

The notebook aims to:

  1. Load multiple model assets
  2. Evaluate performance across models
  3. Ensemble models

Load Models

library(tidyverse)      # data manipulation

mod_rds <- dir(params$outpath, pattern = ".rds", full.names = TRUE)
mod_nms <- mod_rds %>% str_extract("^[^.]*") %>% basename() 
mod_ls <-  mod_rds %>% 
  map(readRDS) %>% 
  set_names(mod_nms)

Compare Metric Scores across Models

transpose(mod_ls)$scores %>% 
  bind_rows(.id = 'Model') %>% 
  split(.$Metric) %>% 
  map(arrange, desc(`Test Score`)) %>% 
  map(knitr::kable)
## $mae
## 
## 
## Model      Metric    Train Score   Test Score
## ---------  -------  ------------  -----------
## baseline   mae         3.3471832     3.104941
## lm-ft      mae         3.1019493     2.846203
## h2o        mae         0.4687236     2.353799
## rf         mae         0.9559566     2.221420
## 
## $rmse
## 
## 
## Model      Metric    Train Score   Test Score
## ---------  -------  ------------  -----------
## baseline   rmse        4.7302427     4.542532
## lm-ft      rmse        4.3149490     3.880764
## h2o        rmse        0.5812407     3.453688
## rf         rmse        1.4917752     3.048917
## 
## $rsq
## 
## 
## Model      Metric    Train Score   Test Score
## ---------  -------  ------------  -----------
## rf         rsq         0.9780070    0.8950499
## h2o        rsq         0.9962012    0.8625395
## lm-ft      rsq         0.7781203    0.8263787
## baseline   rsq         0.7333552    0.7601197

Question: choose champion or ensemble everyone?