Autotuning Methods

This page contains lists of concepts, methods and key ideas for autotuning in the context of this project. This is a living page, where links for resources and new items are added as the project progresses. The purpose of this page is to provide an easily accessible reference for the concepts and tools involved in autotuning.

Overview

The list below summarizes available methods for autotuning, and is based on this tree:

Search Heuristics
- Does not estimate a surrogate function
- There is no model, underlying hypotheses guide exploration
- Examples: Simulated Annealing, Gradient Descent, Genetic Algorithms
- Introductory book: Stochastic Local Search: Foundations & Applications
  - Free chapters
Statistical Learning
- Estimates a surrogate function
- Introductory book: An Introduction to Statistical Learning (free during COVID-19)
  - Online course by the authors (free during COVID-19)
- More in depth: The Elements of Statistical Learning (free during COVID-19)
- Parametric Learning
  - Higher biases, simpler models, focus on finding relationships between parameters and response
  - Examples: Optimal Design, Linear Model, Bandit Algorithms
  - DoE Book: Design and Analysis of Experiments (free during COVID-19)
  - DoE Book: Statistics for Experimenters
  - Classic DoE Book: The Art of Computer Systems Performance Analysis
- Nonparametric Learning
  - Lower biases, more complex models, focus on function minimization
  - Examples: Gaussian Process Regression, Decision Trees, Neural Networks
  - GPR Book: Gaussian Processes for Machine Learning (free)

Tools

Statistical Learning

The R Project: standard library with powerful statistical functions
The Julia language

R packages
- tidyverse: Data plotting and manipulation
- randtoolbox: Quasi random, low discrepancy sampling
- DiceKriging: Gaussian Process Regression
- DiceOptim: Efficient Global Optimization
- DoE.base: Design of Experiments
- AlgDesign: Optimal Design Construction
- quantreg: Quantile Regression