Obsidian Source: Drafts / Evals Cookbook
Summary
Pending synthesis from local Obsidian source.
Original source title: Evals Cookbook
Extracted Preview
Q: ML as applied science
On surface, ML looks a lot like abstract magic, it is just glorified linear algebra. When a newbie is introduced to this, they feel there is no standard way to approach the problem, and there are many heuristics they have to rely on. Eval is an attempt to make ML feel more science-y.
In prototyping, we try out different models, when in production, we test of live data => feedback loop to improve.
Q: Why model eval?
This is applying ML in an applied science context. When not in check, can have can have catastrophic effects.
Talk about IID assumption.
Why thinking about evals up front is nice(help set up realistic goals and measure the success of the project)
Q: What is stratification/cross validation?
- Stratification : When target class/another feature isn't distributed equally.
- Tell about cross validation, and hold-out validation
That's why choosing of metrics should be done carefully. If the classes are not balanced, a model may output a heavily weighed class regardless of input and me more accurate(but less useful!) . Metrics like F1 score, Cohen's Kappa Coefficient or Matthew's Correlation coefficient should be used(which are robust to class imbalance)
Q: Why cross validation sometimes fails with time series/spatial data?
Write the whole long ass story.
Q: What about online and offline learning?
Integration Notes
- Source folder:
/home/yashs/Documents/Docs/Obsidian/Research-Notes - Local source:
/home/yashs/Documents/Docs/Obsidian/Research-Notes/Drafts/Evals Cookbook.md - Raw copy:
raw/obsidian/research-notes/Drafts/Evals Cookbook.md