Obsidian Source: Notes / Scaling Scaling Laws For Board Games.
Summary
Pending synthesis from local Obsidian source.
Original source title: Scaling Scaling Laws For Board Games.
Extracted Preview
Andy Jones, Anthropic gives his best on how scale up works.
Experiments are all about abstractions. But how well do they scale up?
- The main reason behind conducting experiments is to see exactly what factors affect the problem.
- However, the question often comes down to how good your experiment is conducted, and at what scale.
- Experiments are all about abstractions(reducing search space). A good experimental design should scale up reasonably well.
- For the most part, a problem is defined already, and we abstract the boundations to fit our resource budget(think about cooking for 2, 200, 2000 people).
- This paper is different in the sense that apart from increasing the model size, they also increase the problem size(basically, the independent variables for their analysis are board size and compute.)
There is an important question that arises when we think about it. By varying two independent variables - how do we make sure they scale up in the same way.
- The parameter board size can be kept as different orders of magnitude to guess the ballpark region as per our needs.
- Compute is different. We can scale it up in so many ways, so we need to have some constraints to bound our experiment design.
In this paper, they explored three "axes" of compute variation - the depth of network, width of network and the training time.
Key Learnings
- How to exactly conduct experiments. Here two independent variables were there, and how to keep the tradeoff plus how to do preliminary studies.
- How your judgement helps guide you towards making assertions.
Integration Notes
- Source folder:
/home/yashs/Documents/Docs/Obsidian/Research-Notes - Local source:
/home/yashs/Documents/Docs/Obsidian/Research-Notes/Notes/Scaling Scaling Laws For Board Games..md - Raw copy:
raw/obsidian/research-notes/Notes/Scaling Scaling Laws For Board Games..md