PyTorch: a reference language
Edward Yang offers a clarifying model for large-scale ML work: keep approachable PyTorch code as an executable reference, while specialized kernels and explicit forward-backward paths become the production implementation. That separates research ergonomics from hardware-specific optimization without discarding the former. The reference is not dead weight; it is the thing against which faster code can be checked.
The Hacker News objection that this sounds like giving up on compilers is fair, but it frames the trade too narrowly. A compiler remains valuable where it can reliably optimize and preserve intent. The harder cases need explicit control over data movement and fusion, plus a verifier stronger than "the loss looked plausible." Treating the reference implementation as a spec makes that specialization accountable rather than merely faster.