Actually, I’ll walk that half a step back. I think “just word prediction machines” has some value for introducing people to the idea of an LLM from scratch. But it’s a bit of a “whales are like fish that breastfeed” thing – an orienting statement that is true-ish but really not a good place to stop.
Like, “Keeping orcas in captivity is unethical because they’re basically just hairy sharks” is not a good argument.