As someone who also began in the hard syntactic parrot hypothesis and kept up with the progress, here's my take as a linguist and cognitive scientist, and NLP enthusiast since before the age of LLMs: It's syntax all the way down
But this is now quite old news. I’ve been noticing this arithmetic thing since 2023 and posting about it. And of course this is an inefficient way to do arithmetic. I agree! But it’s not an *impossible* way to do arithmetic. If your model implies that it is then you have a problem.