It's not about having a calculator. It's about making quick connections when encountering new facts or numbers.
Without sizeable core of memorized facts, results and numbers you will miss a lot of those. It's the very definition of being not that smart.