Really, there is NOTHING wrong with not starting math until the kid is 5 or even later. Nothing nothing nothing. Just pick something that YOU can work with (and if you don't like manipulatives, feel free to not get a program that involves them!) and do it at the pace your kid is comfortable with. You are not behind.
If a kid is the type of kid who could have been doing grade 2 math in K, well, I would expect him to plow through K-1 math pretty rapidly, but you absolutely don't need to feel *behind* because you didn't start him at 3.