Yes, indeed
You've summed it up well.
The new guy should keep reminding himself that his new employer must be doing a couple of things right...like making enough money to hire more help, for example.
So make haste slowly. That bad policy or inefficient procedure which you want to throw out the window might well be someone's pride and joy. It might also be vastly better than the previous iteration ("You should have seen this before we fixed it!"). Or it might be one of many links in a chain which cannot be broken, in which case your recommendation to throw it out the window will make you look like a fool.
So learn the lay of the land. Then ask a lot of leading questions. If you can get people to understand ON THEIR OWN that something needs to be fixed, they might well be able to do a better job of fixing it than you can.