Those are pretty much the three trickiest places to find, honestly. You've hit all of them, now, so the rest of the game should be easy navigating for you.
The first time I played through, I didn't even know you could find the Bakers. The second time, I thought you could only find one.
But you've got a really good map, now, that Styg et al put serious work into, so you can fumble through and find your way without using guides. But, anyway - with those three taken care of, your frustration should pretty much be over. Everything else is rather easy to find, honestly, until you get to Deep Caverns. That is a whole 'nother ball game, there.