Long story short, I aggroed the pirates on the other side of the shore, I left the area during combat while the pirates were standing on the Ferryman's boat, when I came back, the Ferryman was gone, as expected, but also the pirates, they were gone too, pressumely with the Ferryman.
If I'm not mistake you can also lose dropped stuff if you leave it on the Ferryman's boat and then he leaves, unless it goes from a Backpack to a Flotsam
Also you can get your Jet Ski stuck if you disembarked on the Ferryman's boat and then he leaves, you will have your jet ski away from the shoreline and you wont be able to get on it again.
