I can tell you how I think they work.
Every shotgun shell has different armor bypass values - the less the pellet count, the more the threshold bypass. Shotguns shoot in a cone and generally have low accuracy, each target in the cone has a chance to be hit by a number of pellets. The closer they are, the more pellets that will hit them, although, and again, I think that the final damage shows in a single number no matter how many pellets hit the target. All I have ever shot is a short barreled shotgun so I may be missing something, but I'm guessing this is the general idea behind shotguns. They are pseudo melee weapons.
