Does the mercantile skill also increase the quality of items sold by vendors?
Not directly. It unlocks new tier of special merchandise. This tier gives you more item to choose from, greatly increasing chances of finding good quality item you need. It is also said that this tier has higher quality items in its pool, but someone more knowledgable than me should confirm/deny that. New tier does not override old non-merchantile related items, you see both in trade window and they are expicitely marked as merchantile/non-merchantile items.
Does the mercantile check passes ONCE in dialogue only? For example, when Hypercerebrix used and requirements met.
For merchants you get multiple attempts. For quest rewads, only once.
Side note: IMO Merchantile, up to minimum breakpoints for speical merchandise tiers, is essential for crafting top tier items. Do not attempt crafting min-maxing without it.