I'm not against it since people making use of whatever they get their hands on sounds like something they would do in this post apocalyptic setting, and it gives the option of neat apparel. The concept of using different materials isn't new, look at the Trapper's helm or the Ironhead having fur interior. I think the difficulty lies on implementation, balancing (having many options instead of just having one optimal above others) and still making leather armor viable for Nimble builds and others.
Let's say, how much an insulating vest sewed with bison leather would render ineffective cold damage suffered? And compared to infused bison (can it use infused leather as material?). Although there's already the existence of acid immunity with wyrm scales, without even counting the Tempered: Acid feat.
Lastly, there's the complication of making even less appealing the use of fabrics other than the black cloth for its zero armor penalty and stealth bonus. And consequently, the Clothier feat.
