Contacts full of redundant addresses

I synced my contacts from an Exchange account (365), but along with all my actual contacts in Outlook it's synced the address of just about everyone I've ever emailed or received an email from.
How the hell is this happening?
It's definitely syncing ONLY Contacts from the Exchange account.

Anyone else get this problem?