My friends are really cynical. Any time I tried to show them a trick or anything, they'd actively try to screw it up (well, some just work out how I did it, and that's cool with me). My Dad's kinda interested and curious, being a poker player who's never touched a deck worth more than $1.50 in his life up to this point. My Mum would hate it if she knew hoe much I was spending.
Ahh the cynical crowd. The ones that think magic is for attempting to mess with and interrupt rather than sit back and let the illusion take you in. People don't understand how much better the world is before you know how a magic trick is done. Once you learn, you can't unlearn, and suddenly things are far less magnificent since the answer is never shapeshifting, invisibility cloaks or thousands of miniature jetpacks on your shoes engineered by Nazi scientists in WWII that you managed to procure and now use in your levitation stunt.
Instead, the answers are simple, and extremely boring to non-magicians.
"You mean all you did was lift two cards? That's IT? THAT'S STUPID!"
I think with many "trick breakers" it has a bit less to do with wanting to be smartasses and intentionally messing with you, and a bit more to do with the fact that you're messing with their reality a little. Do some magic to a type-A personality control freak who considers himself the master of his domain and thinks he understands the world left, right and sideways, and it can weird him out a bit. Some people with esteem issues will actually get angry because they think you're trying to make them look bad, especially if you come off as arrogant when performing.
Then again, I had one kid I performed for - his entire extended family were in this dinky hospital room, maybe a dozen people, and when I tried performing for him, his family was telling him in his native language to mess with the trick. By the third time performing that trick, I used a stripper deck and intentionally pulled the card out of the deck mid-trick to ask "Are you SURE you remember your card now? Because I'd hate for you to forget it - the trick's not as much fun that way." By the time I finished that trick, he conceded that I got his card correct.
After that, I went with a WOW trick. I asked them to pick a card and showed them the wrong card intentionally. The kid said it was his card. I wished him well and left without finishing my routine!
Fortunately, little pricks like this kid and his family are few and far between.
My mother actually has a deep understanding of why I enjoy magic and cards so much. I've enjoyed playing cards since childhood, and her father was an inveterate card player, as were most of her brothers and brothers-in-law. Apparently he's the person in the family that I seem to have gotten the most from in terms of my innate personality, and genetically I shared more of my mother's physical traits than any of my brothers (no blood-relative sisters, just one adopted).
I've also been noticing that as my skill has improved, people are more impressed with my magic. I've been doing this one three-trick routine over and over again for so long now, I could probably do it in my sleep - it only requires a standard deck with two identical (or seemingly identical) jokers, and I'm always carrying at least one deck that fits the bill, usually Bicycle Guardians. It feels a little boring at times, but I've done it for people who've seen the tricks individually when I first learned them, and they see it as totally new and very impressive. I remember reading a quote from some famous magician that it's not how many different tricks you know, it's how well you know the ones you do know. One magician mentioned in the same article where I read that quote had a repertoire of exactly five tricks. But he did those five tricks with such utter perfection, more weren't necessary.