There's a group going around Facebook right now called, "I say yes sir and no ma'am because I was raised in the south."
So far, everyone I know who's in it still lives in the south.
I say "yes sir" occasionally, if addressing an old veteran, for example. But that's about it.
In many parts of the country, relentless saying of "sir" and "ma'am" makes the above group's name redundant. They'll immediately know you're from the south. They'll think it's adorable. Precious, even. But they're just as likely to think, "Oh, it's a dyed-in-the-wool southerner" as, "Now there's a respectful person."
The main reason I've never liked saying "sir/ma'am" is because of the regimentation behind it. You're simply supposed to say it. Growing up in the south, you get drilled that the proper way to address people is in this manner. Parents scream at their children when they mess it up. Legislators actually pass laws mandating students say it to their teachers and administrators. It connotes respect — more accurately, the sort of automatic, synthetic respect you're expected to give anyone older and/or above you. If you don't, you're a raging jerk, even if everything else you do is the nicest thing in the world. It's a difficult habit to shake, even if you were only a mild practitioner to begin with, for that reason.
But the truth is, it's not spoken so reverently in all parts of the U.S. It's a colloquial thing, just like numerous other patterns of speech. Every American understands it, but it's not always automatic. And if you say "yes" to someone as opposed to "yes sir" in these areas, you aren't going to be made to feel like a bad person.
I try to treat everyone with the respect they deserve. I do it in ways that are genuine and, hopefully, are noticed. Regimentation and forced pleasantries strictly optional.