This one's easy. Golf is pretty much a one-percent sport at this point.
I wouldn't even say culture has as much to do with it as the simple fact that it's an expensive undertaking. Just to reserve time on a golf course is to take a substantial bite out of many wallets. For most people of modest means, like myself, it has always been out of the question. When I was growing up, we could strike up a game of street football or yard baseball, or any number of other sports and activities, anytime. All we needed was a ball or cheap racquet, or a viable substitute (I once made an adjustable hurdle out of three broomsticks and a couple of nails), and much fun could be had. If we were itching to play golf (which, granted, is pretty fun), well, we could fake that too, whether in the front yard (with golf balls or discs) or at a mini-golf course.
But as far as playing real golf goes, you need a huge plot of land and pricey clubs, to say nothing of golf carts, other equipment and exorbitant green fees. That's not casual stuff for many adults, let alone children. (Indeed, look at any rich-person sport such as polo, dressage or sailing, and the common thread is a steep entry level. There aren't pickup polo matches, and in one respect that's the appeal to its base. Golf, to its credit, isn't quite so steep in that regard — but in an economy where many industries are declining with little immediate hope of resurrection, it's no surprise to see golf slumping as well.)
I was probably 31 years old the first time I ever saw a green close-up, and I marveled then at how beautiful and well-maintained it was. It looked better than most lawns I've seen, which is saying something. That kind of care doesn't come cheap, or even moderately affordable for many people.
That elitism feeds upon itself, because few people will go golfing once; it's either a regular thing or never. Buying the equipment is expensive. Renting the equipment is expensive. Chances are, people who play golf want to be at least reasonably decent at it, which takes practice, which most people can't hack financially.
(Similarly, I'd love to take a crack at skiing. But I know next to nothing about it and it requires plenty of expensive equipment. And I suspect that I'd be as good at skiing as I am at skating, which is not very, since I am much better at sliding left than right. That hasn't kept me from skating, though, because it's fun and relatively cheap. I could probably get good at it without going broke. The idea of spending skiing money to suck at skiing is much more of a deterrent. Such is the case with golf.)
I'd love to see both golf and skiing become more proletarian, remote as that possibility might be. But for golf especially, a revamp might be what it needs to survive in a 99-percent world.