My wife thinks what I enjoy watching is worse, and to most folks it is. But, I can watch billiards being played by pros for hours, because I love to play myself.
Much as I enjoy billiards (grew up with a table in my bedroom) I have not got to the point of being able to watch it on TV. I am not a weekend regular in front of the tv wacthing sports as so many are, but as you rightly point out, a competence in a sport vastly increases the likelihood of enjoying watching it.
I'll watch golf and tennis occasionally. Baseball never (never played but I did play cricket) although I like listening to it on the radio and used to have season tickets to the Blue Jays. Basketball.....never. I didn't play it. Football once in a while. Hockey during the playoffs. Alpine skiing I love to watch. Good slalom watersking I could watch for hours. I like watching the competence and knowledge of the sport allows me to appreciate that competence.
And while reruns of sporting events, where the outcome is known, seems pointless to me, I can watch the 1973 Preakness and the 1973 Belmont Stakes endlessly.