I think it depends on the game...
For waterfowl and clays I am learing to shoot an 8 lb Model 21 Duck.
My best shots however, those "how could anyone hit that!" shots have been made with my lightest SxS, a 12 ga Merkel 47E at 6 lbs 3 ozs. The 47E comes to the shoulder and fires almost subconsciously. Perhaps this is why some shooters perfer a light gun, the mind and body accepts them as part of the body. I don't think a pro tennis player would use a very heavy racket or Tiger Woods would use a driver when a 5 iron would be best. I have shot geese with the 47E using 1.5 oz 3" loads but it is not something I like to do very often! I've also shot clays with it but after 100 rounds or so I've had enough.
As for physical condition; I'm a retired Marine who knows how to walk and walk and walk while carrying far too much gear. I'm also in better physical condition that most men my age. But, during my annual 10 day duck hunting trip to North Dakota I hunt pheasants and sharpies in the afternoon after our morning's duck hunts. After half a dozen or more 3AM wakeups both my retriever and this hard dog can get a litte worn out. For those long afternoons I want a light gun. I'm very certain my afternoon ND field walks are nothing as hard as a few western chukar hunts!
If someone wants and can shoot a 5 pound gun then good for him!!!