It might make a big difference depending on the type of shooting and the weight of the barrel. That is why I ask about the how and what. I have maybe one lightweight SS, it's a .25-Krag on a Borchardt and I shoot that one off the barrel as well. I don't have any single shots that I shoot fast or that get really hot. The majority of my SS shooting is all with heavy barreled rifles shooting lead bullets from rest.
None of my pre-war bolt guns have a free floated barrel and I found that they seem to keep their zero for about the first three or four shots, after that the group starts to move. These are hunting rifles and you should have the job done after three ;-).