If the stock is finished, you're going to have to take it off the stock to remove any more. Then, it will bother you more if you take a hair too much anywhere. Also, it will bother you more when you take it off and try to line it up perfectly everytime you replace it. I'd just break the knife edge with a light pass of 400 paper and call it good.