Oil damages wood by breaking down the cellulose and lignin structure. It also softens it and provides entry and a source for bacterial and fungal growth. Over time, it becomes blackened and punky.
Wood finishes use petroleum distillates as thinners and carriers, but most of these solvent flash off as the finish cures or catalyzes. The solvent soaks often recommended to remove oil from saturated gunstocks also damage the structure of wood. But this damage is probably less than the damage caused by long term oil soaking. Still, solvent soaks are best avoided or minimized if possible.
Use oil on your metal to lubricate moving parts and to prevent corrosion, but keep the oil off of your wood at all costs. As Miller says, a bit of oil on a finished surface isn't likely to penetrate the wood as easily as getting it into end grain. But even the glossy polyurethane finishes are slightly permeable. Wood might look shiny after an oil wipe down, but it isn't doing a damn bit of good.
