Removing color from casehardened parts may not be removing metal but it certainly is moving metal. Casehardening, in the way that produces varying colors, does so by producing a micro-crystalline surface on the metal that reflects some of the colors in the light spectrum. Oscar Gaddy's great two part writings in the DGJ explained this and showed photos of the crystalline surface taken with an electron microscope, as I recall. Steel wool can certainly wear these tiny surface irregularities in a way that reduces or eliminates their "prismatic" abilities to reflect colors.
Think about it ....... if wear from your hands carrying a gun can eventually wear the color away, certainly anything like steel wool can and will do so. And, it really doesn't matter if the s. wool, or whatever other product one chooses to use, wears the metal away or if it just alters it's structure, when the color goes away the surface micro-structure has been irreparably altered.