I wear it all the time - vest & hatband. Like to keep track of my hunting partners as well as anyone else who is in the area I am not aware of.
Some hunters feel birds will flare away from you due to the color or glare from orange, but I have not noticed that myself to any great degree.
Always wondered why only hunters are required to wear it when out in hunting season. Hikers, fishermen,workers etc. would be at risk also.