If I shoot really light loads it doesn't do it.
The Fox safety is simple and straight forward with only three parts Since it doesn't occur with light loads that leads me to believe that the safety spring is not exerting enough force to hold the safety securely in the safety detent in the upper receiver. Pull the stock off and see if the spring is broken, is not properly aligned or there is not enough force put on the spring to hold the safety in the detent. There are no other plausible causes I am aware of.
