While bucks grow their antlers every year, their testosterone level is slowly increasing. This testosterone surge causes them to lose their velvet and begin the rut (mating). Once the rut is over, however, their testosterone decreases rapidly, causing them to shed their antlers.
Fun fact: Although it's rare, some does actually grow antlers; they're called hermaphrodites.
