Thee should be a spring holding the peg in place. It is part #8 in the attached picture. If it's there but just not holding just bend it a little. If its not there go to your local H-D dealer for replacement.
Some of them are crap as in poor heat treatment so the springiness is none existant. Some rust way causing the same thing. What I've done a couple of times is custom grind a flat washer to give a friction fit when the pin is installed. My rear pegs and mounts are removed so don't have that issue any more.Perfect, thanks. I did see that piece in there when I looked but didn't realize that's what it is.