Baddek on Zokassa Ridge will only let you take the glowing crystal if you arrive riding the giant horse.

Quest Location

Go to Ishokin Shrine and activate the teleport to make a beam of light shine to its crystal. Follow the beam of light and talk to Baddek. He won't give you the crystal unless he sees you riding the giant horse. This horse can be found east of the Horse God Bridge, which is in the area south of Highland Stable. It will require a lot of stamina to soothe the horse.