If you want to see some shit, go ahead and watch chris froome's descent on stage 8 of this year's tdf. 6'1" and 157 lbs on a 17lb top of the line aero 56cm pinarello. I forget his top speed, and you probably should never try to do what he's doing but it's evidence that it can be done.

Most likely a combination of factors. Not really sure of the order of importance, but rider weight, aero, technique, rolling resistance, and power are all factors (probably missing a few as well).

Power becomes a nonfactor when you spin out. Weight means the heavier you are, the faster you descend. I weight 145 so on group rides I seem to always make it to the top fastest and people can close the gap on the descents. Bike weight might be a factor, I'm 145 and my bike is an extra 17~18lbs but for you at 185 your bike is probably a smaller part of your weight, but I guess every pound counts.

Aero is a big factor for wind resistance. Most of the drag comes from the rider so your position can really impact and then your frame, wheels, helmet, kit, and a few other things play a descent role.

Tires & rolling resistance also can affect it because if your tires are faster, then all else equal you will be faster as well.

Technique is also important because if you can take the fastest line and avoid braking while still being safe (I have no idea what kind of roads you are on) then the best technique will help you. I think there is a "rule" about how descending technique increases slowly until it abruptly decreases.