It's illegal, at least in a lot of places. Usually goes under something like "patronizing a prostitute" (and prostitute in this case is a bit misleading, since it doesn't need to be a prostitute).
Washington State as an example:
Patronizing a prostitute. (1) A person is guilty of patronizing a prostitute if:
(a) Pursuant to a prior understanding, he or she pays a fee to another person as compensation for such person or a third person having engaged in sexual conduct with him or her; or
(b) He or she pays or agrees to pay a fee to another person pursuant to an understanding that in return therefor such person will engage in sexual conduct with him or her; or
(c) He or she solicits or requests another person to engage in sexual conduct with him or her in return for a fee.
(2) For purposes of this section, "sexual conduct" has the meaning given in RCW 9A.88.030.
(3) Patronizing a prostitute is a misdemeanor.