Your body generates heat even at minimum activity, and if it can't release that heat into the outside environment, it builds up.

To release the heat, the temperature outside has to be lower than body temperature. If it's too high, then the body has to use special mechanisms to cool it anyway, such as sweating, so that evaporation will produce a lower effective temperature on your skin than the ambient environment. But unless you're accustomed to it, being hot enough to sweat is not the most comfortable situation.

The most comfortable temperatures are ones where the difference between the environment and your body is enough that the rate of heat loss through your skin closely matches the rate of heat generation in your body - where you don't lose heat too quickly or too slowly. But you do need to lose heat.

We are not reptiles, and cannot function by just equalizing with a warm outside temperature. Our bodies produce heat no matter what we do, so it has to be gotten rid of or it builds up; but not too fast, or we become hypothermic. The most comfortable ranges tend to be in the 60s through 80s F, averaging into the 70s.