If you were to a throw a ball straight up into the air it would come back down, if you were to throw a ball outward it would go down the field, if you were to throw that ball so fast outword it would go around the planet.

You go up above the atmosphere into a (mostly) vaccuum so you do not experience air drag that slows you down or heats you up, that is the vertical (Altitude aspect) how ever to not be pulled back down you need to start moving horizontally so that you're going so fast that the gravity of earth cant pull you down fast enough so you keep missing the earth rather than hitting it by falling back down.

Altitude is a funny definition of space because for us its basically where our atmosphere sorta ends. If you're going fast enough horizontally on the moon and nothing gets in your way you could maybe be in orbit very low down because there isnt any (or much ) atmosphere to slow you down so gravity can pull you.