Spinning things have momentum just like things moving in a straight line. It isn't as intuitive as straight-line movement, but it is analogous.

Every object with mass has a resistance to changing velocity (speed in a set direction). This depends only on it's mass. Every object also has a resistance to changing how much it is spinning and in what direction. What bikes and unicycles take advantage of is how the torque (turning force) required to change the direction of a spinning object is at a right angle to the change in direction.

Imagine the front wheel of the bike spinning as you ride forwards. With your right had make a 'thumbs up' and then move your hand so the fingers go in a circle in the same direction as the spin making your thumb point to the left. This is known as the right hand rule which is the conventional way of describing the direction of the spin with your thumb being the arrow.

Now imagine the bike starting to fall to the right. As the wheel tips, the direction the thumb is pointing goes up. This change in momentum requires a torque. The direction of this torque we can find using the right hand rule: point your thumb up and your fingers follow the direction of the torque needed. As the wheel is free from the wider world the reaction torque from the tipping wheel acts on the fork turing the handles in the other direction, making the wheel turn towards the right.

Now that the bike is turning, it resists falling in the following way:

The bike is following a circular path. It's direction is constantly changing so, although the speed is the same, it's velocity is changing. This needs a force to cause and it's direction is towards the centre of the circle. This force comes from the contact with the wheel and the ground. This force is out of line with the centre of mass of the bike and rider so causes a torque. This balances the torque from the leaning bike making the gravity and thrust from the ground being out of line. The bike is therefore in balance as it make a turn.

The faster the bike is going, the greater the angular momentum of the wheels and so the greater the amount of torque needed to tip the bike over by the same angle. When the bike is still, there is no angular momentum so the bike is free to tip easily.