You are correct, there's nothing to stop anyone from doing that. But, like a lot of stuff in C++, we shouldn't it just because the language allows.

Doing so is usually bad design. There's no reason (that I can see) to virtually inherit from a class if there are no virtual functions to inherit/override.

There are use cases for non-virtual inheritance that would not work efficiently under that rule of thumb, e.g. CRTP.

If a class is supposed to be inherited from, but never used by itself, you can declare the destructor as protected, so only the derived classes can destroy it. If you're not declaring your class final, there should be a good reason for inheritance, and if it doesn't have other virtual functions, it's probable you won't need a virtual destructor either.