I'm still very indecisive yet. How about this :

It seems to me that the most important yet architecture dependent stuff, is the size of the pointers. Most of the time, there are efficient add/mul/shift whatever for this word size available. Assuming we can cast between size_t and void, that basically reduce the whole mess to only two types (size_t and size_t). Now one might get a performance hit when using say u8, but he could also now go for u4 when needed without thinking too much about it. Because it will all be software anyway. When using the native size_t*/size_t, things will go fast, and when one want to manage memory with a more fine grain or fixed grain, then one use a software layer. Also it will be quite easy to check that the program is correct both for say sizeof(size_t)==1 and sizeof(size_t)==8, just running the unit tests in those two virtual machines.

Of course you'd need a sizeof operator available at runtime, in order to know the size of the architecture you are currently running on. So because size_t* is an index anyway, only one type is really needed : size_t. It can both represent a number, and an index to something (like the 0 memory cell), depending on how it is used.

Having only one type, certainly makes things more straight forward.