I often think about this and the only thing that makes sense to me would be a smaller Switch. Perhaps something the size of a Vita or a little larger without detachable joycons. Maybe a clamshell design, but with just 1 screen like the GBA SP.

The problem I can never get past with a theoretical 3DS successor is that a 3DS successor would almost certainly have to be more powerful than the current 3DS. If the 3DS is between N64 and GameCube power the next one might be somewhere between GameCube and Wii. The Switch is somewhere above the WiiU. I think that would put a 3DS successor too close to the Switch.

Plus with a more powerful system that would almost certainly decrease battery life and increase price. Every Nintendo portable has become more expensive while decreasing in battery life. Again that would put a 3DS successor closer to the Switch in price and battery life. Also a higher price would eliminate the 3DS' current selling point of being a low cost alternative to the Switch as both a new 3DS and Switch would likely be closer to each other in cost than the current 3DS and Switch... especially if the Switch goes down in price at some point.