I've been building up 4*'s like you suggested and I currently have:
40+8 Subaki
40+2 Virion
30+1 Gunter that could have been 30+3 if not for HORSH. Will be 30+2 if I decide I don't need another Hone Cavalry.
23+5 Barst that will get his +6 once the feathers roll in.
23+1 Donnel
20+2 Jakob
20+1 Serra
I could also have a 40+10 Cecilia, but I want to 5* her because HORSH.
Only issue with that plan is it requires another Nino, because I merged the one I upgraded instead of stealing her book.
Hell, I even have a few merged 3* units for shits and giggles.
36+4 Draug
20+1 Setsuna
20+1 Clarine
20+1 Laslow
20+1 Wrys
Most things are level 20 because I like to level up in batches of 10 until they hit level 30. I think my favourite thing about this game is how many long-term projects it gives me that require very little actual effort. Always something to look forwards to. Only thing I really dread is deciding what skills to give to who. Would appreciate some help with that, honestly.