I guess this depends on what you prioritize. There are kind of diamond scenes in Blades :

- Lore tablets

- Social quests

- Side quests

- Legendary weapons

- Elite skills

The Lore tablets, the legendary weapons and the Elite skills help you gain XP and thus Level Up and gain more skills. Lore tablets help you understand the Blades world, the way each race lives, their politics, relationships etc. The Legendary weapons and Elite skills allow you to defeat enemies etc.

Side quests don't increase your XP but you find valuable items that help you in future chapters. You may get out of whatever situation without these items but they do make it easier for you.

Social quests are basically scenes with the LIs, they allow you to spend time with them and get to know them.

So basically, it depends on what you want.