Am anticipating downvotes for how long winded this is... but, to sort of explain to you what I think this means...

I'm not the OP, but a Lit and Writing student in love with metafiction, one of the more "formally experimental" genres of novel out there. If its an MFA application, and he said "formal experimentation" generally that means an experiment with the "form" of the work.

This usually means subverting typical literary expectations the (implied) reader naturally brings with them. These anything from generic conventions (ex. An authoritarian state in a dystopian novel), to narratorial reliability (think how unreliable of a story Great Gatsby would be if Gatsby narrated and not Nick Caraway), to plot expectations (ex. The Empire Strikes back).

Sometimes formal experimentation is literary in nature (though this is defined by others, not the author), sometimes simply generic in nature (which does not make it nonliterary). Occasionally it is even as simple as the way the language is used (think E. E. Cummings's poem: reppohssarg [Grasshopper]).

It is a vague term still, because OP didn't define it, but those are just some examples of what it could mean, just for future reference!