I can tell you without a doubt, hands down, that this stuff is absolutely imperative in intelligent and high-traffic tech.

So you sped up a Javascript Web form to knock up a growth metric by 6x (hooray growth team! nothing else matters!) and launched into a tirade based on that about breadth/depth-first search and bandit algorithms, while simultaneously making the point that simple CRUD apps -- like a Javascript Web form, say -- don't really benefit from these sorts of optimizations. That was a lot of words to genuinely confuse the shit out of me. You also left out a lot: is what you wrote maintainable? Does the rest of your team understand it, and buy in to what you've done? Is the increase in conversations attributable to something else? In my (much higher traffic than your) experience, latency in a form like that doesn't drive 6x gains. More like 2.5-3x.

Your story about Twitter was fun, but you also omitted the transition from a monolithic Rails app to a SOA Thrift (primarily Scala) architecture, which paid them more dividends than clever algorithms by breaking up the problem space across separations of concern. Yes, their primary problem is efficient pubsub and yes, they have a lot of investment into making that scale, but no, it's not out of the realm of mere mortals.

Instead, it's usually "permute this massive state space" where there are dozens of subroutines being called at different substages, and awareness/skills in discrete math are the difference between winning and losing.

I have no idea what this means, nor how discrete math helps. It sounds like you're implementing gdb stack frame analysis or maybe Zipkin (hey, Twitter's back!) and are describing it poorly.