Contrary to popular belief, banks don't hold back your money for fun or to deliberately make intrest. Inter-bank money transfer is actually quite complicated. But since this is ELI5, I'll try to keep it simple.

When you transfer money within the same bank, by all sense the transaction should be instantaneous. If both sending and receiving accounts are managed by the same system, it should be as simple as "substract X from account A, add X to account B" in one transaction. The only limit here is when X is above a certain amount. In that case it can be that your bank will withhold the transaction for a day or two because red lights went off. Large currency transfers can be indicative of fraud or similar illegal stuff after all, and they must be investigated. In my country for example, any transfer of 5000€ or more will automatically trigger this.

Now, transfer between banks, tons more complicated. See, every bank can't know every other bank. This is why there are sime intermediate parties involved, such as so called "clearing houses", that help with the transaction. Money gets transfereed from party to party untill it reaches the destination, and this can take quite some time depending on how often the involved parties process transactions. (some do it ad-hoc, others do it only with nightly batch processing). Here too, any party along the way can decide the transaction triggers a red light (source, destination, amount, transaction history, etc) and decide to hold the transaction for further inspection.

Source: work for a bank, and although I don't work in the payments department, I've gotten a crashcourse of how inter-bank payments work.