Not OP but I used to own both cars. Most all parts are shared between the V8 E39 models, from the interior to suspension parts, transmission, etc.

The most noticeable difference is the lack of a limited slip diff in the 540i which was a huge deal for me when you push the cars. The steering design is the same but the M5 ratio is quicker so it feels a bit sportier in that regard.

Stock, engine power feels about the same under 3000 RPM but past that the M62 in the 540 starts to feel a bit wheeze while the M5 keeps pulling past 6500.

I dare say the M5 engine is more reliable as well in many ways as the intake design avoids most chances of vacuum leaks, the cooling system design is bulletproof and there's just a lot less plastic hanging off.

So yeah, the cars are more similar than they are different, but the refinements BMW made on the M5 make a significant difference in feeling once you start pushing them. If you can afford it, go for the M5 but if you have almost no money the 540i is still a great car. In both cases though, best to know how to wrench a bit.