What does the 'memory pressure' view say? Is it red or still green? If it's green your problem is not RAM usage. Also how much memory is swapped?

I don't know why Apple still has the 'memory' pane in Activity Monitor. It is confusing for developers already - average end users have no hope to grasp the implications at all (not condescending here but average joe probably has no knowledge about modern kernel design and implementations).

A lot things contribute to "high memory usage" like caching - which OS X performs very aggressively but also readily frees those caches when needed. Or 64bit address spaces which are so huge that applications don't need to "release" allocated memory addresses and can just grab as much as they want - which shows up as memory usage even though no real memory is allocated for that. Unused but claimed address space is reported as allocated memory but is not really allocated by the MMU - a real allocation only happens when you write something to that memory address. With 32bit you had to watch out because you could run out of the ~4GB address space - so applications released claimed address space when it was not needed anymore. With 64bit you have such a huge available address space that it's not worth the hassle managing address space. You could never* run out of 64bit addresses anyway.

The only way to find out how much memory a process really uses is to inspect it with tools like instruments. For example Activity Monitor tells me right now that TextEdit is using up 60MB of Memory - but at the same time Instruments shows me that active allocated memory only amounts to 5.5MB.

tl;dr - you can't trust Activity Monitor anymore when it comes to memory usage. The only thing remotely meaningful there is the 'memory pressure' view and the 'Swap Used' cell. If you're not using significant swap memory then you are not out of memory. Everything else could be just random numbers as well.