even for a month of data with over 200m record

this is less than 1 week for me - for just apache logs (nothing else), but for the most part ELK rocks. I'm having some issues drawing histograms over 1 week (300m records) and i gave up when the ES server was at 100 load (i had resubmitted request a few times cause i lack patience), will try again tomorrow.

To answer OP, your devs need to cooperate to some extent with their log formatting. Logstash has pretty good filtering capabilities... and what you do is basic filtering on your actual server, and ship that off to a forwarder to do the rest of the "work", and then that ships the message to ES. To get really good parsing takes some time and testing, and I'm sure these same things hold true across similar systems. There is no out of the box solution that will do this for you. Your best performance will come from not searching whole log messages for fields and have it already split and indexed on those fields.

One situation I hit is logging x-forwarded-for headers. Im apache log config I log %{X-Forward-For}i, %a so it shows up as internal client, clientproxy1, clientproxy2, ourproxy. I want to only record remote. Also finding a lot of proxies out there dont exactly follow standards (hostnames in the list, lack of spaces after comma, two spaces after comma, etc...). So what I did is if the 'remote_addr' is on an internal ip, we replace remote_addr with the last thing provided in the x-forwarded-for (rightmost is closest to you)

my grok was something like to get the right most thing before the last IP.

( *(?<last_xff>(%{IP}|%{HOSTNAME})),)*( *%{IP:remote_addr})

and as far as i can tell that gets the last IP in the comma separated list.

,? because i've seen some proxies do that.

if [remote_addr] =~ /10\./ and [last_xff] 
   mutate { update => ["remote_addr", "%last_xff"] }

What kind of tuning do you do on # of shards etc? Right now I just have one ES server running master/data.

I enabled huge/large pages for java on my ES, seemed to help a bit, but I dont have any good before and after numbers really.