Here's my setup. I welcome any comments or suggestions.
It all starts with some DOCSIS 3.0 modem from ISP. The routing capabilities are disabled and basically it just acts as a switch; it's in bridge mode. The service was recently upgraded to 300/30 Mbps.
Next is a PC Engines apu system running Debian GNU/Linux 7 (Wheezy). I believe the hardware is identical to the low-end pfsense firewall. I used these simple instructions to install the OS to a large mSATA drive. The total cost was approximately $300, although it will be greater if you include parts for 802.11 capability.
iptables is used to create a basic firewall with a NAT. TCP port 80 is redirected to a squid proxy running in intercept mode. It rewrites the user-agent header to a generic Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0, blocks domains by regular expression, and offers other privacy and performance improvements.
squid is chained to privoxy, a filtering proxy. In addition to blocking most ads and malware, I have custom rules to redirect certain sites to https, block anything mentioning the word "facebook", and do all sorts of crazy stuff.
lighttpd runs on port 80 just serving nostalgic GIFs from the 90's web. privoxy is set up to redirect ad images and other block patterns to this server, resulting in a pleasant web browsing experience for all clients. privoxy can do much more, but you get the idea!
dnsmasq is used for DNS and DHCP. It talks to a local dnscrypt-proxy process sending out all DNS queries to my secret server. dnsmasq is also configured with a big blacklist of domains, a varity of which you can find for free online. There are also options to log all queries, cache replies, redirect entire TLDs, and never forward plain names (like hostnames of machines on the local network).
In the past, I've tried snort and other IDS, but for a home network I have found it sufficient just to do basic kernel logging of incoming/outgoing packets. If there's anything of particular interest, I will usually just run tcpdump.
The board has three gigabit ports. One is used by the external interface. I connect another port to a desktop machine (effectively a DMZ) and the last port to my previous router -- an Asus RTN66U running TomatoUSB shibby and Entware. Although it is a very capable box and great wireless router, I was not able to exceed ~100 Mbps download speeds due to the firmware. The wireless network is obviously secured with WPA2-PSK (AES) and the password is rotated every few months with something like openssl rand -base64 20. You can save yourself the hassle and set up a RADIUS server with certificate authentication, but I've found it to be just as much of a hassle.
Since it's my home network, I know about all the devices on it. Because the router is running GNU/Linux, there are a million more things you can do and a lot of potential opportunities to learn.
