Look at it from a simple Network Engineer perspective. If you spend a little time on the older Certification paths, and I don't know what they are anymore, say Cisco, Bay, MCSE, Novell, time studying the protocol stack, it's clear what's happening, though it's not told to 'regular people'. The whole idea behind IP is connectivity and the ability to track. Built into every layer down to Data Link. It's the point. Every routing protocol above a static route is a 'meta data' database. The database is not static, so perhaps it's not a base, but it has all the features of a database record. Where from, when, where to, when. Routing tables exist, of course, and they die or expire. It's an active process to get the death of the data tables.

Not every packet is recorded but they can be. At this point, it's merely a process of handling Big Data. Trend tables, visualization, etc, possible algorithms of what unusual data goes where. Any basic security audit involves just looking at what unusual activities are present. Drill down into that and you automate the record-keeping. Open source tools exist to do this. It's all regular business.

Cell phone 'meta data' keeping is now not a surprise. It would be remiss of a carrier to not have a record of what kind of data comes from who, where, and when.

Now you're just throwing this at the developing Big Data scene.

It's all very logical and systematic. The surprising thing is that people see it from the other side. As if the NSA needs to develop some secret technology to get this done. I can't believe that a huge repository exists to keep this information. They are actively using it, and that is clearly about terrorism and hostile-nation data security.

It is a dangerous and awful tool. The USG has no choice but to proceed on this path, would be terribly remiss if they did not. Big Data Analysis is a Nuclear Weapon of the Information Age. To change our rightly held notions of personal anonymity, the natural protection of the crowd, we'd have to go all the way back and change the fundamentals of data communication. That seems incredibly unlikely. The real 'dangers' on the internet remain terrorists, cyber-criminals, sexual exploiters, and other scum. But the tool used to combat them is so powerful and has so many implications to our personal freedom, it clearly has to be watched. Changes in the base model of information exchange, changes that Aarown Swartz wanted to be a part of and that I support, are included in this.

It's a dangerous and volatile combo.