Hey /u/13steinj!
Sorry about the surprise URL change. The fuller context of why we are doing this change is to help crawlers understand that:
/r/{subreddit}/comments/{id}/{title}/
is the same as
/r/{subreddit}/comments/{id}/{some-other-title}/
Which is itself identical to an infinite number of other urls. For this purpose we have also recently included canonical meta tags. The word we have received though is that canonical meta tags don't perform as well as simply having the same urls.
The reason this code hasn't been announced or even merged into opensource quite yet is because as you said it's a bit buggy right now. I knew when launching the code that since we have millions of urls, it was going to be difficult to catch every single issue without going to production with it. As soon as most of the kinks have been worked out (hopefully monday) I will be merging the code back into open source.
Your last point about it being an unnecessary lookup isn't correct though. You can see here https://github.com/reddit/reddit/blob/master/r2/r2/controllers/front.py#L258 that subreddit lookups always happen regardless of this change.
If you have any other questions or find any other bugs just let me know!