The regex basically requires the title to have <some character or punctuation> by <some character or punctuation> .
You've got some bad syntax here. Don't use | inside of [] unless you're trying to match the | character.
This should accomplish what you want using only \s which represents whitespace:
[^\s]\s+by\s+[^\s]
In English: Any non-whitespace character, followed by any amount of whitespace, then the string "by", then any amount of whitespace, then some more non-whitespace.
Rules are case-insensitive by default so no need to accommodate that. But if you did need to, you'd use [bB][yY].
You might find this site helpful for experimenting with Regex. Set it to Python (on the left), then click the flag icon on the right of the regular expression box and make sure only these options are selected: global, insensitive. That will mimic Automoderator's behavior pretty well. You can then paste in test strings to see what matches and what doesn't, and there's even a handy breakdown of what your expression is doing in the right hand sidebar.
Here's the example I wrote above, with some test data: