if left on for an extremely long time, like for 20+years, a handful of signs of independent thought may arise. But it's not 100% an AI, just a super accurate way of predicting players' behavior and after a long enough time, you get spooky coincidences. Like if you leave a closed container with organic matter inside for a long time, it ferments inside and forms a small ecosystem that increases in complexity over time. I dont think Paul's an AI. Matpat is partially wrong. I think Paul started playing it, and Petscop learned from his inputs. But I do think that the episodes in which he doesn't speak (the demos and that last part) is Petscop trying to solve itself using records of Paul's behavior for reference.

Thinking about it in this way makes it very simple. Paul plays Petscop. Petscop learns from his inputs. Since it's left on, Petscop plays itself. Actually, it's probably better that the game have limited memory/storage because it doesn't have a choice but to re-use inputs from previous players. And the players who played the game the most would of course have the most influence on the game's decisions.