I think the key part is "Agency"

People would love to stretch the definitions of "Interactivity" so that they argue that a comicbook is interactive because you can read the panels in whatever order you like (actual argument I have been presented).

When we talk about "Game" i think it is 2 things

It is first of all a sliding scale on how much agency you are able/willing to give the player. So by that definition Dwarf Fortress is a VERY GAMEY GAME and Visual Novels are not very gamey games. Reason for it being a sliding scale is because the individual may look upon the scale and say "Everything beyond this point isn't a game", where another individual may point to second place and say the same thing.

The second part is "Author's intent". I think it becomes fairly obvious in some cases that some game devs dont want the filthy plebs having a say with their perfect vision. They desire a viewer/reader/listener rather than a player. I think that has some say in what people perceive as a game.

Ultimately I think you should go with what is "Public use of the term" as a definition. Essentially words exist in an effort to relay information and their definitions are made by their use (hence why the whole Literally/figuratively changed definitions through repeated use). The defition of game is so unspecific that you could technically define "Being stuck in traffic" as a game. But when people go "That is not a game!" it is more a rejection of the individual instance rather than a set in stone definition. What they are actually saying is "By the authority of me I reject X on a cultural level". In many ways it says more about the person rejecting that it does about the product. But then again if you already understand the person rejecting it still serves as a working identifier.