So, I don't want to give away where I live, but let's say my Mom had a deep Texas accent. She is from a "country" state and had a pretty thick accent. You might call it "hick."

She moved to "Oregan" and the accent just changed from who she hung out with, who she admired and wanted to be like. She went from a drawl to a pretty mild accent.

My Mom is from [Southern State] and had a thick, thick accent. In-home videos of her talking when I was a baby, she sounded like a whole new person. But she has lived in "Oregon" for so long she doesn't sound "Texan" any longer. Only when we visit family does it come out again.

So it is very likely you developed a pattern of talking with people you admired, which might be British people. If i were you, I'd make a video of myself speaking and post it on a subreddit about accents and see if someone can guess it. It might point you in the direction of where you picked it up.

But from one person to another, I'd own it. Since you like Harry Potter, I would just say, "Well, I am a Hogwarts Graduate, I must have picked it up while I was there. Muggles don't always understand."