It might depend on who picks up the book. The HP series consists of some excellent books, and I'm certain it had a lot of readers who weren't usually attracted to fantasy books. - But like @dying_pteradactyl says, it isn't the first of its kind, and it won't be the last. Even if you disregard books, there are cartoons and series with a similar setting. Personally, I'm not sure I would have thought of HP straight away, unless the description made it sound like it was practically based in the same universe. And it would most definitely spark my interest; I'm not sure I'll ever get tired of magic.

I think you need to worry less about someone calling you a copycat and try to focus on making your book(s) the best they can be. Easier said than done, I know. But the truth is that there will always always be someone who will criticize your book for whatever reason; it's just not possible to please everyone. And even if someone calls you a copycat, it doesn't mean that everyone will agree. You are already so aware of any similarities there might be, I think you are well set to avoid becoming one.