To be honest, if I were you I would have cut all contact with him a long time ago. It's actually something I've done with my own mother in some regards because she was similarly manipulative, although not what I would describe as a bully. I understand that you may want your daughter to have a relationship with her grandparents, but as someone who grew up with only a grandmother present I can say that these kinds of relationships can be found in better, more deserving people. My grandmother was a huge factor in why my mother is so messed up and I grew up hating her because of the stories my mother would tell as well as because of how she treated anyone who wasn't a priest. My dad had a 'father figure'/ best friend who in some ways was like a granddad to me and similarly as a child I claimed his aunt and uncle as my grandparents because that's how they acted in my mind. My mothers side of the family is incredible distinction all, but my father's made up for it in most regards. So I guess that what I'm trying to say is you don't have to put up with how your parents are, and even if you don't want to do what I did, you can still try to reduce the amount of contact and time you spend with them. Obviously I am a lot less forgiving than you are, so this may not be something you want to consider, but I just want to know that it will not harm your daughter if she doesn't know her grandparents very well, it may actually be good for her in some ways. As for how it will affect you, obviously that's for you to think about and decide on, but I think it's worth considering.

I hope it all works out for you and that you realise that by having your daughter's best interests placed above what your parents want you are already being a far better parent than they were.