I actually think Illumi is a really cool character, but I won't lie and say that I don't find him completely detestable. You say Killua has given no indication that he hates Illumi, but I would argue the opposite. He has given no indication that he loves Illumi, or some of the other members of his family such as Milluki or Kikyo. Recall the Zoldyck Family Relationships diagram that Togashi drew in the Election Arc. The point of that diagram was to illustrate that the Zoldyck Family aren't held together by strong familial bonds, but by loyalty to the family itself. The Zoldycks are a group of seriously twisted human beings who have complex relationships with one another based on their individual personalities and compatibility with other members of the family.

Any "love" the Zoldycks have for one another is either for the good of the Family, or selfish in nature. Silva let Killua leave the mansion in order to travel the world with Gon, but still knew that Illumi's needle would eventually force him to come home, and would never truly allow Killua to abandon the assassin lifestyle. Milluki is pure scum, only seeming to genuinely care for his parents, and showing no concern for Killua's life when he was dealing with Nanika. Kikyo and Illumi should go without saying. The latter is a complete monster who only really loves Killua for his immense potential, and openly admitted to himself that he planned to use him as his puppet to control the Family and Nanika's power.

The only real exceptions are Killua and Alluka, who unconditionally love each other (and perhaps Kalluto too, but that remains to be seen).