I am not sure if I should continue to care for IE8 or not.

The real way to tell is look at your analytics and see how many people are using IE8. I know you can't do that with a new project, but you can gauge based off of other projects or maybe the demographic of the site. And then you can look at online statistics and make the decision from there.

The top comment in your post is spot on. If you're doing it in your own time, whatever. But if you're billing clients for that time (and that fine-tune work on older IEs can take a lot of time to iron out), that's likely a waste of time and money to benefit maybe 1-2% of users. But as an example, I used to work for an agency and we had ~20% of users on IE8 and IE9 (windows XP) so not supporting them would mean losing a big chunk of revenue. So it's a business decision to support them, and everything gracefully degraded in those browsers. We spend about as much time on it as the users make in terms of revenue.

I also did a site for NatGeo once and could make what I was doing for them work in IE6/7, but I knew it would take me a half day or so of additional work. At the time anyways (~2012), NatGeo only supported IE8, so it wasn't a requirement for the project/client. When I asked my project manager what to do, he said that if I could do it without impacting the overall project functionality, page weight, etc, and wanted to do it on my own time, that would be fine, he's didn't care. But he emphasized not to let it impact my core time on the project because - while it's nice and gives me a warm fuzzy to know my stuff works basically everywhere - it does not matter and isn't productive for the client or our business. He was a great boss though and loved seeing me happy/excited about stuff. Most bosses probably would have said absolutely not, and from a business perspective, I'd agree with them, too.

Another good guideline to go by is the 80/20 rule. Is what I'm doing going to benefit the 80%? If so, do it. If not, either don't worry about it, and let the 20% slide... or if you really care about it for some reason, do it, but don't let it impact what you're doing for the 80%. It's one of the reasons Apple products are so intuitive and don't come with instruction manuals. 80/20 is literally written into their UX guidelines. If 80% of the people want a button or feature, add it. If not, either drop the feature entirely, or if you really care about it, stuff it in some hidden menu that you have to find to see, and don't let it impact the work you're doing for the 80%.