How much is lost by not reading the books?

This question and its kin have proven repeatedly difficult to answer. In short: you are missing an immense wealth of storytelling.

You can watch the show and not read the books, that's totally fine, but you will not be able to contemplate the nuance and ambiguity surrounding the issues on screen.

Remember that the show must condense the text into 10 hours per season. The tomes are monolithic, a lot of stuff is inevitably shortened or radically altered if not completely removed.

Is it worth it to start reading them again even if I know most plot details

I began as a show only watcher about 2 years ago. I barely knew anything about anything and I desperately wanted to get involved with discussion so I could understand what everyone else was talking about. So in my naivety I went to awoiaf.westeros.org and gorged on the major plot developments.

Almost in retaliation to my ignorance, spoiling myself actually hurt my understanding further. I knew what had happened but not the how or why, or even who most of the characters involved were.

Then I had a sudden thought, a eureka moment, "I should read ADWD! Then all this nonsense will make sense!" Needless to say I was grievously wrong: I bought the books from amazon and began reading ADWD.

To my surprise byy the time I had finished I came out more confused than when I began: who was Jeyne Westerling? Alys Karstark? Bowen Marsh? Who the hell is that? What's his past relationship like with Jon? Why is it that way? What the hell is a Wun Wun or a Moon Boy or Patchface? Don't even get me started on the Skahaz zo Kondazoo zo Zarkoo zo Whatever nonsense. Needless to say It was quite a traumatising period of my life.

Creating a cacophony of resigned sighs I acknowledged my defeat and opened AGOT pg 1 and started where I should've in the first place.

Tl;dr read the books.