Do you know what color Harry's eyes really are and why they are important?
Do you know all the Weasley kids?
Do you know who Peeves is?
Do you really know and understand why Snape mistreated Neville so much? Did you even notice that Snape mistreated Neville more than others?
Do you know what an animagus is?
Do you know and understand Voldemort's history?
Do you know what happens to Peter Pettigrew in the end and why?
Do you know who are all the Marauders?
Do you know who was the Half-Blood Prince and why he was called so?
Do you know who Squibs are?
Do you know why Rita Skeeter's character is important and where it was important?
Do you know and understand Snape's history?
Do you understand Dumbledore's character, his behavior towards Harry right from the 1st book, and what was his overall importance to the series?
Do you know the history behind Lupin and Tonks? How they ended up getting married? Do you know who Teddy is?
Do you know who Phineus Nigellus is and why he was important?
Do you know who Barty Crouch Jr. is and what his backstory is? Do you know who Winky is?
Do you know what happens to Harry's wand in the end?
Do you know why the following scene is the complete opposite of the entire series storyline?
It's quite simple. Do you care to know about the above? I have not even skimmed half of the things you have missed out because of not reading the books.
If you have seen the movies numerous times, I would assume you liked them.
If you really did like them then I would assume you were fascinated by the story and the entire wizarding world as well.
If you were fascinated by the story and that world then I would assume you have questions regarding a few things like some of the questions above.
This is the point that makes it "simple" for you.
Do not judge a book by its movie!
If you really have questions, then yes, you should read the books. You won't appreciate the story otherwise and you surely wouldn't think the movies are good enough. It's a rule of thumb for almost any scenario - The original books are better than the movie adaptations. Those questions are answered in the books. You would understand and appreciate the characters much much better than the movies.
If you don't have any questions, then you don't really care about the story, the fantasy world, the magic behind it all. You don't have to bother reading the books. But remember the following -
The above represents how much you will lose out on by not reading the books.So it's quite simple. Read them.