Oh, how I empathise with you! I had (and to some extent still have) this problem. Watching French movies in particular is pretty taxing. When talking to people in everyday situations, however, things work much more smoothly, even when listening in on conversations two French speaker have among themselves.
The only remedy is to immerse yourself in the language as much as possible. Focus on your personal progress and not on your absolute level of knowledge and understanding. What I did a lot was watch movies and series on, e.g., Netflix, and choosing French as the audio language with French subtitles.
And first and foremost: don't give up. This is supposed to be fun!
