I have had this for the last couple of weeks, but definitely only witnessed this after Victor was added to the game at some point.
I watched my VRAM usage on my second monitor using Btop++ and it was creeping up over time. When the fps dropped I was at full VRAM usage. At the start of the game...