I think I know what was wrong. The screen was sleeping before the screensaver started. Or maybe it went to sleep before the screensaver started?
I forgot which of those 2 it was, but anyways I changed it and I haven't had this problem again. Although to be fair I'm not the only person using the computer, so it might take a few more days to reveal if the problem is still there.