Rebuilding the desktop file on my system takes almost an hour.
The Desktop file is not the "Desktop" itself... The number of items on the desktop is not the determining factor for how long it will take.
Basically the more applications on the drive the long it will take, I've found...