I thought using the iSync permission method was a bit broad so I looked for the specific DL permission. There is a file:
/Users/"your account"/Library/Application\ Support/SyncServices/Local/clientdata/"obscure directory"/clientname.txt
which can tell you which "obscure directory" corresponds to DL. I removed that directory, logged out/in, and then was able to remove the iCal library without any complaints. Checking out more items in DL did not recreate the iCal library. To be honest, I did not check this without first logging out/in so that step may be optional
I did not see the DL preference question panel again, so this may have been irreversible.