The short answer is that option-enter creates a new line in a text field. This is Apple's standard behavior, since enter alone commits changes to the field.
When sorting by genre, it helps to consider that, as far as Mac OS X and, by extension, Delicious Library, is concerned, multiline text fields are just one line, the "newline" being just another character. As such, the entire field is taken into consideration when alphabetizing.
Assuming you manage to keep your genres identical, which you can do by selecting multiple items and editing them at once, sorting by genre will then sort by creator, then title, and so forth. If you have a movie by Alfred Hitchcock and a movie by M. Night Shyamalan, and they both have a genre of:
Sorting by genre will put Hitchcock first. However, the moment you update Hitchcock to:
Mystery
Suspense
Classics
Shyamalan will come first, the rules of alphabetization being what they are. With this in mind, you can bend the genres to your will. For example, by leading the genre with the director's name:
Hitchcock
Mystery
Suspense
Classics
Shyamalan
Mystery
Suspense