I'm not sure if MacNN is the right place to ask this (or if this is the right forum on MacNN if it is), but I'm curious about an issue I'm having with wget. I'm trying to download all of the files for a homework project my professor posted
here, and I'm using this line for wget:
wget
http://science.kennesaw.edu/~jgarrid...otes/mcarwash/ -r -l 1 -nd -R "*.gif","*.html*","*.txt"
The robots.txt file is still being downloaded, despite the exclusion option. Is this normal? It's not a big deal, as I can easily delete it, but it's annoying my obsessive side...
Thanks,
-Scott