Welcome to the MacNN Forums.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

You are here: MacNN Forums > Software - Troubleshooting and Discussion > Mac OS X > scripting: Mirroring multiple urls

scripting: Mirroring multiple urls
Thread Tools
Dedicated MacNNer
Join Date: Mar 2002
Status: Offline
Reply With Quote
Nov 6, 2005, 07:00 AM
 
Hi,
I'm trying to mirror a website that looks something like:
http://www.domain.com/design/x/
The site has 2300 directories from 1 to 2300 that x designates above eg:http://www.domain.com/design/1/ , http://www.domain.com/design/2/ etc.... The site does NOT have any page that link all these directories so wget -m won't work as there are no links to these folders.What I'm thinking is I need a shell script that does a while loop something like:
x=1
while x < 2300
wget -m http://www.domain.com/design/x/

I'm quite a novice at shell scripting , is the above solution achievable ? Thanks in advance.
http://winlab.csbnet.se Visit the Mac Demo Scene.
     
DevNine  (op)
Dedicated MacNNer
Join Date: Mar 2002
Status: Offline
Reply With Quote
Nov 6, 2005, 11:37 AM
 
I'm getting some progress now:
#!/bin/bash
x=1
echo $x
while [ $x -ne 2301 ]
do
echo $x
x=$((x+1))
done

This loops 2300 times, now to test it with wget directory name variable.I only want to get .zip files but I can't seem to get the wget syntax correct

wget -A .zip http://www.domain.com/dir/ doesn't work
(Last edited by DevNine; Nov 6, 2005 at 12:14 PM. )
http://winlab.csbnet.se Visit the Mac Demo Scene.
     
DevNine  (op)
Dedicated MacNNer
Join Date: Mar 2002
Status: Offline
Reply With Quote
Nov 6, 2005, 04:03 PM
 
Got it working:
#!/bin/bash
x=1
while [ $x -ne 2301 ]
do
echo $x
wget -r --no-parent http://www.domain.com/design/$x/
x=$((x+1))
done
http://winlab.csbnet.se Visit the Mac Demo Scene.
     
DevNine  (op)
Dedicated MacNNer
Join Date: Mar 2002
Status: Offline
Reply With Quote
Nov 6, 2005, 04:04 PM
 
Got it working:
#!/bin/bash
x=1
while [ $x -ne 2301 ]
do
echo $x
wget -r --no-parent http://www.domain.com/design/$x/
x=$((x+1))
done
http://winlab.csbnet.se Visit the Mac Demo Scene.
     
DevNine  (op)
Dedicated MacNNer
Join Date: Mar 2002
Status: Offline
Reply With Quote
Nov 6, 2005, 04:05 PM
 
Got it working:
#!/bin/bash
x=1
while [ $x -ne 2301 ]
do
echo $x
wget -r --no-parent http://www.domain.com/design/$x/
x=$((x+1))
done
http://winlab.csbnet.se Visit the Mac Demo Scene.
     
DevNine  (op)
Dedicated MacNNer
Join Date: Mar 2002
Status: Offline
Reply With Quote
Nov 6, 2005, 04:05 PM
 
Got it working:
#!/bin/bash
x=1
while [ $x -ne 2301 ]
do
echo $x
wget -r --no-parent http://www.domain.com/design/$x/
x=$((x+1))
done
http://winlab.csbnet.se Visit the Mac Demo Scene.
     
   
Thread Tools
Forum Links
Forum Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Top
Privacy Policy
All times are GMT -5. The time now is 01:21 PM.
All contents of these forums © 1995-2011 MacNN. All rights reserved.
Branding + Design: www.gesamtbild.com
vBulletin v.3.8.7 © 2000-2011, Jelsoft Enterprises Ltd., Content Relevant URLs by vBSEO 3.3.2