Welcome to the MacNN Forums.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

You are here: MacNN Forums > Software - Troubleshooting and Discussion > macOS > How Can I mirror an FTP directory?

How Can I mirror an FTP directory?
Thread Tools
burro
Fresh-Faced Recruit
Join Date: Apr 2011
Status: Offline
Reply With Quote
Apr 17, 2011, 02:17 AM
 
Hi,

I work with data that is available online on a couple of FTP sites.

I'd like to "mirror" those sites so that I always have a current copy on my own computer whenever files are added or updated on those sites.

I am wondering if there is any good FTP client, other software package, or other set-up, that can do that automatically.

Thank you.
     
besson3c
Clinically Insane
Join Date: Mar 2001
Location: yes
Status: Offline
Reply With Quote
Apr 17, 2011, 03:22 AM
 
If you are talking straight FTP you can mirror these directories with wget. If these directories are available via SSH you can do the same with rsync.
     
burro  (op)
Fresh-Faced Recruit
Join Date: Apr 2011
Status: Offline
Reply With Quote
Apr 17, 2011, 03:41 AM
 
Thank you.

Those are new tools to me.

Is there a "how to" you could recommend, or a piece of software that provides and easy-to-use GUI for them?

Thanks, again.
     
besson3c
Clinically Insane
Join Date: Mar 2001
Location: yes
Status: Offline
Reply With Quote
Apr 17, 2011, 03:44 AM
 
I can help you with getting one or both working, I don't know of a GUI.

Do you have SSH available to you on the FTP server?
     
burro  (op)
Fresh-Faced Recruit
Join Date: Apr 2011
Status: Offline
Reply With Quote
Apr 17, 2011, 11:21 PM
 
Thank you.

I think the answer is "no" about SSH.

It is a public FTP site -- no password required.
     
besson3c
Clinically Insane
Join Date: Mar 2001
Location: yes
Status: Offline
Reply With Quote
Apr 17, 2011, 11:36 PM
 
Okay, well you can compile and install wget yourself using Macports if you are comfortable with this (you'll also need to have XCode installed to do this), or else you can download and install this guy's prepackaged version of wget:

Status-Q Blog Archive wget for Mac OS X Leopard

With wget installed, all you have to do is enter the following into your terminal:

wget -r ftp://yourftpsite.com

the -r means "recursive", meaning it will download the contents of any folders it finds. If you have access to Linux many distros will include wget as well.
     
burro  (op)
Fresh-Faced Recruit
Join Date: Apr 2011
Status: Offline
Reply With Quote
Apr 17, 2011, 11:41 PM
 
Thanks,

Do you know of a way to automate, e.g. to run the comparison/sync once per day or once per week?
     
burro  (op)
Fresh-Faced Recruit
Join Date: Apr 2011
Status: Offline
Reply With Quote
Apr 17, 2011, 11:43 PM
 
Oh -- I should clarify that it is a BIG dataset -- I want to be sure only to download and replace local files with new files and files that have changed.
     
turtle777
Clinically Insane
Join Date: Jun 2001
Location: planning a comeback !
Status: Offline
Reply With Quote
Apr 18, 2011, 12:01 AM
 
Originally Posted by besson3c View Post
Okay, well you can compile and install wget yourself
Dude, WTF

The OP doesn't even know what SSH and rsync is, do you really think suggesting that he'd compile his own port is useful ?

-t
     
turtle777
Clinically Insane
Join Date: Jun 2001
Location: planning a comeback !
Status: Offline
Reply With Quote
Apr 18, 2011, 12:03 AM
 
Originally Posted by burro View Post
I am wondering if there is any good FTP client, other software package, or other set-up, that can do that automatically.
I have never used that functionality, but have a look at those, they both support synchronizations (I'm not sure to what extent automated):

FTP Client Software to Upload and Download Files

FTP, SFTP, WebDAV, Cloud Files, Google Docs & Amazon S3 Browser for Mac & Windows.

-t
     
besson3c
Clinically Insane
Join Date: Mar 2001
Location: yes
Status: Offline
Reply With Quote
Apr 18, 2011, 12:04 AM
 
Originally Posted by burro View Post
Thanks,

Do you know of a way to automate, e.g. to run the comparison/sync once per day or once per week?

If you add the -N option (i.e. wget -rN ftp://yoursite.com) the download will be incremental, only downloading what has changed. As far as automating this, you can do so via a cronjob so that this job runs as often as you'd like.
     
besson3c
Clinically Insane
Join Date: Mar 2001
Location: yes
Status: Offline
Reply With Quote
Apr 18, 2011, 12:06 AM
 
Originally Posted by turtle777 View Post
Dude, WTF

The OP doesn't even know what SSH and rsync is, do you really think suggesting that he'd compile his own port is useful ?

-t

Which is why I also provided the link to download the prepackaged wget binary.

If you have a better idea for the OP, share it. It's kind of dumb and annoying that we (collective "we") have to have these incredibly immature little competitions to see who can be the most helpful. This is a brainstorming session, if an idea is not attractive it is not attractive, it's not a big deal. Having options are better than not having any, even if you are only aware of them existing.
     
turtle777
Clinically Insane
Join Date: Jun 2001
Location: planning a comeback !
Status: Offline
Reply With Quote
Apr 18, 2011, 12:15 AM
 
Originally Posted by besson3c View Post
If you have a better idea for the OP, share it.
I did.

-t
     
besson3c
Clinically Insane
Join Date: Mar 2001
Location: yes
Status: Offline
Reply With Quote
Apr 18, 2011, 12:21 AM
 
Originally Posted by turtle777 View Post
I did.

-t

Cool!

I hope that one or both of these suggestions are useful to you, original poster...
     
   
Thread Tools
 
Forum Links
Forum Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Top
Privacy Policy
All times are GMT -4. The time now is 05:22 PM.
All contents of these forums © 1995-2017 MacNN. All rights reserved.
Branding + Design: www.gesamtbild.com
vBulletin v.3.8.8 © 2000-2017, Jelsoft Enterprises Ltd.,