The FreeBSD Diary
Providing practical examples since 1998If you buy from Amazon USA, please support us by using this link.
Creating an FTP Mirror site 14 February 1999
|This article tells you how to set up an FTP mirror site. It will allow you to keep the files on one system up to date with the files on another system. It's very easy to do and pretty quick and painless.|
My failed attempt
|You are welcome to try this method. If you get it to work or you know how to
make it work, please add your comments.
At the moment, the successful attempt is what you
should try if you want it to work.
My first attempt involved the mirror port. Please don't confuse this with the cvsup-mirror port. The latter is a "kit for easily setting up a FreeBSD mirror site using CVSup" and which is what I used to create my article on Setting up a FreeBSD mirror site. The mirror port is a separate tool which can be used for any type of mirroring. It sounds like a great tool. I just couldn't get it to work.
I did the normal make install. And some configuration. But when it came time to the running, I couldn't enter PASV (passive) mode for my connection. I'm behind a firewall, so I needed PASV mode. Here is the output:
[root@ns:/usr/local/mirror] # mirror -n packages/ftp.au.freebsd.org su: mirror: command not found [root@ns:/usr/local/mirror] # ./mirror -n packages/ftp.au.freebsd.org lchat.pl: using hardwired in network constantants at lchat.pl line 90package=fb-rel ftp.au.freebsd.org:/pub/FreeBSD/releases -> /var/ftp/FreeBSD/releases ftp timeout set to 300 Scanning local directory /var/ftp/FreeBSD/releases /var/ftp/FreeBSD/releases no such directory - creating it Connecting to ftp.au.freebsd.org 220- 220- Welcome to Mirror.AARNet.EDU.AU in Australia 220- 220 Mirror.AARNet FTP server ready. login as anonymous ---> USER anonymous 331 Guest login ok, send your complete e-mail address as password. ---> PASS <somestring> 230- 230- Welcome to Mirror.AARNet.EDU.AU in sunny Brisbane, located at the 230- Prentice Centre, University of Queensland, Australia. 230- 230- Providing fast FTP/WWW access and bandwidth cost savings to AARNet 230- members using Sun hardware, Optus bandwidth and archive management 230- by the DSTC core mirror team of Paul Young and Jason Andrade. 230- 230- Questions/Suggestions/Feedback: <email@example.com> 230- 230- *** DISCLAIMER: The AVCC/DSTC/UQ are not liable for any use *** 230- *** storage or transmission of any files stored here. *** 230- *** Un-authorised use of this server is strictly prohibited *** 230- 230- 97 out of 200 in this class, from: 210-55-210-87.ipnets.xtra.co.nz 230- 230- *** For a list of hosted mirrors, please get: /pub/MIRRORS *** 230- *** For a search engine look at http://mirror.aarnet.edu.au *** 230- 230 Guest login ok, access restrictions apply. ---> REST 0 350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer. Can do restarts ---> TYPE I 200 Type set to I. Scanning remote directory /pub/FreeBSD/releases ---> CWD /pub/FreeBSD/releases 250-Please read the file README.TXT 250- it was last modified on Sun Jan 17 12:42:00 1999 - 28 days ago 250 CWD command successful. ---> TYPE A 200 Type set to A. ---> PASV 227 Entering Passive Mode (203,22,86,145,147,142) Cannot open pasv ftp to ftp.au.freebsd.org ---> LIST -lRat 425 Can't open data connection. Cannot get remote directory listing because: 425 Can't open data connection. Cannot get remote directory details (/pub/FreeBSD/releases) disconnecting from ftp.au.freebsd.org ---> QUIT 221 Goodbye.
|My successful attempt to set up an FTP involved using the rsync port.
This should be familiar to you now. Since I have all the
ports installed, all I needed to do was:
[root@ns:~] # cd /usr/ports/net/rsync [root@ns:/usr/ports/net/rsync] # make [root@ns:/usr/ports/net/rsync] # make install
Configuring the FTP server
|My first attempt at mirroring using rsync failed. I ran out of disk
space. When people connect to an FTP server, the FTP daemon (ftpd) is given
a root directory. This is where ftpd will find all of its files. On
my machine, this was set to /var/ftp, which doesn't have much space. I
needed to change to another directory, so I ran vipw and changed the directory.
Here's what it looks like now:
ftp:*:14:5::0:0:Anonymous FTP Admin:/pub/ftp:/nonexistent
The bit I changed was /pub/ftp. Remember to create the subdirectory. Then I moved my files from the old location to the new location.
I then created the subdirectories (/pub/ftp/pub/FreeBSD/releases/) into which I was going to receive the mirror files.
Then I was to run rsync. Here are some of the commands I use to maintain my mirror:
rsync -av --stats mirror.aarnet.edu.au::freebsd/releases/3.0-RELEASE /pub/ftp/pub/FreeBSD/releases/ rsync -av --stats mirror.aarnet.edu.au::freebsd/releases/2.2.8-RELEASE /pub/ftp/pub/FreeBSD/releases/ rsync -av --stats mirror.aarnet.edu.au::freebsd/releases/XFree86 /pub/ftp/pub/FreeBSD/releases/ rsync -av --stats mirror.aarnet.edu.au::freebsd/tools /pub/ftp/pub/FreeBSD/tools rsync -av --stats mirror.aarnet.edu.au::freebsd/CERT /pub/ftp/pub/FreeBSD/
Here is the /etc/crontab entry I use to run the above every day (those are tabs, not spaces):
0 3 * * * root /usr/local/etc/rsync/update.sh
And here is the rsync shell script which the above will run. I stole this from my cvsup-mirror.