The FreeBSD Diary

The FreeBSD Diary (TM)

Providing practical examples since 1998

If you buy from Amazon USA, please support us by using this link.
[ HOME | TOPICS | INDEX | WEB RESOURCES | BOOKS | CONTRIBUTE | SEARCH | FEEDBACK | FAQ | FORUMS ]
Creating your own cvsupd server 17 December 2000
Need more help on this topic? Click here
This article has 1 comment
Show me similar articles
With the upcoming work I'm going to be doing on FreshPorts, I thought it best to start up a cvsup server for the source code.  This server will be used to share the data amongst the other FreshPorts developers.  This article starts that process.

I first wrote about cvsupd on 6 August 1999.  That article is still around, but now that I'm installing a cvsupd server for my own needs, instead of creating a mirror, I found my existing notes a bit lacking.  Hence, this article.

I used John Polstra's CVSUP FAQ for this exercise.  Of note is the section on setting up a test server.   It's what I used as the basis for this article.

The install
Well, seeing as I had all the port skeletons installed, I just had to do the following:
cd /usr/ports/net/cvsupd-bin
make install

NOTE: in more recent versions of FreeBSD, cvsupd can be found in net/cvsup-without-gui. Yet cvsupd is started by /usr/local/etc/rc.d/cvsupd which is installed by net/cvsup-mirror cvsupd_enable="YES" in /etc/rc.conf cvsupd_flags="-c sup:/home/repositories/sup" Turn off your old cvsupd to avoid accidents.

Setting up a test server
In this section, I'm going to show you the bare minimum required to get the server running so you can talk to it from a client.

This is the main directory which cvsupd uses and its where the repositories will reside.  In this example, we'll use /usr/local/etc/cvsup (which is the default base directory for cvsupd).  See the -b option.

(i.e. mkdir /usr/local/etc/cvsupd/sup).  See the -c option.

  • Create the test collection directory:

This directory will contain the test collection
(i.e. mkdir /usr/local/etc/cvsupd/sup/test)

The releases file identifies the releases which are associated with this collection.
Create /usr/local/etc/cvsupd/sup/test/releases to contain the following:

cvs list=list.cvs prefix=/home/repositories

This defines the prefix.  Our repository will exist at /home/repositories

  • Create the list.cvs file:

This file contains the rules for use when processing the cvs release.  In this case, it contains the freshports-db collection.  Here's what's in that file:

upgrade freshports-db This file should be in the directory /usr/local/etc/cvsupd/sup/test/.

This is your repository and it is relative to the prefix supplied in the releases file.  In our example, the repository will exist at /home/repositories/freshports-db.   It is the contents of this subtree which will be transferred.

Start the server
Here's where we start the server with a very simple situation:
/usr/local/sbin/cvsupd

We have not specified a base directory.  cvsupd will use the default directory, which we also used to create our base directory.  We have not specified the -e option, which means cvsupd will run from the command line and not go into the background.  Here is what it looks like when cvsupd starts:

# /usr/local/sbin/cvsupd
2000.12.17 16:54:40 NZDT [75872]: CVSup server started
2000.12.17 16:54:40 NZDT [75872]: Software version: REL_16_1
2000.12.17 16:54:40 NZDT [75872]: Protocol version: 16.1
2000.12.17 16:54:40 NZDT [75872]: Ready to service requests

This method is fine for testing, but I actually run cvsupd as the nobody and ensure that everything it serves is readable by everyone.  cvsupd does not create or write files, so from a security point of view, the risk of cvsupd being tricked into damaging your system is actually quite low.

I start cvsupd automatically at system startup by creating /usr/local/etc/rc.d/cvsupd.sh, which is chmod 770:

#!/bin/sh
[ -x /usr/local/sbin/cvsupd ] && \
su -m nobody -c "/usr/local/sbin/cvsupd -e -C 8 -l @daemon" && \
echo -n ' cvsupd'
Setting up a test client
We will now set up the test client to use our test server.  We are doing this on the same box as cvsupd is running. Create the directory ~/cvs-test.  In this directory, create supfile and place the following in it:
*default host=localhost
*default base=.
*default release=cvs
*default delete use-rel-suffix
test

This is our cvsup configuration file.  We have specified that we will be connecting to the cvsup server on localhost (i.e. this box) and that we will be obtaining the cvs release.  From that release, we want the test collection.

To run cvsup on the client and pull down the collection, issue the following command:

cvsup supfile

Here is a successful connection and cvsup:

$ cvsup supfile
Connected to localhost
Updating collection test/cvs
 Mkdir freshports-db
 Create freshports-db/freshports.db.sql,v
 Create freshports-db/permissions.txt,v
 SetAttrs freshports-db
Finished successfully

In this case, it's brought down the files associated with my repository.

In this example, I'm going to add a very simple collection, test2, which contains only contrib/top.   This isn't really a practical example, but it might help to illustrate how the various components of cvsup fit together.

I'll create the collection directory:

mkdir /usr/local/etc/cvsup/sup/test2

In that directory, I'll specify the releases associated with this collection by creating releases which contains the following (all on one line):

cvs list=list.cvs prefix=/home/repositories/FreeBSD/ncvs/src/contrib/

This indicates that the repository actually exists at /home/repositories/FreeBSD/ncvs/src/contrib/.

Then I create the list.cvs file, the same directory as releases, to contain this:

upgrade test2

I can use the same supfile as in the previous example, but I change test to top.

Now when I cvsup, I get this:

$ cvsup supfile
Connected to localhost
Updating collection test2/cvs
Checkout top/ADVERTISEMENT
Checkout top/Changes
Checkout top/Configure
Checkout top/DISCLAIMER
Checkout top/FAQ
Checkout top/INSTALL
Checkout top/Make.desc.X
Checkout top/Makefile.X
Checkout top/Porting
Checkout top/README
Checkout top/boolean.h
Checkout top/commands.c
Checkout top/display.c
Checkout top/display.h
Checkout top/getans
Checkout top/getopt.c
Checkout top/install
Checkout top/layout.h
Checkout top/loadavg.h
Checkout top/m-template
Checkout top/machine.h
Checkout top/metatop
Checkout top/os.h
Checkout top/patchlevel.h
Checkout top/prime.c
Checkout top/screen.c
Checkout top/screen.h
Checkout top/sigconv.awk
Checkout top/top.X
Checkout top/top.c
Checkout top/top.h
Checkout top/top.local.H
Checkout top/username.c
Checkout top/utils.c
Checkout top/utils.h
Checkout top/version.c
Finished successfully

Compare the above list of files you can find in the CVS repository for contrib/top.

And if I check my directory, I see this:

$ ls
sup supfile top

And if I look in top, I see the same files as are listed above.

Adding another bit to test2
Now lets try another example.  Let's add contrib/awk to the test2 collection.

I'll assumed you have already created the example from the previous section.  To add awk to the collection, modify /usr/local/etc/cvsup/sup/test2/list.cvs to contain this:

upgrade top
upgrade awk

Now run cvsup again.  You should see this:

$ cvsup supfile
Connected to localhost
Updating collection test2/cvs
Checkout awk/ACKNOWLEDGMENT
Checkout awk/COPYING
Checkout awk/ChangeLog

[snip]

Checkout awk/test/subslash.awk
Checkout awk/test/subslash.ok
Checkout awk/test/zeroflag.awk
Checkout awk/test/zeroflag.ok
Checkout awk/version.c
Finished successfully

and your main directory should contain this:

$ ls
awk sup supfile top

Putting things in a different place
The supfile example contains a base specifiation of base=. which puts everything from cvsup into the current directory.  That's not the ideal situation, but it's what we used for our testing.  Now let's change that.  Let's put everything into /home/dan/mysupfiles.  The first step is to modify our supfile from the previous examples and change base to look like this:

*default base=/home/dan/mysupfiles

Don't forget to create the directory before you run cvsup:

mkdir /home/dan/mysupfiles

Then we cvsup again:

$ cvsup supfile
Connected to localhost
Updating collection test2/cvs
Checkout awk/ACKNOWLEDGMENT
Checkout awk/COPYING
Checkout awk/ChangeLog
[snip]
Checkout top/top.h
Checkout top/top.local.H
Checkout top/username.c
Checkout top/utils.c
Checkout top/utils.h
Checkout top/version.c
Finished successfully

Now if I check my directory, I'll find this:

$ ls /home/dan/mysupfiles/
awk sup top

NOTE: If you get this error:

$ cvsup supfile
Nonexistent base directory "~/mysupfiles" for collection "test2"

Then you didn't create ~/mysupfiles before running cvsup.

This section will concentrate on restricting access to your cvsup server by authenticating the connections.  Authentication is controlled by the existence of the file cvsupd.access which must appear in the base directory (in the example above, the base directory is /usr/local/etc/cvsup).  If this file does not exist, no authentication takes place.

The file cvsupd.passwd (again, in the base directory) contains the authentication rules.  Here's the contents I was testing with:

cvsup.example.org:mysecretkey
dan@develop.example.org:$md5$c2e5a85c8042ce9f8c71bcc0f52876c8::

mysecretkey can be any key you choose.  The second line was created using cvpasswd:

$ cvpasswd dan@develop.example.org cvsup.example.org
Enter password:
Enter same password again:

Send this line to the server administrator at cvsup.example.org:
----------------------------------------------------------------
dan@develop.example.org:$md5$c2e5a85c8042ce9f8c71bcc0f52876c8::
----------------------------------------------------------------
Be sure to send it using a secure channel!

Add this line to your file "$HOME/.cvsup/auth", replacing "XXX"
with the password you typed in:
----------------------------------------------------------------
cvsup.example.org:dan@develop.example.org:XXX:
----------------------------------------------------------------
Make sure the file is readable and writable only by you!

The password I entered was "secret".  So this is what I placed in ~/.cvsup/auth on my client machine:

cvsup.example.org:dan@develop.example.org:secret:

That should be enough to authenticate everyone.

Access Control
Perhaps you don't want everyone accessing your cvsup server. I know I don't. So I've added some rules to my packet filter to restrict access. People scanning the box can't even tell there is a cvsup server there. But multiple layers of protection can be a good thing. That's why I added more. If you read the man page for cvsupd, you'll see that that cvsupd.access is your friend. I added a few rules to this file, which normally resides under /usr/local/etc/cvsup, which permitted certain trusted host unlimited access to my server.
#
# allow anything from our local network

+10.0.0.0/24

# allow connections from local host
+127.0.0.1/32

# everything else must authenticate
*0.0.0.0/0
Using the above scenario, I didn't have to do anything special for boxes on my 10.0.0.* network and the cvsup server could access itself. Everyone else, including my remote boxes, must authenticate using the methods described in the previous section.
Secrecy
From man cvsupd:
If secrecy is desired then the connection can be tunneled through ssh.

I tried to get this to work, but I failed.  If anyone has a practical example, please add it to the comments section. Thanks.

#!/bin/sh
ssh -f -L 32884:example.org:5999 dan@example.org sleep 30
cvsup -h localhost -p 32884 supfile
Common problems and solutions
If you encounter this message:
$ cvsup supfile
Connected to localhost
Server message: Collection "test" release "cvs" is not available here
Skipping collection test/cvs
Finished successfully

...it means the information contained in the releases file is incorrect.  In my case, I had put my data at /usr/repositories instead of /home/repositories as specified in releases.  Once I moved repositories to /home, all was well.

If you encounter this message:

$ cvsup supfile
Connected to localhost
Server message: Unknown collection "test"
Skipping collection test/cvs
Finished successfully

...it means the collection specified in your client's supfile does not exist on the server.  Once of them is wrong.  In our example, it means /usr/local/etc/cvsupd/sup/test does not exist.

This message:

$ cvsup supfile
Cannot connect to localhost: Connection refused
Will retry at 16:53:45

...means that cvsupd isn't accepting connections.  Make sure it's running.  Make sure you are allowed to connect.  Check your authentication.

If you get this:

$ cvsup supfile
Connected to cvsup.freshports.org
Server error: Authentication failed

...then you have the wrong authentication information at either the client or the server.  Check both and try again.


Need more help on this topic? Click here
This article has 1 comment
Show me similar articles