The FreeBSD Diary

Jails are a useful tool for achieving a certain level of virtualization. I have used jails in the past for creating sandboxes for testing of the Bacula project. Today, I will be setting up 7 jails, each for regression tests on different configurations. In the process, I'll show you some shortcuts that Ryan Lortie (desrt) told me about. It saves time. I'll also share some scripts that will help with the setup of various jail items.

This article should be read in conjunction with my other jail articles. It won't stand alone unless you are already familiar with jails and how to configure them.

According to man jail, these are the basic steps for building a jail:

D=/here/is/the/jail
cd /usr/src
mkdir -p $D
make world DESTDIR=$D
make distribution DESTDIR=$D
mount -t devfs devfs $D/dev

desrt pointed out, and I think I've heard this before, after the first build, you don't have to do a full make world. For the subsequent jails, you can get away with this:

D=/here/is/the/OTHER/jail
cd /usr/src
mkdir -p $D
make installworld DESTDIR=$D
make distribution DESTDIR=$D
mount -t devfs devfs $D/dev

The key differences are:

1. Change the value of D to point to the new jail.
2. Do a make installworld, not a make world.

"make world" consists of two main steps: make buildworld && make installworld. There is no advantage in doing the buildworld part again. That has already be done.

During the building of my first jail, I tried this:

export D=mysql41.example.org

This fail, because the install could not find the directory mysql41.example.org. I should have included the full path to the directory: /usr/jail/mysql41.example.org. To complete the installation of the jail, I issued this series of commands:

export D=/usr/jail/mysql41.example.org
make installworld DESTDIR=$D
make distribution DESTDIR=$D
mount -t devfs devfs $D/dev

After the above completed, I had the following mount points:

# mount
/dev/ad0s1a on / (ufs, local)
/devfs on /dev (devfs, local)
/dev/ad0s1e on /tmp (ufs, local, soft-updates)
/dev/ad0s1f on /usr (ufs, local, soft-updates)
/dev/ad0s1d on /var (ufs, local, soft-updates)
/devfs on /usr/jail/mysql41.example.org/dev (devfs, local)

There are a few things that I like to configure within the jail before starting it. The files in the jail environment are accessible from the host environment. So it is a simple matter of using your favourite editor to make the changes. Here are the files I altered in /usr/jail/mysql41.example.org:

• etc/rc.conf - added: sshd_enable="YES"
• etc/resolv.conf - added entries to get DNS working
• etc/syslog.conf - changed the "*.err;kern.warning;auth.notice;mail.crit" line to echo to /var/log/messages instead of the console

It would be easy to script that. For example, here is the script I used to do the installworld for each jail:

#!/bin/sh
JAILDIR="/usr/jail"
JAILS=`ls ${JAILDIR}`
for jail in ${JAILS}
do
   echo $jail
   D=${JAILDIR}/${jail}
   make installworld DESTDIR=$D
   make distribution DESTDIR=$D
   mount -t devfs devfs $D/dev
done

JAILS="mysql51.example.org pg73.example.org pg74.example.org pg80.example.org"

Similar scripts can be created for populating the ports tree. You may think that a single shared ports tree might be sufficient for multiple jails. I've decided to go with a distinct tree for each jail. I'll be building different ports concurrently in each jail. I don't want any interference caused by another jail.

Setting up multiple jails involves adding several entries to /etc/rc.conf, some of which specify defaults for all jails. The default settings in my server are:

# Defaults for all jails:
jail_interface="fxp0"    # Interface to create the IP alias on
jail_devfs_enable="YES"  # mount devfs in the jail
jail_procfs_enable="YES" # mount procfs in jail

I also specify a list of the jails on this machine:

# list of jails on this machine
jail_list="mysql41 mysql50 mysql51 pg73 pg74 pg80 pg81 pg82"

For each of the above mentioned jails, entries similar to this appear:

# values for each jail listed above
jail_mysql41_rootdir="/usr/home/jails/mysql41.example.org"  # Jail's root directory
jail_mysql41_hostname="mysql41.example.org"  # Jail's hostname
jail_mysql41_ip="192.168.0.100"              # Jail's IP number

It can be tedious to setup thse values. So a script can be useful and it could be based upon the script that appears above.

Running PostgreSQL in a jail is interesting. There are shared memory issues which can be resolved by setting security.jail.sysvipc_allowed via sysctl. I tried that. I failed. The problem I encountered during initdb is:

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating directory /usr/local/pgsql/data/global ... ok
creating directory /usr/local/pgsql/data/pg_xlog ... ok
creating directory /usr/local/pgsql/data/pg_xlog/archive_status ... ok
creating directory /usr/local/pgsql/data/pg_clog ... ok
creating directory /usr/local/pgsql/data/pg_subtrans ... ok
creating directory /usr/local/pgsql/data/pg_twophase ... ok
creating directory /usr/local/pgsql/data/pg_multixact/members ... ok
creating directory /usr/local/pgsql/data/pg_multixact/offsets ... ok
creating directory /usr/local/pgsql/data/base ... ok
creating directory /usr/local/pgsql/data/base/1 ... ok
creating directory /usr/local/pgsql/data/pg_tblspc ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 50
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... FATAL:  could 
not create shared memory segment: Function not implemented
DETAIL:  Failed system call was shmget(key=1, size=1327104, 03600).
child process exited with exit code 1
initdb: removing data directory "/usr/local/pgsql/data"

I initially thought I had to set this flag on each of the jails. I was wrong. After a bit of trial and error, I discovered the problem. I had added this value to /etc/sysctl.conf:

# For PostgreSQL jails
security.jail.sysvipc_allowed=1

After a reboot, the above entry ensures this setting after a reboot:

# sysctl security.jail.sysvipc_allowed
security.jail.sysvipc_allowed: 1

When starting a jail, I noticed this:

# /etc/rc.d/jail start pg82
Configuring jails: sysvipc_allow=NO.
Starting jails: pg82.unixathome.org.

# sysctl security.jail.sysvipc_allowed
security.jail.sysvipc_allowed: 0

Not only was the jail not getting the right value, the value in the host system was being reset.

Looking at /etc/rc.d/jail for "Configuring jails", I found more clues leading me to /etc/defaults/rc.conf, where I found this setting:

jail_sysvipc_allow="NO" # Allow SystemV IPC use from within a jail

Ahh, OK, so now I needed to set this in /etc/rc.conf

jail_sysvipc_allow="YES" # For PostgreSQL

After restarting the pg82 jail, I found the security.jail.sysvipc_allowed was correctly set, in both the jail and the host. I tried the initdb again:

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating directory /usr/local/pgsql/data/global ... ok
creating directory /usr/local/pgsql/data/pg_xlog ... ok
creating directory /usr/local/pgsql/data/pg_xlog/archive_status ... ok
creating directory /usr/local/pgsql/data/pg_clog ... ok
creating directory /usr/local/pgsql/data/pg_subtrans ... ok
creating directory /usr/local/pgsql/data/base ... ok
creating directory /usr/local/pgsql/data/base/1 ... ok
creating directory /usr/local/pgsql/data/pg_tblspc ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 50
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... FATAL:  could 
not create semaphores: No space left on device
DETAIL:  Failed system call was semget(1, 17, 03600).
HINT:  This error does *not* mean that you have run out of disk space.
       It occurs when either the system limit for the maximum number of 
semaphore sets (SEMMNI), or the system wide maximum number of semaphores
(SEMMNS), would be exceeded.  You need to raise the respective kernel parameter.
Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its 
max_connections parameter (currently 10).
The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.
child process exited with exit code 1
initdb: removing data directory "/usr/local/pgsql/data"

kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers/max_fsm_pages ... 24MB/153600
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

Success. You can now start the database server using:

    postgres -D /usr/local/pgsql/data
or
    pg_ctl -D /usr/local/pgsql/data -l logfile start

I was then able to start PostgrSQL, create users, etc. I repeated the process for two other jails, then ran into this problem for the third jail:

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating directory /usr/local/pgsql/data/global ... ok
creating directory /usr/local/pgsql/data/pg_xlog ... ok
creating directory /usr/local/pgsql/data/pg_xlog/archive_status ... ok
creating directory /usr/local/pgsql/data/pg_clog ... ok
creating directory /usr/local/pgsql/data/pg_subtrans ... ok
creating directory /usr/local/pgsql/data/base ... ok
creating directory /usr/local/pgsql/data/base/1 ... ok
creating directory /usr/local/pgsql/data/pg_tblspc ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 50
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... FATAL:  could
not create shared memory segment: Cannot allocate memory
DETAIL:  Failed system call was shmget(key=1, size=1122304, 03600).
HINT:  This error usually means that PostgreSQL's request for a shared memory
segment exceeded available memory or swap space. To reduce the request size 
(currently 1122304 bytes), reduce PostgreSQL's shared_buffers parameter 
(currently 50) and/or its max_connections parameter (currently 10).
The PostgreSQL documentation contains more information about shared memory
configuration.
child process exited with exit code 1
initdb: removing data directory "/usr/local/pgsql/data"

I decided to double the settings in /etc/sysctl.conf on the host system:

kern.ipc.semmni=512
kern.ipc.semmns=1024
kern.ipc.semmnu=512

But that did not solve the problem. Stopping PostgreSQL on the other two jails did. This allowed the initdb to complete. But I was still unable to get all the PostgreSQL servers running in each jail.

After a few more trial and error attempts, I discovered that the settings I had in /boot/loader.conf contained syntax errors. Instead of SETTING=VALUE, I had SETTING: VALUE. Fixing that, and rebooting, allowed me to get all five instances of PostgreSQL running concurrently. But there was a bit more to it than just that.

I tried various things. I tried running each postmaster on a different port (didn't help). I also tried running each postmaster as a different GID (that did help). That is the key. I will explain.

PostgreSQL makes use of shared memory. When running multiple instances of PostgreSQL the shared memory for one instance can be stomped on by another instance. That's not nice. The key to avoiding this is using a different UID for each instance. You can see that here:

$ grep -h pgsql /usr/jail/*.unixathome.org/etc/passwd
pgsql:*:1073:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1074:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1080:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1081:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1082:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh

I used a UID that would relate to the version of PostgreSQL that was running. For example, UID=1073 is PostgreSQL version 7.3. There is no need to follow this convention. I did it merely because I could.

When I changed the UID in each jail, I first made sure PostgreSQL was not running in that jail. Then I ran vipw and altered the UID and exited. Then I did a chown to ensure the file attributes were owned by the new UID:

chown -R pgsql /usr/local/pgsql

$ psql -l
psql: FATAL: semctl(262144, 4, SETVAL, 0) failed: Invalid argument

The above type of problem would arise only when I ran multiple PostgreSQL servers with the same UID.

These are the various sysctl settings I'm using on this server:

[dan@polo:~] $ cat /boot/loader.conf
kern.ipc.semaem=32767
kern.ipc.semvmx=65534
kern.ipc.semusz=184
kern.ipc.semume=80
kern.ipc.semopm=200
kern.ipc.semmsl=120
kern.ipc.semmnu=4096
kern.ipc.semmns=8192
kern.ipc.semmni=32767
kern.ipc.semmap=60

[dan@polo:~] $ cat /etc/sysctl.conf
# For PostgreSQL jails
security.jail.sysvipc_allowed=1

# for more shared memory for jails/PostgreSQL
kern.ipc.shmall=65536
kern.ipc.shmmax=134217728
kern.ipc.semmap=4096
[dan@polo:~] $

I am confident that I am over allocating resources here. I kept increasing the values and trying them, without success. Then I discovered my syntax error. I didn't want to reverse-engineer my changes so I have left them as they appear above. You may wish to use lesser values.

You may be interested in the host system dmesg output. As you can see, this isn't a very high spec system. It has only 512M of RAM.

/var/run/dmesg.boot

When running top, I noticed something I had not see before. Number in the username field.

last pid: 82356;  load averages:  2.47,  1.30,  1.01   up 1+21:46:07  10:25:39
285 processes: 6 running, 277 sleeping, 1 stopped, 1 lock
CPU states: 29.2% user,  5.8% nice, 33.1% system,  1.3% interrupt, 30.7% idle
Mem: 173M Active, 100M Inact, 118M Wired, 14M Cache, 60M Buf, 81M Free
Swap: 989M Total, 282M Used, 708M Free, 28% Inuse

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
  81261   1080      1 105    0 10432K  4728K *Giant 1   0:28 16.85% postgres
  82147   1074      1   4    0 10392K  3136K sbwait 0   0:07 16.58% postgres
  81830   1073      1   4    0 11016K  4548K sbwait 0   0:16 15.87% postgres
  81322   1081      1 103    0 11664K  4244K CPU0   1   0:23 13.57% postgres
  81826 dan         6 107   10 11832K  3744K RUN    0   0:04  3.13% bacula-dir
  82143 dan         6 108   10 11856K  3852K RUN    0   0:02  3.03% bacula-dir
  81252 dan        12 106   10 15700K  4296K ucond  0   0:07  1.51% bacula-dir
  81257 dan        12 106   10 15700K  4304K ucond  1   0:07  1.12% bacula-dir

With a bit of thought, you can figure this out. top was running in the host environment. The UID for pgsql has been changed in the jail. The host environment has no knowledge of UIDs from the jails. When top cannot convert a UID to a username, it displays the UID. This is fine. An added benefit is seeing exactly which jail is still running. In this case, it's PostgreSQL 8.0, 7.4, 7.3, and 8.1.

Jails are just right what for what I'm trying here. I certainly do not want 8 servers sitting around the house, just for regression testing of Bacula. I'll be writing up some scripts to automate much of this testing and to upload the results to a website.

Jails: sometimes the right tool for the job.