Overview
This article describes how to use lsyncd as a sync’ing mechanism for multiple web servers data directories. This is only one scenario in which this program could be very useful for our customers.
These notes will provide instructions on how to setup lsyncd to sync the web directory on one server to two other web servers. This guide was setup on our default CentOS 5.5 LAMP build servers, but should work with a few modifications on Ubuntu.
Replicate from master to two slaves:
Master —> slave1
—> slave2
Preparation
Lsyncd requires passwordless SSH from the master to slave systems for the root user.
Generate SSH keypair on master:
# ssh-keygen -t rsa
Then copy /root/.ssh/id_rsa.pub on master to /root/.ssh/authorized_keys on the slave servers.
Then ensure the following lines are uncommented in /etc/ssh/sshd_config on the slave servers:
PermitRootLogin yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
If you have to create the /root/.ssh directory on the slave, it should be owned root:root and have permissions 0700.
Restart SSH after changes are made.
Test ssh from the master to the slave as root to make sure it isn’t throwing a message that prevents lsyncd from being able to connect
On Ubuntu, you may see %h/.ssh/authorized_keys. This is fine as well, the %h is just an alias for the home directory of the user logging in, i.e. /root/ for root.
Supporting Package Installation
First you need to ensure the following packages are installed, that are not within the default LAMP build
Dependencies
CentOS and RHEL
# yum -y install lua lua-devel pkgconfig gcc asciidoc
Ubuntu
# apt-get install -y lua5.1 liblua5.1-dev pkg-config rsync asciidoc
Lsync installation
Be sure to manually check the destination directory on the slave server if it already exists. Customers might have NFS or CloudFuse mounts which would end up wiped by the lsync process.
1. Then grab the source tarball from http://code.google.com/p/lsyncd/, and run the standard (on CentOS/RHEL, you can bypass this step and do ‘yum install lsyncd’ to install 2.1.4 from IUS)
tar xzvf lsyncd-2.1.5.tar.gz
cd lsyncd-2.1.5
./configure && make && make install
2. Create the Lsyncd log directory.
mkdir /var/log/lsyncd
3. Add init scripts for starting and stopping the service.
RHEL/CentOS
Create the following file at /etc/init.d/lsyncd: (skip if you did ‘yum install lsyncd’ above)
#!/bin/bash
#
# lsyncd: Starts the lsync Daemon
#
# chkconfig: 345 99 90
# description: Lsyncd uses rsync to synchronize local directories with a remote
# machine running rsyncd. Lsyncd watches multiple directories
# trees through inotify. The first step after adding the watches
# is to, rsync all directories with the remote host, and then sync
# single file buy collecting the inotify events.
# processname: lsyncd
. /etc/rc.d/init.d/functions
config=”/etc/lsyncd.lua”
lsyncd=”/usr/local/bin/lsyncd”
lockfile=”/var/lock/subsys/lsyncd”
pidfile=”/var/run/lsyncd.pid”
prog=”lsyncd”
RETVAL=0
start() {
if [ -f $lockfile ]; then
echo -n $”$prog is already running: ”
echo
else
echo -n $”Starting $prog: ”
daemon $lsyncd -pidfile $pidfile $config
RETVAL=$?
echo
[ $RETVAL = 0 ] && touch $lockfile
return $RETVAL
fi
}
stop() {
echo -n $”Stopping $prog: ”
killproc $lsyncd
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f $lockfile
return $RETVAL
}
case “$1” in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
status)
status $lsyncd
;;
*)
echo “Usage: lsyncd {start|stop|restart|status}”
exit 1
esac
exit $?
Configure Lsyncd to start at boot:
chkconfig –add lsyncd
chkconfig lsyncd on
Ubuntu
Create the following file at /etc/init/lsyncd.conf
description “lsyncd file syncronizer”
start on (starting network-interface
or starting network-manager
or starting networking)
stop on runlevel [!2345]
expect fork
respawn
respawn limit 10 5
exec /usr/local/bin/lsyncd -pidfile /var/run/lsyncd.pid /etc/lsyncd.lua
Then create a symlink to preserve functionality when called as an old style init script
ln -s /lib/init/upstart-job /etc/init.d/lsyncd
If you want to disable lsyncd from starting automatically at bootup (I.E. customer scaled down to one server), you can create a file at /etc/init/lsyncd.override
manual
Log rotation
Add this file at /etc/logrotate.d/lsyncd:
/var/log/lsyncd/*log {
missingok
notifempty
sharedscripts
postrotate
if [ -f /var/lock/lsyncd ]; then
/sbin/service lsyncd restart > /dev/null 2>/dev/null || true
fi
endscript
}
You can test this using the command
logrotate -d /etc/logrotate.d/lsyncd
Finally, for both distributions, set permissions and ownership on init.d file appropriately:
**NOTE** Don’t do this if you used the Upstart-Job method above for Ubuntu. As it is a symlink, this step is unnecessary.
# chmod 775 /etc/init.d/lsyncd
# chown root:root /etc/init.d/lsyncd
Configuration
Lsyncd configuration file examples. /etc/lsyncd.lua
Legacy: 2.0.X
Here is an example of a master, single slave configuration file (10.x.x.x is the private IP of the slave server) for 2.0.x
settings = {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20
}
sync{
default.rsyncssh,
source=”/var/www/html”,
host=”10.x.x.x”,
targetdir=”/var/www/html”,
rsyncOpts=”-avz”
}
Here is a more advanced example, which uses some Lua magic to make adding new servers easier. In this configuration, there are two slave servers, and new slaves can be added by including them in the “servers” block below. Also note that StrictHostKeyChecking is explicitly set to no, so you shouldn’t need to SSH to each server individually to accept the host key after adding them to your configuration.
settings = {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20
}
servers = {
“web02”,
“web03″
}
for _, server in ipairs(servers) do
sync {
default.rsync,
source=”/var/www/vhosts/”,
target=server..”:/var/www/vhosts/”,
rsyncOpts={“-e”, “/usr/bin/ssh -o StrictHostKeyChecking=no”, “-avz”}
}
end
Current: 2.1.X
Starting with 2.1.0 the configuration has become less stable. In 2.1.0, the last setting is ‘rsyncOps’ instead of ‘rsyncOpts’ (note the missing ‘t’).
With 2.1.1 the syntax change entirely.
Here is a 2.1.1 example with two hosts. Note that StrictHostKeyChecking is explicitly set to no, so you shouldn’t need to SSH to each server individually to accept the host key after adding them to your configuration.
If you installed via yum, the config file is /etc/lsyncd.conf instead of /etc/lsyncd.lua
settings {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20
}
sync {
default.rsync,
source=”/var/www/”,
target=”10.x.x.x:/var/www/”,
rsync = {
compress = true,
archive = true,
verbose = true,
rsh = “/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no”
}
}
sync {
default.rsync,
source=”/var/www/”,
target=”10.X.X.X:/var/www/”,
rsync = {
compress = true,
archive = true,
verbose = true,
rsh = “/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no”
}
}
2.0.4
Lsyncd version 2.0.4 has a known issue with permissions, see the following bug reports. Always make sure you install either 2.0.7 (stable) or 2.1.4+ (latest)
https://gist.github.com/axkibe/lsyncd/issues/94
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=639148
2.1.2
Version 2.1.2 has a bug in it that it will not use your server specified in the target, nor can you specify a host due to an issue with the way it’s passing parameters. This is filed in this issue:
https://github.com/axkibe/lsyncd/issues/168
You can work around this by using
rsh = “/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no”
until the fix goes into production.
2.1.4
On RedHat distros, the init script in lsync 2.1.4 may contain a bug to where the service will fail to start and you will be presented with “USAGE:” instructions. If this is the case, edit /etc/sysconfig/lsyncd and add the following line:
LSYNCD_OPTIONS=”-pidfile /var/run/lsyncd.pid /etc/lsyncd.conf”
Alternatively, you can modify the following line under the start() function in /etc/init.d/lsyncd:
daemon /usr/bin/lsyncd $LSYNCD_OPTIONS
Change this line to the following:
daemon /usr/bin/lsyncd -pidfile /var/run/lsyncd.pid /etc/lsyncd.conf
Exclusions
If you want to exclude a directory, use an exclude file and the excludeFrom directive as seen below.
The excludeFrom feature is not working in 2.1.1 (see https://github.com/axkibe/lsyncd/issues/164). The “exclude” function still lets you list every file you want to exclude, but trying to read the file from a list with excludeFrom will give an error on reastarting Lsyncd saying that excludeFrom is not a valid variable. (This bug has been fixed. You can now use exclude from as you normally would.)
Note that the paths to the files are relative from the source directive:
/etc/lsyncd-excludes.txt:
cache/
uploads/
dontcopymebro/
/etc/lsyncd.lua (this is a 2.0.x example, but the syntax of excludeFrom does not change between versions)
settings = {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20
}
sync{
default.rsyncssh,
source=”/var/www/html”,
host=”10.x.x.x”,
targetdir=”/var/www/html”,
excludeFrom=”/etc/lsyncd-excludes.txt”,
rsyncOpts=”-avz”
}
CAUTION! When using excludeFrom and a file, make sure the file has no empty lines in it. If it does, lsyncd treats this as “exclude /” and then everything gets excluded.
Delay considerations
Sync/propagation delay is generally significantly more noticeable on busy sites. The default delay (15 seconds) can be overridden in a couple of ways:
Added as an option to lsyncd process (at start) via the inclusion of ‘-delay SECONDS’
Fine grained control per sync block in the format of ‘delay=SECONDS’
An example of managing/overriding sync delay (2.0.x syntax)
sync{
default.rsyncssh,
source=”/var/www/html”,
host=”10.x.x.x”,
targetdir=”/var/www/html”,
delay=5,
rsyncOpts=”-avz”
}
The listed approach is likely to address sync times however bear in mind that it will lead to significantly more rsync (SSH) connections to the remote hosts in the ‘cluster’. A decent delay value should deal with unavailable resources on busy/volatile sites (to some extent at least – until we settle on an alternative).
So that’s about it. You should then be able to start it up at using ‘service lsyncd start’ and monitor the log file /var/log/lsyncd/lsyncd.log for any specific problems
Lsync Gotchas
There is a kernel parameter fs.inotify.max_user_watches which limits the number of directories which can be monitored by inotify. By default, it is set to 8192, but if the customer has a lot of directories, this will likely need to be increased. The Lsyncd log will typically notify you when this limitation is reached.
To increase the max_user_watches, use the following sysctl command:
sysctl -w fs.inotify.max_user_watches=16384 >> /etc/sysctl.conf
sysctl -p
To determine the number that this parameter should be set to, count the number of directories (including subdirectories) that will be monitored by lsync and multiply that by 2 or 3 to accommodate growth. You can can execute the following command:
find /var/www/vhosts/ -type d | wc -l | awk ‘{print $1″ directories needs monitoring, set [ fs.inotify.max_user_watches = ” $1*3″ ]”}’
Sample output:
33577 directories needs monitoring, set [ fs.inotify.max_user_watches = 100731 ]
If combining Lsyncd with NFS, be sure to read Lsyncd + NFS Solutions
