vickistan.com

Git stuff

Conceptual description: http://www.sbf5.com/~cduan/technical/git/git-1.shtml

**HEAD is the name given to the commit from which the working tree’s current state was initialized.
git rebase allows you to pull changes from master into your development branch, but leave all of your development work “on top of” (later in the commit log) the stuff from master. When your new work is complete, the merge back to master is then very straightforward.

#Make a local copy of the git repo that you want to work on
git clone URL

make changes in local repository

git add changed.file

git commit -m “new changes”

git status

git push origin master

When you want to update your local copy with a special branch, do the following:

#Pull a fresh copy of develop
git checkout develop

#Look for last minute changes
git pull

#Check out the feature branch
git checkout feature/my_special_branch

#Merge with develop branch you just updated
git merge develop

**Switching to develop and doing a pull will ensure your local copy is up to date with the most recent develop branch. Then you add in the new feature branch by pulling it and then merging with develop.

Overview

This article describes how to use lsyncd as a sync’ing mechanism for multiple web servers data directories. This is only one scenario in which this program could be very useful for our customers.
These notes will provide instructions on how to setup lsyncd to sync the web directory on one server to two other web servers. This guide was setup on our default CentOS 5.5 LAMP build servers, but should work with a few modifications on Ubuntu.

Replicate from master to two slaves:
Master —> slave1
—> slave2
Preparation

Lsyncd requires passwordless SSH from the master to slave systems for the root user.

Generate SSH keypair on master:

# ssh-keygen -t rsa

Then copy /root/.ssh/id_rsa.pub on master to /root/.ssh/authorized_keys on the slave servers.

Then ensure the following lines are uncommented in /etc/ssh/sshd_config on the slave servers:

PermitRootLogin yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

If you have to create the /root/.ssh directory on the slave, it should be owned root:root and have permissions 0700.

Restart SSH after changes are made.

Test ssh from the master to the slave as root to make sure it isn’t throwing a message that prevents lsyncd from being able to connect

On Ubuntu, you may see %h/.ssh/authorized_keys. This is fine as well, the %h is just an alias for the home directory of the user logging in, i.e. /root/ for root.

Supporting Package Installation

First you need to ensure the following packages are installed, that are not within the default LAMP build
Dependencies

CentOS and RHEL

# yum -y install lua lua-devel pkgconfig gcc asciidoc

Ubuntu

# apt-get install -y lua5.1 liblua5.1-dev pkg-config rsync asciidoc

Lsync installation
Be sure to manually check the destination directory on the slave server if it already exists. Customers might have NFS or CloudFuse mounts which would end up wiped by the lsync process.

1. Then grab the source tarball from http://code.google.com/p/lsyncd/, and run the standard (on CentOS/RHEL, you can bypass this step and do ‘yum install lsyncd’ to install 2.1.4 from IUS)

tar xzvf lsyncd-2.1.5.tar.gz
cd lsyncd-2.1.5
./configure && make && make install

2. Create the Lsyncd log directory.

mkdir /var/log/lsyncd

3. Add init scripts for starting and stopping the service.
RHEL/CentOS

Create the following file at /etc/init.d/lsyncd: (skip if you did ‘yum install lsyncd’ above)

#!/bin/bash
#
# lsyncd: Starts the lsync Daemon
#
# chkconfig: 345 99 90
# description: Lsyncd uses rsync to synchronize local directories with a remote
# machine running rsyncd. Lsyncd watches multiple directories
# trees through inotify. The first step after adding the watches
# is to, rsync all directories with the remote host, and then sync
# single file buy collecting the inotify events.
# processname: lsyncd

. /etc/rc.d/init.d/functions

config=”/etc/lsyncd.lua”
lsyncd=”/usr/local/bin/lsyncd”
lockfile=”/var/lock/subsys/lsyncd”
pidfile=”/var/run/lsyncd.pid”
prog=”lsyncd”
RETVAL=0

start() {
if [ -f $lockfile ]; then
echo -n $”$prog is already running: ”
echo
else
echo -n $”Starting $prog: ”
daemon $lsyncd -pidfile $pidfile $config
RETVAL=$?
echo
[ $RETVAL = 0 ] && touch $lockfile
return $RETVAL
fi
}

stop() {
echo -n $”Stopping $prog: ”
killproc $lsyncd
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f $lockfile
return $RETVAL
}

case “$1” in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
status)
status $lsyncd
;;
*)
echo “Usage: lsyncd {start|stop|restart|status}”
exit 1
esac

exit $?

Configure Lsyncd to start at boot:

chkconfig –add lsyncd
chkconfig lsyncd on

Ubuntu

Create the following file at /etc/init/lsyncd.conf

description “lsyncd file syncronizer”

start on (starting network-interface
or starting network-manager
or starting networking)

stop on runlevel [!2345]

expect fork

respawn
respawn limit 10 5

exec /usr/local/bin/lsyncd -pidfile /var/run/lsyncd.pid /etc/lsyncd.lua

Then create a symlink to preserve functionality when called as an old style init script

ln -s /lib/init/upstart-job /etc/init.d/lsyncd

If you want to disable lsyncd from starting automatically at bootup (I.E. customer scaled down to one server), you can create a file at /etc/init/lsyncd.override

manual

Log rotation

Add this file at /etc/logrotate.d/lsyncd:

/var/log/lsyncd/*log {
missingok
notifempty
sharedscripts
postrotate
if [ -f /var/lock/lsyncd ]; then
/sbin/service lsyncd restart > /dev/null 2>/dev/null || true
fi
endscript
}

You can test this using the command

logrotate -d /etc/logrotate.d/lsyncd

Finally, for both distributions, set permissions and ownership on init.d file appropriately:

**NOTE** Don’t do this if you used the Upstart-Job method above for Ubuntu. As it is a symlink, this step is unnecessary.

# chmod 775 /etc/init.d/lsyncd
# chown root:root /etc/init.d/lsyncd

Configuration

Lsyncd configuration file examples. /etc/lsyncd.lua
Legacy: 2.0.X

Here is an example of a master, single slave configuration file (10.x.x.x is the private IP of the slave server) for 2.0.x

settings = {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20

}

sync{
default.rsyncssh,
source=”/var/www/html”,
host=”10.x.x.x”,
targetdir=”/var/www/html”,
rsyncOpts=”-avz”

}

Here is a more advanced example, which uses some Lua magic to make adding new servers easier. In this configuration, there are two slave servers, and new slaves can be added by including them in the “servers” block below. Also note that StrictHostKeyChecking is explicitly set to no, so you shouldn’t need to SSH to each server individually to accept the host key after adding them to your configuration.

settings = {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20
}

servers = {
“web02”,
“web03″
}

for _, server in ipairs(servers) do
sync {
default.rsync,
source=”/var/www/vhosts/”,
target=server..”:/var/www/vhosts/”,
rsyncOpts={“-e”, “/usr/bin/ssh -o StrictHostKeyChecking=no”, “-avz”}
}
end

Current: 2.1.X

Starting with 2.1.0 the configuration has become less stable. In 2.1.0, the last setting is ‘rsyncOps’ instead of ‘rsyncOpts’ (note the missing ‘t’).

With 2.1.1 the syntax change entirely.

Here is a 2.1.1 example with two hosts. Note that StrictHostKeyChecking is explicitly set to no, so you shouldn’t need to SSH to each server individually to accept the host key after adding them to your configuration.

If you installed via yum, the config file is /etc/lsyncd.conf instead of /etc/lsyncd.lua

settings {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20
}

sync {
default.rsync,
source=”/var/www/”,
target=”10.x.x.x:/var/www/”,
rsync = {
compress = true,
archive = true,
verbose = true,
rsh = “/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no”
}
}

sync {
default.rsync,
source=”/var/www/”,
target=”10.X.X.X:/var/www/”,
rsync = {
compress = true,
archive = true,
verbose = true,
rsh = “/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no”
}
}

2.0.4

Lsyncd version 2.0.4 has a known issue with permissions, see the following bug reports. Always make sure you install either 2.0.7 (stable) or 2.1.4+ (latest)

https://gist.github.com/axkibe/lsyncd/issues/94

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=639148

2.1.2

Version 2.1.2 has a bug in it that it will not use your server specified in the target, nor can you specify a host due to an issue with the way it’s passing parameters. This is filed in this issue:

https://github.com/axkibe/lsyncd/issues/168

You can work around this by using

rsh = “/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no”

until the fix goes into production.

2.1.4

On RedHat distros, the init script in lsync 2.1.4 may contain a bug to where the service will fail to start and you will be presented with “USAGE:” instructions. If this is the case, edit /etc/sysconfig/lsyncd and add the following line:

LSYNCD_OPTIONS=”-pidfile /var/run/lsyncd.pid /etc/lsyncd.conf”

Alternatively, you can modify the following line under the start() function in /etc/init.d/lsyncd:

daemon /usr/bin/lsyncd $LSYNCD_OPTIONS

Change this line to the following:

daemon /usr/bin/lsyncd -pidfile /var/run/lsyncd.pid /etc/lsyncd.conf

Exclusions

If you want to exclude a directory, use an exclude file and the excludeFrom directive as seen below.

The excludeFrom feature is not working in 2.1.1 (see https://github.com/axkibe/lsyncd/issues/164). The “exclude” function still lets you list every file you want to exclude, but trying to read the file from a list with excludeFrom will give an error on reastarting Lsyncd saying that excludeFrom is not a valid variable. (This bug has been fixed. You can now use exclude from as you normally would.)

Note that the paths to the files are relative from the source directive:

/etc/lsyncd-excludes.txt:

cache/
uploads/
dontcopymebro/

/etc/lsyncd.lua (this is a 2.0.x example, but the syntax of excludeFrom does not change between versions)

settings = {
logfile = “/var/log/lsyncd/lsyncd.log”,
statusFile = “/var/log/lsyncd/lsyncd-status.log”,
statusInterval = 20

}

sync{
default.rsyncssh,
source=”/var/www/html”,
host=”10.x.x.x”,
targetdir=”/var/www/html”,
excludeFrom=”/etc/lsyncd-excludes.txt”,
rsyncOpts=”-avz”
}

CAUTION! When using excludeFrom and a file, make sure the file has no empty lines in it. If it does, lsyncd treats this as “exclude /” and then everything gets excluded.
Delay considerations

Sync/propagation delay is generally significantly more noticeable on busy sites. The default delay (15 seconds) can be overridden in a couple of ways:

Added as an option to lsyncd process (at start) via the inclusion of ‘-delay SECONDS’
Fine grained control per sync block in the format of ‘delay=SECONDS’

An example of managing/overriding sync delay (2.0.x syntax)

sync{
default.rsyncssh,
source=”/var/www/html”,
host=”10.x.x.x”,
targetdir=”/var/www/html”,
delay=5,
rsyncOpts=”-avz”
}

The listed approach is likely to address sync times however bear in mind that it will lead to significantly more rsync (SSH) connections to the remote hosts in the ‘cluster’. A decent delay value should deal with unavailable resources on busy/volatile sites (to some extent at least – until we settle on an alternative).

So that’s about it. You should then be able to start it up at using ‘service lsyncd start’ and monitor the log file /var/log/lsyncd/lsyncd.log for any specific problems
Lsync Gotchas

There is a kernel parameter fs.inotify.max_user_watches which limits the number of directories which can be monitored by inotify. By default, it is set to 8192, but if the customer has a lot of directories, this will likely need to be increased. The Lsyncd log will typically notify you when this limitation is reached.

To increase the max_user_watches, use the following sysctl command:

sysctl -w fs.inotify.max_user_watches=16384 >> /etc/sysctl.conf
sysctl -p

To determine the number that this parameter should be set to, count the number of directories (including subdirectories) that will be monitored by lsync and multiply that by 2 or 3 to accommodate growth. You can can execute the following command:

find /var/www/vhosts/ -type d | wc -l | awk ‘{print $1″ directories needs monitoring, set [ fs.inotify.max_user_watches = ” $1*3″ ]”}’
Sample output:
33577 directories needs monitoring, set [ fs.inotify.max_user_watches = 100731 ]

If combining Lsyncd with NFS, be sure to read Lsyncd + NFS Solutions

To change the admin username in WordPress:

1)Go into your phpmyadmin, and click on “Databases.”
2)Click on the name of your database.
3)Click the “Browse” icon next to wp_users.
4)Click the edit icon next to admin – should be first button on first row (like a notepad).
Change it.
5)Go down to the bottom and click “GO.”

Explaining the following regular expression:

(\d(3,4)[.-]?)+

( starts a capturing group
\ escapes the following character
d end character shorthand (match any digit in the range 0 through 9 with \d)
{ open qualifier
3 minimum quantity to match
, separate quantities
4 maximum quantity to match
} close quantifier
[ open character class
. match literal dot
– literal hyphen to match hyphen
] close character class
? zero or one quantifier
) close character class
+ one or more quantifier

Improved version:
(\d{3}[.-]?){2}\d{4}

php directives are listed here: http://www.php.net/manual/en/ini.list.php

Modes determine when and where a PHP directive may or may not be set. This is explained here: http://www.php.net/manual/en/ini.list.php

A virtual host may be configured to use a local php.ini file in place of the system one. The local one would override the entire system php.ini file in this case. Do this as follows:

copy the php.ini file to a local directory such as /var/www/vhosts/mydomain.com
add the following to the virtual host file:

PHPINIDir /var/www/vhosts/mydomain.com

You could simply make your changes in an .htaccess file, if your host doesn’t allow you to touch the php.ini file or you want to change only a few values leaving the values in php.ini as default, manage changes in a local .htaccess file:

For reference, the proper lines for an .htaccess file would have to be prepended with php_values as follows:

php_value suhosin.post.max_vars 100000
php_value suhosin.request.max_vars 100000
php_value memory_limit 128

Note: php_flag and _php_value can be written in httpd.conf for FastCGI, but not in .htaccess file. if PHP is running as module, you can do anything with .htaccess

Still another way to do it is to use PHP’s native ini_set() function.

Edit the /etc/ssh/sshd_config:

1. Comment out the Subsystem sftp line
2. Create line as follows:
Subsystem sftp internal-sftp
3. Add the following at the bottom of the sshd_config file:
Match Group sftponly
ChrootDirectory %h
X11Forwarding no
AllowTCPForwarding no
ForceCommand internal-sftp
4. Restart ssh
5. Create a file called create_chroot.sh:
useradd -d /home/contractor3 -s /bin/false -G sftponly contractor3
chown root:root /home/contractor3/
chmod 755 /home/contractor3/
mkdir /home/contractor3/upload
chown contractor3:contractor3 /home/contractor3/upload
passwd contractor3