Rocks Utilities

From BeSTGRID

1 Locations
2 Sharing across compute nodes
3 Scripts
4 Compute nodes configuration
5 Swaping interfaces on head node during the install
6 Rocks Database

[edit] Locations

rocks utility invokes commands written in python, located at /opt/rocks/lib/python2.4/site-packages/rocks/commands. Every command has its own directory with source code located at __init__.py So for example

rocks list host profile

Is located at opt/rocks/lib/python2.4/site-packages/rocks/commands/list/host/profile/__init__.py

[edit] Sharing across compute nodes

Sharing Configuration Files With 411 Service (the same works with 5.0)

Sharing Applications

[edit] Scripts

insert-ethers can be given additional parameters and set to run in batch mode. For example:

insert-ethers --hostname="power-0-0" --rack=0 --rank=0 --cpus=1 --appliance="Power Units" --mac="00:30:48:97:fb:e5" --device="None" --module="None" --ipaddr="10.0.1.6" --netmask="255.0.0.0" --norestart --batch --verbose

cluster-fork is useful to automate installation of software not available on rolls, and other administration tasks. Examples:

# print network configuration of every node
cluster-fork ifconfig
# print "hello" on all nodes compute-1-x (query runs against [Rocks Database])
cluster-fork --query="select name from nodes where name like 'compute-1-%'" echo hello
# run job in background
cluster-fork --bg echo hello

script to propagate updates from yum to compute nodes. stolen from rocks mailing list, author Travis Daygale. [1]

#!/bin/bash

# 
# this script copies packages cached by yum so that the rocks distro can be updated too (head node rev levels ! = rocks distro rev levels)
# 
# there might be better ways to do this, but for now...

# Change this as necessary:
RocksDistro=/home/install/contrib/4.2.1/x86_64/RPMS/

if [ ! -d $RocksDistro ]
then
echo "No directory $RocksDistro, are you sure this is a rocks node?"
echo "No changes made, aborting"
exit
fi


for yumcache in `find /var/cache/yum/ -name packages -type d -print`
do
cd $yumcache
tar cf - . | (cd $RocksDistro; tar xf -)
echo "Copied all packages from $yumcache to $RocksDistro"
done

echo "Recreating the Rocks distro using rocks-dist"
cd /home/install
/opt/rocks/bin/rocks-dist dist

To re-install all compute nodes:

 cluster-fork /boot/kickstart/cluster-kickstart

To synchronize users on compute nodes:

rocks-user-sync

To synchronize all the changes in cluster database:

rocks sync config

Script to install new roll on existing cluster (given for bio roll, but others are similar)

 su - root
 mount -o loop bio.iso /mnt/cdrom
 cd /home/install
 rocks-dist --install copyroll
 umount /mnt/cdrom
 rocks-dist dist
 kroll bio | bash
 init 6
 cluster-fork /boot/kickstart/cluster-kickstart

Useful torque commands:

qsub
qstat

Rocks example script to submit job via Grid engine (from Rocks manual).

#!/bin/bash
#$ -S /bin/bash
#
# set the P4_GLOBMEMSIZE
#$ -v P4_GLOBMEMSIZE=10000000
#
# Set the Parallel Environment and number of procs.
#$ -pe mpi 2 
# Where we will make our temporary directory.
BASE="/tmp" 
#
# make a temporary key
#
export KEYDIR=`mktemp -d $BASE/keys.XXXXXX`
#
# Make a temporary password.
# Makepasswd is quieter, and presumably more efficient.
# We must use the -s 0 flag to make sure the password contains no quotes.
#
if [ -x `which mkpasswd` ]; then
	export PASSWD=`mkpasswd -l 32 -s 0`
else
	export PASSWD=`dd if=/dev/urandom bs=512 count=100 | md5sum | gawk '{print $1}'`
fi
/usr/bin/ssh-keygen -t rsa1 -f $KEYDIR/tmpid -N "$PASSWD"
cat $KEYDIR/tmpid.pub >> $HOME/.ssh/authorized_keys
#
# make a script that will run under its own ssh-agent 
#
cat > $KEYDIR/launch-script <<"EOF"
#!/bin/bash
expect -c 'spawn /usr/bin/ssh-add $env(KEYDIR)/tmpid' -c \
	'expect "Enter passphrase for $env(LOGNAME)@$env(HOSTNAME)" \
		{ send "$env(PASSWD)\n" }' -c 'expect "Identity"' 
echo

#
# Put your Job commands here.
#
#------------------------------------------------ 
/usr/bin/mpirun -np $NSLOTS -machinefile $TMP/machines \
	/opt/hpl/openmpi-hpl/bin/xhpl

#/opt/mpich/gnu/bin/mpirun -np $NSLOTS -machinefile $TMP/machines \
	#/opt/hpl/mpich-hpl/bin/xhpl

#------------------------------------------------
EOF
chmod u+x $KEYDIR/launch-script
# 
# start a new ssh-agent from scratch -- make it forget previous ssh-agent
# connections
#
unset SSH_AGENT_PID
unset SSH_AUTH_SOCK

[edit] Compute nodes configuration

The compute nodes are bootstraped with redhat kickstart script that controls how the questions during OS install are answered, and runs some pre and post install scripts.

To see this script, execute: