Compute Nodes Setting
From BeSTGRID
Contents |
[edit] BMC on compute nodes
[edit] MACs:
| IP Address | MAC Address |
|---|---|
| Headnode | 00:30:48:97:f6:33 |
| 10.0.1.1 | 00:30:48:97:fc:c5 |
| 10.0.1.2 | 00:30:48:94:9e:a5 |
| 10.0.1.3 | 00:30:48:97:fb:ed |
| 10.0.1.4 | 00:30:48:97:fc:bc |
| 10.0.1.5 | 00:30:48:94:9e:b3 |
| 10.0.1.6 | 00:30:48:97:fb:e5 |
| 10.0.1.7 | 00:30:48:97:fb:8a |
| 10.0.1.8 | 00:30:48:97:f6:32 |
| 10.0.1.9 | 00:30:48:97:f6:69 |
| 10.0.1.10 | 00:30:48:97:fb:87 |
[edit] Headnode's iptables configuration to access BMC from public network
By default those run DHCP client on a local network. Therefore to make them available inside local network, there have to be DHCP server on the frontnode. Rocks runs dhcp server to configure cluster appliances so it is possible to assign addresses to BMC by running insert-ethers and selecting any appliance other then compute node. The Rocks wiki suggests that power appliance does not try to kickstart nodes, so it is most suitable. However the experience shows that insert-ethers does try to kickstart even for power appliance, so the choice does not matter much. I still selected power appliance, so the names assigned look like power-x-x. After the original setup it is possible to access BMC from web interface just like for the front node, assign static address, and remove the appliances from Rocks by running insert-ethers remove power-x-x.
To make BMC available on public network, we decided to configure iptables on front node to forward connections to 2001-2010 ports to port 80 on appropriate private addresses i.e.
/sbin/iptables -t nat -A PREROUTING -p tcp -i eth1 -d 130.216.189.80 --dport 2001 -j DNAT --to 10.0.1.1:80 /sbin/iptables -t nat -A PREROUTING -p tcp -i eth1 -d 130.216.189.80 --dport 2002 -j DNAT --to 10.0.1.2:80 /sbin/iptables -t nat -A PREROUTING -p tcp -i eth1 -d 130.216.189.80 --dport 2003 -j DNAT --to 10.0.1.3:80 /sbin/iptables -t nat -A PREROUTING -p tcp -i eth1 -d 130.216.189.80 --dport 2004 -j DNAT --to 10.0.1.4:80 /sbin/iptables -t nat -A PREROUTING -p tcp -i eth1 -d 130.216.189.80 --dport 2005 -j DNAT --to 10.0.1.5:80
/sbin/iptables -A FORWARD -p tcp -i eth1 -o eth0 -d 10.0.1.1 --dport 80 -j ACCEPT /sbin/iptables -A FORWARD -p tcp -i eth1 -o eth0 -d 10.0.1.2 --dport 80 -j ACCEPT /sbin/iptables -A FORWARD -p tcp -i eth1 -o eth0 -d 10.0.1.3 --dport 80 -j ACCEPT /sbin/iptables -A FORWARD -p tcp -i eth1 -o eth0 -d 10.0.1.4 --dport 80 -j ACCEPT /sbin/iptables -A FORWARD -p tcp -i eth1 -o eth0 -d 10.0.1.5 --dport 80 -j ACCEPT
Note that this setting does not give access to remote console. Another way to access BMC console is to set up SOCKS proxy on the client machine and configure your browser to use it. For example the following command creates proxy on client machine:
ssh -fND localhost:10001 root@hpc-bestgrid.auckland.ac.nz
Port can be any free port, username and domain name are for the front node. To setup firefox to use this proxy, go to Edit -> Preferences -> Advanced -> Connections and add SOCKS proxy for localhost and port 10001 (or any other as above). Then local address for BMC can be used (i.e. 10.0.1.x etc.)
[edit] Configuring BMC with ipmitool
to install IPMI on compute nodes the following is needed:
RPMS (I used version 2.0.6-5 but can be any version):
- OpenIPMI-2.0.6-5.el5.4.x86_64.rpm
- OpenIPMI-libs-2.0.6-5.el5.4.x86_64.rpm
- ipmitool (downloaded sources and compiled)
Then do /etc/init.d/ipmi start
Useful ipmitool commands:
To print configuration:
ipmitool lan print 1
To set an IP address
ipmitool lan set 1 ipaddr <x.x.x.x>
To set netmask
ipmitool lan set 1 netmask <x.x.x.x>
To set gateway IP
ipmitool lan set 1 defgw ipaddr <x.x.x.x>
[edit] BMC Problem
On unknown reason only 6 nodes from ten send DHCP requests to acquire IPs for IPMI BMC. The internal switch doesn't block DHCP requests on certain ports because DHCP requests of PXE boot from 4 nodes come to the Headnode.
Compute nodes have no CD drives and ISO image for USB flash drives to set IPMI/BMC unusable. The only way to set/check IPMI/BMC configuration is to boot a node from an external USB CDROM drive. We are looking around to find one.
Update: The problem was due to some of the nodes set up using static ip with gateway 0.0.0.0. Therefore they do not ask for DHCP address and were not accessible.
[edit] Cluster specification of Compute nodes on 7/05/2008
| System | Rack | Position | Serial Number | MACs | IP | BMC MAC | BMC IP | Hostname |
| Head Node Altix XE 250 | P9 | 28,29 | X0004868 | 00-30-48-7F-71-88 00-30-48-7F-71-89 |
130.216.189.80 10.1.1.1 |
00-30-48-97-F6-33 | 130.216.189.81 | Cluster |
| Node1 Altix XE 320 Node2 Altix XE 320 | P9 | 30 | X0004872 | 00-30-48-7F-46-D2 00-30-48-7F-46-D3 00-30-48-7F-47-7E 00-30-48-7F-47-7F |
10.255.255.252 10.255.255.253 |
00-30-48-97-FC-C5 00-30-48-97-FC-BC |
10.0.1.1 10.0.1.4 |
compute-0-1 compute-0-0 |
| Node3 Altix XE 320 Node4 Altix XE 320 | P9 | 31 | X0004874 | 00-30-48-7F-A8-F6 00-30-48-7F-A8-F7 00-30-48-7F-A8-F8 00-30-48-7F-A8-F9 |
10.255.255.244 10.255.255.245 |
00-30-48-94-9E-A5 00-30-48-94-9E-B3 |
10.0.1.2 10.0.1.5 |
compute-0-9 compute-0-8 |
| Node5 Altix XE 320 Node6 Altix XE 320 | P12 | 23 | X0004873 | 00-30-48-7F-42-52 00-30-48-7F-42-53 00-30-48-7F-47-7A 00-30-48-7F-47-7B |
10.255.255.247 10.255.255.246 |
00-30-48-97-FB-ED 00-30-48-97-FB-E5 |
10.0.1.3 10.0.1.6 |
compute-0-6 compute-0-7 |
| Node7 Altix XE 320 Node8 Altix XE 320 | P12 | 24 | X0004871 | 00-30-48-7F-2F-88 00-30-48-7F-2F-89 00-30-48-7F-2F-84 00-30-48-7F-2F-85 |
10.255.255.248 10.255.255.249 |
00-30-48-97-FB-8A 00-30-48-97-FB-87 |
10.0.1.7 10.0.1.10 |
compute-0-5 compute-0-4 |
| Node9 Altix XE 320 Node10 Altix XE 320 | P12 | 25 | X0004870 | 00-30-48-7F-48-9E 00-30-48-7F-48-9F 00-30-48-7F-48-B0 00-30-48-7F-48-B1 |
10.255.255.250 10.255.255.251 |
00-30-48-97-F6-69 00-30-48-97-F6-32 |
10.0.1.9 10.0.1.8 |
compute-0-3 compute-0-2 |
