General

This document describes how to set up a typical web application environment under openQRM. The sample environmnet is a 3-tier environmnet, seperating the web server from the application server and the database. This document assumes an already up and running openQRM server, with at least 4 nodes connected to it. For more complex scenarios, like load balancing and multiple failures testings, additional nodes will be needed. For installing the openQRM server, refer to the Installation Manual, or the “Installing openQRM Guide”.

Environment 

• Database Server - e.g. MySQL
  We will use a single database server, with high availability provided by openQRM
• Application Server - e.g. Apache Tomcat
  We will use multiple instances of the application server, scaling according  to load. Please refer to the Tomcat guide on How To set up Tomcat clustering. 
• Web Server - e.g. Apache HTTP Server
  The web server will serve only static content, so multiple instances of it  can run concurrently. 
• Load Balanacer - Level 4 switch which will direct the web requests to the  different servers. At the minimum, it is needed in front of the web  server. Preferably, it should also be between the web server and the application server. If a hardware load balancer is unavailable, it is possible to set it up in software -
  Use a software load balancer such as Pound in  front of the web servers and configure the connection between the web server  and the application servers as described on Load Balancer How To.

Setup Overview

We will set up multiple Virtual Environments sharing the same filesystem-image. Our filesystem image will reside on NFS.
By sharing the same filesystem we make the maintenance of the system easier – to install a security patch or change some part of the base configuration all we have to do is modify the shared filesystem image, and it will apply to all nodes.
The network configuration and started applications will differ between the instances based on the specific virtual environment and the specific instance. To take more instance specific actions, we will use the node-parameters provided by the QRM to the node (located in /var/qrm/conf/node-parameter.conf on the running node).

In this document, we assume for each of the nodes has two network interfaces - eth0 connected to the openQRM management network and eth1 connected to the "external" network, accessible to the users.
We will set all the external addresses on the subnet 10.1.0.0/16 .
We will use an instance initialization script to make sure each new instance will have the right configuration.

Base Filesystem Image

This image is created from a running server. E.g. from the openQRM server.
In the document we'll assume RedHat conventions for file locations.
Start by create a new filesystem image. You can choose whatever name you want for it, in this document we'll use the name “image1”. This is done by running the qrm-filesystem-image with the –-shared flag. See the user manual for detailed instructions on using this command.
We will also create the corresponding kernel image, using the qrm-boot-image command. In this document we'll use the name “kernel1”. See the user manual for detailed instuctions on creating and managing boot images.

To make sure the image starts our custom configurations we'll add to it our start script.

>  cd <path-to-images>/image1
>  ln -sf /custom/ve/init.sh etc/rc.d/rc3.d/S99ve
>  ln -sf /custom/node/init.sh etc/rc.d/rc3.d/S99node
>  ln -sf /custom/ve/init.sh etc/rc.d/rc1.d/K10ve
>  ln -sf /custom/ve/init.sh etc/rc.d/rc6.d/K10ve
>  ln -sf /custom/node/init.sh etc/rc.d/rc6.d/K10node
>  ln -sf /custom/node/init.sh etc/rc.d/rc6.d/K10node
>  # We'll also set the eth1 interface to have a node-specific configuration:
>  ln -sf /custom/node/ifcfg-eth1 etc/sysconfig/network-scripts/ifcfg-eth1

Note that while the first parameter is an absolute path (the origin of the symbolic link), the target is a relative path, under the filesystem image root directory. This will cause the start and stop script that are specific to each VE and to each instance to run when the system starts up (reaches runlevle 3) and shuts down or reboots (at runlevels 1 and 6).

openQRM Server

We will now create the Virtual Environments. Each will have the filesystem image set to “image1” and the kernel image set to “kernel1”. You can leave the other parameters at their default for now.
The VEs will have the following names:
- "db" - for the database. Set as a Single-Server.
- "app-server" - for the application server. Set as a Multi-Server.
- "web" - for the web server. Set as a Multi-Server.

If you're using a software load balancer, create one more VE:
- "lb" - for the load balancer. Set as a Single-Server (or Multi-Server if you
  plan on using DNS Round Robin).
We'll then start all VEs to make sure that they initialize their shared resources. This is done from the VE's “control” menu on the right.

Database Server

External IP Address: 10.1.1.1
Looking at the openQRM Server interface, we can see the managment IP address of the node currently running the “db” Virtual Environment. We'll use it to login to it and set it up.
After installing and configuring the database on it, we'll create start and stop scripts for them:

Edit (on the node) the file /custom/ve/init.sh and add interface configuration
and database startup and shutdown commands for this VE. Since we're using the regular MySQL scripts, we might as well use them directly:

>  ln -sf /etc/init.d/mysqld /custom/ve/init.sh

Put this network configuration into /custom/node/ifcfg-eth1 :

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
PEERDNS=no
TYPE=Ethernet
IPADDR=10.1.1.1
NETMASK=255.255.0.0
NETWORK=10.1.0.0
BROADCAST=10.1.255.255

Check the start and stop scripts to make sure they work ok.

Next go to the openQRM VE configuration for this environment and configure it for High Availability - we want a database server running at all times.
Go to the VE screen, open the VE Configuration Tab and choose the High Availability button. Enable the Automatic Hardware Failover capability, with a minimum resource amount of 1 (so there will always be a backup server
available). 

Application Server

IP Addresses: 10.1.2.X
Log into the node running this VE, then install and configure the application server.
The only difference between instances of the application image will be their IP address. To have each instance get an address according to their instance, we will use the "node parameters" supplied by the QRM server.
The control script will now look like:

>  cat > /custom/ve/init.sh << EOF
case $1 in
 start)
  cd /opt/tomcat/bin && ./startup.sh
  ;;
 stop)
  cd /opt/tomcat/bin && ./shutdown.sh
  ;;
 *)
  echo "Usage: $0 start|stop"
  ;;
esac
EOF

We will now set the VE to configure each node with its own IP address - Get the id of the VE from the node-parameter.conf file (it appears as "VE_ID=<value>"). Now edit the file /private/$VE_ID/init_vem as follows:

>  cat > /private/$VE_ID/init_vem << EOFF
#!/bin/bash
VE_ID=$1
CS_VEM=$2
cat > /private/$VE_ID/$CS_VEM/custom/ifcfg-eth1 << EOF
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
PEERDNS=no
TYPE=Ethernet
IPADDR=10.1.2.$CS_VEM
NETMASK=255.255.0.0
NETWORK=10.1.0.0
BROADCAST=10.1.255.255
EOF
EOFF

Now, we will configure the Application Server Virtual Environment to start with 2 nodes and add another one if the load increases.
Go to the openQRM VE configuration for this environment, open the VE Configuration Tab and choose "Provisioning and Policy" button. Set the VE to start with 2 nodes, and have a minimum of 1 and a maximum of 3 nodes.
Set the policy to Internal, Low-Load policy, and set the Handling to "Enforce" instead of only recommending.
Save the changes, and add Automatic Hardware Failover, just in case.

Web Server

IP Addresses: 10.1.3.X
This machine will be configured the same as the application server, but will start with 3 nodes and set with a maximum of 10 nodes. Here we will leave the "Automatic" scaling policy.

The control script will look use the Apache webserver control script:

>  ln -sf /etc/init.d/httpd /custom/ve/init.sh

We will now set the VE to configure each node with its own IP address - Get the id of the VE from the node-parameter.conf file (it appears as "VE_ID=<value>"). Now edit the file
/private/$VE_ID/init_vem as follows:

>  cat > /private/$VE_ID/init_vem << EOFF
#!/bin/bash
VE_ID=$1
CS_VEM=$2
# set up networking
cat > /private/$VE_ID/$CS_VEM/custom/ifcfg-eth1 << EOF
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
PEERDNS=no
TYPE=Ethernet
IPADDR=10.1.3.$CS_VEM
NETMASK=255.255.0.0
NETWORK=10.1.0.0
BROADCAST=10.1.255.255
EOF
EOFF

Finally, we'll configure the Web Virtual Environment to start with 3 nodes and add another one if the load increases.
Go to the openQRM VE configuration for this environment, open the VE Configuration Tab and choose "Provisioning and Policy" button. Set the VE to start with 3 nodes, and have a minimum of 1 and a maximum of 10 nodes.
Set the policy to Internal (leave it as "Automatic") and set the Handling to "Enforce" instead of only recommending.
Save the changes, and add Automatic Hardware Failover, to make sure a new server replaces a failed server.  

Comments (0)

Subscribe to this comment's feed