Wednesday, October 2, 2013

Simple clustering proof of concept with Glassfish, NGINX, and HAPROXY


IRL I do Java EE6 work.   My latest project uses several glassfish (3.1.1/ perhaps even 4.01) servers to support a JSF application.   The application is state-full and I use regular (well CDI session scoped beans) server based session tracking.    Our network engineering team has all sorts of traffic balancing devices in multiple layers in front of my application and there are some odd-ball case where session affinity doesn't work  (or there is some network voo-doo that make setting up affinity hard).

A Possible Solution

Cluster the web application servers so that if affinity breaks it really doesn't matter. 

My Task

So I was tasked to do a POC with our web app and glassfish to see how realistic this might be, here is my story (inset the "Law and Order" sound here).


  • Workstation (in my case, fedora 19x64)
  • GF
  • JDK 1.7_25
  • NGINX 

Some Useful Resources  (on GF HA or Cluster Setup)

Now To GlassFish Setup

CLI & asadmin is your friend:

cd /opt/glassfish-
./asadmin create-domain ClusterDAS
cd ../glassfish/domains/ClusterDAS/config/
keytool -delete -alias gtecybertrust5ca -keystore cacerts.jks
cd /opt/glassfish-
./asadmin start-domain ClusterDAS
##test using admin port and browser
##slim/trim down ClusterDAS
./asadmin delete-http-listener http-listener-1
./asadmin delete-http-listener http-listener-2
./asadmin delete-virtual-server server
##Cluster stuffs
./asadmin create-cluster MyCluster
./asadmin create-local-instance --cluster MyCluster ClusterInstance1
./asadmin create-local-instance --cluster MyCluster ClusterInstance2
##Now start it
./asadmin start-cluster MyCluster 
##GoTo Admin Console of ClusterDAS and see what is there!

Quick Description Pictures of Some Stuff

This is what  the ClusterDAS should look like:
This is an image of the DAS with Clustering setup

Some Specs About This Cluster (assuming no changes from default  and above)

Simple Cluster Test JSF App/WAR

Now a simple WAR can be used to test the above cluster.   The critical element needed is a session value that is set with some value that indicates something about the server that was used to set it and some display of data of the current server servicing the request. Here is some example stuff/code that provides this.

Simple Java Session Scoped Bean

package fhw;
import javax.inject.Named;
import javax.enterprise.context.SessionScoped;
import javax.annotation.PostConstruct;
import javax.faces.context.FacesContext;
import javax.servlet.http.HttpServletRequest;
@Named(value = "sesBean")
public class SesBean implements Serializable
    private String hostVal;
    public SesBean()
    private void init()
    private void mutateHostVal()
        HttpServletRequest r = (HttpServletRequest)FacesContext.getCurrentInstance().getExternalContext().getRequest();
        hostVal = String.format("%s:%d",r.getLocalName(),r.getLocalPort());        
    public String getHostVal()
        String s = hostVal;
        return s;
    public void setHostVal(String hostVal)
        this.hostVal = hostVal;

Facelet Using the Session Scoped Bean

<?xml version='1.0' encoding='UTF-8' ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns=""
        <title>Facelet Title</title>
        <h:outputText value="Session ID:   #{}"/>
        <h:outputText value="Server requested name:  #{request.serverName}" />
        <h:outputText value="Server local name:  #{request.localName}" />        
        <h:outputText value="Server requested Port:  #{request.serverPort}" />
        <h:outputText value="Server Local Port:  #{request.localPort}" />        
        <h:outputText value="#{}" >
            <f:convertDateTime pattern="dd.MM.yyyy HH:mm:ss" />
        <br />
        <h:outputText value="Current session value:  #{sesBean.hostVal}" />

Simple web.xml for sample

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="" xmlns:xsi="" xsi:schemaLocation="">
    <description>Simple JSF thing for tinkering with clustering...</description>
        <servlet-name>Faces Servlet</servlet-name>
        <servlet-name>Faces Servlet</servlet-name>
The rest of the WAR is pretty simple and no magic.

Cluster Deployment in GF

./asadmin deploy --target MyCluster  --availabilityenabled=true /home/blahblah/VerySimple.war 
##To view deployed apps
./asadmin list-applications  MyCluster

Custer Undeploy

./asadmin undeploy --target MyCluster VerySimple

Quick Test of Cluster

Open some browsers and test 2 URLS:
Test Case (perform in order):
  1. open browser go to: http://localhost:28080/VerySimple/
  2. open another browser window or tab (same browser instance) go to: http://localhost:28081/VerySimple/
  3. observe screen shot #1
  4. in browser 2 refresh browser 2.
  5. observe screen shot #2
  6. in browser 1 refresh
  7. observe screen shot #3
Note: code changed slightly to include the 'local' values in the session bean and display. However, that doesnt' really change this simple cluster test that doesn't use a LB.

Screen Shot #1
Screen Shot #2
Screen Shot #3

Shot #1: Note session id is shared! Note current session value ends with 80. Recall from above, the existing session value is saved, then changed to match the current server, and then the previous value was returned.

Shot #2: Notice that session value is 81 in browser 2. That is the value of the 'previous' server to touch the value.

Shot #3: Notice that in browser 1 now says 81, because 81 was last person to 'get' the session value.

Now To NGINX Setup Install

Since there are some hardware load balancers in front of my real app, some sort of stand-in so I could better simulate my target environment.  So I tried NGINX.
yum install nginx
systemctl start nginx.service
##go test that it works!
systemctl stop nginx.service

NGINX HA/LB Resources

Configure NGINX to HA/LB Over '80' and '81'

Make your nginx.conf file look like this (and restart NGINX):
user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log;
#error_log  /var/log/nginx/error.log  notice;
#error_log  /var/log/nginx/error.log  info;
pid        /run/;
events {
    worker_connections  1024;

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    #keepalive_timeout  0;
    keepalive_timeout  65;
    #gzip  on;
    include /etc/nginx/conf.d/*.conf;
    server {
        listen       80;
        server_name  localhost;
        #access_log  /var/log/nginx/host.access.log  main;
 location /VerySimple/ {
  proxy_pass http://my_cluster/VerySimple/; 
  proxy_redirect  http://my_cluster http://localhost/VerySimple/; 
    upstream my_cluster  {
 server;  ###UPDATE IP!!!
 server;  ###UPDATE IP!!!

Oh and BTW SELINUX and NGINX may be allergic to each other. And/Or it maybe SELINUX and stuff on port 80 has issues.    Disabling SELINUX helped. If I had to guess I think yumming up apache; may have done a few SELINUX tweaks to make using port 80 a little less sensitive to SELINUX issues; YMMV.

A Quick Test with NGINX as LB

Open up a browser and test http://localhost/VerySimple/ Press refresh a few times and notice that basically it will RR (round robin). Look at the screen shots.  Basically examine the local port and local name and the host session val and notice how it changes after each refresh.  Observe in following pictures. 

HAPROXY (Because NGINX Doesn't Stick)

NginX doesn't have session affinity w/o some plugins (does have some rudimentary ip_hash routing; but not true affinity) , but due to some pot-holes I had set up HAPROXY; which does have basic session affinity. So quickly here is HAPROXY NowTo:
yum install haproxy
##hack up haproxy.cfg to look like the sample below
systemctl start haproxy.service 
Sample haproxy.cfg:
 log local2
 chroot      /var/lib/haproxy
 pidfile     /var/run/
 maxconn     4000
 user        haproxy
 group       haproxy
        log global
        mode http
        option httplog
        option dontlognull
        retries 3
        option redispatch
        maxconn 2000
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
frontend http-in
        bind *:80
        default_backend servers
backend servers
        option httpchk OPTIONS /
        option forwardfor
        option http-server-close
        appsession JSESSIONID len 52 timeout 3h
        server ClusterInstance1 localhost:28080 check inter 5000
        server ClusterInstance2 localhost:28081 check inter 5000

And Then The HA Proxy Test

Goto http://localhost/VerySimple.  Refresh and notice Server Local Port is sticking to a specific port. Also notice that the session value isn't changing.

Now tear down a cluster instance and re-test.

If the local port was '80' use:  ./asadmin stop-instance ClusterInstance1

Otherwise use:  /asadmin stop-instance ClusterInstance2
## to restart an instance
##./asadmin start-instance ClusterInstance[1|2]
Examine the session IDs, local port value and session value and watch for things to change. Fail over works just as expected. Fail back or restarting, for example instance1, seems to start a new session, not clear if HAProxy on GF is behind that. Investigating....but resolving that maybe different task; off the radar for now.


  1. hello i tried your tutoriel but i got a problem with the session id. on the same browser it does not show the same id changing the instance on the same cluster

    Thank you for your help

  2. This comment has been removed by the author.