Monday, October 29, 2012

Glassfish 3.1.x Behind SSL Terminating Load Balancer

What's The Problem 

I am building a SSL only JSF 2.x web application that sits behind a SSL terminating load balancer and an Apache/mod_proxy instance.  Currently neither Apache nor GF has any idea that the original requests were https, and the load balancer will do an initial http ->https redirect when the originating request was not ssl.   Mostly this is OK; currently I don't have technical or business requirements that requires that the application to know that requests were https.  But...

...I am using Apache CODI to provide some CDI extensions.   CODI has this nice feature enabled by default that provides session like behavior but on a browser window or tab basis.   It will do this by appending 'windowId' parameter to all requests and this windowId will be used to qualify other beans in other scopes when they are accessed.  

And now the problem:   CODI will generate a redirect on absence of the windowId and since GF doesn't know a thing about https, nor does CODI, and it will generate a windowId redirect with http instead of https; thus causing an extra redirect...

A Bunch of Links

Before cataloging some solutions, here are some interesting links that are somewhat related to this or that I found as I was researching my options:

Turn Off WindowID Stuff In CODI

Turn off this feature and the windowId redirect goes away, right?  Yep.  CODI's documentation doesn't spell  out how to do this in black and white.  But after a bit of research I found that you provide an extension of  
org.apache.myfaces.extensions.cdi.core.api.scope.conversation.config.WindowContextConfig, annotate it with @Specializes, and include in WAR.   

This seems to work OK, and here is my sample:

package fhw;

import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.inject.Specializes;
import org.apache.myfaces.extensions.cdi.core.api.scope.conversation.config.WindowContextConfig;

public class MyWinConfig
    extends WindowContextConfig
    public boolean isAddWindowIdToActionUrlsEnabled()
        return false;

    public boolean isUrlParameterSupported()
        return false;

By doing this, I am pretty sure the window/tab based session behavior is disabled.    I have tested the above config specialization and the windowId redirect goes away (as does parameter addition).  I have not done any testing or tinkering to see if CODI does some other magic to continue to support window context stuff without the windowId parameter addition.

Use a Servlet Filter To Wrap Request

This should be pretty simple:  write a filter to wrap the inbound request with an extension that overrides a few methods so that downstream code will see https instead of http.     I haven't written a test app yet for this, but seems like it should work; I hope to whittle up a test shortly


This header actually exists specifically for communicating the originating protocol deeper in the application stack (i.e. my problem almost exactly).   Basically,
  • set up the ssl terminator to set the X-Forwareded-Proto after de-ssling.  This header will contain the original protocol.
  • set up or make sure Apache passes this on too
  • set up or make sure that GF 'sees' this value and instruments the dispatched http/s request to indicate the value implied by X-Forwarded-Proto
Assuming all the layers will pay attention to this stuff, it would be a matter of just setting it up. 

Glassfish Settings and Tweaks

There are a few settings in GF that may help:
  • Auth Pass Through  (authPassThrough) -- This http-listener switch is specifically for GF instances behind SSL terminators.
  • scheme-mapping in the http definition of domain.xml.   

GF has bugs!

The trouble with AuthPassThrough and a few other GF things related to early SSL termination is that it doesn't work and/or is buggy (in version <3.1.1 which I am using).  Here is a list of bugs related to this:

Apache Configuration to Rewrite Redirects

It also should be possible to teach Apache to rewrite redirects only.

Glassfish ProxyHandler Extension

Basically provide an extension of com.sun.appserv.ProxyHandler and have this implementation do whatever it can to influence the request to look like HTTPS.    This is a good resource to follow up on:

Hmm, there is more to learn on this subject,  turns out that this may work in conjunction with authPassThrough.

Load Balancer/SSL Terminator Rewrite Redirects

I wonder if we can teach the LBs to re-write 302 and 301 requests to be https rather the http?

Switch Apache to GF Connection to JK

ModJK is a solid alternative to mod-proxy, when the traffic is targeted to a Tomcat, Glassfish or other application server.   ModJK has all sorts of tuning and switches and maybe there is something that help expose the terminated SSL.    (Depending on who you ask, ModJK is 'better' than just mod-proxy.)

Swap Apache out for NGINX

NGINX has a strong reputation of being better reverse proxy and faster content server than Apache (while consuming less resources).  NGINX doesn't change anything related to SSL termination; it wouldn't know more or less than Apache currently knows.    However NGINX has lots more proxy features and rewriting features than Apache.   I don't really view this as a solution to this issue, but rather if NGINX has some additional value to my project; then it may have a few easier steps/tweaks to dealing with the redirect issues.

Turns out NGINX has some support for this sort of thing, via some "proxy_redirect http:// https://" like setting.  Maybe worth exploring.

I Need to Proto-type and Test Some Things Out

Due situations beyond my control; the importance of this has be 'de-emphasized'.   However, I'd still like to solve it; so I will chip away at it from time to time.  

The first thing I need is a work bench or test environment.   So I setup the following on my chaos 1212:

The nginX instance handled the SSL termination; it acted as a stand-in for my real project's HW based SSL terminator.   The apache instance was just configured much like my project's real apache.   And, GF will run a war to simulate my application.   I used NetBeans 7.2 to generate a simple JSF/PF web app and then just tinkered with the code some and added CODI to the web short order I had a working simulation.

Test 1 -- success in reproducing the problem!

  • nginx config and ssl config
  • gf.conf in my /etc/httpd/conf.d dir (i.e. apache conf to forward to gf; apache listens on 8000)
  • glassfish (nothing much of interest here; just http listening on 8080)
  • my test war
So, if you look at the next set of images, you can see the extra redirects that GF and CODI are causing in absence of knowing the real protocol.    I captured these using chrome's debugging tools.   The original request was http://localhost/CodiFun.
Initial Request and nginX's redirect response

Glassfish redirection to index page of war (see the http!)
nginx corrects the http to https

CODI and JSF redirection to inject session id and window id (there is http again!
nginX fixing the problem again


Test 2 - NGINX Http/Https Rewrite

While this test (or solution) doesn't really help me; my load balancers are F5/BigIP appliances.  It does show a neat nginx feature:  proxy rewrites of redirects from downstream servers.   Notice the only change from Test 1, is enabling the proxy_redirect directive in the ssl.conf file. 

Here is the chrome 'net-dump' for the same url as above:
Notice the rewritten response from GF for the index page has been SSL by nginx

Test 3 - GF's authPassThrough

  • turn off nginx redirect/rewirte (back to Test 1 state)
  • turn on GF auth pass through enabled  (changed http-listener-1, the one listening on 8080, to have auth-pass-through-enabled="true" )
The test got the same results as Test 1.  Basically I expected this, as nginx wasn't really passing any terminating information to apache much less GF.

Test 4 - GF's authPassThrough and nginx headers

  • like test 3 but have nginx add some X-Forwarded headers for downstream servers to use; here is the ssl.conf file enabling this. 
I had hoped this would work;  It didn't.  Here is the Chrome network traces including a dump of request headers that the JSF war received:

Note the x-forwarded-proto and the location redirect

Test 5 - scheme-mapping="X-Forwared-Proto"

Some success!!!  But not quite right!    I turned off out the auth pass through on the http listener and added the scheme-mapping directive to my domain.xml in GF and got this:

Got rid of the bogus http redirect; but GF of JSF/Codi specified the wrong port!

Test 6 -- Success with Apache mod_header!

It has been a couple of weeks since I visited this problem.  Real life needed attention and it took me a bit of searching to find how mod_header can be used to help with this problem.

The test config was just like the above and:
  • gf.conf was doctored to edit location headers from a redirects.
With this tweak, I got this:
True Success!

In Sum

Well the mod_headers/edit tweak for the time being, solves my issue.  Further I can apply this without a GF upgrade.    The final header edit directive in my gf.conf file turned out like this:

Header edit Location  ^http://(.*):8080(.*) https://$1$2

This directive cared for both the port and the protocol in the redirects from glassfish.  

No comments:

Post a Comment