Logging real remote IPs in Tomcat logs with Google Load Balancer

As part of our technical support operations, we have noticed that our customer’s Tomcat server on Google Cloud Platform is not using the correct remote IP address in the access log.

This was, of course, due to the fact it was behind Google’s fabulous load balancer, the load balancer appends the client remote IP address to the X-Forwarded-For Http header.

It took us a few hours to figure out the correct Tomcat configuration so we have decided to post this publicly so you hopefully won’t need to spend as much time on this as we did:

We wanted to turn this:

remoteHostname: 130.211.2.139 // Internal load balancer address
x-fwd-for= 192.115.200.197, 35.186.199.42

Into this:

remoteHostname: 192.115.200.197 // Real client IP
x-fwd-for= 192.115.200.197, 35.186.199.42

The class “RemoteIpValve” replaces the apparent client remote IP address and hostname for the request with the IP address list presented by a proxy or a load balancer via a request headers (e.g. “X-Forwarded-For”).

How does this work?

• RemoteIpValve Loops on the comma delimited list of IPs and hostnames passed by the preceding load balancer or proxy in the given request’s Http header named $remoteIpHeader (default value x-forwarded-for). Values are processed in right-to-left order.
• For each ip/host of the list in internalProxies:
• if it matches the internal proxies list, the ip/host is swallowed/removed
• otherwise, the ip/host is declared to be the remote ip and looping is stopped.

Now we just need come up with the “internalProxies” regex to use the following IP range: 130.211.0.0/22 and 35.191.0.0/16

This range is published by google here:

https://cloud.google.com/load-balancing/docs/https/#firewall_rules

"130\.211\.\d{1,3}\.\d{1,3}|35\.191\.\d{1,3}\.\d{1,3}"

We also need to add our load balancer external IP, since it is added to the X-Forwarded-For header.

In our case it is : 35.186.199.42

The resulting regex is:

"130\.211\.\d{1,3}\.\d{1,3}|35\.191\.\d{1,3}\.\d{1,3}|35\.186\.199\.42"

The final result should be something like this:

<Valve className="org.apache.catalina.valves.RemoteIpValve"
internalProxies="130\.211\.\d{1,3}\.\d{1,3}|35\.191\.\d{1,3}\.\d{1,3}|35\.186\.199\.42"
 />

<Valve className="org.apache.catalina.valves.AccessLogValve"
 directory="logs"
 prefix="access_log" 
 suffix=".txt"
 pattern="combined"
 requestAttributesEnabled="true" />

Now, just add this to your server.xml and restart the Tomcat service.

Happy logging :)