Websockets support for Apache httpd 2.2.15 and Tomcat 7.0.27

Objective

We have poor performance using http via apache mod_jk to talk to tomcat. Upon researching different options such as comet, ajp, long polling, websockets appeared to be the most efficient method to communicate with tomcat. The issue we encountered in our production environment is that the software release is set to specific version and we need to have it support websockets.

Websocket Basics

Websocket connection starts out as a ws or wss protocol request from a client such as browser:

Request URL:ws://server:8080/examples/websocket/chat

Server Responses and makes websocket connection:

Status Code: 101 Switching Protocols

Client Header:

connection: upgrade
Date: Fri, 21 Mar 2014 15:06:58 GMT
Sec-WebSocket-Accept: XQBn52BiXRMiO0t5qCdQ+v8V9nM=
Server: Apache-Coyote/1.1
Transfer-Encoding: hunked
upgrade: websocket

Environment

Centos 6.3 x86_64
Apache httpd-2.2.15
Java 1.6.0_31 Download
Tomcat 7.0.27 Download

We will install all the software on hostname "server"

Tomcat 7.0.27

As of version 7.0 tomcat has built in support and example application of websocket.

Install Tomcat

tar -xvf apache-tomcat-7.0.27.tar.gz
cd apache-tomcat-7.0.27/bin
sh startup.sh

I recommend using chrome as the test browser. It has tools to debug the connection. Click Customize and Control Google Chrome > Tools > Developer tools

Testing websockets in tomcat

http://server:8080/examples/websocket/chat.html

On the server, you can see the detail package by using tcpdump

tcpdump -X port 8080 and greater 100

14:27:28.134980 IP client.58673 > server.webcache: Flags [P.], seq 3090567567:3090568050, ack 2992001024, win 16458, length 483
        0x0000:  4500 020b 3c31 4000 7f06 9c66 0a00 06fa  E...<1@....f....
        0x0010:  0a00 065c e531 1f90 b836 518f b256 5000  ...\.1...6Q..VP.
        0x0020:  5018 404a 373e 0000 4745 5420 2f65 7861  P.@J7>..GET./exa
        0x0030:  6d70 6c65 732f 7765 6273 6f63 6b65 742f  mples/websocket/
        0x0040:  6368 6174 2e68 746d 6c20 4854 5450 2f31  chat.html.HTTP/1
        0x0050:  2e31 0d0a 486f 7374 3a20 3130 2e30 2e36  .1..Host:.10.10.6
        0x0060:  2e39 323a 3830 3830 0d0a 436f 6e6e 6563  .192:8080..Connec
        0x0070:  7469 6f6e 3a20 6b65 6570 2d61 6c69 7665  tion:.keep-alive
        0x0080:  0d0a 4361 6368 652d 436f 6e74 726f 6c3a  ..Cache-Control:
        0x0090:  206d 6178 2d61 6765 3d30 0d0a 4163 6365  .max-age=0..Acce
        0x00a0:  7074 3a20 7465 7874 2f68 746d 6c2c 6170  pt:.text/html,ap
        0x00b0:  706c 6963 6174 696f 6e2f 7868 746d 6c2b  plication/xhtml+
        0x00c0:  786d 6c2c 6170 706c 6963 6174 696f 6e2f  xml,application/
        0x00d0:  786d 6c3b 713d 302e 392c 696d 6167 652f  xml;q=0.9,image/
        0x00e0:  7765 6270 2c2a 2f2a 3b71 3d30 2e38 0d0a  webp,*/*;q=0.8..
        0x00f0:  5573 6572 2d41 6765 6e74 3a20 4d6f 7a69  User-Agent:.Mozi
        0x0100:  6c6c 612f 352e 3020 2857 696e 646f 7773  lla/5.0.(Windows
        0x0110:  204e 5420 362e 313b 2057 4f57 3634 2920  .NT.6.1;.WOW64).
        0x0120:  4170 706c 6557 6562 4b69 742f 3533 372e  AppleWebKit/537.
        0x0130:  3336 2028 4b48 544d 4c2c 206c 696b 6520  36.(KHTML,.like.
        0x0140:  4765 636b 6f29 2043 6872 6f6d 652f 3333  Gecko).Chrome/33
        0x0150:  2e30 2e31 3735 302e 3135 3420 5361 6661  .0.1750.154.Safa
        0x0160:  7269 2f35 3337 2e33 360d 0a41 6363 6570  ri/537.36..Accep
        0x0170:  742d 456e 636f 6469 6e67 3a20 677a 6970  t-Encoding:.gzip
        0x0180:  2c64 6566 6c61 7465 2c73 6463 680d 0a41  ,deflate,sdch..A
        0x0190:  6363 6570 742d 4c61 6e67 7561 6765 3a20  ccept-Language:.
        0x01a0:  656e 2d55 532c 656e 3b71 3d30 2e38 0d0a  en-US,en;q=0.8..
        0x01b0:  4966 2d4e 6f6e 652d 4d61 7463 683a 2057  If-None-Match:.W
        0x01c0:  2f22 3338 3535 2d31 3333 3332 3035 3131  /"3855-133320511
        0x01d0:  3230 3030 220d 0a49 662d 4d6f 6469 6669  2000"..If-Modifi
        0x01e0:  6564 2d53 696e 6365 3a20 5361 742c 2033  ed-Since:.Sat,.3
        0x01f0:  3120 4d61 7220 3230 3132 2031 343a 3435  1.Mar.2012.14:45
        0x0200:  3a31 3220 474d 540d 0a0d 0a              :12.GMT....
14:27:28.136326 IP server.webcache > client.58673: Flags [P.], seq 1:124, ack 483, win 15544, length 123
        0x0000:  4500 00a3 28a6 4000 4006 f059 0a00 065c  E...(.@.@..Y...\
        0x0010:  0a00 06fa 1f90 e531 b256 5000 b836 5372  .......1.VP..6Sr
        0x0020:  5018 3cb8 21eb 0000 4854 5450 2f31 2e31  P.<.!...HTTP/1.1
        0x0030:  2033 3034 204e 6f74 204d 6f64 6966 6965  .304.Not.Modifie
        0x0040:  640d 0a53 6572 7665 723a 2041 7061 6368  d..Server:.Apach
        0x0050:  652d 436f 796f 7465 2f31 2e31 0d0a 4554  e-Coyote/1.1..ET
        0x0060:  6167 3a20 572f 2233 3835 352d 3133 3333  ag:.W/"3855-1333
        0x0070:  3230 3531 3132 3030 3022 0d0a 4461 7465  205112000"..Date
        0x0080:  3a20 4672 692c 2032 3120 4d61 7220 3230  :.Fri,.21.Mar.20
        0x0090:  3134 2031 383a 3237 3a32 3820 474d 540d  14.18:27:28.GMT.
14:27:28.179957 IP client.58679 > server.webcache: Flags [P.], seq 3023899341:3023899808, ack 2187644622, win 16458, length 467
        0x0000:  4500 01fb 3c34 4000 7f06 9c73 0a00 06fa  E...<4@....s....
        0x0010:  0a00 065c e537 1f90 b43d 0acd 8264 cece  ...\.7...=...d..
        0x0020:  5018 404a cd0e 0000 4745 5420 2f65 7861  P.@J....GET./exa
        0x0030:  6d70 6c65 732f 7765 6273 6f63 6b65 742f  mples/websocket/
        0x0040:  6368 6174 2048 5454 502f 312e 310d 0a55  chat.HTTP/1.1..U
        0x0050:  7067 7261 6465 3a20 7765 6273 6f63 6b65  pgrade:.websocke
        0x0060:  740d 0a43 6f6e 6e65 6374 696f 6e3a 2055  t..Connection:.U
        0x0070:  7067 7261 6465 0d0a 486f 7374 3a20 3130  pgrade..Host:.10
        0x0080:  2e30 2e36 2e39 323a 3830 3830 0d0a 4f72  .0.6.192:8080..Or
        0x0090:  6967 696e 3a20 6874 7470 3a2f 2f31 302e  igin:.http://10.
        0x00a0:  302e 362e 3932 3a38 3038 300d 0a50 7261  10.6.192:8080..Pra
        0x00b0:  676d 613a 206e 6f2d 6361 6368 650d 0a43  gma:.no-cache..C
        0x00c0:  6163 6865 2d43 6f6e 7472 6f6c 3a20 6e6f  ache-Control:.no
        0x00d0:  2d63 6163 6865 0d0a 5365 632d 5765 6253  -cache..Sec-WebS
        0x00e0:  6f63 6b65 742d 4b65 793a 2066 3862 7a62  ocket-Key:.f8bzb
        0x00f0:  532b 2f70 4437 5776 6d66 314b 6c35 6279  S+/pD7Wvmf1Kl5by
        0x0100:  673d 3d0d 0a53 6563 2d57 6562 536f 636b  g==..Sec-WebSock
        0x0110:  6574 2d56 6572 7369 6f6e 3a20 3133 0d0a  et-Version:.13..
        0x0120:  5365 632d 5765 6253 6f63 6b65 742d 4578  Sec-WebSocket-Ex
        0x0130:  7465 6e73 696f 6e73 3a20 7065 726d 6573  tensions:.permes
        0x0140:  7361 6765 2d64 6566 6c61 7465 3b20 636c  sage-deflate;.cl
        0x0150:  6965 6e74 5f6d 6178 5f77 696e 646f 775f  ient_max_window_
        0x0160:  6269 7473 2c20 782d 7765 626b 6974 2d64  bits,.x-webkit-d
        0x0170:  6566 6c61 7465 2d66 7261 6d65 0d0a 5573  eflate-frame..Us
        0x0180:  6572 2d41 6765 6e74 3a20 4d6f 7a69 6c6c  er-Agent:.Mozill
        0x0190:  612f 352e 3020 2857 696e 646f 7773 204e  a/5.0.(Windows.N
        0x01a0:  5420 362e 313b 2057 4f57 3634 2920 4170  T.6.1;.WOW64).Ap
        0x01b0:  706c 6557 6562 4b69 742f 3533 372e 3336  pleWebKit/537.36
        0x01c0:  2028 4b48 544d 4c2c 206c 696b 6520 4765  .(KHTML,.like.Ge
        0x01d0:  636b 6f29 2043 6872 6f6d 652f 3333 2e30  cko).Chrome/33.0
        0x01e0:  2e31 3735 302e 3135 3420 5361 6661 7269  .1750.154.Safari
        0x01f0:  2f35 3337 2e33 360d 0a0d 0a              /537.36....
14:27:28.180593 IP server.webcache > client.58679: Flags [P.], seq 1:222, ack 467, win 15544, length 221
        0x0000:  4500 0105 b61e 4000 4006 627f 0a00 065c  E.....@.@.b....\
        0x0010:  0a00 06fa 1f90 e537 8264 cece b43d 0ca0  .......7.d...=..
        0x0020:  5018 3cb8 224d 0000 4854 5450 2f31 2e31  P.<."M..HTTP/1.1
        0x0030:  2031 3031 2053 7769 7463 6869 6e67 2050  .101.Switching.P
        0x0040:  726f 746f 636f 6c73 0d0a 5365 7276 6572  rotocols..Server
        0x0050:  3a20 4170 6163 6865 2d43 6f79 6f74 652f  :.Apache-Coyote/
        0x0060:  312e 310d 0a75 7067 7261 6465 3a20 7765  1.1..upgrade:.we
        0x0070:  6273 6f63 6b65 740d 0a63 6f6e 6e65 6374  bsocket..connect
        0x0080:  696f 6e3a 2075 7067 7261 6465 0d0a 5365  ion:.upgrade..Se
        0x0090:  632d 5765 6253 6f63 6b65 742d 4163 6365  c-WebSocket-Acce
        0x00a0:  7074 3a20 317a 4973 3474 7a2f 7073 5149  pt:.1zIs4tz/psQI
        0x00b0:  6f57 7035 774d 686d 3237 4a72 4465 343d  oWp5wMhm27JrDe4=
        0x00c0:  0d0a 5472 616e 7366 6572 2d45 6e63 6f64  ..Transfer-Encod
        0x00d0:  696e 673a 2063 6875 6e6b 6564 0d0a 4461  ing:.chunked..Da
        0x00e0:  7465 3a20 4672 692c 2032 3120 4d61 7220  te:.Fri,.21.Mar.
        0x00f0:  3230 3134 2031 383a 3237 3a32 3820 474d  2014.18:27:28.GM
        0x0100:  540d 0a0d 0a

Event summary

client : get /examples/websocket/chat.hmtl
server : 304
client : get /examples/websocket/chat, upgrade:websocket
server : 101 Switching procotol 

Tomcat logs : localhost_access_log
server_ip - - [21/Mar/2014:15:51:02 -0400] "GET /examples/websocket/chat.html HTTP/1.1" 304 -
server_ip - - [21/Mar/2014:15:51:02 -0400] "GET /examples/websocket/chat HTTP/1.1" 101 -

 

Apache httpd server

Now we established that tomcat websockets works, we will add the additional layer of the http server. Websockets was implemented in Apache version 2.4.5 but we only have version 2.2.15. We will need to build httpd apache proxy_wstunnel from source.

Build backport from source

Reference : https://gist.github.com/vitkin/6661683

svn co http://svn.apache.org/repos/asf/httpd/httpd/tags/2.2.15 httpd-2.2.15

wget https://gist.github.com/vitkin/6661683/raw/873dd8b4de4ad1ff69757ffe48fc574374aedc57/apache-2.2-wstunnel.patch
cd httpd-2.2.15
patch -p1 -i ../apache-2.2-wstunnel.patch
svn co http://svn.apache.org/repos/asf/apr/apr/branches/1.5.x/ srclib/apr
svn co http://svn.apache.org/repos/asf/apr/apr-util/branches/1.5.x/ srclib/apr-util
  #autoconf not work
./buildconf
./configure --enable-so --enable-proxy --enable-proxy_wstunnel=shared
make

./modules/proxy/.libs/mod_proxy_wstunnel.so

cp ./modules/proxy/.libs/mod_proxy_wstunnel.so /usr/lib64/httpd/modules/

Edit apache configuration to support websockets

vi /etc/httpd/conf/httpd.conf

LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

Listen 9090
    ServerName server

        ProxyPass        /    ws://localhost:8080/
        ProxyPassReverse /    ws://localhost:8080/
        ProxyPass        /    wss://localhost:8080/
        ProxyPassReverse /    wss://localhost:8080/
        ProxyPass        /  http://localhost:8080/
        ProxyPassReverse /  http://localhost:8080/

Note ws, wss need to go before http

service httpd restart

apachectl -t -D DUMP_MODULES | grep proxy
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_connect_module (shared)
 proxy_wstunnel_module (shared)

Testing websockets through apache httpd server

 http://server:9090/examples/websocket/chat.html

TroubleShooting

When implementing the apache server, apache server would not return the websocket connection

tomcat directly
client : get /examples/websocket/chat.hmtl
client : get /examples/websocket/chat, upgrade:websocket
server : 101 Switching procotol

tomcat proxy via apache
client : get /examples/websocket/chat.hmtl
server : 200
client : get /examples/websocket/chat, upgrade:websocket
server : 101 Switching procotol
server : 502 Proxy.Error

Turns out the order of the  proxypass is important

          ProxyPass        /  http://localhost:8080/
          ProxyPassReverse /  http://localhost:8080/
        ProxyPass        /    ws://localhost:8080/
        ProxyPassReverse /    ws://localhost:8080/
        ProxyPass        /    wss://localhost:8080/
        ProxyPassReverse /    wss://localhost:8080/

When I had ProxyPass http first, the request were redirected to http and not ws. Once switch http to the bottom and restart httpd, all worked fine.

References:
http://blog.cafarelli.fr/post/2013/04/26/Backporting-Apache-support-for-websockets-reverse-proxy-%28aka-getting-GateOne-to-work-behind-Apache%29

http://stackoverflow.com/questions/19522635/how-to-get-apache2-reversproxy-with-websockets-mod-proxy-wstunnel-tomcat7-run?rq=1

https://gist.github.com/vitkin/6661683

http://stackoverflow.com/questions/15443550/how-to-deploy-tomcats-example-websocket-applications