From DevRandom

TCP timers and keepalives

Netstat -o includes tcp timers which are useful for Apache keepalive analysis.

netstat -ntpo | grep ESTAB | egrep ":80|:443"

tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.7:41086   ESTABLISHED 19380/httpd         keepalive (7196.19/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.7:41089   ESTABLISHED 25812/httpd         keepalive (7196.71/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.7:41088   ESTABLISHED 19385/httpd         keepalive (7196.71/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.45:41090  ESTABLISHED 26995/httpd         keepalive (7196.71/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.37:41087  ESTABLISHED 25813/httpd         keepalive (7196.70/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.7:41088   ESTABLISHED 19385/httpd         keepalive (7196.71/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.27:41090  ESTABLISHED 26995/httpd         on (7196.71/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.17:41088  ESTABLISHED 19385/httpd         off (7196.71/0/0)
tcp        0      0 ::ffff:98.xxx.xxx.106:80    ::ffff:83.138.139.17:41090  ESTABLISHED 26995/httpd         off (7196.71/0/0)

The last column denotes what the connection is doing.

• 'on' - Actively transfering data.

• 'off' - Currently disconnecting

• 'keepalive' - Connections are using TCP keepalives. The first number denotes the time in seconds from when the last data was transferred until when the next TCP keepalive probe will be sent. By default this starts at 7200s, and resets again every time more data is sent. If the value is low , for eg. 4000 seconds , it means some of the keep alive connections are hanging or doing nothing for a long period. Note, connections to internal proxy or other internal processes might hang longer but this should not happen to web based connection.

The defaults are dictated by the sysctl values.

• net.ipv4.tcp_keepalive_probes - How many keepalive probes TCP sends out, until it decides that the connection is broken. Default value: 9.

• net.ipv4.tcp_keepalive_time - How often TCP sends out keepalive messages when keepalive is enabled. Default: 2hours (7200 seconds)

• net.ipv4.tcp_keepalive_intvl - How frequently the probes are send out. Multiplied by tcp_keepalive_probes it is time to kill not responding connection, after probes started. Default value: 75sec i.e. connection will be aborted after ~11 minutes of retries.

More information on the sysctly values can be found in the kernel documentation /usr/share/doc/kernel-doc-<VERSION>/Documentation/networking/ip-sysctl.txt