IPSEC: Implementing Server-to-Server Encryption of TCP/IP Connections



Under a variety of circumstances, it is desirable to encrypt data between multiple computers. One of the beauties of open source is that there is a multiplicity of answers. Unfortunately, with those answers comes some confusion. Often times, people overlook some solutions, and one of those solutions is the use of IPSEC transport mode for server to server communications.

Target Audience

IPSEC is an advanced security topic. Unlike firewalls and other Linux security topics, understanding IPSEC requires at least familiarity with TCP/IP, the kernel, the TCP/IP stack, the OSI model, cryptography, etc. Therefore this article is targeted at users interested in “getting their feet wet.” This article is by no means authoritative or complete.

IPSEC: Reader’s Digest short explanation

Short for IP security, Internet Protocol Security is one of the many standards for secure communications on IP Networks. Standardized under several RFC specifications. IPSEC has the ability to use several different cryptographic systems. IPSEC is a mandatory part of IPV6. IPSEC can be used to create tunneled Under some cases, it is desirable to encrypt data between computers. Much has been written on VPNs, but few know about IPSEC and how easy it is to create a private VPN. This feature article describes how to create a private VPN between two computers using the existing IP infrastructure. VPNs, configured for Road Warrior mode, and transport mode.


IPSEC in transport mode has some serious advantages over other solutions. Compared to other technologies, IPSEC is built into to the Linux kernel. In other words there is no daemon running in the background. Better yet, IPSEC does not require port-forwarding; some people elect to use SSH, stunnel, and other technologies that rely on port forwarding. With IPSEC, you simply have to run a program and it’s configuration file. After running it, encryption between hosts is mandatory. Connections will be refused if the other connection does not have the appropriate keys. Groups of computers can share the same key, and it can even be done on a per-port setting (for example securing VNC, etc).


IPSEC in transport mode does have a couple draw backs. In transport mode you cannot have any dynamic setups where the IP addresses change from time to time. In other words, IPSEC is usually insufficient for workstation environments or dynamically assigned networks. Also, if you want to do a per-port setup the configuration becomes harder.

Security Implications

A very astute user can use IPSEC to bypass firewalls and other security measures. Since IPSEC uses cryptography, information is passed between machines in encrypted format. If the keys are not known, there is no practical way to decrypt the information (it is virtual impossible due to the sheer amount of time it would take).

Machine-to-Machine IPSEC installations should be considered as Virtual Private Networks (VPNs) for security considerations. Please check with your system administrator, business policies, and laws and regulations of your locality in order to establish whether or not to institute IPSEC.


  • ipsec-tools package
  • static IP addresses for each machine

Configuration file

The configuration file, /etc/setkey.conf, contains the information about the IPSEC policy. Below is a sample configuration policy (i.e. don’t implement this policy because it is insecure).

Example file


	add ah 0x200 -A hmac-md5
	add ah 0x300 -A hmac-md5

	add esp 0x201 -E aes-cbc
	add esp 0x301 -E aes-cbc

	spdadd any -P out ipsec

	spdadd any -P in  ipsec

Sections Explained


The first part of the configuration file simply flushes the keys and then the policy’s.

	add ah 0x200 -A hmac-sha-256
	add ah 0x300 -A hmac-sha-256

	add esp 0x201 -E aes-cbc
	add esp 0x301 -E aes-cbc

These lines are the actual keys and the encryption that will be used. The first block has the keys that will be used for authentication. In this case, it is the “hmac-md5” algorithm. The second block contains the keys that will be used for privacy, and the method of encryption. In the example, AES-CBC will be used, which is probably stronger than should be required; the key that we will be using is 194bits, meaning that it is good enough for US Government Secret and below classifications.

	spdadd any -P out ipsec

	spdadd any -P in  ipsec

The final block includes the actual policy. This is where you can put port numbers and even define whether it will be TCP or UDP.

Generating the keys

The more random the key, the better. Obviously, the example above is insufficient to secure a network. The following command will generate a random key. While running this command, you’ll need to wiggle the mouse to make it run faster. Or, if you are using a terminal use /dev/urandom instead.

	dd if=/dev/random count=16 bs=1| xxd -ps

Depending on the size of the key that you want, adjust the count (16 will produce a 128 bit key, 24 will produce a 196 bit key, and 32 will produce a 512 bit key)

The size of the key is important. If you really paranoid or just have CPU cycles to burn on cryptography, use a 256 bit key. Generally speaking 256-bit encryption is becoming the standard for sensitive data. For most applications, it is overkill. In this example, we are using the AES-CBC cipher which is used by the US Government. At the time of this writing AES 128 bit-key lengths is the defacto standard for anything that is “Secret” or below, while AES 196 or 256 bit-key lengths are required for “Top Secret.” Needless to say, the choice of the cryptographic strength and the cipher is largely your choice.

For the tinfoil hat crowd, the following are viable options for the cipher and the usable strengths:


algorithm keylen (bits)
des-cbc 64 esp-old: rfc1829, esp: rfc2405
3des-cbc 192 rfc2451
null 0 to 2048 rfc2410
blowfish-cbc 40 to 448 rfc2451
cast128-cbc 40 to 128 rfc2451
des-deriv 64 ipsec-ciph-des-derived-01
3des-deriv 192 no document
rijndael-cbc 128/192/256 rfc3602 (also known as aes-cbc)
twofish-cbc 0 to 256 draft-ietf-ipsec-ciph-aes-cbc-01
aes-ctr 160/224/288 draft-ietf-ipsec-ciph-aes-ctr-03


Loading the Config

After you have hashed out the configuration file, you can load it using the “setkey -f /etc/setkey.conf” command. If there are any errors, it will tell you. Otherwise, after the command has been run, it is immediately active. Any connection to or from the machine in question that meets a rule will have to be appropriately encrypted.

If you should find that you have rendered your system unable to connect to the world, type “setkey -FP.”


The testing is rather simple — if after loading the configuration file you can access the other server, then it worked. Otherwise, there is a problem. If you are particularly interested to see that the data is indeed being encrypted, a simple tcpdump will show it to you:

# tcpdump -i eth0  | egrep "AH|ESP"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
18:52:37.459643 IP bhoward1 > bhoward2: AH(spi=0x00000200,seq=0x2a): ESP(spi=0x00000201,seq=0x2a), length 104
18:52:37.459761 IP bhoward2 > bhoward1: AH(spi=0x00000300,seq=0x2a): ESP(spi=0x00000301,seq=0x2a), length 104
18:52:38.463752 IP bhoward1 > bhoward2: AH(spi=0x00000200,seq=0x2b): ESP(spi=0x00000201,seq=0x2b), length 104
18:52:38.463899 IP bhoward2 > bhoward1: AH(spi=0x00000300,seq=0x2b): ESP(spi=0x00000301,seq=0x2b), length 104
18:52:39.467689 IP bhoward1 > bhoward2: AH(spi=0x00000200,seq=0x2c): ESP(spi=0x00000201,seq=0x2c), length 104
18:52:39.467809 IP bhoward2 > bhoward1: AH(spi=0x00000300,seq=0x2c): ESP(spi=0x00000301,seq=0x2c), length 104
18:52:40.471733 IP bhoward1 > bhoward2: AH(spi=0x00000200,seq=0x2d): ESP(spi=0x00000201,seq=0x2d), length 104
18:52:40.471841 IP bhoward2 > bhoward1: AH(spi=0x00000300,seq=0x2d): ESP(spi=0x00000301,seq=0x2d), length 104
18:52:41.475738 IP bhoward1 > bhoward2: AH(spi=0x00000200,seq=0x2e): ESP(spi=0x00000201,seq=0x2e), length 104
18:52:41.475841 IP bhoward2 > bhoward1: AH(spi=0x00000300,seq=0x2e): ESP(spi=0x00000301,seq=0x2e), length 104
140 packets captured
284 packets received by filter
0 packets dropped by kernel

If you see the AH(…) and the ESP(…) lines, then the information is encrypted. In this case it was a simple ping operation.

Boot start

In order to load the rules at boot, add the following line to /etc/init.d/boot.local:

	# /etc/init.d/boot.local
	/usr/sbin/setkey -f /etc/setkey.conf

/etc/init.d/boot.local will start before the network connections are activated.

Adjust your firewall

Since IPSEC will only encrypt traffic from one IP address to another, you may need to change you firewall rules to reflect this behavior. For example, if an application is listening on 5901 (VNC) and all the machines that will access the ports need have IPSEC, it would be advisable to shut out everyone except the IP addresses you want. Otherwise a user could connect from a non-IPSEC secured machine and then transmit the data unencrypted.

In other words, IPSEC is only part of the equation.


If you are using a desktop IPSEC client like Raccoon you may have your rules cleared. You may need to edit your configuration files in order to include the rules.

Side note

On a practical level IPSEC is great to secure applications with out additional overhead. At the time of writing this, I had three problems:

  • Securing several hosts with out using a full-fledged VPN solution like OpenVPN
  • Running Synergy
  • Securing VNC
  • Concern for telnet

Obviously, the first item was an outcrop of the last three. IPSEC soon became the best choice for me. Other people might find that solutions like “stunnel” and ssh will work for them. The biggest issue was that I wanted something that would just work. I didn’t want something that I would have to monkey around or have to drop into a super-user in order to fix a problem here and a problem there. I found IPSEC to be faster than any of the other solutions although I don’t have hard numbers on it.

(Visited 1 times, 1 visits today)

Leave a Reply

Your email address will not be published. Required fields are marked *

No comments yet