IPSec between Solaris 8 and solaris 9

IPSec setup between two systems (solaris 8)
(Also supports tunnel setup between Solaris 8 and Solaris 9)

Created by: Rakesh Gupta
Date: 02-Oct-2007
=========================================================================

To setup IPSec between the above two systems, follow the steps below:

1. Install following Encryption packages which can be downloaded from

http://www.sun.com/software/solaris/encryption/download.html

system SUNWcrman Encryption Kit On-Line Manual Pages
system SUNWcry Crypt Utilities
system SUNWcry64 Prototype package for Crypt Library (64-bit)
system SUNWcryr Solaris Root Crypto
system SUNWcryrx Solaris Root Crypto (64-bit)

2. Reboot the system (this step is necessary otherwise you will get the following error while doing a SA (Security Association):

One of entered the values is incorrect.
return message (in doaddup): Invalid argument

Note: Please make it sure that no body is working on the system you want to reboot.

3. Update each machine's /etc/hosts table to include the target machine's IP and hostname. .
Please notice ">>" (to append at the end of existing file) in the command below. Don't replace it with ">" (delete old content) as you will lose you old content in the /etc/hosts file
You can also edit the files manually to add the required info.

So, on host2

echo " " >> /etc/hosts

on host1

echo " " >> /etc/hosts

4. Go to each box and create /etc/inet/keyfile as root and chmod it to 600
we will have to exchange the keys now and the key info should be same for both the systems.

Note: 8-character random hex numbers are needed for spi
48-character random hex number is needed for ESP
32-character random hex number is needed for AH

Random Hexadecimal strings can be generated using:

od -X -A n /dev/random | head -2

(under unix shell and not under ipseckey prompt)

230a2369 64656e74 09224028 23296970
73656369 6e69742e 73616d70 6c650931

if /dev/random is not available any file with random numbers can be used to generate random hexadecimal strings using the folloing command:

od -X -A n

230a2369 64656e74 09224028 23296970
73656369 6e69742e 73616d70 6c650931

Edit the conent below with the newly generated hexadecimal strings and put the edited content in /etc/inet/keyfile on both the systems.

#####OUTBOUND TRAFFIC
#for ESP 192-bit encryption

add esp spi 0x2c928d04 \
src dst \
encr_alg 3DES \
encrkey d41fb74470271826a8e7a80d343cc5aae9e2a7f05f13730d


#for ah 128-bit encryption

add ah spi 0x6c1ac83f \
src dst \
auth_alg MD5 \
authkey e896f8df7f78d6cab36c94ccf293f031


#####INBOUND TRAFFIC
#for esp 192-bit encryption

add esp spi 0x065ce4d1 \
src dst \
encr_alg 3DES \
encrkey dd325c5c137fb4739a55c9b3a1747baa06359826a5e4358e

#for ah 128-bit encryption

add ah spi 0x1bcc21e7 \
src dst \
auth_alg MD5 \
authkey ad9ced7ad5f255c9a8605fba5eb4d2fd

5. Add the key file using ipseckey command (as root)

ipseckey flush
ipseckey -f /etc/inet/keyfile
ipseckey dump

#you should see the following output (output below is for reference only)

Base message (version 2) type DUMP, SA type AH.
Message length 136 bytes, seq=1, pid=27338.
SA: SADB_ASSOC spi=0x1923, replay=0, state=MATURE
SA: Authentication algorithm = HMAC-MD5
SA: flags=0x0 < >
SRC: Source address (proto=0/)
SRC: AF_INET: port = 0, 10.89.144.243 (vpn1).
DST: Destination address (proto=0/)
DST: AF_INET: port = 0, 10.89.144.244 (vpn2).
AKY: Authentication key.
AKY: f123bce6583132cf68dacb9fc8339d4b/128
LT: Lifetime information
CLT: 0 bytes protected, 0 allocations used.
CLT: SA added at time Wed Aug 28 16:14:15 2002
CLT: Time now is Wed Aug 28 16:14:16 2002

6. Create the ipsecinit.conf file as root(if doesn't exist)

cp etc/inet/ipsecinit.sample /etc/inet/ipsecinit.conf

#add these lines at the end of the file on

{saddr daddr } apply {auth_algs MD5 encr_algs 3DES sa shared}
{saddr daddr } permit {auth_algs MD5 encr_algs 3DES sa shared}

#add these lines at the end of the file on

{saddr daddr } apply {auth_algs MD5 encr_algs 3DES sa shared}
{saddr daddr } permit {auth_algs MD5 encr_algs 3DES sa shared}

7. Use the conf file created/modified above (as root)

ipsecconf -f
ipsecconf -qa /etc/inet/ipsecinit.conf

8. A simple startup script can be used to ensure that the IPsec SA database survives across a reboot:

cat /etc/init.d/ipseckey

# Simple boot script to ensure that the IPsec security association (sa)
# database survives across reboots.

if [ -f /etc/inet/keyfile -a -f /etc/inet/ipsecinit.conf ]; then
/usr/sbin/ipseckey -f /etc/inet/keyfile
fi
#save and exit from script

#change directory to ../rc2.d directory
pwd

/etc/inet

cd ../rc2.d

#Create a link in /etc/rc2.d directory

ln -s ../init.d/ipseckey S99ipseckey

ls -l S99ipseckey
-rw-r--r-- 2 root other 225 Aug 28 16:18 S99ipseckey

9. To test the encryption is working as expected. Use snoop (as root) and ENCRYPTED DATA in the output will confirm that the encryption is working.
on do :

snoop -v

AH: ----- Authentication Header -----
AH:
AH: Next header = 50 (ESP)
AH: AH length = 4 (24 bytes)
AH:
AH: SPI = 0x6c1ac83f
AH: Replay = 3
AH: ICV = 8011337ce9a0be9a9bad9843
AH:
ESP: ----- Encapsulating Security Payload -----
ESP:
ESP: SPI = 0x2c928d04
ESP: Replay = 3
ESP: ....ENCRYPTED DATA....