Encapsulating Security Payload

The sender may add from 0 to 255 bytes of padding, and by default fills it with the bytes 1, 2, 3 and so on, so the receiver can check it.

Worked example. A 20-byte payload under AES-CBC: 20 + 2 = 22 bytes of payload and trailer; the next multiple of 16 is 32; so 10 bytes of padding. Under AES-GCM, which needs only four-byte alignment: 22 rounds up to 24, so 2 bytes of padding.

ESP in the two modes

Figure 74.1 ESP encrypts what follows its header and authenticates from its header to its trailer; the IP header in front is covered by neither.

Transport mode. The ESP header follows the original IP header. The TCP or UDP segment and the trailer are encrypted; the original IP header is neither encrypted nor covered by ESP's check. An observer sees the real addresses, and nothing else.

Tunnel mode. The whole original packet, header included, is encrypted inside a new packet whose header names the two IPsec endpoints. An observer sees only that two gateways are exchanging ESP; which hosts behind them are talking, and on which ports, is hidden. This is the standard arrangement for a VPN.

The run: ESP built byte by byte

The listing assembles ESP packets exactly as RFC 4303 lays them out, with HMAC-SHA2-256-128 as the integrity check. It works the padding for the two ciphers RFC 8221 requires, then shows what an observer sees, runs ESP with the NULL cipher, and passes a packet through a NAT. The cipher itself is a stand-in, but every size is the size AES would give.

# ESP built byte by byte: padding, what an observer sees, NULL encryption, and NAT.
import hashlib, hmac, random

rng = random.Random(740)
KEY_E, KEY_A, SPI = rng.randbytes(16), rng.randbytes(32), 0x0000B002

def keystream(iv, n):                     # a stand-in for AES; the sizes below are AES's
    return b''.join(hashlib.sha256(KEY_E + iv + i.to_bytes(4, 'big')).digest()
                    for i in range(n // 32 + 1))[:n]

def esp(payload, next_header, seq, block=16, iv_len=16, cipher=True):
    """RFC 4303: SPI, sequence, [IV], payload, padding, pad length, next header, ICV."""
    pad = (-(len(payload) + 2)) % block
    trailer = bytes(range(1, pad + 1)) + bytes([pad, next_header])   # default padding 1, 2, 3
    plain = payload + trailer
    iv = rng.randbytes(iv_len) if cipher else b''
    body = iv + (bytes(a ^ b for a, b in zip(plain, keystream(iv, len(plain)))) if cipher
                 else plain)
    head = SPI.to_bytes(4, 'big') + seq.to_bytes(4, 'big')
    icv = hmac.new(KEY_A, head + body, hashlib.sha256).digest()[:16]  # HMAC-SHA2-256-128
    return head + body + icv, pad

def esp_open(packet, iv_len=16, cipher=True):
    head_body, icv = packet[:-16], packet[-16:]
    if not hmac.compare_digest(icv, hmac.new(KEY_A, head_body, hashlib.sha256).digest()[:16]):
        return None
    body = head_body[8:]
    iv, data = body[:iv_len], body[iv_len:]
    plain = bytes(a ^ b for a, b in zip(data, keystream(iv, len(data)))) if cipher else data
    pad, nh = plain[-2], plain[-1]
    return nh, plain[:-2 - pad]

# ---- 1. padding, for the ciphers RFC 8221 names ----------------------------------------
print('PADDING: payload + padding + 2 must fill whole cipher blocks, and end on 4 bytes')
print('  payload   AES-CBC (16-byte blocks)      AES-GCM (4-byte alignment only)')
for n in (1, 14, 20, 100, 1400):
    _, p_cbc = esp(bytes(n), 6, 1, block=16, iv_len=16)
    _, p_gcm = esp(bytes(n), 6, 1, block=4, iv_len=8)
    cbc = 8 + 16 + n + p_cbc + 2 + 16
    gcm = 8 + 8 + n + p_gcm + 2 + 16
    print('  %5d     pad %2d, ESP adds %2d bytes      pad %d, ESP adds %2d bytes'
          % (n, p_cbc, cbc - n, p_gcm, gcm - n))

# ---- 2. what an observer on the path sees ----------------------------------------------
tcp = (8080).to_bytes(2, 'big') + (80).to_bytes(2, 'big') + bytes(16) + b'GET /marks HTTP/1.1\r\n'
packet, _ = esp(tcp, 6, seq=7)
print()
print('WHAT AN OBSERVER SEES OF ONE ESP PACKET (%d bytes)' % len(packet))
print('  SPI %s  sequence %d  : in clear, so the receiver can find the SA and check replays'
      % (packet[:4].hex(), int.from_bytes(packet[4:8], 'big')))
print('  the TCP ports 8080 and 80 are visible   :', tcp[:4] in packet)
print('  the text "GET /marks" is visible        :', b'GET /marks' in packet)
print('  the receiver recovers protocol and data :', esp_open(packet) == (6, tcp))

# ---- 3. ESP with NULL encryption: integrity without secrecy ----------------------------
null, _ = esp(tcp, 6, seq=8, block=4, iv_len=0, cipher=False)
print()
print('ESP WITH THE NULL CIPHER (RFC 8221: MUST be implemented)')
print('  the text is visible                     :', b'GET /marks' in null)
print('  the receiver verifies and recovers it   :',
      esp_open(null, iv_len=0, cipher=False) == (6, tcp))
altered = bytearray(null)
altered[30] ^= 1
print('  one changed byte is detected            :', esp_open(bytes(altered), 0, False) is None)

# ---- 4. a NAT on the path ---------------------------------------------------------------
def outer_ip(src, dst, protocol, payload):
    h = bytearray(20)
    h[0], h[8], h[9] = 0x45, 64, protocol
    h[2:4] = (20 + len(payload)).to_bytes(2, 'big')
    h[12:16], h[16:20] = bytes(map(int, src.split('.'))), bytes(map(int, dst.split('.')))
    return bytes(h) + payload

sent = outer_ip('10.1.4.7', '203.0.113.2', 50, packet)
translated = sent[:12] + bytes([198, 51, 100, 1]) + sent[16:]      # the NAT's new source
print()
print('A NAT REWRITES THE SOURCE ADDRESS 10.1.4.7 TO 198.51.100.1')
print('  the outer header changed                :', translated[12:16] != sent[12:16])
print('  ESP still verifies and decrypts         :', esp_open(translated[20:]) == (6, tcp))
print('  (AH covers the source address, so the same NAT breaks it: the previous chapter)')

What the run establishes, in order.

Padding depends on the cipher. Under AES-CBC every payload is padded to a multiple of 16: a 1-byte payload gets 13 bytes of padding, a 20-byte payload 10, and a 14-byte payload none, because 14 + 2 is exactly 16. Under AES-GCM only four-byte alignment is needed, so padding never exceeds 3 bytes. Counting the SPI, sequence number, IV, padding, trailer and ICV, ESP adds 42 to 55 bytes under AES-CBC and 34 to 36 under AES-GCM, one reason RFC 8221 now makes GCM a MUST.

An observer learns almost nothing. Of an 88-byte packet only the SPI and the sequence number are readable. Even the TCP ports are hidden, which AH could never do; the web request is invisible; and the receiver recovers the protocol and the data exactly.

ESP with the NULL cipher is AH's job done by ESP. The text is readable, but the receiver verifies it, and a single changed byte is detected.

A NAT does not break ESP. Rewriting the outer source address leaves ESP verifying and decrypting correctly, because ESP's check starts at the ESP header. In the previous chapter the same rewrite broke AH.

ESP against AH

Why the modern world uses ESP alone

It does everything needed. Confidentiality and integrity together, or integrity alone with the NULL cipher.

It survives NAT. AH does not, and NAT is everywhere.

Combined algorithms do both jobs in one pass. AES in GCM mode encrypts and authenticates at once, producing its own integrity tag. RFC 8221 made AES-GCM a MUST for ESP; used with it, ESP needs no separate integrity algorithm.

Current algorithm requirements, RFC 8221 (October 2017):

The textbooks' ESP, with DES or Triple DES and HMAC-MD5-96, is today's MUST NOT and SHOULD NOT.

What beginners get wrong here

Saying ESP always authenticates. Integrity is a service ESP provides when it is selected; an SA without it is legal but unwise. RFC 8221 forbids the combination of a non-combined cipher with no integrity algorithm.

Saying ESP protects the IP header. In transport mode the IP header is outside ESP; in tunnel mode the outer header is. Neither is encrypted or covered by ESP's check.

Forgetting what padding is for. It completes the cipher's blocks and aligns the trailer; hiding the length is TFC's job.

Thinking the sequence number is encrypted. The SPI and sequence number must be read before decryption and travel in the clear, protected only by the integrity check.

Quoting DES and MD5 as ESP's algorithms. RFC 8221 says DES and HMAC-MD5 must not be used.

Quick revision

• ESP (RFC 4303): IP protocol 50; confidentiality, integrity, origin authentication, anti-replay, limited traffic flow confidentiality.
• Format: SPI, Sequence Number (both clear), Payload Data (with IV), Padding (0 to 255), Pad Length, Next Header, ICV.
• Integrity covers SPI to trailer; encryption covers payload and trailer; the IP header in front is covered by neither.
• Padding: block size, four-byte alignment; not for hiding length (use TFC padding). AES-CBC pads to 16; AES-GCM to 4.
• Transport hides the payload and ports; tunnel hides the whole inner packet, real addresses included.
• ESP alone: does AH's job with NULL encryption, survives NAT (RFC 3948, UDP port 4500), and AES-GCM does both jobs in one pass. ESP MUST, AH MAY.
• RFC 8221: AES-GCM, AES-CBC, NULL MUST; HMAC-SHA2-256-128 MUST; 3DES SHOULD NOT; DES and HMAC-MD5 MUST NOT.

Test yourself

1. Draw the ESP packet format and state which fields are encrypted and which are authenticated. It consists of the Security Parameters Index and the Sequence Number, each 32 bits; the Payload Data, beginning with an initialisation vector when the cipher needs one; the Padding, from 0 to 255 bytes; the 8-bit Pad Length; the 8-bit Next Header; and the Integrity Check Value. The payload and the trailer (padding, pad length and next header) are encrypted, the IV excepted. The integrity check covers everything from the SPI to the end of the trailer. The SPI and sequence number are not encrypted, and the IP header before ESP is covered by neither.

2. Give the reasons for the Padding field. To make the plaintext, payload plus padding plus pad length plus next header, a whole number of blocks for a block cipher; and to make the trailer end on a four-byte boundary so that the ICV is aligned, even when the cipher has no block size. Extra padding could also hide the payload's length, but RFC 4303 says the field is too limited for that and provides separate TFC padding.

3. How much padding does a 20-byte payload need under AES-CBC, and under AES-GCM? Under AES-CBC, 20 bytes of payload and 2 bytes of trailer make 22, and the next multiple of the 16-byte block is 32, so 10 bytes of padding. Under AES-GCM only four-byte alignment is needed, so 22 is rounded up to 24, and 2 bytes of padding are added.

4. Compare ESP with AH. ESP provides confidentiality as well as integrity, origin authentication and anti-replay, and can hide the transport protocol and ports; AH provides no confidentiality. AH's check covers the unchanging fields of the IP header in front of it, which ESP's does not. Because of that, AH fails through NAT while ESP passes through it. RFC 4301 requires ESP and makes AH optional.

5. Why is ESP now used on its own? Because it can do everything required: with the NULL cipher it provides integrity and origin authentication alone, which is AH's job; it passes through NAT, which AH cannot; and combined algorithms such as AES-GCM give encryption and integrity in one operation. RFC 8221 keeps NULL encryption mandatory for exactly this purpose.

6. What does an observer on the path see of an ESP packet in tunnel mode? The outer IP header, which names only the two IPsec endpoints, usually gateways, and the SPI and sequence number. The original packet, including the real source and destination addresses, the transport protocol, the ports and the data, is encrypted.