Virtual Fragmentation Reassembly

In this post we will look at virtual fragmentation reassembly and a few examples of where it is useful.  The topology and configurations for this example are shown below:

R1:
interface FastEthernet0/0
 ip address 10.1.1.1 255.255.255.0
!
interface Serial0/0
 ip address 10.0.12.1 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 Serial0/0


R2:
interface Serial0/0
 ip address 10.0.12.2 255.255.255.0
!
interface Serial0/1
 ip address 10.0.23.2 255.255.255.0
!
ip route 10.0.34.0 255.255.255.0 Serial0/1
ip route 10.1.1.0 255.255.255.0 Serial0/0


R3:
interface Serial0/0
 ip address 10.0.23.3 255.255.255.0
!
interface Serial0/1
 ip address 10.0.34.3 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 Serial0/0

R4:
interface Serial0/0
 ip address 10.0.34.4 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 Serial0/0


With virtual fragmentation reassembly enabled, a router holds fragments received on an interface until all fragments of a packet have been received and the packet can be reassembled.  The router can then make decisions based on the L4 and other information available in the reassembled packet that would otherwise not be available in the non-initial fragments.  The original fragments are then forwarded based on the information in the reassembled packet, and the reassmbled packet is discarded. 

 

First we will look at how this can be useful in CBAC, also known as IOS classic firewall.  R1 is set up as a TFTP and HTTP server and has a 250 KB file in flash, which R4 will download.  R3 will inspect TFTP and HTTP inbound on S0/1.  R2 S0/1 MTU is lowered to 300, causing fragmentation to occur at R2 for return traffic:

R1:
ip http server
ip http path flash:
tftp-server flash:asdf.txt


R3:
ip inspect audit-trail
ip inspect name asdf http
ip inspect name asdf tftp
!
access-list 100 deny ip any any
!
interface Serial0/0
 ip access-group 100 in
!
interface Serial0/1
 ip inspect asdf in


R2:
interface Serial0/1
 ip mtu 300


Next we will start the download on R4:

R4#copy http://10.0.12.1/asdf.txt null:
Loading http://10.0.12.1/asdf.txt !
%Error reading http://10.0.12.1/asdf.txt (Broken pipe)


The transfer fails.  Both R1 and R4 have specified a TCP MSS of 536 bytes in their first packet:

 

The rest of the first part of the exchange looks as shown below Wireshark.  R4 sends an HTTP GET to R1, and the first data packet sent by R1 is an unfragmented 292 byte packet.  After this, all packets  sent by R1 are fragmented.  The majority of them are a 304 byte initial fragment followed by a 300 byte 2nd fragment due to the MSS of 536 on R4 and MTU of 300 on R2 S0/1 (536B of data + 20B TCP header + 20B IP header = 576; fragmented by R2 into 300B and 276B; add 20B IP header to 2nd fragment and 4B HDLC to both fragments = 304B and 300B):

Let’s compare this sequence of packets that we see in Wireshark to the CBAC debug output:

R3#debug ip inspect protocol tcp
Mar 1 01:22:07.895: %FW-6-SESS_AUDIT_TRAIL_START: Start http session: initiator (10.0.34.4:21143) -- responder (10.0.12.1:80)
Mar 1 01:22:07.895: CBAC* sis 66F0A118 pak 663FE6F4 SIS_CLOSED/LISTEN TCP SYN SEQ 1282798572 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:07.895: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 48, proto=6
Mar 1 01:22:07.895: CBAC: data length for this packet is zero, returning
Mar 1 01:22:08.003: CBAC* sis 66F0A118 pak 663FE340 SIS_OPENING/SYNSENT TCP SYN ACK 1282798573 SEQ 590243577 LEN 0 (10.0.12.1:80) <= (10.0.34.4:21143)
Mar 1 01:22:08.003: CBAC* Details of Pak 663FE340 IP: s=10.0.12.1 (Serial0/0), d=10.0.34.4 (Serial0/1), len 48, proto=6
Mar 1 01:22:08.007: CBAC: data length for this packet is zero, returning
Mar 1 01:22:08.135: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPENING/SYNRCVD TCP ACK 590243578 SEQ 1282798573 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:08.139: CBAC* sis 66F0A118 http L7 inspect result: PASS packet
Mar 1 01:22:08.139: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:08.139: CBAC: data length for this packet is zero, returning
Mar 1 01:22:08.143: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590243578 SEQ 1282798573 LEN 127 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:08.143: CBAC* sis 66F0A118 http L7 inspect result: PASS packet
Mar 1 01:22:08.143: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 171, proto=6
Mar 1 01:22:08.147: CBAC: pak seq = 1282798573, rcvwnd = 4128 rcvnxt = 590243578
Mar 1 01:22:08.375: CBAC* sis 66F0A118 pak 663FE340 SIS_OPEN/ESTAB TCP ACK 1282798700 SEQ 590243578 LEN 248 (10.0.12.1:80) <= (10.0.34.4:21143)
Mar 1 01:22:08.375: CBAC* sis 66F0A118 http L7 inspect result: PASS packet
Mar 1 01:22:08.375: CBAC* Details of Pak 663FE340 IP: s=10.0.12.1 (Serial0/0), d=10.0.34.4 (Serial0/1), len 292, proto=6
Mar 1 01:22:08.375: CBAC: pak seq = 590243578, rcvwnd = 4001 rcvnxt = 1282798700
Mar 1 01:22:08.379: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590243826 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:08.379: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:08.379: CBAC: data length for this packet is zero, returning
Mar 1 01:22:08.475: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590244115 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:08.475: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:08.475: CBAC: data length for this packet is zero, returning
Mar 1 01:22:08.495: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590244115 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:08.495: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:08.499: CBAC: data length for this packet
is zero, returning
Mar 1 01:22:08.571: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590244611 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:08.571: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:08.571: CBAC: data length for this packet is zero, returning
Mar 1 01:22:08.691: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590245147 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:08.691: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:08.691: CBAC: data length for this packet is zero, returning


By comparing the source/destination address and the length shown in the debug output, it’s easy to correlate them to the Wireshark output.  The first several packets match Wireshark exactly, but after the 292 byte unfragmented packet from R1, we see only ACKs from R4.  The fragmented packets from R1 are forwarded to R4 without even being checked by CBAC as long as the L3 information matches the L3 information of the session.  This is somewhat similar to how ACLs handle IP fragments, except that in this case not even the initial fragment is examined by CBAC to determine the L4 information.  Part way through the transfer, R1 happens to send a 161-byte packet that does not need fragmentation at R2:

This packet also shows up in the debug output since it is unfragmented and can be inspected by CBAC:

Mar 1 01:22:10.347: CBAC* sis 66F0A118 pak 663FE340 SIS_OPEN/ESTAB TCP ACK 1282798700 SEQ 590257277 LEN 117 (10.0.12.1:80) <= (10.0.34.4:21143)
Mar 1 01:22:10.347: CBAC* sis 66F0A118 L4 inspect result: DROP packet 663FE340 (10.0.12.1:80) (10.0.34.4:21143) bytes 117 ErrStr = Out-Of-Order Segment http
Mar 1 01:22:10.351: CBAC* Details of Pak 663FE340 IP: s=10.0.12.1 (Serial0/0), d=10.0.34.4 (Serial0/1), len 161, proto=6
Mar 1 01:22:10.351: CBAC: pak seq = 590257277, rcvwnd = 4001 rcvnxt = 1282798700
Mar 1 01:22:10.351: CBAC Storing the packet 663FE340 into OOO queue of session 66F0A118


As shown in the log message, CBAC believes this is an out of order segment based on the TCP sequence number since it did not inspect any of the intermediate packets.  As explained here, CBAC allows the packet to pass but stores a copy in memory.  If it turns out to be an out of order packet, future packets should be able to fill in the gap in sequence numbers.  Shortly after this, we begin seeing this debug message for R4’s ACKs:

Mar 1 01:22:11.035: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590261146 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:11.039: CBAC* sis 66F0A118 L4 inspect result: DROP packet 663FE6F4 (10.0.34.4:21143) (10.0.12.1:80) bytes 0 ErrStr = Invalid Ack (or no Ack) http
Mar 1 01:22:11.039: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6


CBAC thinks this is an invalid ACK by R4 since it does not believe that sequence number has been sent yet by R1.  As shown in the message, the packet is dropped.  R1 can still send packets to R4, but R4 can no longer acknowledge them.  Since R1 does not receive an ACK it continues trying to resend sequence number 590,260,074:

R4 receives them all and after each one tries to tell R1 it already received that sequence number and wants 590,262,754 instead, but the ACKs continue to be dropped:

Mar 1 01:22:11.615: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590262754 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:11.615: CBAC* sis 66F0A118 L4 inspect result: DROP packet 663FE6F4 (10.0.34.4:21143) (10.0.12.1:80) bytes 0 ErrStr = Invalid Ack (or no Ack) http
Mar 1 01:22:11.615: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:12.759: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590262754 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:12.763: CBAC* sis 66F0A118 L4 inspect result: DROP packet 663FE6F4 (10.0.34.4:21143) (10.0.12.1:80) bytes 0 ErrStr = Invalid Ack (or no Ack) http
Mar 1 01:22:12.763: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6
Mar 1 01:22:14.363: CBAC* sis 66F0A118 pak 663FE6F4 SIS_OPEN/ESTAB TCP ACK 590262754 SEQ 1282798700 LEN 0 (10.0.34.4:21143) => (10.0.12.1:80)
Mar 1 01:22:14.363: CBAC* sis 66F0A118 L4 inspect result: DROP packet 663FE6F4 (10.0.34.4:21143) (10.0.12.1:80) bytes 0 ErrStr = Invalid Ack (or no Ack) http
Mar 1 01:22:14.367: CBAC* Details of Pak 663FE6F4 IP: s=10.0.34.4 (Serial0/1), d=10.0.12.1 (Serial0/0), len 44, proto=6


25 seconds after R3 received the second unfragmented packet from R1, we see the following log messages:

Mar 1 01:22:35.619: CBAC: Responder's OOO Idle timeout handler with Session 66F0A118
Mar 1 01:22:35.619: CBAC: TCP OOO timeout handling method called for session 66F0A118
Mar 1 01:22:35.619: CBAC: OOO queue timer exceeded the retry threshold. Deleting the session 66F0A118
Mar 1 01:22:35.623: %FW-6-SESS_AUDIT_TRAIL: Stop http session: initiator (10.0.34.4:21143) sent 127 bytes -- responder (10.0.12.1:80) sent 248 bytes
Mar 1 01:22:35.627: %FW-4-TCP_OoO_SEG: Deleting session as expected TCP segment with seq:590243826 has not arrived even after 25 seconds - session 10.0.34.4:21143 to 10.0.12.1:80


R3 deletes the session from it’s state table because the expected out-of-order segments have not arrived.  The sequence number of the first 292-byte unfragmented packet was 590,243,578; subtracting HDLC, IP, and TCP headers leaves 248 bytes of data, which explains why R3 was still looking for 590,243,826.  The audit trail log message also proves that CBAC did not inspect initial fragments, as it believes R1 sent 248 bytes of data when in fact it has sent about 19,000 bytes.  Let’s enable virtual fragmentation reassembly on R3 S0/0 and try the transfer again:

R3:
interface Serial0/0
 ip virtual-reassembly


R4#copy http://10.0.12.1/asdf.txt null:
Loading http://10.0.12.1/asdf.txt !!
255324 bytes copied in 17.828 secs (14322 bytes/sec)


R3 is now able to inspect the reassembled packets correctly and the transfer completes successfully. An example of reassmbled packet being inspected by R3 is shown below:

R3#debug ip inspect protocol tcp
Mar 1 02:27:42.155: CBAC sis 66F0A118 pak 6678EDF8 SIS_OPEN/ESTAB TCP ACK 168810932 SEQ 1043666778 LEN 536 (10.0.12.1:80) <= (10.0.34.4:53749)
Mar 1 02:27:42.159: CBAC sis 66F0A118 http L7 inspect result: PASS packet
Mar 1 02:27:42.159: CBAC sis 66F0A118 http L7 inspect result: PASS packet
Mar 1 02:27:42.159: CBAC Details of Pak 6678EDF8 IP: s=10.0.12.1 (Serial0/0), d=10.0.34.4 (Serial0/1), len 304, proto=6
Mar 1 02:27:42.159: CBAC: pak seq = 1043666778, rcvwnd = 4001 rcvnxt = 168810932


As shown by the debug, the 536-byte reassembled TCP segment is inspected, which results in both fragments being forwarded.  The reassembled packet is then dropped.

 

Next we will try a TFTP transfer.  Virtual fragmentation reassembly has been disabled again on R3:

R4#copy tftp://10.0.12.1/asdf.txt null:
Accessing tftp://10.0.12.1/asdf.txt...
Loading asdf.txt from 10.0.12.1 (via Serial0/0): !... [timed out]
%Error reading tftp://10.0.12.1/asdf.txt (Timed out)

The start of the transfer in Wireshark is shown below:

The transfer appears to be working in both directions.  Notice that every single packet from R1 gets fragmented by R2 since they are all a fixed size.  Since CBAC does not examine fragmented packets, it does not even see an initial response from R1.  The session remains in the SIS_OPENING state and R3 has a pre-generated session in the state table where it is still waiting for a response from R1 on a port above 1023:

The transfer continues to work until the UDP session timeout expires (30 seconds by default).  At this point, the session is deleted and the fragmented packets are dropped since their L3 information does not match anything in the session table:

Mar 1 03:57:50.927: %FW-6-SESS_AUDIT_TRAIL: Stop tftp session: initiator (10.0.34.4:63208) sent 17 bytes -- responder (10.0.12.1:69) sent 0 bytes


Let’s try it again with virtual fragmentation reassembly enabled on R3 S0/0:

R3:
interface Serial0/0
 ip virtual-reassembly 


R4#copy tftp://10.0.12.1/asdf.txt null:
Accessing tftp://10.0.12.1/asdf.txt...
Loading asdf.txt from 10.0.12.1 (via Serial0/0): !
[OK - 255324 bytes]
255324 bytes copied in 159.280 secs (1603 bytes/sec)


R3#debug ip inspect protocol udp
R3#debug ip inspect protocol tftp
Mar 1 03:20:25.211: CBAC UDP: sis 66F09E40 pak 66F554A0 SIS_OPEN UDP packet (10.0.12.1:50791) => (10.0.34.4:65521) datalen 516
Mar 1 03:20:25.215: TFTP DATA Channel 66F09E40 state SIS_OPEN
Mar 1 03:20:25.215: TFTP Code : DATA, packet length : 516
Mar 1 03:20:25.215: CBAC sis 66F09E40 tftp-data L7 inspect result: PASS packet
Mar 1 03:20:25.423: CBAC* UDP: sis 66F09E40 pak 663FE6F4 SIS_OPEN UDP packet (10.0.12.1:50791) <= (10.0.34.4:65521) datalen 4
Mar 1 03:20:25.423: TFTP DATA Channel 66F09E40 state SIS_OPEN
Mar 1 03:20:25.423: TFTP Code : ACK
Mar 1 03:20:25.423: CBAC* sis 66F09E40 tftp-data L7 inspect result: PASS packet


Again, CBAC is now able to correctly inspect the TFTP traffic and the transfer is successful.

 

 

Next we will remove the CBAC configuration and try the same two transfers using the zone-based firewall.  Virtual fragmentation reassembly has also been removed.  The ZBFW configuration on R3 is:

R3:
class-map type inspect match-all tftp
 match protocol tftp
class-map type inspect match-all web
 match protocol http
!
policy-map type inspect asdf
 class type inspect web
  inspect
 class type inspect tftp
  inspect
 class class-default
!
zone security inside
zone security outside
zone-pair security inside-outside source inside destination outside
 service-policy type inspect asdf
!
interface Serial0/0
 zone-member security outside
!
interface Serial0/1
 zone-member security inside


We’ll try the HTTP transfer first:

R4#copy http://10.0.12.1/asdf.txt null:
Loading http://10.0.12.1/asdf.txt !
%Error reading http://10.0.12.1/asdf.txt (Broken pipe)


The transfer fails.  I’m not aware of any equivalent debugs for ZBFW that show per-packet information and inspection results, but Wireshark is sufficient for finding the problem.  Edit: I found out afterward that CBAC debugs will also work for ZBFW.  Shown below is a capture on R3 S0/1 followed by a capture on R3 S0/0:

Every packet in the transfer is the same on each interface up until #139 on S0/1, at 13:40:47.135.  This ACK is never seen on R3 S0/0.  Instead, we see only traffic coming from R1 after this point.  The sequence number in every packet sent by R1 after this point is 1,590,239,484 and the acknowledgement number sent by R4 is 1,590,242,700, which also indicates that R4 is receiving data from R1 but R1 is not receiving ACKs from R4.  In the sessions table, we see the exact same number of bytes sent in each direction as we saw for the HTTP transfer with CBAC:

The problem is the same that we had with CBAC.  Fragmented packets are not examined, and R3 therefore believes that R4 is sending invalid ACKs and begins dropping them.  Like before, enabling virtual fragmentation reassembly fixes the problem:

R3:
interface Serial0/0
 ip virtual-reassembly


The transfer now works, and R3 shows the correct number of bytes transferred in the session:

R4#copy http://10.0.12.1/asdf.txt null:
Loading http://10.0.12.1/asdf.txt !!
255324 bytes copied in 15.232 secs (16762 bytes/sec)


Mar 1 04:07:48.907: %FW-6-SESS_AUDIT_TRAIL: Stop http session: initiator (10.0.34.4:60779) sent 127 bytes -- responder (10.0.12.1:80) sent 255572 bytes


 

 

Next we will try the TFTP transfer with virtual fragmentation reassembly disabled:

R4#copy tftp://10.0.12.1/asdf.txt null:
Accessing tftp://10.0.12.1/asdf.txt...
%Error opening tftp://10.0.12.1/asdf.txt (Timed out)


The transfer fails.  Shown below are captures on R3 S0/1 and then R3 S0/0:

These captures show that the inspection logic used by ZBFW is a little more detailed than CBAC (at least for TFTP).  With ZBFW, none of the acknowledgements from R4 to R1 are allowed because R3 does not believe that it is a valid acknowledgement since it is not aware of R1 sending data.  With CBAC, the acknowledgements were always allowed even though the firewall had not seen the data block that was being acknowledged pass through it.  Therefore the transfer fails almost immediately with the ZBFW, while with CBAC it takes a while and could actually finish if the transfer completes before the UDP timeout expires.  Once again enabling virtual fragmentation reassembly fixes the problem:

R3:
interface Serial0/0
 ip virtual-reassembly


R4#copy tftp://10.0.12.1/asdf.txt null:
Accessing tftp://10.0.12.1/asdf.txt...
Loading asdf.txt from 10.0.12.1 (via Serial0/0): !
[OK - 255324 bytes]
255324 bytes copied in 146.804 secs (1739 bytes/sec)


 

 

Next we will look at some of the types of invalid fragments and fragment attacks that virtual fragmentation reassembly can prevent as well as some of it’s configurable settings.  The ZBFW configuration has been removed from R3.  Colasoft Packet Builder will be used to send packets from a PC at 10.1.1.10 connected to R1 to R4.  First, a correctly fragmented ICMP echo with 100 bytes of data will be sent.  The ICMP echo is sent in 2 fragments 1 second apart, the first containing the 8-byte ICMP header and 80 bytes of data with the More Fragments flag set, and the second containing the last 20 bytes of data and a Fragment Offset of 11 (88 octets).

R3:
interface Serial0/0
 ip virtual-reassembly


First fragment:

Second fragment:

R3#debug ip virtual-reassembly
Mar 1 00:24:47.747: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:0, offset:0, len:88) in fast path...
Mar 1 00:24:47.747: IP_VFR: created frag state for sa:10.1.1.10, da:10.0.34.4, id:0...
Mar 1 00:24:47.747: IP_VFR: pak incomplete cpak-offset:0, cpak-len:88, flag: 1
Mar 1 00:24:47.747: IP_VFR: dgrm incomplete, returning...
Mar 1 00:24:48.747: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:0, offset:88, len:20) in fast path...
Mar 1 00:24:48.747: IP_VFR: cpak-offset:0, cpak-len:88, npak-offset:88
Mar 1 00:24:48.747: IP_VFR: dgrm complete, switching the frags.
Mar 1 00:24:48.747: IP_VFR: switching fragment (sa:10.1.1.10, da:10.0.34.4, id:0, offset:0, len:88)
Mar 1 00:24:48.751: IP VFR:Enqueing packet to IP queue
Mar 1 00:24:48.751: IP_VFR: switching fragment (sa:10.1.1.10, da:10.0.34.4, id:0, offset:88, len:20)
Mar 1 00:24:48.751: IP VFR:Enqueing packet to IP queue
Mar 1 00:24:48.751: IP_VFR: all fragments have been switched.
Mar 1 00:24:48.755: IP_VFR: pak_subblock_free - pak 0x667907E4
Mar 1 00:24:48.759: IP_VFR: deleted frag state for sa:10.1.1.10, da:10.0.34.4, id:0
Mar 1 00:24:48.759: IP_VFR: pak_subblock_free - pak 0x6678AB50


R4#debug ip packet detail
Mar 1 00:24:48.843: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 00:24:48.843: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 108, rcvd 3
Mar 1 00:24:48.847: IP Fragment, Ident = 0, fragment offset = 0
Mar 1 00:24:48.847: ICMP type=8, code=0
Mar 1 00:24:48.847: IP: recv fragment from 10.1.1.10 offset 0 bytes
Mar 1 00:24:48.871: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 00:24:48.871: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 40, rcvd 3
Mar 1 00:24:48.875: IP Fragment, Ident = 0, fragment offset = 88
Mar 1 00:24:48.875: IP: recv fragment from 10.1.1.10 offset 88 bytes
Mar 1 00:24:48.875: IP: tableid=0, s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), routed via FIB
Mar 1 00:24:48.875: IP: s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), len 128, sending
Mar 1 00:24:48.879: ICMP type=0, code=0


The fragments pass virtual reassembly on R3 and R4 is able to reassemble the ICMP echo and send an unfragmented reply.  We can also see that although the fragments arrive 1 second apart at R3, they arrive almost simultaneously at R4 because R3 does not forward any of the fragments until it has received all of them and verified that they are valid.  Let’s try lowering the fragment offset of the second fragment from 11 (88 octects) to 10 (80 octets) so that part of the data in the second fragment overlaps the first fragment:

R3#debug ip virtual-reassembly
Mar 1 00:37:50.339: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:0, offset:0, len:88) in fast path...
Mar 1 00:37:50.339: IP_VFR: created frag state for sa:10.1.1.10, da:10.0.34.4, id:0...
Mar 1 00:37:50.339: IP_VFR: pak incomplete cpak-offset:0, cpak-len:88, flag: 1
Mar 1 00:37:50.339: IP_VFR: dgrm incomplete, returning...
Mar 1 00:37:51.303: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:0, offset:80, len:20) in fast path...
Mar 1 00:37:51.303: IP_VFR: cpak-offset:0, cpak-len:88, npak-offset:80
Mar 1 00:37:51.303: %IP_VFR-3-OVERLAP_FRAGMENTS: Serial0/0: from the host 10.1.1.10 destined to 10.0.34.4
Mar 1 00:37:51.307: IP_VFR: err while checking for compltns of dgrm
Mar 1 00:37:51.307: IP_VFR: deleted frag state for sa:10.1.1.10, da:10.0.34.4, id:0


R3 generates a log message reporting the overlapping fragments and drops them both without forwarding them to R4. 

 

Another type of attack that could be performed is sending several incomplete fragments.  With VFR disabled on R3 S0/0, we will send 5 different copies of the second fragment from the previous example, each with a different IP Identification number:

R4#debug ip packet detail
Mar 1 00:59:10.395: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 00:59:10.395: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 40, rcvd 3
Mar 1 00:59:10.399: IP Fragment, Ident = 1, fragment offset = 80
Mar 1 00:59:10.399: IP: recv fragment from 10.1.1.10 offset 80 bytes
Mar 1 00:59:11.403: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 00:59:11.403: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 40, rcvd 3
Mar 1 00:59:11.407: IP Fragment, Ident = 2, fragment offset = 80
Mar 1 00:59:11.407: IP: recv fragment from 10.1.1.10 offset 80 bytes
Mar 1 00:59:12.447: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 00:59:12.447: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 40, rcvd 3
Mar 1 00:59:12.447: IP Fragment, Ident = 3, fragment offset = 80
Mar 1 00:59:12.451: IP: recv fragment from 10.1.1.10 offset 80 bytes
Mar 1 00:59:13.403: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 00:59:13.403: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 40, rcvd 3
Mar 1 00:59:13.407: IP Fragment, Ident = 4, fragment offset = 80
Mar 1 00:59:13.407: IP: recv fragment from 10.1.1.10 offset 80 bytes
Mar 1 00:59:14.419: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 00:59:14.419: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 40, rcvd 3
Mar 1 00:59:14.423: IP Fragment, Ident = 5, fragment offset = 80
Mar 1 00:59:14.423: IP: recv fragment from 10.1.1.10 offset 80 bytes


R4 receives all the fragments and holds them in memory while waiting for the other fragments.  After about 40 seconds, they have not arrived so R4 sends an ICMP Fragmentation Reassembly Time Exceeded message back to R1 for each fragment:

R4#debug ip packet detail
Mar 1 00:59:51.563: IP: tableid=0, s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), routed via FIB
Mar 1 00:59:51.563: IP: s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), len 56, sending
Mar 1 00:59:51.563: ICMP type=11, code=1
Mar 1 00:59:51.567: IP: tableid=0, s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), routed via FIB
Mar 1 00:59:51.567: IP: s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), len 56, sending
Mar 1 00:59:51.571: ICMP type=11, code=1
Mar 1 00:59:51.571: IP: tableid=0, s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), routed via FIB
Mar 1 00:59:51.571: IP: s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), len 56, sending
Mar 1 00:59:51.575: ICMP type=11, code=1
Mar 1 00:59:51.575: IP: tableid=0, s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), routed via FIB
Mar 1 00:59:51.575: IP: s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), len 56, sending
Mar 1 00:59:51.579: ICMP type=11, code=1
Mar 1 00:59:51.579: IP: tableid=0, s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), routed via FIB
Mar 1 00:59:51.579: IP: s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), len 56, sending
Mar 1 00:59:51.583: ICMP type=11, code=1


This results in wasted resources on R4 as well as wasted bandwidth on the R3-R4 link.  With VFR on R3, the fragments would never have been forwarded because the remaining fragments never arrived.  The timeout and max-reassemblies options can be used to tune the amount of resources that VFR uses for this.  We will set a maximum of 3 reassemblies and a timeout of 10 seconds and send the same 5 non-initial fragments again:

R3:
interface Serial0/0
 ip virtual-reassembly max-reassemblies 3 timeout 10


R3#debug ip virtual-reassembly
Mar 1 01:19:05.111: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:1, offset:80, len:20) in fast path...
Mar 1 01:19:05.111: IP_VFR: created frag state for sa:10.1.1.10, da:10.0.34.4, id:1...
Mar 1 01:19:05.111: IP_VFR: dgrm incomplete, returning...
Mar 1 01:19:06.123: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:2, offset:80, len:20) in fast path...
Mar 1 01:19:06.123: IP_VFR: created frag state for sa:10.1.1.10, da:10.0.34.4, id:2...
Mar 1 01:19:06.123: IP_VFR: dgrm incomplete, returning...
Mar 1 01:19:07.095: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:3, offset:80, len:20) in fast path...
Mar 1 01:19:07.095: IP_VFR: created frag state for sa:10.1.1.10, da:10.0.34.4, id:3...
Mar 1 01:19:07.095: IP_VFR: dgrm incomplete, returning...
Mar 1 01:19:08.151: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:4, offset:80, len:20) in fast path...
Mar 1 01:19:08.151: %IP_VFR-4-FRAG_TABLE_OVERFLOW: Serial0/0: the fragment table has reached its maximum threshold 3
Mar 1 01:19:09.135: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:5, offset:80, len:20) in fast path...
Mar 1 01:19:15.111: IP VFR: frag state expired -src-addr:10.1.1.10, dst-addr:10.0.34.4, ip-id:1...
Mar 1 01:19:15.111: IP_VFR: deleted frag state for sa:10.1.1.10, da:10.0.34.4, id:1
Mar 1 01:19:16.123: IP VFR: frag state expired -src-addr:10.1.1.10, dst-addr:10.0.34.4, ip-id:2...
Mar 1 01:19:16.123: IP_VFR: deleted frag state for sa:10.1.1.10, da:10.0.34.4, id:2
Mar 1 01:19:17.095: IP VFR: frag state expired -src-addr:10.1.1.10, dst-addr:10.0.34.4, ip-id:3...
Mar 1 01:19:17.095: IP_VFR: deleted frag state for sa:10.1.1.10, da:10.0.34.4, id:3


The first three fragments arrive and are stored in memory for reassembly.  When the fourth fragment arrives, the max-reassemblies threshold is exceeded and a log message is generated, and the fourth and fifth fragments are dropped.  After 10 seconds, the first three fragments are also dropped since they have not been fully reassembled yet. 

 

The max-fragments option controls the max amount of fragments allowed for a single datagram.  To test this out we will first disable VFR on R3 and send an ICMP echo to R4 that is split into 4 fragments.  The first fragment contains the 8-byte ICMP header and the remaining 3 fragments each contain 8 bytes of data.  First fragment:

Second fragment:

Third fragment:

Fourth fragment:

R4 reassembles the fragments and sends a 52-byte reply (20B IP header + 8B ICMP header + 24B of data):

R4#debug ip packet detail
Mar 1 01:43:28.883: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 01:43:28.883: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 28, rcvd 3
Mar 1 01:43:28.883: IP Fragment, Ident = 1, fragment offset = 0
Mar 1 01:43:28.887: ICMP type=8, code=0
Mar 1 01:43:28.887: IP: recv fragment from 10.1.1.10 offset 0 bytes
Mar 1 01:43:29.851: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 01:43:29.851: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 28, rcvd 3
Mar 1 01:43:29.855: IP Fragment, Ident = 1, fragment offset = 8
Mar 1 01:43:29.855: IP: recv fragment from 10.1.1.10 offset 8 bytes
Mar 1 01:43:30.831: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 01:43:30.835: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 28, rcvd 3
Mar 1 01:43:30.835: IP Fragment, Ident = 1, fragment offset = 16
Mar 1 01:43:30.835: IP: recv fragment from 10.1.1.10 offset 16 bytes
Mar 1 01:43:31.855: IP: tableid=0, s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), routed via RIB
Mar 1 01:43:31.855: IP: s=10.1.1.10 (Serial0/0), d=10.0.34.4 (Serial0/0), len 28, rcvd 3
Mar 1 01:43:31.859: IP Fragment, Ident = 1, fragment offset = 24
Mar 1 01:43:31.859: IP: recv fragment from 10.1.1.10 offset 24 bytes
Mar 1 01:43:31.859: IP: tableid=0, s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), routed via FIB
Mar 1 01:43:31.859: IP: s=10.0.34.4 (local), d=10.1.1.10 (Serial0/0), len 52, sending
Mar 1 01:43:31.863: ICMP type=0, code=0


Let’s try it again with VFR configured on R3 for a max of 2 fragments per datagram:

R3:
interface Serial0/0
 ip virtual-reassembly max-fragments 2 timeout 10


R3#debug ip virtual-reassembly
Mar 1 01:45:18.091: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:1, offset:0, len:8) in fast path...
Mar 1 01:45:18.091: IP_VFR: created frag state for sa:10.1.1.10, da:10.0.34.4, id:1...
Mar 1 01:45:18.091: IP_VFR: pak incomplete cpak-offset:0, cpak-len:8, flag: 1
Mar 1 01:45:18.091: IP_VFR: dgrm incomplete, returning...
Mar 1 01:45:19.055: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:1, offset:8, len:8) in fast path...
Mar 1 01:45:19.055: IP_VFR: cpak-offset:0, cpak-len:8, npak-offset:8
Mar 1 01:45:19.059: IP_VFR: pak incomplete cpak-offset:8, cpak-len:8, flag: 1
Mar 1 01:45:19.059: IP_VFR: dgrm incomplete, returning...
Mar 1 01:45:20.087: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:1, offset:16, len:8) in fast path...
Mar 1 01:45:20.087: %IP_VFR-4-TOO_MANY_FRAGMENTS: Serial0/0: Too many fragments per datagram (more than 2) - sent by 10.1.1.10, destined to 10.0.34.4
Mar 1 01:45:20.087: IP_VFR: deleted frag state for sa:10.1.1.10, da:10.0.34.4, id:1
Mar 1 01:45:21.087: IP_VFR: fragment (sa:10.1.1.10, da:10.0.34.4, id:1, offset:24, len:8) in fast path...
Mar 1 01:45:21.091: IP_VFR: created frag state for sa:10.1.1.10, da:10.0.34.4, id:1...
Mar 1 01:45:21.091: IP_VFR: dgrm incomplete, returning...
Mar 1 01:45:31.091: IP VFR: frag state expired -src-addr:10.1.1.10, dst-addr:10.0.34.4, ip-id:1...
Mar 1 01:45:31.091: IP_VFR: deleted frag state for sa:10.1.1.10, da:10.0.34.4, id:1


When the third fragment is received, R3 drops all three fragments and generates a log message that too many fragments were received for the datagram.  The fourth fragment arrives a second later and R3 creates a new entry for it.  Since the first three fragments were already dropped and the datagram cannot be reassembled, the fourth fragment ends up getting dropped 10 seconds later when the timeout expires.

 

Finally, VFR can also be configured to drop all incoming fragments on an interface whether or not they are part of a valid datagram with the drop-fragments option.  The same 4 fragments are sent again 1 second apart:

R3:
interface Serial0/0
 ip virtual-reassembly drop-fragments


R3#debug ip virtual-reassembly
Mar 1 01:55:49.911: IP_VFR: in drop mode
Mar 1 01:55:50.887: IP_VFR: in drop mode
Mar 1 01:55:51.919: IP_VFR: in drop mode
Mar 1 01:55:52.883: IP_VFR: in drop mode


Reflexive ACLs, Classic IOS Firewall (CBAC), and Zone-Based Policy Firewall

This example is designed to show a simple example of using reflexive ACLs, the Cisco IOS ‘classic’ firewall (CBAC), and the newer zone-based policy firewall to accomplish the same thing.  Although each of these technologies have significant differences, this example is only designed to show a basic configuration of each for comparison.  The topology and initial configurations are shown below:

 

hostname FW
!
interface FastEthernet1/0
 ip address 192.168.1.1 255.255.255.0
 duplex full
 speed 100
!
interface FastEthernet1/1
 ip address 10.1.1.1 255.255.255.0
 duplex full
 speed 100


hostname Inside
!
no ip domain lookup
ip host Outside 10.1.1.2
!
interface FastEthernet1/0
 ip address 192.168.1.2 255.255.255.0
 duplex full
 speed 100
!
ip route 0.0.0.0 0.0.0.0 192.168.1.1
!
line vty 0 4
 privilege level 15
 no login


hostname Outside
!
no ip domain lookup
ip host Inside 192.168.1.2
!
interface FastEthernet1/0
 ip address 10.1.1.2 255.255.255.0
 duplex full
 speed 100
!
ip route 0.0.0.0 0.0.0.0 10.1.1.1
!
line vty 0 4
 privilege level 15
 no login


For this example, we want to be able to initiate telnet connections from Inside to Outside, and allow Outside to respond only if the connection was initiated from Inside. All other traffic should be denied.  We will create a configuration to meet these requirements using each of the previously mentioned 3 technologies, with each one starting with the base configs shown above.

 

Reflexive ACL Configuration

FW:
ip access-list extended InsideACL
 permit tcp 192.168.1.0 0.0.0.255 gt 1024 10.1.1.0 0.0.0.255 eq telnet reflect TelnetResponse
 deny ip any any log
!
ip access-list extended OutsideACL
 evaluate TelnetResponse
 deny ip any any log
!
interface FastEthernet1/0
 ip access-group InsideACL in
!
interface FastEthernet1/1
 ip access-group OutsideACL in


The above configuration  creates an ACL named InsideACL which allows telnet traffic from Inside to Outside.  Any traffic permitted by the ACL causes a temporary entry in the reflexive ACL named TelnetResponse to be created, with the source and destination addresses and port numbers reversed.  All other traffic from Inside is denied and logged.  An ACL named OutsideACL was also created, which first evaluates the reflexive ACL we created.  Anything not permitted by the reflexive ACL is denied and logged.  Let’s test it out.  Outside attempting to telnet to Inside is denied and generates a log message on FW:

Inside attempting to telnet to Outside is successful:

We can also view the reflexive ACL entry that has been created on FW when we telnetted from Inside to Outside:

 

Classic IOS Firewall / CBAC

FW:
ip access-list extended InsideACL
 permit tcp 192.168.1.0 0.0.0.255 gt 1024 10.1.1.0 0.0.0.255 eq telnet
 deny ip any any log
!
ip access-list extended OutsideACL
 deny ip any any log
!
ip inspect name AllowTelnet telnet
!
interface FastEthernet1/0
 ip access-group InsideACL in
 ip inspect AllowTelnet in
!
interface FastEthernet1/1
 ip access-group OutsideACL in

The above configuration creates an ACL named InsideACL which allows telnet traffic from Inside to Outside and denies everything else.  Next, it creates an ACL named OutsideACL which denies everything coming from Outside.  It also creates an inspection rule named AllowTelnet which inspects Telnet traffic from Inside to Outside and allows response traffic from Outside, which our OutsideACL would have otherwise denied.  Let’s test the CBAC configuration out.  Outside attempting to telnet to Inside is denied again and generates a log message on FW:

Inside attempting to telnet to Outside is successful:

We can view information about the telnet connection in the session table on FW:

 

Zone-Based Policy Firewall

FW:

zone security Inside
!
zone security Outside
!
interface FastEthernet1/0
 zone-member security Inside
!
interface FastEthernet1/1
 zone-member security Outside
!
class-map type inspect match-all TelnetClass
 match protocol telnet
!
policy-map type inspect AllowTelnet
 class type inspect TelnetClass
  inspect
!
zone-pair security Inside-Outside source Inside destination Outside
 service-policy type inspect AllowTelnet


 The above configuration creates the security zones Inside and Outside and places F1/0 into Inside and F1/1 into Outside.  Next, it creates the class-map TelnetClass which matches telnet traffic and the policy-map AllowTelnet which specifies that telnet traffic should be inspected in order to allow return traffic.  With the zone-based firewall, all traffic between different zones is denied by default, so no ACLs are necessary.  The last step is to create a zone pair named Inside-Outside with Inside as the source and Outside as the destination and apply our policy-map.  Let’s test the zone-based firewall configuration.  Outside attempting to telnet to Inside is denied, just like the last 2 examples:

Inside attempting to telnet to Outside is successful:

Just as with CBAC, we can view information about the telnet session: