Cisco NAT

Cisco make their NAT translations really hard to understand. I believe this is because they use terminology that really does not make any sense.

Cisco defines these terms:

  • Inside local address—The IP address assigned to a host on the inside network. This is the address configured as a parameter of the computer OS or received via dynamic address allocation protocols such as DHCP. The address is likely not a legitimate IP address assigned by the Network Information Center (NIC) or service provider.
  • Inside global address—A legitimate IP address assigned by the NIC or service provider that represents one or more inside local IP addresses to the outside world.
  • Outside local address—The IP address of an outside host as it appears to the inside network. Not necessarily a legitimate address, it is allocated from an address space routable on the inside.
  • Outside global address—The IP address assigned to a host on the outside network by the host owner. The address is allocated from a globally routable address or network space.

Now forget about these for a moment and consider the diagram below. All it talks about are “original ip addresses” and “translated ip address”.

Cisco NATUsing this terminology you can easily workout what the required NAT statement is. Cisco use the inside/outside local/global terms to include the direction the packet is traversing. For network engineer you only need to add a source and a destination interface and it all works out without any tears.

 

 

 

Advertisements

Netapp Concepts (for network engineers)

This blog is written from a a network engineers point view. I’m going to try relate the concepts to tasks/commands that you usually run on a new network device to understand what its doing.

Hardware

Like Cisco devices the Netapp filer has a concept of slots and ports. The slots start their numbering from zero and can contain either storage adapters (called HBAs) or network adapters (called line cards or NICs).

The Netapp Simulator below has no storage adapters and one quad port ethernet line card

netapp01*> sysconfig
        NetApp Release 8.2.1 7-Mode: Fri Mar 21 14:48:58 PDT 2014
        System ID: 4082368508 (netapp01)
        System Serial Number: 4082368508 (netapp01)
        System Storage Configuration: Multi-Path
        System ACP Connectivity: NA
        slot 0: System Board
                Model Name:         SIMBOX
                Processors:         2
                Memory Size:        1599 MB
                Memory Attributes:  None
        slot 0: 10/100/1000 Ethernet Controller V
                e0a MAC Address:    00:0c:29:56:d2:4a (auto-1000t-fd-up)
                e0b MAC Address:    00:0c:29:56:d2:54 (auto-1000t-fd-up)
                e0c MAC Address:    00:0c:29:56:d2:5e (auto-1000t-fd-up)
                e0d MAC Address:    00:0c:29:56:d2:68 (auto-1000t-fd-up)

 

 

While a real Netapp filer shows:

netapp-a> sysconfig
        NetApp Release 8.1.1 7-Mode: Mon Jul 30 12:49:46 PDT 2012
        System ID: xxxxxxxx (netapp-a); partner ID: yyyyyyy (netapp-b)
        System Serial Number: zzzzzzz (netapp-a)
        System Rev: D0
        System Storage Configuration: Multi-Path HA
        System ACP Connectivity: Full Connectivity
        slot 0: System Board
                Processors:         4
                Processor type:     Intel(R) Xeon(R) CPU           C3528  @ 1.73GHz
                Memory Size:        6144 MB
                Memory Attributes:  Hoisting
                                    Normal ECC
                Controller:         A
        Service Processor           Status: Online
        slot 0: Internal 10/100 Ethernet Controller
                e0M MAC Address:    00:a0:98:38:26:13 (auto-100tx-fd-cfg_down)
                e0P MAC Address:    00:a0:98:38:26:12 (auto-100tx-fd-up)
        slot 0: Quad Gigabit Ethernet Controller 82580
                e0a MAC Address:    00:a0:98:38:26:0e (auto-1000t-fd-up)
                e0b MAC Address:    00:a0:98:38:26:0f (auto-1000t-fd-up)
                e0c MAC Address:    00:a0:98:38:26:10 (auto-1000t-fd-up)
                e0d MAC Address:    00:a0:98:38:26:11 (auto-1000t-fd-up)
        slot 0: Interconnect HBA:   Mellanox IB MT25204
        slot 0: SAS Host Adapter 0a
                72 Disks:            120629.8GB
                1 shelf with IOM3, 1 shelf with IOM6, 1 shelf with IOM6E
        slot 0: SAS Host Adapter 0b
                72 Disks:            120629.8GB
                1 shelf with IOM3, 1 shelf with IOM6, 1 shelf with IOM6E
        slot 0: Intel ICH USB EHCI Adapter u0a (0xdf101000)
                boot0   Micron Technology Real SSD eUSB 2GB, class 0/0, rev 2.00/11.10, addr 2 1936MB 512B/sect (4DF0022700247875)
        slot 1: Dual 10 Gigabit Ethernet Controller IX1-SFP+
                e1a MAC Address:    00:a0:98:37:2f:78 (auto-10g_twinax-fd-up)
                e1b MAC Address:    00:a0:98:37:2f:79 (auto-10g_twinax-fd-up)

The filer has two slots: slot0 & slot1. Slot 0 has four adapters two network (ports numbered e0x) & two storage adapters (ports numbered 0a and 0b).

 

Network Interfaces

The network interface config can be viewed using the standard ifconfig / netstat commands. However the command ifgrp shows the etherchannel config for the interfaces.

netapp-a> ifgrp status
default: transmit 'IP Load balancing', Ifgrp Type 'multi_mode', fail 'log'
lvif1: 2 links, transmit 'IP Load balancing', Ifgrp Type 'multi_mode' fail 'default'
         Ifgrp Status   Up      Addr_set
        up:
        e1b: state up, since 07Nov2014 08:28:44 (19+02:08:46)
                mediatype: auto-10g_twinax-fd-up
                flags: enabled
<SNIP>
        e1a: state up, since 18Jan2013 16:33:49 (676+18:03:41)
                mediatype: auto-10g_twinax-fd-up
                flags: enabled
<SNIP>
lvif0: 4 links, transmit 'IP Load balancing', Ifgrp Type 'lacp' fail 'default'
         Ifgrp Status   Up      Addr_set
        up:
        e0d: state up, since 08Sep2014 11:06:16 (78+22:31:14)
<SNIP>

There are three types of etherchannel:

  • single-mode – only one of the interfaces in the interface group is active. The other interfaces are on standby
  • static multimode – all links are bundled manually (in cisco’ese it basically says etherchannel mode on)
  • dynamic multimode – links are bundled using lacp

 

Storage interfaces

The storage adapters details can be seen as follows:

netapp-a> storage show adapter -a
Slot:            0a
Description:     SAS Host Adapter 0a (PMC-Sierra PM8001 rev. C)
Firmware Rev:    01.11.00.00
Base WWN:        5:00a098:0012b0e:70
State:           Enabled
In Use:          Yes
Redundant:       Yes
Phy State:       [0] Enabled, 6.0Gb/s (10)
                 [1] Enabled, 6.0Gb/s (10)
                 [2] Enabled, 6.0Gb/s (10)
                 [3] Enabled, 6.0Gb/s (10)

Slot:            0b
Description:     SAS Host Adapter 0b (PMC-Sierra PM8001 rev. C)
Firmware Rev:    01.11.00.00
Base WWN:        5:00a098:0012b0e:74
State:           Enabled
In Use:          Yes
Redundant:       Yes
Phy State:       [0] Enabled, 3.0Gb/s (9)
                 [1] Enabled, 3.0Gb/s (9)
                 [2] Enabled, 3.0Gb/s (9)
                 [3] Enabled, 3.0Gb/s (9)

The storage adapters connect to “shelves” which contains the storage media. A “pure” SSD shelf contains SSDs only; a “mixed” shelf contains a combination of SSDs and HDDs. The connections are done in a daisy chain fashion which means you can see the same shelf on multiple ports. Each shelf has a unique serial number and is given a unique shelf id like shelf1 which is set physically on the hardware. In front of this ID is added the storage port adapter though which you can “see” the shelf. EG 0a.shelf1 and 0b.shelf1 means you can see the same shelf through both adapters.

netapp-a> storage show shelf 0a.shelf1
Shelf name:    0b.shelf1
Channel:       0b
Module:        A
Shelf id:      1
Shelf UID:     50:05:0c:c1:02:03:9c:6b
Shelf S/N:     SHJ00000000xxxx
Term switch:   N/A
Shelf state:   ONLINE
Module state:  OK


               Partial Path   Link    Invalid   Running     Loss    Phy       CRC     Phy
Disk    Port   Timeout        Rate     DWord    Disparity   Dword   Reset     Error   Change
Id     State   Value (ms)    (Gb/s)    Count    Count       Count   Problem   Count   Count
--------------------------------------------------------------------------------------------
[SQR0] OK             7        6.0        0           0       0        0         0       3
<SNIP>
[SIL3] DIS/UNUSD      7         NA        0           0       0        0         0       1

Shelf name:    0a.shelf1
Channel:       0a
Module:        B
Shelf id:      1
Shelf UID:     50:05:0c:c1:02:03:9c:6b
Shelf S/N:     SHJ00000000xxxx
Term switch:   N/A
Shelf state:   ONLINE
Module state:  OK


               Partial Path   Link    Invalid   Running     Loss    Phy       CRC     Phy
Disk    Port   Timeout        Rate     DWord    Disparity   Dword   Reset     Error   Change
Id     State   Value (ms)    (Gb/s)    Count    Count       Count   Problem   Count   Count
--------------------------------------------------------------------------------------------
[SQR0] OK             7        6.0        0           0       0        0         0       3
<SNIP>
[SIL3] DIS/UNUSD      7         NA        0           0       0        0         0       1

 

Storage Concepts

Physical disks (in shelves or onboard) are organized into aggregates which provides pools of storage. In each aggregate, one or more flexible volumes can be created. Each volume has a default qtree (called qtree0). A qtree creates a subset of a volume to which a quota can be applied to limit its size. As a special case, a qtree can be the entire volume. A qtree is flexible because you can change the size of a qtree at any time. In addition to a quota, a qtree possesses a few other properties (mainly file security permissions).

A plex is a physical copy of a filesystem or the disks holding the data. A volume normally consists of one plex (called plex0).   A mirrored volume has two or more plexes, each with a complete copy of the data in the volume.  Multiple plexes provides safety for your data as long as you have one complete plex, you will still have access to all your data. So bottom-line, unless you mirror an aggregate, plex0 is just a placeholder that should remind you of the ability to create a mirror if needed.

On a brand new system you would first need to create and aggregate and then a volume that lives on that aggregate.  You can then attach CIFS or NFS to this volume to make it available to end users. The default qtree0 and plex0 are created automatically.

Here is the what this looks like on a Netapp Simulator

netapp01*> aggr status -v
           Aggr State           Status                Options
          aggr0 online          raid_dp, aggr         root, diskroot, nosnap=off, raidtype=raid_dp,
                                64-bit                raidsize=16, ignore_inconsistent=off,
                                                      snapmirrored=off, resyncsnaptime=60,
                                                      fs_size_fixed=off, lost_write_protect=on,
                                                      ha_policy=cfo, hybrid_enabled=off,
                                                      percent_snapshot_space=0%,
                                                      free_space_realloc=off

                Volumes: vol0

                Plex /aggr0/plex0: online, normal, active
                    RAID group /aggr0/plex0/rg0: normal, block checksums

netapp01*> vol status -v
         Volume State           Status                Options
           vol0 online          raid_dp, flex         root, diskroot, nosnap=off, nosnapdir=off,
                                64-bit                minra=off, no_atime_update=off, nvfail=off,
                                                      ignore_inconsistent=off, snapmirrored=off,
                                                      create_ucode=off, convert_ucode=off,
                                                      maxdirsize=16291, schedsnapname=ordinal,
                                                      fs_size_fixed=off, guarantee=volume,
                                                      svo_enable=off, svo_checksum=off,
                                                      svo_allow_rman=off, svo_reject_errors=off,
                                                      no_i2p=off, fractional_reserve=100, extent=off,
                                                      try_first=volume_grow, read_realloc=off,
                                                      snapshot_clone_dependency=off,
                                                      dlog_hole_reserve=off, nbu_archival_snap=off
                         Volume UUID: 19647a8b-5a5c-4bd7-b67e-37a78fb4108c
                Containing aggregate: 'aggr0'

                Plex /aggr0/plex0: online, normal, active
                    RAID group /aggr0/plex0/rg0: normal, block checksums

        Snapshot autodelete settings for vol0:
                                        state=off
                                        commitment=try
                                        trigger=volume
                                        target_free_space=20%
                                        delete_order=oldest_first
                                        defer_delete=user_created
                                        prefix=(not specified)
                                        destroy_list=none
        Volume autosize settings:
                                mode=off
        Hybrid Cache:
                Eligibility=read-write
netapp01*> qtree status -v
Volume   Tree     Style Oplocks  Status
-------- -------- ----- -------- ---------
vol0              unix  enabled  normal

Finally show which file systems have been “advertised” by NFS and can be mounted by clients:

netapp01> exportfs
/vol/vol0/home  -sec=sys,rw,nosuid
/vol/vol0       -sec=sys,rw,anon=0,nosuid

 

 

 

 

Running the Netapp Ontap 7-Mode 8.2.1 Simulator

The below is copied from a set of notes taken a while ago. I’ll update these as I go.

For VMware Workstation

  • Grab the vsim_netapp-7m.tgz from the Netapp Support site, untar / unzip it
  • This will uncompress a bunch of vmdk files. Most of these files are “individual” disk which will appear on the storage controller
  • Load the VMX file in VMware Workstation

 For ESXi

  • Grab the vsim_esx-7m.tgz from the Netapp Support site, untar / unzip it
  • Enable ssh on the exsi server
  • Copy the tar to the datastore1
  • Uncompress the image (tar -xvzf)
  • Run vmkload_mod multiextent  (https://communities.netapp.com/thread/24329)

 

Common Instructions

  • Boot the vm
    Ctrl-c during boot

1

Select option 4 to start with a fresh config

Provide a hostname, don’t use IPv6 or interface groups
Then setup an IP address and default gateway on e0a, the remaining interfaces can be setup later

When requested don’t provide an admin host (otherwise you will be restricted to use this machine configure the netapp)

netapp_ip

Setup root password when requested
Log in as root
Change the network interfaces setting to connect the correct physical network (Usually e0a is mapped to network adaper 1)

netapp_settings

  • You should now be able to ping and ssh to the netapp using root / pwd
  • Now install OnCommand System Manager 3.1  (I tired to use 3.1.1 but it refused to authenticate correctly stick with the older version)
  • Add the netapp to OnCommand System Manager using the root username/password

If the command manager does not work (connection refused) then its probably because the httpd wasn’t enabled during initial config. Fix as follows:

1) ssh to netapp

netapp1> options httpd
httpd.access legacy
httpd.admin.access legacy
httpd.admin.enable off
httpd.admin.hostsequiv.enable off
httpd.admin.max_connections 512
httpd.admin.ssl.enable off
httpd.admin.top-page.authentication on
httpd.autoindex.enable off
httpd.bypass_traverse_checking off
httpd.enable off
httpd.ipv6.enable off
httpd.log.format common
httpd.method.trace.enable off
httpd.rootdir /vol/vol0/home/http
httpd.timeout 300
httpd.timewait.enable off
3) options httpd.admin.enable true (enable access)
4) options httpd.admin.ssl.enable true (enable secure access)
netapp1> options httpd
httpd.access legacy
httpd.admin.access legacy
httpd.admin.enable on
httpd.admin.hostsequiv.enable off
httpd.admin.max_connections 512
httpd.admin.ssl.enable on
httpd.admin.top-page.authentication on
httpd.autoindex.enable off
httpd.bypass_traverse_checking off
httpd.enable off
httpd.ipv6.enable off
httpd.log.format common
httpd.method.trace.enable off
httpd.rootdir /vol/vol0/home/http
httpd.timeout 300
httpd.timewait.enable off

Now install the licenses from the OnCommand System Manager.  Click Config -> System tools -> Licences -> Add then paste the codes. Note, the ESXi version has different codes to the Vmware Workstation version.

The Netapp is now ready to configure and use.

Read my Netapp Storage Concepts (for network engineers) for more information on how to use the netapp.

 

Disk Structure in the Simulator

The 8.2.1 simulator starts off with:

  • 28 disks (2 shelves with 14 disks each)
netapp01*> storage  show disk
DISK                  SHELF BAY SERIAL           VENDOR   MODEL      REV
--------------------- --------- ---------------- -------- ---------- ----
v4.16                   ?    ?  08561200         NETAPP   VD-1000MB- 0042
v4.17                   ?    ?  08561201         NETAPP   VD-1000MB- 0042
v4.18                   ?    ?  08561202         NETAPP   VD-1000MB- 0042
v4.19                   ?    ?  08561203         NETAPP   VD-1000MB- 0042
v4.20                   ?    ?  08561204         NETAPP   VD-1000MB- 0042
v4.21                   ?    ?  08561205         NETAPP   VD-1000MB- 0042
v4.22                   ?    ?  08561206         NETAPP   VD-1000MB- 0042
  • pool 0 with 14 assigned disks (leaving 14 unowned disks)
  • aggr0, containing plex0, and rg0 (RAID group) with 3 disks in a RAID-DP configuration (1 data disk)
netapp01*> aggr status -v
           Aggr State           Status                Options
          aggr0 online          raid_dp, aggr         root, diskroot, nosnap=off, raidtype=raid_dp,
                                64-bit                raidsize=16, ignore_inconsistent=off,
                                                      snapmirrored=off, resyncsnaptime=60,
                                                      fs_size_fixed=off, lost_write_protect=on,
                                                      ha_policy=cfo, hybrid_enabled=off,
                                                      percent_snapshot_space=0%,
                                                      free_space_realloc=off

                Volumes: vol0

                Plex /aggr0/plex0: online, normal, active
                    RAID group /aggr0/plex0/rg0: normal, block checksums
  • vol0 in aggr0 – thick provisioned 871.916MB in size
netapp01*> vol size vol0
vol size: Flexible volume 'vol0' has size 871916k.


In onCommand click Storage -> Disks

netapp_disk

 

 

Enable access to the OS

Enter advanced mode and unlock the diagnostic user. This will allow you to look at the operating system files/logs

 ssh as root users, Enter a password and confirm.
 priv set advanced
 useradmin diaguser unlock
 useradmin diaguser password

Then launch the systemshell and login as diag and enter the password you have just set:

systemshell

netapp_diag

 

 

References

Netapp Cheat Sheet – Lists most basic cli commands

ESXi Install guide

Add Shelves to the simulator

SSH Key Exchange fails to Cisco devices

I upgraded my VM to use FreeBSD 10.1 and included with this was an upgrade to OpenSSH (OpenSSH_6.6.1p1). When you ssh to some Cisco devices using this version you cannot connect.

Debugging on the client side does not show much other than the connection is dropped.

OpenSSH_6.6.1p1, OpenSSL 1.0.1j-freebsd 15 Oct 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: ciphers ok: [aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc]
debug3: macs ok: [hmac-md5,hmac-sha1,hmac-ripemd160]
debug2: ssh_connect: needpriv 0
debug1: Connecting to rtr01.lpr [10.202.70.151] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/xx/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/xx/.ssh/id_rsa type 1
debug1: identity file /home/xx/.ssh/id_rsa-cert type -1
debug1: identity file /home/xx/.ssh/id_dsa type -1
debug1: identity file /home/xx/.ssh/id_dsa-cert type -1
debug1: identity file /home/xx/.ssh/id_ecdsa type -1
debug1: identity file /home/xx/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/xx/.ssh/id_ed25519 type -1
debug1: identity file /home/xx/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420
debug1: Remote protocol version 2.0, remote software version Cisco-1.25
debug1: no match: Cisco-1.25
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_load_hostkeys: loading entries for host "rtr01.lpr" from file "/home/xx/.ssh/known_hosts"
debug3: ssh_load_hostkeys: found key type RSA in file /home/xx/.ssh/known_hosts:11
debug3: ssh_load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: kex_parse_kexinit: aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5
debug1: kex: server->client aes256-cbc hmac-md5 none
debug2: mac_setup: setup hmac-md5
debug1: kex: client->server aes256-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<8192<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Connection closed by 10.202.70.151

Debugging on the server site (ie on the cisco device) shows

Nov 24 05:46:22.944: SSH1: starting SSH control process
Nov 24 05:46:22.948: SSH1: sent protocol version id SSH-2.0-Cisco-1.25
Nov 24 05:46:22.952: SSH1: protocol version id is - SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420
Nov 24 05:46:22.960: SSH2 1: SSH2_MSG_KEXINIT sent
Nov 24 05:46:22.960: SSH2 1: SSH2_MSG_KEXINIT received
Nov 24 05:46:22.968: SSH2:kex: client->server enc:aes256-cbc mac:hmac-md5
Nov 24 05:46:22.972: SSH2:kex: server->client enc:aes256-cbc mac:hmac-md5
Nov 24 05:46:22.980: SSH2 1: SSH2_MSG_KEX_DH_GEX_REQUEST received
Nov 24 05:46:22.980: SSH2 1: Range sent by client is - 1024 < 8192 < 8192
Nov 24 05:46:22.980: SSH2 1:  Client DH key range mismatch with max built-in DH key on server!
Nov 24 05:46:23.084: SSH1: Session disconnected - error 0x00

The Cisco device supports a maximum key length of 2048 where as the client is requesting a keylength of 8192. A debug from an older client shows that is request a smaller key:

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 145/256
debug2: bits set: 505/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

This change seems to have been introduced in OpenSSH v6.6 and I can’t find a way to turn this off. Cisco has registered a bug CSCuo76464 for this.

The simplest workaround seems to be to reorganize the KexAlgorithms in /etc/ssh/ssh_config by adding the following line:

KexAlgorithms diffie-hellman-group14-sha1,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

 

 

 

 

 

 

 

 

Enabling syslogs in Cisco ASA

I was recently troubling shooting an issue for a client and found that all the syslog messages generated by the ASA were not reaching the syslog server. I could see the syslog generated on the ASA using the “show logg” command however these messages never reached the remote syslog server.

After some reading I found that there were two changes I could make to help this situation:

  1. Turn off log messages to the console port. If this is enabled the ASA syslog subsystem slows down and buffers messages so that it does not overload the 9600 baud console serial port.
  2. If the log queue is full then the rest of the messages are tail dropped

Firstly check if you are tail dropping log messages:

asa-fw-01# show log queue
        Logging Queue length limit : 512 msg(s)
        483 msg(s) discarded due to queue overflow
        0 msg(s) discarded due to memory allocation failure
        Current 0 msg on queue, 512 msgs most on queue

You can see that 483 messages were discarded. The queue length can be increased as follows:

asa-fw-01(config)# logging queue ?
configure mode commands/options:
  <0-8192>  The length of log queue, 512 is default 0 means maximum permitted
            queue size. Max queue size on ASA-5505 is 1024, on ASA-5510 is 2048
            and 8192 on all other platforms

 

Next you can turn off console logging using the “no logging console” command. However this raised a question: What should be the minimal set of commands to provide comprehensive logging? After a bit of research I decided to add the following commands to all our ASAs.

 

!enable logging
logging enable
! add the system time to generated message
logging timestamp
!Optional (allow the standby to send syslogs too)
logging standby
!Turn of sending syslog messages to console port 
no logging console
!make the buffer as large as possible (this buffers stores all the output which is displayed when you run the "show log" command)
logging buffer-size 65536
!Allow log messages to be sent to ssh/telnet sessions. By setting it to debugging I'm allow all messages to be send to ssh session. You could drop it to "informational" if you wanted
logging monitor debugging
!Allow messages up the informational level to be stored in the asa memory buffer. Discard the debugging messages
logging buffered informational
!Allow "all" messages to be send to the syslog server. This is good in-case you want to capture the debug messages for later use
logging trap debugging
!Turn of sending syslog messages to snmp servers. Why would you want to do this? 
no logging history informational
!Allow messages up the informational level to be sent to the ASDM gui
logging asdm informational
!Increase the log queue. Set this as per your ASA hardware type
logging queue 2048
! add the system hostname to generated message
logging device-id hostname
!Tell the ASA about your syslog server
logging host inside 10.202.200.78
!Allow User Authenticaion messages to be send to syslog
logging class auth trap debugging
!Allow interactions with the config prompt to be sysloged
logging class config trap debugging
!Optional: Log IKE and IPSEC messages
logging class vpn trap debugging
!Optional: Log WebVPN client messages
logging class webvpn trap debugging
!Optional: log SSL VPN Client
logging class svc trap debugging

 

My IPSEC Notes

IPSec is used to transfer data between sites in an encrypted manner. There are two basic techniques for encrypting information: symmetric encryption and asymmetric encryption.
With Symmetric encryption both parties share the same key. This allows the sender to encrypt the information using the shared key and the receiver to decrypt this using the same key. The main problem then becomes how the two parties negotiate the same key.
With Asymmetric encryption both parties have their own keys. The sender encrypts the information using one key and the receiver decrypts the information using another key.
IPsec uses symmetric encryption and uses Diffie–Hellman key exchange (DH) to generate the same key at each site. DH works like this:
  1. Alice and Bob agree to use a prime number p = 23 and base g = 5.
  2. Alice chooses a secret integer a = 6, then sends Bob A = ga mod p
    • A = 56 mod 23 = 8
  3. Bob chooses a secret integer b = 15, then sends Alice B = gb mod p
    • B = 515 mod 23 = 19
  4. Alice computes s = Ba mod p
    • s = 196 mod 23 = 2
  5. Bob computes s = Ab mod p
    • s = 815 mod 23 = 2
  6. Alice and Bob now share a secret (the number 2).
If the prime numbers are large enough then the shared secret is considered to be secure as it would take a long time for a computer to test all the options.
However the critical problem with DH is that it’s vulnerable to a man-in-the-middle attack. This basically means that Eve could sit in the middle and negotiate a secret with Alice and different secret with Bob. Eve could then listen to all the traffic between Alice and Bob.
IPSec solves this problem by authenticating Alice and Bob using pre-shared-keys (PSK), Certificates or RSA keys.
This problem of authentication and DH is wrapped up into Internet Key Exchange (IKE) protocol which IPSec uses to ensure secrecy between the parties. IKE has two phases, Phase one negotiates the following:
                  
s    a)  Which prime numbers & base to use
b)      Which encryption to use in Phase two
c)       Generation of the DH shared secret
d)      Authentication of the endpoints using PSK or Certs / RSA
Phase 2 uses the encryption algorithm decided in phase one to encrypt its packets and within this encryption it negotiates the following
 
a)   The encryption method to use for encrypting the data packets between the end points (IPsec
      b)      The shared secret for this encryption
This potentially means that you could have the different encryption methods for Phase 2 and IPsec. It would be wise to set these differently as it increases the overall security of the communications. I would suggest that you use AES256 for Phase 2 and AES 128 for IPsec.
Now consider step 1 of the DH negotiation: “Alice and Bob agree to use a known prime number p and base g”. The values of p and g are selected from a pool of well known numbers which we refer to as Diffie-Hellman Groups. IANA has defined these groups here:
For example Group 2 uses the following well known numbers:

   The prime is 2^1024 – 2^960 – 1 + 2^64 * { [2^894 pi] + 129093 }.

   Its hexadecimal value is

         FFFFFFFF FFFFFFFF C90FDAA2 2168C234 C4C6628B 80DC1CD1

         29024E08 8A67CC74 020BBEA6 3B139B22 514A0879 8E3404DD

         EF9519B3 CD3A431B 302B0A6D F25F1437 4FE1356D 6D51C245

         E485B576 625E7EC6 F44C42E9 A637ED6B 0BFF5CB6 F406B7ED

         EE386BFB 5A899FA5 AE9F2411 7C4B1FE6 49286651 ECE65381

         FFFFFFFF FFFFFFFF

    The generator is 2 (decimal)
Current best practices say to avoid IKE Groups 1, 2, and 5 and use IKE Group 14 or 24, both of which employ 2048-bit DH.
Encryption comes in two flavours DES or AES. The only other option is then the key length used for the encryption.
Protocol
Key Length
Block Size
CAST
40
DES
56
64 Bit
3DES (DES applied three times)
56*3=168 but really ~112 as its not perfect
64 Bit
AES-128
128
128 Bit
AES-196
196
128 Bit
AES-256
256
128 Bit
However, you may encounter some security issues depending on the block size and how much data has to be encrypted. 3DES uses 64-bit blocks, which can lead to trouble after processing 2^(64/2) blocks, i.e. 32 gigabytes; AES uses 128-bit blocks, for a limit of 2^(128/2) blocks, i.e. 2^68 bytes, also known as “quite a lot of data”. 2^68 bytes of data is approximately 20 times the information content of “all human knowledge.”
 
What this really means is if you choose 3DES for your encryption and happen to transmit >32 gigabits of data then your DH key could be extracted. If the keys are never changed, an eavesdropper is listening in on an IPsec session could be easily compromised.
By setting up a lifetime (of time for IKE phase 1, and Time and/or Data for IKE phase 2) we cause the tunnels to be torn down and recreated periodically which reduces the amount of time the attacker would have to decode and take advantage of our tunnels.
IKE phase 2 doesn’t run DH again (since it was run during IKE phase 1 to generate shared secret keys for both peers). So if the Phase 1 key is compromised then the Phase 2 negotiation can viewed. However, if IKE phase 2 doesn’t want to use the keys created in IKE phase 1 there is an option to run DH again in IKE phase 2.  This 2nd DH is called PFS (Perfect Forward Secrecy) and if desired the configuration could add the set pfs xxxx command in the crypto map to implement the PFS.
While this is slower, it makes sure that no keys are dependent on any other previously used keys; no keys are extracted from the same initial keying material. This is to make sure that, in the unlikely event that some key was compromised, no subsequent keys can be derived.
Once the phase-2 negotiation is finished, the VPN connection is established and ready for use.
———————-

Installing Nfsen 1.6.12 on Centos

As you might already know the last time I installed nfsen was in FreeBSD v9 and it used a special NSEL fork for nfdump. This version allowed us to capture the extra information generated by ASA v9 netflow records.

My initial build of nfsen in Feb 2012 has been in continuous use for over two years now. We collect data from 9 sources and only have two two profiles: the default live and one we called services. The services profile basically displays the traffic for each type of service (eg ssh, web, https, dns, vnc, DVR etc). We also have defined a number of alerts which send snmptraps to our network management systems.

image

Since the orignal install two years ago, the main version of nfdump as been rewritten to incorporate NSEL. This post describes how I installed my first prototype system with the latest version of nfdump (1.6.12) and nfsen (1.3.6p1). The only difference will be that I’m going to use CentOS release 6.5 (Final) as the base OS. As this is a prototype build, I’m being very brief with explaining the commands.

(I think the commands should be similar if you wanted to install this in a FreeBSD instance)

I copied the first 7 command from here.

1) Install a new Centos 6.5 System

2) Check iptables is off

iptables -t filter -L -v –n

3) Check SELinux is off

cat /etc/selinux/config
sestatus (show show disabled)
vi /etc/selinux/config
set SELINUX=disabled
reboot

4) Install Apache and other per-requisits
yum install -y httpd php wget gcc make rrdtool-devel rrdtool-perl perl-MailTools perl-Socket6 flex byacc
yum install libtool.x86_64

5) Start HTTPd
service httpd start

6) Enable HTTPd at boot
chkconfig httpd on

7)Get nfdump and nfsen
wget http://sourceforge.net/projects/nfdump/files/stable/nfdump-1.6.12/nfdump-1.6.12.tar.gz (latest version supports nsel)
wget http://sourceforge.net/projects/nfsen/files/stable/nfsen-1.3.6p1/nfsen-1.3.6p1.tar.gz/

8) Compile and install nfdump
tar -zxvf nfdump-1.6.12.tar.gz
./configure –enable-nfprofile –enable-nftrack –enable-sflow –enable-nsel
autoreconf
Make
Make install

9) Add a netflow user
adduser netflow
usermod -a -G apache netflow

10) Compile and install nfsen
tar -xvzf nfsen-1.3.6p1.tar.gz
cd nfsen-1.3.6p1/etc
cp nfsen-dist.conf nfsen.conf

(now make a /data/nfsen folder somwhere on the system, you need lots of space for this)

vi etc/nfsen.conf & change $WWWUSER to apache
vi etc/nfsen.conf & change $WWWGROUP to apache
vi etc/nfsec.conf & change $HTMLDIR    = “/var/www/nfsen/”; to /var/www/html/nfsen
vi etc/nfsec.conf & uncomment $EXTENSIONS = ‘all’; or add $EXTENSIONS = ‘nsel’;

cd ..
./install.pl etc/nfsen.conf

11) Configure ASA: -> https://supportforums.cisco.com/document/30471/netflow-asa

12)Add the data sources as described previously

/data/nfsen/bin/nfsen stop
vi /data/nfsen/etc/nfsen.conf
/data/nfsen/bin/nfsen reconfig
/data/nfsen/bin/nfsen start
chkconfig nfsen on (to make it autostart after reboot)

The web address should now be http:///nfsen/nfsen.php. If you have done everthing right then you can see the following:

image_001

  Once you have some data collected, pick a time slot and and select “List Flows” and hit process. You will then see the extended ASA information (Event / Extended Events)

image_002

One of the new features that the later version have introduced is that filers can now accept additional info other than standard tcpdump parameters. From the man page:

“The filter syntax is comparable to tcpdump  and  extended  for  netflow data.”

This mean you can have a filter that does the following:

“asa event deny and port 80”

image_003

Here is a sample from the nfdump man page:

NSEL/ASA specific filters:

NSEL/ASA Event
asa event
asa event [comp]
select NSEL/ASA event by name or number. If given as number it can be compared with a number

NSEL/ASA denied reason
asa event denied
Select a NSEL/ASA denied event by type

NSEL/ASA extended events
asa xevent [comp]
Select an extended NSELL ASA event by number, or optionally compared by a number.

X-late IP addresses and ports
[src|dst] xip
Select the translated IP address

[src|dst] xnet /
with as a valid translated IPv4 or IPv6 network and as maskbits.  The number of mask bits must
match  the  appropriate  address  familiy in IPv4 or IPv6. Networks may be abreviated such as 172.16/16 if
they are unambiguous.
[src|dst] xport
Select the translated port

NSEL/ASA ingress/egress
ingress [comp] number
Select/compare an ingress ACL

egress ACL [comp]
Select/compare an egress ACL