Backing up FreeNAS to Amazon S3

ZFS is excellent at storing all your data safely but “there are still lots of potential ways for your data to die, and you still need to back up your pool. Period. PERIOD!

The process below shows how you can setup a regular sync to Amazon S3 and was tested on FreeNAS-9.3-STABLE-201509022158.

Setup the S3 Folders and Upload User

Log on to the AWS interface

Select Services -> S3

  • create a bucket
  • configure the logs as requested

Select Services -> Security and Identity -> IAM

  • On the left either click “Users” or click “Create individual IAM users” in the main interface
  • Create a new user and note the AWS Access Key and AWS Secret Key
  • Select the user and give is the permission “AmazonS3FullAccess”

Add Jail for s3cmd

On FreeNAS: Click Jails -> Add Jail and give it a name s3cmd.

So attached to the jail and install s3cmd.

ssh root@freenas.local
jexec <1> /bin/tcsh (where 1 is the jail ID from the previous command)
pkg install py27-dateutil-2.4.2
pkg install py27-magic-5.25
pkg install gnupg
pkg install py27-s3cmd-1.6.0
mkdir /mnt/zfspool

Click Jails -> s3cmd -> Storage -> Add storage

Then configure the following

  • Source is the folder in the zfspool you want to backup
  • Destination is /mnt/zfspool/new_folder
  • Ensure its set as Read Only
  • Select Create Directory

Configure s3cmd

root@s3cmd:~ # s3cmd --configure
Then configure the following:
AWS Access Key
AWS Secret Key
Default Region: US
Encryption password: password safe
Path to gpg: /usr/local/bin/gpg


root@s3cmd:~ # s3cmd ls
2016-02-29 05:09  s3://Bucket1
2015-03-12 04:36  s3://Bucket2
2015-12-07 05:43  s3://Bucket3

Start your first sync

root@s3cmd:~ # s3cmd -v sync /mnt/ s3://Bucket1
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 96943 files, this may take some time...



Layer 3 and Layer 2 VPN Characteristics


The critical information here is that Layer 3 VPNs needs the service provider to get involved in customer routing.

With Layer 2 VPNs the frames are forwarded right through to the other side.

Image 2


In an L2VPN service, the ISP does not require information about the customer’s network topology, policies, routing information, point-to-point links, or network point-to-point links from other ISPs.

There are two fundamentally different kinds of Layer 2 VPN service that a service provider could offer to a customer: Virtual Private Wire Service (VPWS) and Virtual Private LAN Service (VPLS). There is also the possibility of an IP-only LAN-like Service (IPLS).

A VPWS is a VPN service that supplies an L2 point-to-point service. As this is a point-to-point service, there are very few scaling issues with the service as such. Scaling issues might arise from the number of end-points that can be supported on a particular PE.

A VPLS is an L2 service that emulates LAN service across a Wide Area Network (WAN). With regard to the amount of state information that must be kept at the edges in order to support the forwarding function, it has the scaling characteristics of a LAN. Other scaling issues might arise from the number of end-points that can be   supported on a particular PE.

Cisco BFD

Cisco Bidirectional Forwarding (BFD)


BFD provides fast BFD peer failure detection times independently of all media types, encapsulations, topologies, and routing protocols BGP, EIGRP, IS-IS, and OSPF. By sending rapid failure detection notices to the routing protocols in the local router to initiate the routing table recalculation process, BFD contributes to greatly reduced overall network convergence time.

BFD payload control packets will be encapsulated in UDP packets, using destination port 3784.

The closest alternative to BFD in conventional EIGRP, IS-IS, and OSPF deployments is the use of modified failure detection mechanisms for EIGRP, IS-IS, and OSPF routing protocols.

If you set EIGRP hello and hold timers to their absolute minimums, the failure detection rate for EIGRP falls to within a one- to two-second range.

If you use fast hellos for either IS-IS or OSPF, these Interior Gateway Protocol (IGP) protocols reduce their failure detection mechanisms to a minimum of one second.

There are several advantages to implementing BFD over reduced timer mechanisms for routing protocols:

Although reducing the EIGRP, IS-IS, and OSPF timers can result in minimum detection timer of one to two seconds, BFD can provide failure detection in less than one second.

Because BFD is not tied to any particular routing protocol, it can be used as a generic and consistent failure detection mechanism for EIGRP, IS-IS, and OSPF.

Because some parts of BFD can be distributed to the data plane, it can be less CPU-intensive than the reduced EIGRP, IS-IS, and OSPF timers, which exist wholly at the control plane.

BFD has two different modes: Asynchronous (as described above) and Demand.

BFD has two operating modes that may be selected, as well as an
   additional function that can be used in combination with the two
   The primary mode is known as Asynchronous mode.  In this mode, the
   systems periodically send BFD Control packets to one another, and if
   a number of those packets in a row are not received by the other
   system, the session is declared to be down.

   The second mode is known as Demand mode.  In this mode, it is assumed
   that a system has an independent way of verifying that it has
   connectivity to the other system.  Once a BFD session is established,
   such a system may ask the other system to stop sending BFD Control
   packets, except when the system feels the need to verify connectivity
   explicitly, in which case a short sequence of BFD Control packets is
   exchanged, and then the far system quiesces.  Demand mode may operate
   independently in each direction, or simultaneously.

   An adjunct to both modes is the Echo function.  When the Echo
   function is active, a stream of BFD Echo packets is transmitted in
   such a way as to have the other system loop them back through its
   forwarding path.  If a number of packets of the echoed data stream
   are not received, the session is declared to be down.  The Echo
   function may be used with either Asynchronous or Demand mode.  Since
   the Echo function is handling the task of detection, the rate of
   periodic transmission of Control packets may be reduced (in the case
   of Asynchronous mode) or eliminated completely (in the case of Demand

   Pure Asynchronous mode is advantageous in that it requires half as
   many packets to achieve a particular Detection Time as does the Echo
   function.  It is also used when the Echo function cannot be supported
   for some reason.

Giving zcat more CPUs

Troubleshooting for me generally starts with parsing logs and what better way than using a combination of zcat and grep. Here is my most recent example:

zcat messages*.gz | grep “Username = whoisit” | grep “Duration:” >/tmp/matches

This works well on the old single cpu servers, but on a multi-cpu hyperthreading server zcat only consumes one CPU to 100% percent.

top - 12:44:33 up 110 days, 20:03,  2 users,  load average: 1.06, 1.17, 1.16
Tasks: 169 total,   2 running, 167 sleeping,   0 stopped,   0 zombie
Cpu(s): 17.2%us,  2.6%sy,  0.0%ni, 76.6%id,  3.6%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1921968k total,  1846908k used,    75060k free,     5304k buffers
Swap:  4193272k total,        0k used,  4193272k free,  1659364k cached

29319 root      20   0  4428  616  432 R 99.7  0.0   0:10.84 gzip
29320 root      20   0  100m  892  748 S 19.3  0.0   0:02.06 grep

A number of web pages suggest to use pigz however if you look closely at the documentation it can parallelize only for compression tasks. For decompression its quite useless.

Decompression can’t be parallelized, at least not without specially 
prepared deflate streams for that purpose. As a result, pigz uses a 
single thread (the main thread)  for  decompression,  but  will  create
three  other threads  for  reading,  writing,  and  check  calculation, 
which can speed up decompression under some circumstances.  Parallel 
decompression can be turned off by specifying one process ( -dp 1 or 
-tp 1 ).

This is where gnu-parallels steps up.  Its basically like running a “for” loop and assigning one file to each process on its own cpu.

You can install gnu-parallels as shown below otherwise you can select from a number of excuses for not installing GNU Parallels.

(wget -qO - ||  curl | bash

This requires you allow pgpkeyserver TCP 11371  port outbound on your internet connection so that it can check the signed image.

Once parallels is installed your command becomes:

ls mess*.gz | parallel -k 'zcat {}|grep "Username = whoisit" | grep "Duration:"' >/tmp/match

When you look at the top command its got the awesome goodness of 6 CPUs.

top - 13:49:49 up 110 days, 21:08,  2 users,  load average: 7.70, 7.30, 4.98
Tasks: 186 total,   8 running, 178 sleeping,   0 stopped,   0 zombie
Cpu(s): 91.0%us,  7.8%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  1.2%si,  0.0%st
Mem:   1921968k total,  1847476k used,    74492k free,     3400k buffers
Swap:  4193272k total,      176k used,  4193096k free,  1650776k cached

31654 root      20   0  4424  608  432 R 89.4  0.0   0:57.74 gzip
31657 root      20   0  4424  608  432 R 89.1  0.0   0:57.16 gzip
31648 root      20   0  4424  608  432 R 88.4  0.0   1:05.91 gzip
31666 root      20   0  4424  612  432 R 88.4  0.0   0:17.95 gzip
31660 root      20   0  4424  608  432 R 87.4  0.0   0:53.46 gzip
31651 root      20   0  4424  608  432 R 87.1  0.0   1:03.67 gzip
31649 root      20   0  100m  896  752 S 10.6  0.0   0:07.86 grep
31655 root      20   0  100m  892  752 S 10.6  0.0   0:06.72 grep
31658 root      20   0  100m  896  752 S 10.6  0.0   0:06.70 grep
31667 root      20   0  100m  896  752 S 10.6  0.0   0:02.10 grep
31661 root      20   0  100m  896  752 S 10.3  0.0   0:06.40 grep
31652 root      20   0  100m  896  752 S 10.0  0.0   0:07.58 grep




Building Kodi / XBMC

This are some notes I made when setting up Kodi.

The features I’m after (other than default)

  1. Aeon Nox Skin
  2. Multiple libraries (to split my movies by language)
  3. TV Shows
  4. File browser for stuff that does not fit either of these or is short term
  5. Fan to be displayed on the main page


In this step you are going to install Kodi and install your first theme. I like to use Aeon Nox as it excellent on a 3m projector screen.

1) Download and Install Kodi
2) Run Kodi
3) Goto System -> Settings -> Add-ons-> Install from repository ->Skin
4) Select Aeon Nox -> Install & wait for it to download
5) When asked, switch to the new skin and keep the changes
6) Hit “backspace” till you get back to the main window


Import Movies (all of them)

In this step you will import all your movies. We’ll create the sub-libraries later.

This steps assumes your movies have all the meta data stored in the same folders. Do do this you will need to use Ember media manager.

1) Goto Video -> Files -> Add videos
2) Select Browse and add the movie folders (repeat this step for all movie folders you have, eg on multiple HDDs, we’ll filter later)
3) Select “this directory contains: Movies
4) Do not scrape from the movie database and select “Local information only”. (We will use Ember Media manager to scrap)
5) Movies are in separate folders
6) Scan recursively
7) Click Ok
8) Refresh all items in path

Import TV Shows (all of them)

1) Goto Video -> Files -> Add videos
2) Select Browse and add the movie folders (repeat this step for all movie folders you have, eg on multiple HDDs)
3) This directory contains: TV Shows
4) Do not scrape from the movie database and select “Local information only”. (We will use Ember Media manager to scrap)
5) Click Ok
6) Refresh all items in path


Create a sub-libraries


Add a play list (for Bollywood Movies)

1) Goto Video -> Playlists -> New smart playlist
2) Type = Movies
3) Name of play List = Bollywood Movies
4) Add new rule Where (path) contains “Bollywood Movies” – not case sensitive
5) Order by “Sort title”

Repeat this for the following playlists

Hollywood Movies

Bollywood Blu Ray

Hollywood Blu Ray

3D Movies


External Media Player

I prefer to use MPC-HC & VLC rather than the built in media player. This is how you configure it.

Also I setup a rule which sends the “rar” file to MPC-HC to be played rather than decompressed.


1) In C:\Users\<username>\AppData\Roaming\XBMC\userdata
2) Ccreate playercorefactory.xml (or edit if one exists)
3) Add the following XML statements

   <player name="MPC-HC" type="ExternalPlayer" audio="false" video="true">
     <filename>C:\Program Files (x86)\MPC-HC\mpc-hc.exe</filename>
     <args>"{1}" /fullscreen /close</args>

   <player name="VLC" type="ExternalPlayer" audio="false" video="true">
     <filename>C:\Program Files (x86)\VideoLAN\VLC\vlc.exe</filename>
     <args>"{1}" --fullscreen</args>
<rules action="prepend">
 <rule filename=".*rar" player="MPC-HC"/>
 <rule video="true" player="MPC-HC"/>


Turn off some defaults in the Aeon Nox Skin

1) System -> Appearance -> Skin -> Show RSS news feeds (Turn Off)
2) System -> Appearance -> Skin -> settings -> Home window -> position = low (I used to like this but now I prefer to leave it in the center)


Setup the Aeon Nox menus for Playlists

1) System -> Appearance -> Skin -> Settings -> Home window -> Setup Aeox Nox main menu

In this window do the following disable all unused items (leave video / system for future): Pictures / Music / Live TV / Movies / Favorites / Concerts

Now we will create our own Custom Movies menu item and point it to a playlist created above:

1) select Custom 1 -> Change name to Bollywood
2) Default select action -> Video Library -> “Playlist >” -> Display it
3) Set item background -> Video Library -> “Playlist >” -> same playlist as in previous step

NOTE: you need to scroll down the list as there are two menu items which are called playlist. You want the second menu item, which as an expansion.

You can now add two sub menus to this Custom1 -> Manage submenu and then proceed to add two sub menus each which point to a play list. To do this click

a) Change Action -> Video Library -> “Playlist >” -> Display it”
b) Set label -> type in the menu item name

For each movie menu item set it as an “infoWall display” by selecting it on the menu on the left hand side of the screen.

For the TV Shows set it to “logo” which look pretty nice.


Setup an Aeon Nox menus as a File browser (of sorts)

Add a menu item (or sub menu item) with a custom action:

– for local folders
ActivateWindow(10006,”C:\Media\Kids Movies\”,return)

– for network folders

NOTE: This cannot span multiple HDDs, you will need to create a new menu item for each.


Add a program launcher

Like the previous menu, add a custom action:

Start firefox and open netflix: System.Exec(C:\Progra~2\Mozill~1\Firefox.exe


(not necessary) Add a program launcher (eg Firefox / Internet Explorer / Calibre) – Does not work any more

Angelscry’s Repository 1.2.2

1) Obtain the repository installation zip file from above
2) In order to install the Repository go to your system -> addons -> install from zip file
Choose this option and in the dialog that appears and navigate to the repository zip file.
3) Afterwards click on the Button “Get Addons” and choose the newly installed repository
4) Then select program Addons -> Advanced Launcher -> Install

to configure

1) system -> addons -> enabled addons -> program addons -> Advanced Laucher -> configure

Adding a Launcher

1) cick on Apps -> Advanced Launcher
2) click default to create a new “standalone launcher”

Select the “Standalone (normal PC executable)” option
Browse and select the executable file of the application you want to start
In the next window you can pass command-line arguments. If the application you want to run does not need any arguments or does not support them you can leave this field blank. Otherwise just fill in the argument you have to pass.
This title will be used when viewing the application in Advanced Launcher.
Choose the game platform. This indicates the platform of your game-system for the image-/data-scrapers. If you do not add a game but an application, this option does not matter.

If you did not configure Advanced Launcher to use default fanart- and thumbnail-folders you have to do these steps too, otherwise you’re through at this point:

Select the thumbnail path. This is the folder where the thumbnail image of the launcher will be downloaded to. If there is already an image (jpeg, gif, png or bmp format) with the same name as the executable file (can include the _thumb suffix),, this image will be automatically used for the launcher.
Select the fanart path. This is the folder where the fanart image of the launcher will be downloaded to. If there is already an image (jpeg, gif, png or bmp format) with the same name as the executable (can include the _fanart suffix), this image will be automatically used for the launcher.

After adding all the programs, add them to the favouries
Then add the favs to custom menu items


The more easy way the found the right XBMC command line to start a launcher (or an item) of Advanced Launcher directly from XBMC is to add it as an XBMC favourite. For this, highlight a launcher (or an item) and go into its context menu (“C” key) then select Add to Favourites option. Then edit your /xbmx/userdata/favourites.xml file to discover the right command to use.

Item and stand alone launcher command will be like this :


Files launcher command will be like this :


So, in your case, the code to start your Hyperspin application will surely be :




(not necessary) Add random script items

Install zip from (

Follow this for random backgrounds


Cisco NAT

Cisco make their NAT translations really hard to understand. I believe this is because they use terminology that really does not make any sense.

Cisco defines these terms:

  • Inside local address—The IP address assigned to a host on the inside network. This is the address configured as a parameter of the computer OS or received via dynamic address allocation protocols such as DHCP. The address is likely not a legitimate IP address assigned by the Network Information Center (NIC) or service provider.
  • Inside global address—A legitimate IP address assigned by the NIC or service provider that represents one or more inside local IP addresses to the outside world.
  • Outside local address—The IP address of an outside host as it appears to the inside network. Not necessarily a legitimate address, it is allocated from an address space routable on the inside.
  • Outside global address—The IP address assigned to a host on the outside network by the host owner. The address is allocated from a globally routable address or network space.

Now forget about these for a moment and consider the diagram below. All it talks about are “original ip addresses” and “translated ip address”.

Cisco NATUsing this terminology you can easily workout what the required NAT statement is. Cisco use the inside/outside local/global terms to include the direction the packet is traversing. For network engineer you only need to add a source and a destination interface and it all works out without any tears.