Merlin: Network Magic

Simply a better way; a magical way; to collect, transform, and configure network state information from the CLI or REST API

What is Network Automation, Infrastructure as Code, and Merlin?

Watch these videos for an introduction and live demo of the Merlin, Excalibur, and Lancelot features

What Is Network Automation using Infrastructure As Code?: https://www.youtube.com/embed/Wo6GmvwdSdY

Learn Genie Learn: https://youtu.be/-Ns5sGIKRPI

Powered by Genie

And the pyATS framework

Featuring

And now

Also

And for your Asset Management needs - introducting - Excalibur

Welcome!

Merlin is an infrastructure as code and network automation tool that transforms CLI command and REST API data, using the Cisco Genie parsers, the Cisco pyATS Python library, and Python to automatically generate, at scale, better documentation from the output; send #chatbots; #voicebots; even #phonebots!

Merlin First "Learns" the following features and protocols about the device:

Then Merlin runs multiple "show" commands and creates the following:

A Nice JSON file (command_output.json)

A Nice YAML file (command_output.yaml)

A CSV spreadsheet (command_output.csv)

A Markdown file (command_output.md)

An HTML page (command_output.html)

An interactive HTML Mind Map (command_output_mind_map.html)

An interactive HTML NETJSON NetGraph (command_output_netjson.html)

A TinyDB

Instant messages to WebEx, Slack, Discord, and others

Text-to-Speech, in over 200 languages, creating customized MP3 audio files in a human voice

Phone calls to any phone number in the world

Instantly. With the push of a button.

Excalibur

If you are here for the Excalibur utility please follow this guide

Excalibur - Automated Asset Management

Genie and pyATS

The main Genie documentation guide:

https://developer.cisco.com/docs/genie-docs/

The main pyATS documentation guide:

https://developer.cisco.com/docs/pyats/

The Cisco's Test Automation Solution

The Cisco Test Automation GitHub repository

https://github.com/CiscoTestAutomation

Here are the pyATS documentation guides on Testbed files and Device Connectivity:

Testbed and Topology Information: https://pubhub.devnetcloud.com/media/pyats/docs/topology/index.html

Device Connection: https://pubhub.devnetcloud.com/media/pyats-getting-started/docs/quickstart/manageconnections.html

Testbed File Example: https://pubhub.devnetcloud.com/media/pyats/docs/topology/example.html

Device Connections: https://developer.cisco.com/docs/pyats/#!connection-to-devices

Secret Strings (how I encrypted the enable secret in my testbed file): https://pubhub.devnetcloud.com/media/pyats/docs/utilities/secret_strings.html

Getting Started

Requirements (instructions below)

• pyATS
• Rich
• markmap (CLI)
• netjson netgraph

Virtual Environment

We recommend running Merlin in a Python virtual environment while testing or developing. This will help keep your host system clean and allow you to have multiple environments to try new things. If you are not using a virtual environment, start at the download/clone step below.

You will also need Python 3, pip, and venv installed on your host system.

In your project directory, create your virtual environment

python3 -m venv env

Activate (use) your new virtual environment (Linux):

source env/bin/activate

Download or clone the Merlin repository:

git clone https://github.com/automateyournetwork/merlin

Install pyATS, Rich, markmap, netjsongraph, xmltodict and TinyDB into your environment:

pip install -r requirements.txt

sudo apt update
sudo apt install npm
sudo npm install markmap-cli -g
npm install netjsongraph.js --save

If you run into any installation issues with pyATS, please see the installation guide here: https://pubhub.devnetcloud.com/media/pyats-getting-started/docs/install/installpyATS.html

When you are finished with your Merlin session, enter the deactivate command to exit the python virtual environment.

Merlin's Magic

How to update the appropriate testbed/testbed_*.yaml file to reflect your device inventory and configuration:

Devices:

    4500: <-- Update to your router hostname (MUST MATCH HOSTNAME)

        alias: <-- Update your alias; this can be the hostname or any friendly name you want

        type: <-- This should be "router" for L3 routers with ARP tables and VRFs and such 

        platform: <-- Your Catalyst IOS-XE Platform

        username: <-- Your Cisco device username

        password: <-- Your Enable Secret; It is strongly recommended to follow the "Secret Strings" link to encrypt your secrets

        ip: <-- The management IP address of your router 

    Copy & Paste and make each device unique to scale this up to many routers

Devices:

    3850: <-- Update to your switch hostname (MUST MATCH HOSTNAME)
    9300: <-- Update to your switch hostname (MUST MATCH HOSTNAME)

        alias: <-- Update your alias; this can be the hostname or any friendly name you want

        type: <-- This should be "switch" for L2 switches without ARP tables or VRFs but features like PoE 

        platform: <-- Your Catalyst IOS-XE Platform

        username: <-- Your Cisco device username

        password: <-- Your Enable Secret; It is strongly recommended to follow the "Secret Strings" link to encrypt your secrets

        ip: <-- The management IP address of your router 

    Copy & Paste and make each device unique to scale this up to many routers    

Take Merlin to the Cloud

Merlin now connects to the following Cisco.com APIs:

• Recommended Software Release (using the real version of software from your devices)

• Serial Number 2 Infomation (using the real serial numbers, for every part, from your devices)

• Product Security Incident Response Team (PSIRT) (using the real version of software from your device)

Take Merlin to the Cloud

Merlin on ISE

Merlin now supports the Cisco Identity Services Engine (ISE) External RESTful Service (ERS) and Monitoring and Troubleshooting (MnT) APIs !

Merlin on ISE

Pro tip:

You can create a testbed file using an Excel spreadsheet (see testbed/SampleTestbedFile.xlsx), this is useful if you are not familiar with YAML and/or have a long list of devices where copying and pasting in Excel would be easier.

1. Edit the relevant fields in the sample file.

2. Run the following command:

pyats create testbed file --path SampleTestbedFile.xlsx --output my_testbed.yaml

3. That's it. You're good to go!

Ensure SSH connectivity at from your host's CLI, and run the pyATS job:

Cisco IOS-XE:

pyats run job IOS_XE_merlin_job.py --testbed-file testbed/testbed_ios_xe.yaml

Cisco DevNet Sandbox for CSR1000v (see more here)

pyats run job DevNet_Sandbox_CSR_merlin_job.py --testbed-file testbed/testbed_DevNet_CSR1000v_Sandbox.yaml

Cisco ISE:

pyats run job pyats run job ISE_merlin_job.py

JunOS 17 / 18 / 19:

pyats run job JUNOS_merlin_job.py --testbed-file testbed/testbed_juniper.yaml

F5 BIG-IP:

python3 F5_merlin.py

cd Camelot

ls 

To view the pyATS log in a web browser Locally

pyats logs view

To view the pyATS log in a web browser remotely

pyats logs view --host 0.0.0.0 --port 8080 -v

Explore Camelot!

Here is an example of just one of the treasures you will find: the show ip route command!

Here is what a Global Routing Table looks like in JSON:

The same routing table, but in YAML:

The JSON and YAML outputs are incredible representations of the routing table and can be used for further pyATS testing or data modeling.

"Business-ready" documentation includes the incredibly powerful and versatile Comma-Separated Values (csv) spreadsheet format.

Markdown, the format this README file is written in, can also be used to express the data in a lightweight format that renders nicely in modern browsers.

What about a full-blown HTML webpage? Merlin also creates at least one of these per command.

Which renders nicely like this in your browser:

Another HTML page, an interactive mindmap, is also created from the Markdown file!

Another Interactive HTML page, this time a NETJSON NetGraph, is also crteated

To launch a Python web server and make Camelot available in a browser, where you can view the HTML pages:

Launch a local web server available on the same host:

cd Camelot
pushd;  python3 -m http.server --bind 127.0.0.1 8888; popd;

Launch your web browser and visit

http://127.0.0.1:8888/

Launch a local web server available to remote hosts:

cd Camelot
pushd;  python3 -m http.server --bind 0.0.0.0 8888; popd;

Launch your web browser and visit

http://{{ your server IP }}:8888/

Sharepoint 2016 Plug-in

Every artifact can also be uploaded dynamically to a Sharepoint 2016 Document Library !

The CSV files work with the Office extension and present as version controlled Excel-online files!

Add the code from the SharePoint2016_Plugin.py file. The show version command is used in the example.

Simply add the SHAREPOINT code, updating to reflect the appropriate command:

#Enter your SharePoint site and target library
APIs:
  inventory:
    sharePointUrl: 'https://{{ mydomain.com }}'
    folderUrl: '/Inventory'
    sharePointFileName: "Inventory/Inventory.csv"
    uploadFileName: "Inventory.csv"
    sharepoint_domain: mydomain
    sharepoint_user: {{ domain_user_name }}
    sharepoint_password: {{ domain_password }}
  contracts:
    sharePointUrl: 'https://{{ mydomain.com }}'
    folderUrl: '/Inventory'
    sharePointFileName: "Inventory/Contracts.csv"
    uploadFileName: "Contracts.csv"
    sharepoint_domain: mydomain
    sharepoint_user: {{ domain_user_name }}
    sharepoint_password: {{ domain_password }}

Don't have your own lab gear to test with?

Test Merlin using the Cisco DevNet IOS-XE Sandbox

Test Merlin using the Cisco DevNet Nexus 9k Sandbox

Test Merlin using the Cisco DevNet Identity Services Engine 3.0 + pxGrid Sandbox

Supported Command Index

Cisco IOS-XE:

learn ACL

learn ARP

learn BGP

learn CONFIG

learn Dot1X

learn Interface

learn LLDP

learn NTP

learn OSPF

learn Routes

learn STP

learn VLAN

learn VRF

show access lists

show access session

show access session interface {{ interface }} detail

show authentication sessions

show authentication session interface {{ interface }} detail

show cdp neighbors

show cdp neighbors details

show environment all

show etherchannel summary

show interfaces

show interfaces status

show interfaces trunk

show inventory

show ip arp

show ip arp vrf {{ vrf }}

show ip interface brief

show ip ospf

show ip ospf database

show ip ospf interface

show ip ospf neighbor

show ip ospf neighbor detail

show ip route

show ip route vrf {{ vrf }}

show issu state detail

    * 4500X IOS-XE in VSS 

show mac address-table

show ntp associations

show power inline

show version

show vlan

show vrf

Cisco NXOS:

learn ACL

learn ARP

learn BGP

learn CONFIG

learn Dot1X

learn Interface

learn LLDP

learn NTP

learn OSPF

learn PLATFORM

learn Routes

learn STP

learn VLAN

learn VRF

show access lists

show bgp process VRF all

show bgp sessions
   
show interfaces status

show inventory

show ip arp

show ip arp vrf {{ vrf }}

show ip interface brief

show ip ospf

show ip route

show ip route vrf {{ vrf }}

show mac address-table

show version

show vlan

show vrf

show vrf all detail

show vrf all interface

Cisco Identity Services Engine (ISE)

External RESTful Service (ERS) APIs 

    Administrators

    Allowed Protocols

    Authorization Profiles

    downloadable/dynamic Access Control Lists (dACLs)   

    Endpoint Groups 

    Identity Groups 

    Network Devices     

Monitoring and Troubleshooting (MnT) APIs   

    Total Active Sessions   

    Active Sessions 

    MAC Session Details

JunOS:

show chassis hardware

show system information

F5 BIG-IP

/mgmt/tm/ltm/virtual

/mgmt/tm/sys/file/ssl-cert

Cross Platform Tests

Tested on:

Cisco:

    Cisco Catalyst 4500X-16 03.11.03a.E

    Cisco Catalyst 9300-48UXM Gibraltar

    Cisco Catalyst 9200-24P 16.12.03 and 17.03.02a

    Cisco Catalyst 3850-12X48U Gibraltar 16.12.04

    Cisco CSR1000v

    Cisco IOSv

    Cisco vISE 2.7

Juniper:

    JunOS 17, 18, 19

F5: 

    i2600 REST API

The World's First Talking Network is Powered by Merlin