3. OPNFV Fuel User Guide

3.1. Abstract

This document contains details about using OPNFV Fuel Hunter release after it was deployed. For details on how to deploy OpenStack, check the installation instructions in the References section.

This is an unified documentation for both x86_64 and aarch64 architectures. All information is common for both architectures except when explicitly stated.

3.2. Network Overview

Fuel uses several networks to deploy and administer the cloud:

Network name Description
PXE/admin Used for booting the nodes via PXE and/or Salt control network
mcpcontrol Docker network used to provision the infrastructure hosts (Salt & MaaS)
management Used for internal communication between OpenStack components
internal Used for VM data communication within the cloud deployment
public Used to provide Virtual IPs for public endpoints that are used to connect to OpenStack services APIs. Used by Virtual machines to access the Internet

These networks - except mcpcontrol - can be Linux bridges configured before the deploy on the Jumpserver. If they don’t exists at deploy time, they will be created by the scripts as libvirt managed networks (except mcpcontrol, which will be handled by Docker using the bridge driver).

3.2.1. Network mcpcontrol

mcpcontrol is a virtual network, managed by Docker. Its only purpose is to provide a simple method of assigning an arbitrary INSTALLER_IP to the Salt master node (cfg01), to maintain backwards compatibility with old OPNFV Fuel behavior. Normally, end-users only need to change the INSTALLER_IP if the default CIDR ( overlaps with existing lab networks.

mcpcontrol uses the Docker bridge driver, so the Salt master (cfg01) and the MaaS containers (mas01, when present) get assigned predefined IPs (.2, .3, while the jumpserver gets .1).

Host Offset in IP range Default address
jumpserver 1st
cfg01 2nd
mas01 3rd

This network is limited to the jumpserver host and does not require any manual setup.

3.2.2. Network PXE/admin


PXE/admin does not usually use an IP range offset in IDF.


During MaaS commissioning phase, IP addresses are handed out by MaaS’s DHCP.


Default addresses in below table correspond to a PXE/admin CIDR of (the usual value used in OPNFV labs).

This is defined in IDF and can easily be changed to something else.

Host Offset in IP range Default address
jumpserver 1st (manual assignment)
cfg01 2nd
mas01 3rd
prx01, prx02 4th, 5th,
gtw01, gtw02, gtw03 ...
kvm01, kvm02, kvm03    
dbs01, dbs02, dbs03    
msg01, msg02, msg03    
mdb01, mdb02, mdb03    
ctl01, ctl02, ctl03    
odl01, odl02, odl03    
mon01, mon02, mon03, log01, log02, log03, mtr01, mtr02, mtr03    
cmp001, cmp002, ...    

3.2.3. Network management


management often has an IP range offset defined in IDF.


Default addresses in below table correspond to a management IP range of (one of the commonly used values in OPNFV labs). This is defined in IDF and can easily be changed to something else. Since the jumpserver address is manually assigned, this is usually not subject to the IP range restriction in IDF.

Host Offset in IP range Default address
jumpserver N/A (manual assignment)
cfg01 1st
mas01 2nd


prx01, prx02


4th, 5th,,

gtw01, gtw02, gtw03 ...


kvm01, kvm02, kvm03



dbs01, dbs02, dbs03



msg01, msg02, msg03



mdb01, mdb02, mdb03



ctl01, ctl02, ctl03



odl01, odl02, odl03



mon01, mon02, mon03,


log01, log02, log03,


mtr01, mtr02, mtr03

cmp001, cmp002, ...    

3.2.4. Network internal


internal does not usually use an IP range offset in IDF.


Default addresses in below table correspond to an internal CIDR of (the usual value used in OPNFV labs). This is defined in IDF and can easily be changed to something else.

Host Offset in IP range Default address
jumpserver N/A (manual assignment, optional)
gtw01, gtw02, gtw03 1st, 2nd, 3rd,,
cmp001, cmp002, ... 4th, 5th, …,, ...

3.2.5. Network public


public often has an IP range offset defined in IDF.


Default addresses in below table correspond to a public IP range of (one of the used values in OPNFV labs). This is defined in IDF and can easily be changed to something else. Since the jumpserver address is manually assigned, this is usually not subject to the IP range restriction in IDF.

Host Offset in IP range Default address
jumpserver N/A (manual assignment, optional)


prx01, prx02


2nd, 3rd,,

gtw01, gtw02, gtw03 4th, 5th, 6th,,
ctl01, ctl02, ctl03 ...
cmp001, cmp002, ...    

3.3. Accessing the Salt Master Node (cfg01)

The Salt Master node (cfg01) runs a sshd server listening on

To login as ubuntu user, use the RSA private key /var/lib/opnfv/mcp.rsa:

jenkins@jumpserver:~$ ssh -o StrictHostKeyChecking=no \
                          -i /var/lib/opnfv/mcp.rsa \
                          -l ubuntu


User ubuntu has sudo rights.


The Salt master IP ( is not hard set, it is configurable via INSTALLER_IP during deployment.


Starting with the Gambia release, cfg01 is containerized, so this also works (from jumpserver only):

jenkins@jumpserver:~$ docker exec -it fuel bash

3.4. Accessing the MaaS Node (mas01)

Starting with the Hunter release, the MaaS node (mas01) is containerized and no longer runs a sshd server. To access it (from jumpserver only):

jenkins@jumpserver:~$ docker exec -it maas bash

3.5. Accessing Cluster Nodes

Logging in to cluster nodes is possible from the Jumpserver, Salt Master etc.

jenkins@jumpserver:~$ ssh -i /var/lib/opnfv/mcp.rsa ubuntu@


/etc/hosts on cfg01 has all the cluster hostnames, which can be used instead of IP addresses.

/root/.ssh/config on cfg01 configures the default user and key: ubuntu, respectively /root/fuel/mcp/scripts/mcp.rsa.

root@cfg01:~$ ssh ctl01

3.6. Debugging MaaS Comissioning/Deployment Issues

One of the most common issues when setting up a new POD is MaaS failing to commission/deploy the nodes, usually timing out after a couple of retries.

Such failures might indicate misconfiguration in PDF/IDF, TOR switch configuration or even faulty hardware.

Here are a couple of pointers for isolating the problem.

3.6.1. Accessing the MaaS Dashboard

MaaS web-based dashboard is available at http://<jumpserver IP address>:5240/MAAS.

The administrator credentials are opnfv/opnfv_secret.

3.6.2. Ensure Commission/Deploy Timeouts Are Not Too Small

Some hardware takes longer to boot or to run the initial scripts during commissioning/deployment phases. If that’s the case, MaaS will time out waiting for the process to finish. MaaS logs will reflect that, and the issue is usually easy to observe on the nodes’ serial console - if the node seems to PXE-boot the OS live image, starts executing cloud-init/curtin hooks without spilling critical errors, then it is powered down/shut off, most likely the timeout was hit.

To access the serial console of a node, see your board manufacturer’s documentation. Some hardware no longer has a physical serial connector these days, usually being replaced by a vendor-specific software-based interface.

If the board supports SOL (Serial Over LAN) over IPMI lanplus protocol, a simpler solution to hook to the serial console is to use ipmitool.


Early boot stage output might not be shown over SOL, but only over the video console provided by the (vendor-specific) interface.

jenkins@jumpserver:~$ ipmitool -H <host BMC IP> -U <user> -P <pass> \
                               -I lanplus sol activate

To bypass this, simply set a larger timeout in the IDF.

3.6.3. Check Jumpserver Network Configuration

jenkins@jumpserver:~$ brctl show
jenkins@jumpserver:~$ ifconfig -a
Configuration item Expected behavior
IP addresses assigned to bridge ports IP addresses should be assigned to the bridge, and not to individual bridge ports

3.6.4. Check Network Connectivity Between Nodes on the Jumpserver

cfg01 is a Docker container running on the jumpserver, connected to Docker networks (created by docker-compose automatically on container up), which in turn are connected using veth pairs to their libvirt managed counterparts (or manually created bridges).

For example, the mgmt network(s) should look like below for a virtual deployment.

jenkins@jumpserver:~$ brctl show mgmt
bridge name   bridge id           STP enabled   interfaces
mgmt          8000.525400064f77   yes           mgmt-nic

jenkins@jumpserver:~$ docker network ls
NETWORK ID    NAME                              DRIVER   SCOPE
81a0fdb3bd78  docker-compose_mgmt               macvlan  local

jenkins@jumpserver:~$ docker network inspect docker-compose_mgmt
        "Name": "docker-compose_mgmt",
        "Options": {
            "parent": "veth_mcp3"

Before investigating the rest of the cluster networking configuration, the first thing to check is that cfg01 has network connectivity to other jumpserver hosted nodes, e.g. mas01 and to the jumpserver itself (provided that the jumpserver has an IP address in that particular network segment).

jenkins@jumpserver:~$ docker exec -it fuel bash
root@cfg01:~# ifconfig -a | grep inet
    inet addr:     Bcast:  Mask:
    inet addr:   Bcast:  Mask:
    inet addr:  Bcast:  Mask:

For each network of interest (mgmt, PXE/admin), check that cfg01 can ping the jumpserver IP in that network segment.


mcpcontrol is set up at container bringup, so it should always be available, while the other networks are configured by Salt as part of the virtual_init STATE file.

root@cfg01:~# ping -c1  # mcpcontrol jumpserver IP
root@cfg01:~# ping -c1  # mcpcontrol mas01 IP


mcpcontrol CIDR is configurable via INSTALLER_IP env var during deployment. However, IP offsets inside that segment are hard set to .1 for the jumpserver, .2 for cfg01, respectively to .3 for mas01 node.

root@cfg01:~# salt 'mas*' pillar.item --out yaml \
              _param:infra_maas_node01_deploy_address \

root@cfg01:~# ping -c1  # PXE/admin jumpserver IP
root@cfg01:~# ping -c1  # PXE/admin mas01 IP
root@cfg01:~# ping -c1   # mgmt jumpserver IP
root@cfg01:~# ping -c1  # mgmt mas01 IP


Jumpserver IP addresses for PXE/admin, mgmt and public bridges are user-chosen and manually set, so above snippets should be adjusted accordingly if the user chose a different IP, other than .1 in each CIDR.

Alternatively, a quick nmap scan would work just as well.

root@cfg01:~# apt update && apt install -y nmap
root@cfg01:~# nmap -sn     # expected: cfg01, mas01, jumpserver
root@cfg01:~# nmap -sn  # expected: cfg01, mas01, jumpserver
root@cfg01:~# nmap -sn   # expected: cfg01, mas01, jumpserver

3.6.5. Check DHCP Reaches Cluster Nodes

One common symptom observed during failed commissioning is that DHCP does not work as expected between cluster nodes (baremetal nodes in the cluster; or virtual machines on the jumpserver in case of hybrid deployments) and the MaaS node.

To confirm or rule out this possibility, monitor the serial console output of one (or more) cluster nodes during MaaS commissioning. If the node is properly configured to attempt PXE boot, yet it times out waiting for an IP address from mas01 DHCP, it’s worth checking that DHCP packets reach the jumpserver, respectively the mas01 container.

jenkins@jumpserver:~$ sudo apt update && sudo apt install -y dhcpdump
jenkins@jumpserver:~$ sudo dhcpdump -i admin_br


If DHCP requests are present, but no replies are sent, iptables might be interfering on the jumpserver.

3.6.6. Check MaaS Logs

If networking looks fine, yet nodes still fail to commission and/or deploy, MaaS logs might offer more details about the failure:

  • /var/log/maas/maas.log
  • /var/log/maas/rackd.log
  • /var/log/maas/regiond.log


If the problem is with the cluster node and not on the MaaS server, node’s kernel logs usually contain useful information. These are saved via rsyslog on the mas01 node in /var/log/maas/rsyslog.

3.7. Recovering Failed Deployments

The first deploy attempt might fail due to various reasons. If the problem is not systemic (i.e. fixing it will not introduce incompatible configuration changes, like setting a different INSTALLER_IP), the environment is safe to be reused and the deployment process can pick up from where it left off.

Leveraging these mechanisms requires a minimum understanding of how the deploy process works, at least for manual STATE runs.

3.7.1. Automatic (re)deploy

OPNFV Fuel’s deploy.sh script offers a dedicated argument for this, -f, which will skip executing the first N STATE files, where N is the number of -f occurrences in the argument list.


The list of STATE files to be executed for a specific environment depends on the OPNFV scenario chosen, deployment type (virtual, baremetal or hybrid) and the presence/absence of a VCP (virtualized control plane).

e.g.: Let’s consider a baremetal enviroment, with VCP and a simple scenario os-nosdn-nofeature-ha, where deploy.sh failed executing the openstack_ha STATE file.

The simplest redeploy approach (which usually works for any combination of deployment type/VCP/scenario) is to issue the same deploy command as the original attempt used, then adding a single -f:

jenkins@jumpserver:~/fuel$ ci/deploy.sh -l <lab_name> -p <pod_name> \
                                        -s <scenario> [...] \
                                        -f # skips running the virtual_init STATE file

All STATE files are re-entrant, so the above is equivalent (but a little slower) to skipping all STATE files before the openstack_ha one, like:

jenkins@jumpserver:~/fuel$ ci/deploy.sh -l <lab_name> -p <pod_name> \
                                        -s <scenario> [...] \
                                        -ffff # skips virtual_init, maas, baremetal_init, virtual_control_plane


For fine tuning the infrastructure setup steps executed during deployment, see also the -e and -P deploy arguments.


On rare occassions, the cluster cannot idempotently be redeployed (e.g. broken MySQL/Galera cluster), in which case some cleanup is due before (re)running the STATE files. See -E deploy arg, which allows either forcing a MaaS node deletion, then redeployment of all baremetal nodes, if used twice (-EE); or only erasing the VCP VMs if used only once (-E).

3.7.2. Manual STATE Run

Instead of leveraging the full deploy.sh, one could execute the STATE files one by one (or partially) from the cfg01.

However, this requires a better understanding of how the list of STATE files to be executed is constructed for a specific scenario, depending on the deployment type and the cluster having baremetal nodes, implemented in:

  • mcp/config/scenario/defaults.yaml.j2
  • mcp/config/scenario/<scenario-name>.yaml

e.g.: For the example presented above (baremetal with VCP, os-nosdn-nofeature-ha), the list of STATE files would be:

  • virtual_init
  • maas
  • baremetal_init
  • virtual_control_plane
  • openstack_ha
  • networks

To execute one (or more) of the remaining STATE files after a failure:

jenkins@jumpserver:~$ docker exec -it fuel bash
root@cfg01:~$ cd ~/fuel/mcp/config/states
root@cfg01:~/fuel/mcp/config/states$ ./openstack_ha
root@cfg01:~/fuel/mcp/config/states$ CI_DEBUG=true ./networks

For even finer granularity, one can also run the commands in a STATE file one by one manually, e.g. if the execution failed applying the rabbitmq sls:

root@cfg01:~$ salt -I 'rabbitmq:server' state.sls rabbitmq

3.8. Exploring the Cloud with Salt

To gather information about the cloud, the salt commands can be used. It is based around a master-minion idea where the salt-master pushes config to the minions to execute actions.

For example tell salt to execute a ping to on all the nodes.

root@cfg01:~$ salt "*" network.ping
                   ^^^                       target
                       ^^^^^^^^^^^^          function to execute
                                    ^^^^^^^  argument passed to the function


Complex filters can be done to the target like compound queries or node roles.

For more information about Salt see the References section.

Some examples are listed below. Note that these commands are issued from Salt master as root user.

3.8.1. View the IPs of All the Components

root@cfg01:~$ salt "*" network.ip_addrs

3.8.2. View the Interfaces of All the Components and Put the Output in a yaml File

root@cfg01:~$ salt "*" network.interfaces --out yaml --output-file interfaces.yaml
root@cfg01:~# cat interfaces.yaml
   hwaddr: 52:54:00:72:77:12
   - address:
     label: enp1s0
   - address: fe80::5054:ff:fe72:7712
     prefixlen: '64'
     scope: link
   up: true

3.8.3. View Installed Packages on MaaS Node

root@cfg01:~# salt "mas*" pkg.list_pkgs

3.8.4. Execute Any Linux Command on All Nodes (e.g. ls /var/log)

root@cfg01:~# salt "*" cmd.run 'ls /var/log'

3.8.5. Execute Any Linux Command on Nodes Using Compound Queries Filter

root@cfg01:~# salt -C '* and cfg01*' cmd.run 'ls /var/log'

3.8.6. Execute Any Linux Command on Nodes Using Role Filter

root@cfg01:~# salt -I 'nova:compute' cmd.run 'ls /var/log'

3.9. Accessing Openstack

Once the deployment is complete, Openstack CLI is accessible from controller VMs (ctl01ctl03).

Openstack credentials are at /root/keystonercv3.

root@ctl01:~# source keystonercv3
root@ctl01:~# openstack image list
| ID                                   | Name                                          | Status |
| 152930bf-5fd5-49c2-b3a1-cae14973f35f | CirrosImage                                   | active |
| 7b99a779-78e4-45f3-9905-64ae453e3dcb | Ubuntu16.04                                   | active |

The OpenStack Dashboard, Horizon, is available at http://<proxy public VIP>. The administrator credentials are admin/opnfv_secret.


A full list of IPs/services is available at <proxy public VIP>:8090 for baremetal deploys.


3.10. Guest Operating System Support

There are a number of possibilities regarding the guest operating systems which can be spawned on the nodes. The current system spawns virtual machines for VCP VMs on the KVM nodes and VMs requested by users in OpenStack compute nodes. Currently the system supports the following UEFI-images for the guests:

OS name x86_64 status aarch64 status
Ubuntu 17.10 untested Full support
Ubuntu 16.04 Full support Full support
Ubuntu 14.04 untested Full support
Fedora atomic 27 untested Full support
Fedora cloud 27 untested Full support
Debian untested Full support
Centos 7 untested Not supported
Cirros 0.3.5 Full support Full support
Cirros 0.4.0 Full support Full support

The above table covers only UEFI images and implies OVMF/AAVMF firmware on the host. An x86_64 deployment also supports non-UEFI images, however that choice is up to the underlying hardware and the administrator to make.

The images for the above operating systems can be found in their respective websites.

3.11. OpenStack Storage

OpenStack Cinder is the project behind block storage in OpenStack and OPNFV Fuel supports LVM out of the box.

By default x86_64 supports 2 additional block storage devices, while aarch64 supports only one.

More devices can be supported if the OS-image created has additional properties allowing block storage devices to be spawned as SCSI drives. To do this, add the properties below to the server:

root@ctl01:~$ openstack image set --property hw_disk_bus='scsi' \
                                  --property hw_scsi_model='virtio-scsi' \

The choice regarding which bus to use for the storage drives is an important one. virtio-blk is the default choice for OPNFV Fuel, which attaches the drives in /dev/vdX. However, since we want to be able to attach a larger number of volumes to the virtual machines, we recommend the switch to SCSI drives which are attached in /dev/sdX instead.

virtio-scsi is a little worse in terms of performance but the ability to add a larger number of drives combined with added features like ZFS, Ceph et al, leads us to suggest the use of virtio-scsi in OPNFV Fuel for both architectures.

More details regarding the differences and performance of virtio-blk vs virtio-scsi are beyond the scope of this manual but can be easily found in other sources online like VirtIO SCSI or VirtIO performance.

Additional configuration for configuring images in OpenStack can be found in the OpenStack Glance documentation.

3.12. OpenStack Endpoints

For each OpenStack service three endpoints are created: admin, internal and public.

ubuntu@ctl01:~$ openstack endpoint list --service keystone
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                          |
| 008fec57922b4e9e8bf02c770039ae77 | RegionOne | keystone     | identity     | True    | internal  |  |
| 1a1f3c3340484bda9ef7e193f50599e6 | RegionOne | keystone     | identity     | True    | admin     | |
| b0a47d42d0b6491b995d7e6230395de8 | RegionOne | keystone     | identity     | True    | public    |    |

MCP sets up all Openstack services to talk to each other over unencrypted connections on the internal management network. All admin/internal endpoints use plain http, while the public endpoints are https connections terminated via nginx at the VCP proxy VMs.

To access the public endpoints an SSL certificate has to be provided. For convenience, the installation script will copy the required certificate to the cfg01 node at /etc/ssl/certs/os_cacert.

Copy the certificate from the cfg01 node to the client that will access the https endpoints and place it under /etc/ssl/certs/. The SSL connection will be established automatically after.

jenkins@jumpserver:~$ ssh -o StrictHostKeyChecking=no -i /var/lib/opnfv/mcp.rsa -l ubuntu \
  "cat /etc/ssl/certs/os_cacert" | sudo tee /etc/ssl/certs/os_cacert

3.13. Reclass Model Viewer Tutorial

In order to get a better understanding of the reclass model Fuel uses, the reclass-doc tool can be used to visualise the reclass model.

To avoid installing packages on the jumpserver or another host, the cfg01 Docker container can be used. Since the fuel git repository located on the jumpserver is already mounted inside cfg01 container, the results can be visualized using a web browser on the jumpserver at the end of the procedure.

jenkins@jumpserver:~$ docker exec -it fuel bash
root@cfg01:~$ apt-get update
root@cfg01:~$ apt-get install -y npm nodejs
root@cfg01:~$ npm install -g reclass-doc
root@cfg01:~$ ln -s /usr/bin/nodejs /usr/bin/node
root@cfg01:~$ reclass-doc --output ~/fuel/mcp/reclass/modeler \

The generated documentation should be available on the jumpserver inside fuel git repo subpath mcp/reclass/modeler/index.html.