Provisioning Fully Virtualized OpenShift Clusters on OpenShift Virtualization (o3)

This project (pronounced oh-cubed) automates the deployment of fully virtualized OpenShift clusters on OpenShift Virtualization (no HCP) using Ansible and Red Hat's Advanced Cluster Management (ACM).

The automation can be executed from any Linux workstation with Kubernetes API connectivity to an ACM Hub cluster and an OpenShift Virtualization cluster. The playbooks were developed on a Fedora workstation, which is currently the only tested environment.

To begin, create a directory to organize our work. This directory will eventually include the git repository, our inventory file/vault and artifacts generated during playbook execution. We will also need to create a Python Virtual Environment and install the required Python dependencies and Ansible Collections.

export WORKING_DIRECTORY=~/o3

mkdir -p $WORKING_DIRECTORY

Clone this repository to our working directory:

git clone https://github.com/nasx/o3.git $WORKING_DIRECTORY/o3

Using the python command, setup a new virtual environment as shown:

python -m venv $WORKING_DIRECTORY/ansible-venv

Activate the virtual environment by sourcing the activation script as follows:

source $WORKING_DIRECTORY/ansible-venv/bin/activate

After activating, the name of the virtual environment should now be visible at the beginning of your prompt.

Next, install the required Python dependencies (this will include Ansible):

pip install -U -r $WORKING_DIRECTORY/o3/requirements.txt

Also install the required Ansible collections:

ansible-galaxy collection install -r $WORKING_DIRECTORY/o3/collections/requirements.yaml

The playbooks will leverage ACM to drive most of the cluster provisioning. A service account is needed on the Hub cluster to orchestrate various resources like the InfraEnv, ClusterDeployment and Agent resources and and can be created as follows:

First create a namespace, o3 in this case but really the service account can be in any namespace.

oc new-project o3

Create a service account (called o3) in that namespace:

oc create sa -n o3 o3

Next, add the cluster-admin role to this service account:

oc adm policy add-cluster-role-to-user clusteradmin -n o3 -z o3 --rolebinding-name o3-cluster-admin

Finally, create a token for this service account.

oc create token -n o3 o3 --duration=8760h

This token should be saved in your vault using the variable hub_token.

The cluster hosting the virtual machines for OpenShift is referred to as the virtualization cluster. Create a service account on this cluster the same way you did on the hub cluster and save the token in your vault using the variable virtualization_token.

If the clusters are the same, use the same token value for both variables.

We will need to store tokens for the hub and virtualization clusters in a vault along with their respective API end points and a pull secret to access the Red Hat registries. The following table lists supported variables:

Create a vault with the following command:

ansible-vault create $WORKING_DIRECTORY/vault.yaml

Make sure the vault contains the required variables from the table above:

hub_api: https://api.hub.example.com:6443
hub_token: <o3_service_account_token_on_hub>
pull_secret: <pull_secret_json>  
openshift_subscription:  
  subscribe: false
virtualization_api: https://api.virtualization.example.com:6443
virtualization_token: <o3_service_account_token_on_virtualization>

A basic layout of the inventory file is shown below:

---
all:
  children:
    pg:
      hosts:
        node1:
          <node1 vars>  
        node2:  
          <node2 vars>
        nodeN:  
          <nodeN vars>
  vars:
    <global vars>

The pg group (short for provisioning group and can be renamed to any valid group name) contains all of the nodes that will be added to our cluster. Ansible does not actually login to these nodes during the provisioning process. All other variables are stored globally in vars. Specific configurations for the provisioning group, as well as the global variables are detailed in the next sections.

For reference, a completed/working inventory file is available in the clusters directory at the root of this repository.

When finished, save your inventory file in $WORKING_DIRECTORY/inventory.yaml.

Each host in the inventory file can use the variables listed in the table below.

All of the supported inventory variables are listed in the table below.

To kick off the installation, run the following:

ansible-playbook --ask-vault-pass -e @path/to/vault.yaml -i path/to/inventory.yaml create.yaml

To remove the cluster after installation, run the following:

ansible-playbook --ask-vault-pass -e @path/to/vault.yaml -i path/to/inventory.yaml delete.yaml