ansible, automation, aap,

Ansible Automation Platform - Docs & References

Follow · 19 mins read
Ansible Automation Platform - Docs & References
Share this

Disclaimer:
This document represents an upstream version created during my learning process, proof-of-concepts (PoCs), and other experimental practices. The complete and finalized version of this document might be published on the website techbeatly.com and will be periodically updated.

Ansible Automation Platform

(Formerly Ansible Tower)

Enable RHEL and AAP Subscriptions

Make sure you subscribed to Red Hat and added RHEL Subscription.

$ sudo subscription-manager register
$ sudo subscription-manager attach --pool=<pool_id of RHEL subscription>
$ sudo subscription-manager list --consumed

Now, search for Ansible Automation Platform subscription and attach the pool ID.

$ sudo subscription-manager list --available --all | grep "Ansible Automation Platform" -B 3 -A 6
$ sudo subscription-manager attach --pool=<pool_id>
$ sudo subscription-manager list --consumed

Update required repo for AAP 2.1

$ sudo subscription-manager repos \
  --disable=ansible-automation-platform-2.0-early-access-for-rhel-8-x86_64-rpms

$ sudo subscription-manager repos \
  --enable=ansible-automation-platform-2.1-for-rhel-8-x86_64-rpms

Red Hat Container Registry Authentication

AAP Download Options

Extract the AAP Installer

# e.g.
$ tar xvzf ansible-automation-platform-containerized-setup-bundle-2.5-13-x86_64.tar.gz
$ cd ansible-automation-platform-containerized-setup-bundle-2.5-13-x86_64

Get Red Hat Registry Credential

You need access to Red Hat Container Registry to fetch the continer images for automation controller and execution environment.

You can create a service account in Red Hat registry for the same and use the credential in AAP installation inventory.

# Credentials for container registry to pull execution environment images from,
# comment out registry_username if authentication is not required
registry_url='registry.redhat.io'
registry_username='YOUR_SERICE_ACCOUNT_USERNAME'
registry_password='YOUR_SERICE_ACCOUNT_PASSWORD'

Read more Registry Service Account Management Application

SSH Access and Credentials

sudo or root access to the remote machines is required, you need to use any of the below methods

$ ansible_user=root ansible_ssh_pass=”your_password_here” inventory host or group variables
# or
$ ansible_user=root ansible_ssh_private_key_file=”path_to_your_keyfile.pem” inventory host or group variables
# or
$ ANSIBLE_BECOME_METHOD=’sudo’ ANSIBLE_BECOME=True ./setup.sh

Disaster Recovery for the Ansible Automation Platform

Please note DR and High Availability (“HA”) are not currently supported. Current Ansible Automation Platform customers are needing to configure their DR instances to accurately reflect their node usage and need guidance based on changes due to manifest files. This can be for cold, warm, or hot DR/HA. This article regards how manifests are handled in DR and HA environments.

Refer to the documentation.

Ansible Automation Platform - Containerized Setup

$ export ANSIBLE_COLLECTIONS_PATH=/home/devops/ansible-automation-platform-containerized-setup-bundle-2.4-1-x86_64/collections/

$ ansible-playbook -i inventory ansible.containerized_installer.install

To uninstall a containerized deployment, execute the uninstall.yml playbook.

$ ansible-playbook -i inventory ansible.containerized_installer.uninstall

Reference: Ansible Automation Platform containerized installation (2.5)

Removing instance from cluster

bash-4.4$ awx-manage deprovision_instance --hostname aap-rhel-92-2.lab.local
Instance Removed
Successfully deprovisioned aap-rhel-92-2.lab.local
(changed: True)

How to check AAP and patch version

Refer to the Patch releases document for the component details.

You can check the WEBUI and find the versions of,

  • utomation Controller Version
  • vent Driven Automation Version
  • utomation Hub Version

Also you can check the version from CLI as follows

$ podman inspect -f "" automation-controller-web
4.6.9

# or check the controller RPM
$ podman exec -it automation-controller-web /bin/bash

bash-4.4$ rpm -q ansible-automation-platform-common
ansible-automation-platform-common-2.5-2.el8ap.noarch

AAP SSL Certificate Renewal

Renewing the self-signed SSL certificate

The following steps regenerate a new SSL certificate for both automation controller and automation hub.

  • Add aap_service_regen_cert=true to the inventory file in the [all:vars] section:
[all:vars]
aap_service_regen_cert=true
  • Run the installer.

AAP on Clouds

Reset Gateway user password from CLI

# Login to the automation gateway container
$ podman exec -it automation-gateway bash

# Change password for user; e.g: admin
bash-4.4$ aap-gateway-manage changepassword admin
Changing password for user 'admin'
Password:
Password (again):
Password changed successfully for user 'admin'
bash-4.4$

AD/LDAP Integration - AAP 2.5

(In-Progress)

Example LDAP Configuration for Ansible Automation Platform

  • Name: ActiveDirectorySandboxDev
  • Type: LDAP
  • LDAP Server URI: ldap://192.168.57.101:389
  • LDAP Bind DN: cn=ansible_bind,cn=users,dc=sandbox,dc=dev
  • LDAP Bind Password: your-bind-user-password
  • LDAP Group Type: ActiveDirectoryGroupType
  • LDAP User DN Template: Optional

NOTE: This LDAP User DN Template configuration has not tested successfully.

sAMAccountName=%(user)s,OU=AAP,DC=sandbox,DC=dev

For openLDAP style it would be of below format

uid=%(user)s,cn=users,cn=accounts,dc=sandbox,dc=dev
  • LDAP Start TLS: Off/On
  • LDAP Require Group (optional): CN=aap-users,OU=AAP,DC=sandbox,DC=dev
  • LDAP User Search:
[
  "cn=users,dc=sandbox,dc=dev",
  "SCOPE_SUBTREE",
  "(sAMAccountName=%(user)s)"
]

Optional: To allow the user to login with email:

[
  "cn=users,dc=sandbox,dc=dev",
  "SCOPE_SUBTREE",
  "(|(sAMAccountName=%(user)s)(mail=%(user)s))"
]
  • LDAP Group Search:
[
  "cn=users,dc=sandbox,dc=dev",
  "SCOPE_SUBTREE",
  "(objectClass=group)"
]
  • LDAP User Attribute Map:
{
  "email": "mail",
  "last_name": "sn",
  "first_name": "givenName"
}
  • LDAP Group Type Parameters:
{
  "name_attr": "cn"
}

Authentication mapping

Examples:

Superuser:

  • Trigger: Groups
  • Groups: cn=aap-admins,cn=users,dc=sandbox,dc=dev

NW-Org-Admin:

  • Trigger: Groups
  • Groups: cn=network-admins,cn=users,dc=sandbox,dc=dev

NW-Org-Member:

  • Trigger: Groups
  • Groups: cn=network-team,cn=users,dc=sandbox,dc=dev

NW-Team-Admins:

  • Trigger: Groups
  • Groups: cn=network-admins,cn=users,dc=sandbox,dc=dev
  • Team: Network-Admins
  • Organization: Network Dept
  • Role: Team Admin

NW-Org-Member:

  • Trigger: Groups
  • Groups: cn=network-team,cn=users,dc=sandbox,dc=dev
  • Team: Network-Team
  • Organization: Network Dept
  • Role: Team Member

References:

AD/LDAP Integration - AAP 2.4

Example LDAP Configuration for Ansible Automation Platform

  • Name: ActiveDirectoryExampleCom
  • Type: LDAP
  • LDAP Server URI: ldap://192.168.57.101:389
  • LDAP Bind DN: CN=ansible_bind,CN=users,DC=sandbox,DC=dev
  • LDAP Bind Password: your-bind-user-password
  • LDAP Group Type: MemberDNGroupType
  • LDAP User DN Template:

NOTE: This LDAP User DN Template configuration has not tested successfully.

sAMAccountName=%(user)s,OU=AAP,DC=sandbox,DC=dev

For openLDAP style it would be of below format

uid=%(user)s,cn=users,cn=accounts,DC=sandbox,DC=dev
  • LDAP Start TLS: Off/On
  • LDAP Require Group (optional): CN=aap-users,OU=AAP,DC=sandbox,DC=dev
  • LDAP User Search:
[
  "DC=sandbox,DC=dev",
  "SCOPE_SUBTREE",
  "(sAMAccountName=%(user)s)"
]
  • LDAP Group Search:
[
  "DC=sandbox,DC=dev",
  "SCOPE_SUBTREE",
  "(objectClass=group)"
]
  • LDAP User Attribute Map:
{
  "email": "mail",
  "first_name": "givenName",
  "last_name": "sn"
}
  • LDAP Group Type Parameters:
{
  "member_attr": "member",
  "name_attr": "cn"
}

LDAP Organization Map

{
  "Network-Ops": {
    "admins": "CN=network_admins,OU=AAP,DC=sandbox,DC=dev",
    "remove_admins": false,
    "remove_users": false,
    "users": true
  },
  "XYZCorp-CaC": {
    "admins": "CN=ansible_admins,OU=AAP,DC=sandbox,DC=dev",
    "remove_admins": false,
    "remove_users": false,
    "users": true
  }
}

LDAP Team Map

{
  "cac-admins": {
    "organization": "XYZCorp-CaC",
    "remove": true,
    "users": "cn=ansible_admins,ou=AAP,DC=sandbox,DC=dev"
  },
  "cac-operators": {
    "organization": "XYZCorp-CaC",
    "remove": true,
    "users": "cn=ansible_operators,ou=AAP,DC=sandbox,DC=dev"
  },
  "network-admins": {
    "organization": "Network-Ops",
    "remove": true,
    "users": "cn=network_admins,ou=AAP,DC=sandbox,DC=dev"
  },
  "network-operators": {
    "organization": "Network-Ops",
    "remove": true,
    "users": "cn=network_operators,ou=AAP,DC=sandbox,DC=dev"
  }
}

LDAP Troubleshooting and testing

# install ldap client if doe
$ sudo yum install openldap-clients -y

# install nc
$ sudo yum install nmap-ncat -y

Ensure you can reach the LDAP server and port:

$ nc -zv 192.168.57.101 389
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.57.101:389.
Ncat: 0 bytes sent, 0 bytes received in 0.06 seconds.

Test LDAP Bind Credentials

Verify if the ansible_bind account can bind to LDAP:

$ ldapwhoami -x -H ldap://192.168.57.101:389 -D "CN=ansible_bind,CN=users,DC=sandbox,DC=dev" -w 'Welcome123'

Or use environment variables for quick access

export LDAP_SERVER=192.168.57.101
export LDAP_PORT=389
export LDAP_BIND_USER=ansible_bind
export LDAP_BIND_PASSWORD='Welcome123'

ldapwhoami -x \
  -H ldap://${LDAP_SERVER}:${LDAP_PORT} \
  -D "CN=${LDAP_BIND_USER},CN=Users,DC=sandbox,DC=dev" \
  -w "${LDAP_BIND_PASSWORD}"

Test LDAP User Search

Note: The ldapsearch utility is not automatically pre-installed with automation controller, however, you can install it from the openldap-clients package.

If you cannot install the package (if you are running it inside Container or OpenShift), then test the connectivity using curl command as follows.

In this scenario, we have an Automation Controller running using Podman (containerized AAP).

Test 389 or 636 depends on the port you are using.

[devops@aap-rhel-92-1 ~]$ podman exec -it automation-controller-task /bin/bash
bash-4.4$  podman exec -it automation-controller-task /bin/bash
bash-4.4$ curl -kv http://WIN2019.example.com:636
* Rebuilt URL to: http://WIN2019.example.com:636/
*   Trying 192.168.57.137...
* TCP_NODELAY set
* Connected to WIN2019.example.com (192.168.57.137) port 636 (#0)
> GET / HTTP/1.1
> Host: WIN2019.example.com:636
> User-Agent: curl/7.61.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

bash-4.4$ curl -kv http://WIN2019.example.com:389
* Rebuilt URL to: http://WIN2019.example.com:389/
*   Trying 192.168.57.137...
* TCP_NODELAY set
* Connected to WIN2019.example.com (192.168.57.137) port 389 (#0)
> GET / HTTP/1.1
> Host: WIN2019.example.com:389
> User-Agent: curl/7.61.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Validate the user search query:

$ ldapsearch -x -H ldap://192.168.57.101:389 \
-D "CN=ansible_bind,CN=users,DC=sandbox,DC=dev" \
-w 'Welcome123' \
-b "DC=sandbox,DC=dev" \
"(sAMAccountName=john)"

Test LDAP Group Search

Validate the group search query:

$ ldapsearch -x -H ldap://192.168.57.101:389 \
-D "CN=ansible_bind,CN=users,DC=sandbox,DC=dev" \
-w 'Welcome123' \
-b "DC=sandbox,DC=dev" \
"(objectClass=network-team)"

Verify LDAP User Attribute Mapping

Check if email, givenName, and sn attributes are available for a user:

$ ldapsearch -x -H ldap://192.168.57.101:389 \
-D "CN=ansible_bind,CN=users,DC=sandbox,DC=dev" \
-w 'Welcome123' \
-b "DC=sandbox,DC=dev" \
"(sAMAccountName=john)" mail givenName sn

Verify Group Membership

If you want to validate if a user belongs to a specific group:

$ ldapsearch -x -H ldap://192.168.57.101:389 \
-D "CN=ansible_bind,CN=users,DC=sandbox,DC=dev" \
-w 'Welcome123' \
-b "DC=sandbox,DC=dev" \
"(member=CN=user1,CN=users,DC=sandbox,DC=dev)"
  • Replace user1 with an actual LDAP username.

Verify TLS Configuration (if enabled)

If LDAP Start TLS is On, test the TLS connection:

$ ldapsearch -x -H ldap://192.168.57.101:389 \
-Z \
-D "CN=ansible_bind,CN=users,DC=sandbox,DC=dev" \
-w 'Welcome123' \
-b "DC=sandbox,DC=dev" \
"(sAMAccountName=user1)"
  • -Z enables StartTLS.

Test from the Automation Controller Container (if applicable)

If Ansible Automation Platform is running in a container:

$ podman exec -it <container_name> bash \
ldapsearch -x -H ldap://192.168.57.101:389 \
-D "CN=ansible_bind,CN=users,DC=sandbox,DC=dev" \
-w 'Welcome123' \
-b "DC=sandbox,DC=dev" \
"(sAMAccountName=john)"
  • Replace <container_name> with the appropriate container name.
$ ldapsearch -x  -H ldap://192.168.57.101:389 -D "CN=ansible_bind,CN=Users,DC=sandbox,DC=dev" -b "DC=sandbox,DC=dev" -w yourbindpassword

# search for a specific user
$ ldapsearch -x  -H ldap://192.168.57.101:389 -D "CN=ansible_bind,CN=Users,DC=sandbox,DC=dev" -w yourbindpassword -b "cn=devops,cn=Users,DC=sandbox,DC=dev"

Red Hat Ansible Automation Platform Hardening Guide

Migrating, Updating and Upgrading AAP

Ansible Automation Hub

Documentation

Tools for Ansible Automation Platform

Database Replication

(Not officially supported)

Verify Streaming Replication

On primary database node

swtich to postgres user.

[ansible@AAP-DB-1 ~]$ sudo su - postgres

Then,

[postgres@AAP-DB-1 ~]$ psql -c 'select application_name, state, sync_priority, sync_state from pg_stat_replication;'
 application_name |   state   | sync_priority | sync_state
------------------+-----------+---------------+------------
 awx              | streaming |             1 | sync
(1 row)

or

[postgres@AAP-DB-1 ~]$ psql -c 'select client_addr, state, sent_lsn, write_lsn,flush_lsn, replay_lsn from pg_stat_replication;'
  client_addr  |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn
---------------+-----------+-----------+-----------+-----------+------------
 192.168.56.32 | streaming | 0/402AF38 | 0/402AF38 | 0/402AF38 | 0/402AF38
(1 row)

[postgres@AAP-DB-1 ~]$ psql -c 'select * from pg_stat_replication;'
  pid  | usesysid |  usename   | application_name |  client_addr  | client_hostname | client_port |
backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn |    write
_lag    |    flush_lag    |   replay_lag    | sync_priority | sync_state
-------+----------+------------+------------------+---------------+-----------------+-------------+---------
----------------------+--------------+-----------+-----------+-----------+-----------+------------+---------
--------+-----------------+-----------------+---------------+------------
 46392 |    23055 | replicator | awx              | 192.168.56.32 |                 |       37668 | 2022-02-
03 07:23:25.296914+00 |              | streaming | 0/402B4E0 | 0/402B4E0 | 0/402B4E0 | 0/402B4E0  | 00:00:00
.002516 | 00:00:00.006299 | 00:00:00.007456 |             1 | sync
(1 row)

On secondary/standby database node

Switch to postgres user

[ansible@AAP-DB-2 ~]$ sudo su - postgres
Last login: Thu Feb  3 07:59:07 UTC 2022 on pts/1
[postgres@AAP-DB-2 ~]$

Then,

[postgres@AAP-DB-2 ~]$ psql -c 'select * from pg_stat_wal_receiver;'
  pid  |  status   | receive_start_lsn | receive_start_tli | received_lsn | received_tli |      last_msg_send_time
       |     last_msg_receipt_time     | latest_end_lsn |        latest_end_time        | slot_name |

                      conninfo

-------+-----------+-------------------+-------------------+--------------+--------------+------------------------
-------+-------------------------------+----------------+-------------------------------+-----------+-------------
------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------
 51587 | streaming | 0/4000000         |                 1 | 0/4031110    |            1 | 2022-02-03 08:15:14.493
483+00 | 2022-02-03 08:15:14.493939+00 | 0/4031110      | 2022-02-03 08:15:14.493483+00 |           | user=replica
tor passfile=/var/lib/pgsql/.pgpass channel_binding=prefer dbname=replication host=192.168.56.31 port=5432 applica
tion_name=awx fallback_application_name=walreceiver sslmode=prefer sslcompression=0 ssl_min_protocol_version=TLSv1
.2 gssencmode=prefer krbsrvname=postgres target_session_attrs=any
(1 row)

or,

[postgres@AAP-DB-2 ~]$ psql -c 'SELECT pg_last_xact_replay_timestamp();'
 pg_last_xact_replay_timestamp
-------------------------------
 2022-02-03 08:15:54.592737+00
(1 row)

Ansible Tower HA Failover

Check current Tower database configurations.

[root@AnsibleController1x-1 ~]# cat /etc/tower/conf.d/postgres.py
# Ansible Tower database settings.

DATABASES = {
   'default': {
       'ATOMIC_REQUESTS': True,
       'ENGINE': 'awx.main.db.profiled_pg',
       'NAME': 'awx',
       'USER': 'awx',
       'PASSWORD': """aapadmin""",
       'HOST': 'aap-db-1.lab.local',
       'PORT': '5432',
       'OPTIONS': { 'sslmode': 'prefer',
                    'sslrootcert': '/etc/pki/tls/certs/ca-bundle.crt',
       },
   }
}

Database tips

Check Data path

[postgres@AAP-DB-2 ~]$ pg_ctl status
pg_ctl: server is running (PID: 51581)
/usr/bin/postgres "-D" "/var/lib/pgsql/data"

[postgres@AAP-DB-2 ~]$ psql -c "SHOW data_directory;"
   data_directory
---------------------
 /var/lib/pgsql/data
(1 row)

Ansible Tower with database SSL connection enabled

Verify SSL Certificate-Key pair

If all the three match, the SSL certificate matches the Private Key.

# SSL certificate
$ openssl x509 –noout –modulus –in <file>.crt | openssl md5

# RSA private key
$ openssl rsa –noout –modulus –in <file>.key | openssl md5

# CSR
$ openssl req -noout -modulus -in <file>.csr | openssl md5

SSL Certificate with Passphrase

Troubleshooting

References

aap cloud automation containers kubernetes ansible automation platform red hat ansible

Latest Stories

Red Hat Remote Exams – Everything you need to know
In Careers, DevOps, How To,
Ansible Capacity Planning: Ansible contol node and automation controller
In Ansible, Automation, DevOps,
Best Kubernetes Monitoring & Observability Tools (Updated)
In Cloud, DevOps, Featured,
Book Review: Cloud Native Anti-Patterns
In Books, anti-patterns, cloud adoption,
Artificial Intelligence (AI) – Learning and Development
In Artificial Intelligence, Careers, Developer,

Featured

  1. Best Kubernetes Monitoring & Observability Tools (Updated)
    In Cloud, DevOps, Featured,
  2. Integrating Ansible Automation Controller with Splunk
    In Ansible, Automation, Featured,
  3. Mastering Tekton: Your Guide to Cloud-Native CI/CD Workflows
    In Automation, Courses, DevOps,
  4. Learn Ansible – A Comprehensive Guide for Courses and Exams
    In Ansible, Ansible Resources, Automation,
  5. Free GitOps and CI/CD Courses and Certifications
    In ArgoCD, DevOps, Featured,
  6. Terraform Cheat Sheet
    In DevOps, Featured, Infrastructre & Hardware,
  7. Kubernetes vs OpenShift – 15 Facts You Should Know
    In Cloud, DevOps, Featured,
  8. Introduction to OpenShift BootCamp
    In DevOps, Featured, Kubernetes,
  9. How to set up and use Python virtual environments for Ansible
    In Ansible, Automation, DevOps,
  10. 5 Mistakes You Should Avoid During Kubernetes Exam
    In Cloud, DevOps, Featured,
  11. 10 Tips for your Kubernetes Exam – CKA & CKAD
    In DevOps, Featured, Kubernetes,
  12. CKA, CKS & CKAD Learning Path and Certification
    In Featured, Infrastructre & Hardware, Kubernetes,