[Notes] - Configuration Management with Ansible
What is Ansible?
It's a tool to automate IT tasks. Example: to perform a specific task like configure something, restart a service, or check the status of a service on tens or hundreds of servers.
We do it in 4 different ways:
- Execute tasks from your own machine
- Configuration/Installation/Deployment steps in a single YAML file
- Re-use same file multiple times and for different environments
- More reliable and less likely for errors
Supporting all infrastructure: from operating systems to cloud providers.
Ansible is agentless.
How does Ansible work?
Ansible work with Modules(Small programs that do the actual work). They get pushed to the target server, do their work and get removed. Modules are very granular, one module can do one small specific task(like, for user creation one module, for creating a directory one module).
There are multiple modules that can do specific tasks:
- Jenkins Module: we can create and delete Jenkins jobs and many other things
- Docker Module: create container, start container, apply configuration and much more
- Postgres Module: Rename a table,set owner,truncate table and much more
Ansible-playbooks:
Playbook = 1 or more Plays
The sequential modules are grouped into tasks, where each task makes sure the module gets executed with certain arguments and also describes the task with a name.
YAML strictly follows indentation.
tasks:
- name: Rename table
postgresql_table:
table: test1
rename: test2
Where should these tasks execute?
HOSTS (on the list of IPs mentioned in hosts file)
With which user should the tasks execute?
REMOTE_USER (for AWS, ec2-user)
Use variables for repeating values.
A play is a list of tasks, with which user on which hosts it needs to be executed. Playbook = 1 or more plays.
Playbook describes:
- How and in which order
- At what time and where (on which machines)
- What (the modules) should be executed
It is a good practice naming plays.
Where does the "hosts:" value come from in the playbook?
The hosts reference is defined in Ansible inventory list.
The hosts file keeps the list of inventory:
Inventory = all the machines (IPs or hostnames) involved in task executions. That groups multiple IP addresses or hostnames
10.20.1.0
[webserver]
10.22.0.1
10.22.0.2
[database]
10.23.0.1
10.23.0.2
Ansible Tower: UI dashboard from Red Hat.
-Centrally store automation tasks
-Across teams
-Configure permissions
-Manage inventory
Comparable/Alternative Tools: puppet and chef
Ansible:
Simple YAML
agentless
Puppet and Chef:
Ruby more difficult to learn
Installation needed(on target servers)
So need for managing updates on target servers
Install Ansible:
Ansible machine requirement: Python needs to be installed.
We can install Ansible either on the local server (or) on a remote server(recommended)
MAC: brew install ansible
[OR]
pip install ansible
Steps to install it on Windows10:
1. Install Windows Subsystem for Linux (WSL):
Open PowerShell as an administrator: wsl --install
2. Install a Linux distribution from the Microsoft Store:(e.g., Ubuntu, Debian, or openSUSE).
3. Set up your Linux distribution:
Provide a username and password when prompted during the initial setup
4. Update your Linux distribution: sudo apt update && sudo apt upgrade -y
5. Install Ansible: sudo apt install ansible -y
6. Verify the Ansible installation: ansible --version
Ansible connects to the target servers using ssh, it does not need any agent installed on those servers. However note that on Linux servers, there has to be Python installed so that Ansible can execute commands. Since Ansible is written in python, so it needs Python interpreter to configure the server.
Ansible Inventory File:
A file containing data about the ansible client servers
"hosts" means the manged servers
the default location for file: /etc/ansible/hosts
Ansible needs username/password (or) private ssh key to get authenticated with remote hosts.
hosts
[ip1] ansible_ssh_private_key_file=~/.ssh/id_rsa ansible_user=root[ip2] ansible_ssh_private_key_file=~/.ssh/id_rsa ansible_user=root
Ansible ad-hoc commands:
ad-hoc commands are not stored for future uses
A fast way to interact with desired servers
$ ansible [pattern] -m [module] -a "[module options]"
- [pattern] = targeting hosts and group
- "all" = default group, which contains every host.
- Example(execute on all servers): ansible all -i hosts -m ping
- Execute on a specific server(aws group): ansible aws -i hosts -m ping
- Execute on a specific server(target one server): ansible 192.168.0.2 -i hosts -m ping
Grouping hosts:
You can put each host in more than one group.
You can create groups that track:
WHERE - a datacenter/region, e.g. east,west
WHAT - e.g. database servers, web servers etc
WHEN - which stage, e.g. dev, test, prod environment.
Using Group specific variables:
[aws]
15.206.80.78 ansible_user=ubuntu #For ubuntu VM
35.154.182.205
[aws:vars]
ansible_ssh_private_key_file=~/.ssh/id_rsa
ansible_user=ec2-user
ansible_python_interpreter=/usr/bin/python3
Host Key Checking:
It is enabled by default in Ansible.
It guards against server spoofing and man-in-the-middle attacks.
Disable host key checking:
In recent ansible versions, the default directory: /etc/ansible is not getting created. We can disable host key checking in ansible configuration file: /etc/ansible/ansible.cfg (or) we can configure file: ~/ansible.cfg in the current user's home directory as well.
ansible.cfg
[defaults]
HOST_KEY_CHECKING = false
interpreter_python = /usr/bin/python3
#Configure the default hosts file
inventory = hosts
enable_plugins = aws_ec2
#Remote User
remote_user = ec2-user
private_key_file = ~/.ssh/id_rsa
Changes can be made and used in a configuration file which will be searched for in the following order: order of precedence
-
ANSIBLE_CONFIG (environment variable if set)
- ansible.cfg (in the current directory)
- ~/.ansible.cfg (in the home directory)
- /etc/ansible/ansible.cfg
Create a simple Playbook:
Playbook is
-ordered list of tasks
-plays & tasks runs in order from top to bottom
We can't give an IP address in the "hosts" section of play, which is not included in hosts file.
my-playbook.yaml
---
- name: Configure nginx web server
hosts: aws
tasks:
- name: Install nginx server
apt:
name: nginx=1.10.0-0ubuntu1
state: present
- name: Start nginx server
service:
name: nginx
state: started
Run Playbook:
ansible-playbook -i hosts my-playbook.yaml
ansible-playbook my-playbook.yaml (If default hosts file is configured in ansible.cfg)
To get additional troubleshooting info:
ansible-playbook my-playbook.yaml -vv
The "Gather Facts" module is automatically called by playbooks, to gather useful variables about remote hosts that can be used in playbooks. So Ansible provides many facts about the system automatically.
Idempotency: Most Ansible modules check whether the desired state has already been achieved (It will check the Actual State Vs Desired state, just like Terraform). If the desired state is already there on the server, it won't make any changes.
Why collection index is introduced(no module index) in latest Ansible version(2.10).
In Ansible 2.9 and earlier versions, all modules were included. Which means when we installed them we have one Ansible distribution that contains "Ansible code" and "all the modules and plugins" for different use cases. Since Ansible grow in size, Ansible engineers decided to modularize the Ansible code -> Separated Ansible code and "Ansible Module & Plugins". Collections/Modules maintanined and managed by Ansible community itself.
Ansible 2.10 and later:
Ansible/Ansible(ansible-base) repo contains the core Ansible programs.
Modules and Plugins moved into various "collections"
What is a collection compared to a Module?
A packaging format for bundling and distributing Ansible content
Can be released and installed independent of other collections (Collection: Playbook, Plugins, Modules, documentation,etc). Now all modules are part of a collection. You can also create own Collection (with plugins, modules and playbooks).
Collections follow a simple data structure: required a galaxy.yml file (containing metadata) at the root level of the collection.
Ansible Plugins: Pieces of code that add to Ansible's functionality or modules. You can also write your own plugins.
Ansible Galaxy: Collections group basically Ansible content (modules and plugins). Where is this content hosted and shared? where do I get Ansible Collections? One of the main hubs of the collections is Ansible Galaxy. It is the place where the code of the collection lives, when ever you need you can download it from here. It is a command-line utility to install individual collections.
ansible-galaxy collection install <collection name>
Whenever we need to install a collection, we can use the command: ansible-galaxy collection install <collection_name>. Whenever we need to update only a few specific collections, we can only update those few collections(with the new Ansible version) instead of updating all.
========
- To print output of a command in Ansible, we can use "register" module, which can capture the output of the command into register assigned variable.
- The "Debug" module prints statements during execution, which is useful for debugging variables or expressions.
- "command" and "shell" modules are not idempotent. This means whatever commands/tasks we mentioned, will be executed every time we run the playbook. To overcome this, we can use "conditionals" in Ansible.
- Python Vs Ansible: In Python, we need to check the Status (whether the execution is successful or not). Ansible and Terraform handle that state check for us.
- privilege escalation: become. "become_user" = set to user with desired privileges, We have to enable "become: True" before we use "become_user". Default is root(i.e., become:yes).'become' allows you to 'become' another user(different from the user that logged into the machine)
---
- name: Install node and npm
hosts: 43.205.139.175
tasks:
- name: Update apt repo and cache
dnf: update_cache=yes
become: yes
- name: Install nodejs and npm
become: yes
dnf:
name:
- nodejs
- npm
state: present
- name: Create new linux user for node app
hosts: 43.205.139.175
become: yes
tasks:
- name:
user:
name: kishore
comment: Installation user
group: root #root user group on amazon linux
- name: Deploy nodejs app
hosts: 43.205.139.175
become: True
become_user: kishore
tasks:
- name: Unpack the nodejs file
unarchive:
src: ~/ansible/bootcamp-node-project-1.0.tgz
dest: /home/kishore
# remote_src: yes (Incase the file is remote)
- name: Install dependencides
npm:
path: /home/kishore/package
- name: Start the application
command:
chdir: /home/kishore/package
cmd: node server
async: 1000
poll: 0
- name: Ensure app is running
shell: ps aux | grep -w node
register: app_status #To register output of shell, into app_status variable
- debug: msg={{app_status.stdout_lines}}
Registering variables:
Create variables from the output of an Ansible task.
This variable can be used in any later tasks in your play (within curly braces {{ }}).
Referencing variables:
Using double curly braces.
If you start a value with {{ value }} immediately after :(colon), you must quote the whole expression to create valid YAML syntax. (Ex: src; "{{node-file-location}}"
Define Variables:
We can define variables in couple of ways:
- Define in the yaml file itself with "vars:"
- location: ~/ansible
- version: 1.0
- user_home_dir: /home/kishore
- Passing variables on the command line
ansible-playbook -i hosts deploy-node.yaml -e "version=1.0 \
location=~/ansible/bootcamp-node-project-1.0.tgz name=nginx2"
- External variables file (uses YAML syntax)
project-vars:
=============
location: ~/ansible
uname: nginx2
version: 1.0
user_home_dir: /home/{{uname}}
linux_user: newuser
user_groups: adm,docker
Usage: in playbook
=======
- name: Install docker & docker-compose python module
hosts: aws
vars_files:
- project-vars
Naming of Variables:
- Not valid: Python keywords, such as async. Playbook keywords, such as environment
- Valid: Letters, numbers and underscores
- Should always start with a letter
- DON'T define like this: linux-name, linux name, linux.name or 12. Instead we can write it as linux_name (or) linuxname.
Parameterize Playbook:
---
- name: Install node and npm
hosts: 65.1.136.165
tasks:
- name: Update apt repo and cache
dnf: update_cache=yes
become: yes
- name: Install nodejs and npm
become: yes
dnf:
name:
- nodejs
- npm
state: present
- name: Create new linux user for node app
hosts: 65.1.136.165
become: yes
vars_files:
- project-vars
tasks:
- name: "Create linux user"
user:
name: "{{uname}}"
comment: Installation user
group: root #root user group on amazon linux
register: user_status
debug: msg={{user_status}}
- name: Deploy nodejs app
hosts: 65.1.136.165
become: True
become_user: "{{uname}}"
vars_files:
- project-vars
# vars:
# - user_home_dir: /home/{{uname}}
tasks:
- name: Unpack the nodejs file
unarchive:
src: "{{location}}/bootcamp-node-project-{{version}}.tgz" #If you start a value with {{ }}, quotes are must
dest: "{{user_home_dir}}"
# remote_src: yes (Incase the file is remote)
- name: Install dependencides
npm:
path: "{{user_home_dir}}/package"
- name: Start the application
command:
chdir: "{{user_home_dir}}/package"
cmd: node server
async: 1000
poll: 0
- name: Ensure app is running
shell: ps aux | grep -w node
register: app_status #To register output of shell, into app_status variable
- debug: msg={{app_status.stdout_lines}}
Find a directory with regular expression:
- name: Find a folder with the name nexus
find:
paths: /opt
pattern: "nexus-*"
file_type: directory
register: find_result
- debug: msg={{find_result}}
Ansible Conditionals:
- When: applies to a single task
- name: Find folder with name nexus
find:
paths: /opt
pattern: "nexus-*"
file_type: directory
register: find_result
- name: Check nexus folder stats
stat:
path: /opt/nexus
register: stat_result
- name: Rename nexus folder
shell: mv {{find_result.files[0].path}} /opt/nexus
when: not stat_result.stat.exists
"file" module:
Manage files and file properies. For windows targets: "win_file" module
Modules to change contents of a file: There are a couple of modules to change the contents of a file.
- "blockinfile" module: Insert/update/remove a multi-line text surrounded by customizable marker lines
- name: Start nexus with nexus user
hosts: ip
tasks:
- name: set run_as_user nexus
blockinfile:
path: /opt/nexus/bin/nexus.rc
block: | # pipe represents multi line string
run_as_user="nexus"
#other_config="testing"
- "lineinfile" module: Ensures a partifular line is in a file, or replace an existing line using regex. Useful when you want to change a single line in a file only.
- "replace" module: to change multiple lines
Nexus Installation Playbook:
hosts:
ansible.cfg:
[defaults]
HOST_KEY_CHECKING = false
inventory = hosts #Configure default hosts file
install-nexus.yaml: (on Amazon linux machine)
---
- name: Install jdk and net-tools
hosts: aws
become: yes
tasks:
- name: Update apt repo and cache
dnf: update_cache=yes
- name: Install jre and net-tools
dnf:
name:
- java-1.8.0-amazon-corretto
- java-1.8.0-amazon-corretto-devel
- net-tools
state: present
- name: Download and Untar nexus
hosts: aws
become: yes
tasks:
- name: Check nexus folder stats
stat:
path: /opt/nexus
register: stat_result
- name: Download nexus
get_url:
url: https://download.sonatype.com/nexus/3/nexus-3.57.0-01-unix.tar.gz
dest: /opt
register: download_result
#- debug: msg={{archive_file}}
- name: Untar nexus
unarchive:
src: "{{download_result.dest}}"
dest: /opt/
remote_src: yes
when: not stat_result.stat.exists
- name: Find folder with name nexus
find:
paths: /opt
pattern: "nexus-*"
file_type: directory
register: find_result
- name: Rename nexus folder
shell: mv {{find_result.files[0].path}} /opt/nexus
when: not stat_result.stat.exists
- name: Create nexus user to own nexus folders
hosts: aws
become: yes
vars_files:
- project-vars
tasks:
- name: Ensure nexus group exists
group:
name: nexus
state: present
- name: "Create nexus user"
vars:
- uname: nexus
user:
name: "{{uname}}"
comment: Nexus user
group: nexus #root user group on amazon linux
#register: user_status
#debug: msg={{user_status}}
- name: Make nexus user, owner of nexus folder
file:
path: "{{ item }}"
state: directory
owner: nexus
group: nexus
recurse: yes
loop:
- /opt/nexus
- /opt/sonartype-work
- name: Start nexus with nexus user
hosts: aws
become: true
become_user: nexus
tasks:
- name: set run_as_user nexus
# blockinfile:
# path: /opt/nexus/bin/nexus.rc
# block: | # pipe represents multi line string
# run_as_user="nexus"
# #other_config="testing"
lineinfile:
path: /opt/nexus/bin/nexus.rc
regexp: '^#run_as_user=""' #this regexp line will be replaced with below line
line: run_as_user="nexus"
- name: Start nexus
command: /opt/nexus/bin/nexus start
- name: verify nexus running
hosts: aws
tasks:
- name: check with ps
shell: ps -ef|grep -w nexus
register: app_status
- debug: msg={{app_status.stdout_lines}}
- name: wait for one minute
pause:
minutes: 1 #wait for 1min for nexus port listening
- name: Check with netstat
shell: netstat -nlpt
register: app_status
- debug: msg={{app_status.stdout_lines}}
Push the code from non git directory to an existing git repo directory:
- git clone existing git repo to a separate directory
- copy required file from local directory to the cloned git repo directory
- git add .; git commit -m "Ansible practice files"
- git push
Alternatively, we can also try using commands like: git remote add orign <repo url>
Ansible playbook : install docker and docker-compose:
---
- name: Install python3,docker and docker-compose
hosts: aws
become: yes
tasks:
- name: Make sure python3 and docker are installed
vars:
ansible_python_interpreter: /usr/bin/python
dnf: #yum module as of now supports only python2 interpreter
name:
- python3
- docker
update_cache: yes
state: present
- name: Install docker-compose
get_url:
#using "lookup" from jinja template to interpret commands inside url
url: https://github.com/docker/compose/releases/download/1.27.4/docker-compose-Linux-{{lookup('pipe', 'uname -m')}}
dest: /usr/local/bin/docker-compose
mode: +x
- name: Ensure docker is running
systemd:
name: docker
state: started
- name: Add user to docker group
hosts: aws
become: yes
tasks:
- name: Add ec2-user to docker group
user:
name: ec2-user
groups: docker
append: yes
- name: Reconnect to server session
meta: reset_connection #To reset connection immediate after above task
- name: Test docker pull
hosts: aws
tasks:
- name: pull redis
command: docker pull redis
Interactive input: prompts
- Playbook prompts the user for certain input
- Prompting the user for variables lets you avoid recording sensitive data like passwords.
- you could also encrypt the entered values.
Below Ansible snippet prompting for password of docker registry
- name: Start docker containers
hosts: aws
vars_prompt:
- name: docker_password
prompt: Enter password for docker registry
# [OR]
# vars_files:
# - project-vars
tasks:
- name: copy docker compose
copy:
src: /file_path/docker-compose.yaml
dest: /home/ec2-user/docker-compose.yaml
- name: Docker login
docker_login:
registry_url: https://index.docker.io/v1
username: kishoredockr
password: {{docker_password}}
project-vars: stroing docker password in vars file
location: ~/ansible
uname: nginx2
version: 1.0
user_home_dir:
/home/{{uname}}linux_user: newuser
user_groups: adm,docker
docker_password: | paste_password_here
Generic Playbook: Install docker, docker-compose and start container with user defined in vars file
---
- name: Install python3,docker and docker-compose
hosts: aws
become: yes
gather_facts: False #To disable gather facts task in Ansible
tasks:
- name: Make sure python3 and docker are installed
vars:
ansible_python_interpreter: /usr/bin/python
dnf: #yum module as ofnow supports only python2 interpreter
name:
- python3
- docker
- python3-pip
update_cache: yes
state: present
- name: Install docker-compose
get_url:
url: https://github.com/docker/compose/releases/download/1.27.4/docker-compose-Linux-{{lookup('pipe', 'uname -m')}} #using "lookup" from jinja template to interpret commands inside url
dest: /usr/local/bin/docker-compose
mode: +x
- name: Ensure docker is running
systemd:
name: docker
state: started
- name: Create a new user
hosts: aws
vars_files:
- project-vars
become: yes
tasks:
- name: Create a new user
user:
name: "{{linux_user}}"
groups: "{{user_groups}}"
- name: Install docker & docker-compose python module
hosts: aws
vars_files:
- project-vars
become: yes
become_user: "{{linux_user}}"
tasks:
- name: Install docker python module #This is dependency with ansible docker_image module
pip:
name:
- docker
- docker-compose
- name: Start docker containers
hosts: aws
become: yes
become_user: "{{linux_user}}"
vars_prompt:
- name: docker_password
prompt: Enter password for docker registry
# [OR]
# vars_files:
# - project-vars
tasks:
- name: copy docker compose
copy:
src: /file_path/docker-compose.yaml
dest: "{{user_home_dir}}/docker-compose.yaml"
- name: Docker login
docker_login:
registry_url: https://index.docker.io/v1
username: kishoredockr
password: {{docker_password}}
- name: Start container from docker-compose
docker_compose:
project_src: "{{user_home_dir}}"
state: present #default is present, equals to docker-compose up
Manual tasks between provisioning and configuring:
1) We get the IP address manually from TF output
2) Update the hosts file manually with IP
3) Execute Ansible command
We can avoid manual tasks as follows:
In Terrform file:
#Terraform recommends not to use Provisioners
# Avoid manual tasks to cofigure server in ansible once it is provisioned
provisioner "local-exec" {
working_directory = "/path-to-file/"
# comma(,) is mandatory as inventory expect file location (or) comma separated list of IPs
command = "ansible playbook --inventory ${self.public_ip}, --private-key ${var.ssh_key_private} --user ec2-user deploy-docker.yaml"
#Next step will be update hosts: all in deploy.docker.yaml playbook
}
[OR]
#Another way to execute a provisioner if you want to separat it from AWS instance resource
#Using null_resource in terraform
#We can use it for both local-exec & remote-exec
resource "null_resource" "configure_server" {
triggers {
trigger = aws_instance.myapp-server.public_ip
}
provisioner "local-exec" {
working_directory = "/path-to-file/"
# comma(,) is mandatory as inventory expect file location (or) comma separated list of IPs
command = "ansible playbook --inventory ${aws_instance.myapp-server.public_ip}, --private-key ${var.ssh_key_private} --user ec2-user deploy-docker.yaml"
#Next step will be update hosts: all in deploy.docker.yaml playbook
}
}
In Ansible Playbook: Add below task to check the server is ready to SSH
#In Ansible playbook add a task to check server is able to SSH
- name: Ensure server is ready to SSH
hosts: all
gather_facts: false
tasks:
- name: Ensure ssh port open
wait_for:
port: 22
delay: 10
timeout: 100
search_regex: OpenSSH
host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}'
vars:
ansible_connection: local
ansible_python_interpreter: /usr/bin/python
Ansible.cfg before enabling aws_ec2 plugin:
[defaults]
HOST_KEY_CHECKING = false
#Configure default hosts file
inventory = hosts
interpreter_python = /usr/bin/python3
enable_plugins = aws_ec2 #Enabling AWS plugin
ansible-inventory: command is used to display or dump the configured inventory as Ansible sees it
ansible-inventory -i inventory_aws_ec2.yaml --list (to get all the info)
ansible-inventory -i inventory_aws_ec2.yaml --graph (to get only servers)
Ec2 inventory source: config file must end with aws_ec2.yaml
Working with Dynamic Inventory:
1) Inventory Plugins (Ansible recommended: Because plugins make use of Ansible features such as state management, plugins are written in YAML format)
2) Inventory Scripts (written in Python)
Plugins/Scripts specific to the infrastructure provider:
You can use "ansible-doc -t inventory -l" to see a list of available plugins
For AWS infrastructure, you need a specific plugin/script for AWS(aws_ec2: boto3 & botocore python modules are required to use it)
Configure Ansible to use Dynamic Inventory:
Pass dynamic inventory file:
ansible-playbook -i inventory_aws_ec2.yaml deploy-docker-with-newuser.yaml
Issues observed:
===================
Error: updating Security Group (sg-0a99641b7c7e39c08) ingress rules:
authorizing Security Group (ingress) rules: InvalidPermission.Duplicate:
the specified rule "peer: 104.0.0.0/16, TCP, from port: 22, to port:
22, ALLOW" already exists
Error: updating Security Group (sg-0a99641b7c7e39c08) egress rules: authorizing Security Group (egress) rules: InvalidPermission.Duplicate: the specified rule "peer: 0.0.0.0/0, ALL, ALLOW" already exists
To fix it: Add below
lifecycle {
# Use the `ignore_changes` block to specify which attributes to ignore during updates.
ignore_changes = [
# Ignore changes to the ingress rules. This ensures Terraform won't attempt to update them.
ingress,egress
]
}
===================
───────── Credits to: TechWorldwithNana & BestTechReads ─────────
DISCLAIMER
The purpose of sharing the content on this website is to Educate. The author/owner of the content does not warrant that the information provided on this website is fully complete and shall not be responsible for any errors or omissions. The author/owner shall have neither liability nor responsibility to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the contents of this website. So, use the content of this website at your own risk.
This content has been shared under Educational And Non-Profit Purposes Only. No Copyright Infringement Intended, All Rights Reserved to the Actual Owner.
For Copyright Content Removal Please Contact us by Email at besttechreads[at]gmail.com