[Notes] - Infrastructure as Code with Terraform
What is Terraform?
- Automate and manage infrastructure and your platform
- And services that run on that platform
- Open source
- And Declarative language (define what end result you want)
Difference between Ansible and Terraform?
Both: Infrastructure as a Code.
Both automate: Provisioning, configuring and managing the infrastructure.
Terraform is mainly infrastructure provisioning tool. Can deploy apps as well.
Ansible is mainly a configuration tool. Configure the infrastructure, deploy apps, install/update software.
Ansible is more mature and Terraform is relatively new but more advanced in orchestration.
Terraform is better tool for Infrastructure. Ansible is better tool for configuring that infrastructure.
Terraform Architecture:
2 input sources:(2 main components)
1) CORE
Use two input sources. TF-config (what to create/configure?) & Terraform "State" (current state of the setup). Based on these two inputs CORE figure out the plan: what needs to be created/updated/deleted?
2) Providers for specific technologies like AWS, k8s, Azure, etc. There are two components we get from provider i.e., Resources & Data Sources.
Example config file:
provider "aws" {
version = "~> 2.0"
region = "us-east-1"
}
resource "aws-vpc" "example" {
cidr_block = "10.0.0.0/16"
}
Terraform commands for differeent stages:
refresh - Query infrastructure provider to get current state
plan - create an execution plan(determine what actions are necessary to achieve the desired state). This is just a preview, no changes to real resources
apply - execute the plan.
destroy - destroy the resources/infrastructure.
terraform apply -auto-approve (to avoid prompt for yes/no)
Ex: terraform apply -var-file terraform-dev.tfvars --auto-approve
terraform destroy (to destroy all components mentioned in the configuration file)
Ex: terraform destroy -var-file terraform-dev.tfvars --auto-approve
terraform state list - to list resources in the state
terraform state show aws_subnet.dev-subnet-2 - show a resource in the state
Key Takeaways:
Terraform is a tool for creating and configuring infrastructure (but not for installing applications on these provisioned clusters)
it is universal IaC tool (if you are using different cloud provider & different technologies)
use just 1 tool to integrate all those different technologies(like jenkins, aws, kubernetes, vmware,etc).
check version:
terraform -v
Providers in Terraform:
Expose resources for specific infrastructure platform (e.g. AWS)
Responsible for understanding API of that platform
Provider is just code that knows how to talk to specific technology or platform
terraform init - initializes a working directory. Install providers defined in the Terraform coniguration(e.g. main.tf). Please note we need to switch to the exact Terraform directory where we want to init terraform to install providers in case if we have multiple terraform coniguration files.
Creating a resource from a non existing resource id:
resource "aws_subnet" "dev-subnet1" {
vpc_id = aws_vpc.development-vpc.id
cidr_block = "10.0.10.0/24"
availability_zone = "ap-south-1a"
}
Create a subnet in an existing VPC:
"Data Sources" allow data to be fetched for use in TF configuration.
data "aws_vpc" "existing-vpc" {
default = "true"
}
resource "aws_subnet" "dev-subnet-2" {
vpc_id = data.aws_vpc.existing-vpc.id
cidr_block = "172.31.40.0/20"
availability_zone = "ap-south-1a"
tags = {
name = "dev-subnet-2"
}
}
~ indicates the change to the existing resource
+ means creating a new resource
- means destroying an existing resource
Destroy/Delete Terraform resources:
Method 1 (Recommended): Delete the code from terraform configuration file(main.tf) and run terraform apply.
Method 2: terraform destroy -target <resource_type>.<resource_name>
Ex: terraform destroy -target aws_subnet.dev-subnet-2
terraform destroy (to destroy all components mentioned in the configuration file)
---------
Terraform status will be stored in terraform.tfstate file, same info will be compared between current vs desired state.
terraform state list - to list resources in the state
Output Values:
output "output-name" {
value = <resource_type>.<terraform_resource_name>.id
}
E.g.:
output "dev-vpc-id" {
value = aws_vpc.development-vpc.id
}
output "dev-subnet-id" {
value = aws_vpc.dev-subnet-1.id
}
Variables in Terraform:
variable "subnet_cidr_block" {
description = "subnet cidr block"
type = string
nullable = false
}
resource "aws_subnet" "dev-subnet-1" {
vpc_id = aws_vpc.development-vpc.id
cidr_block = var.subnet_cidr_block
availability_zone = "ap-south-1a"
tags = {
Name = "dev-subnet-1"
}
}
#Mention the variable values in variables.tfvar
There are three ways to define values to a variable.
1) terraform apply will prompt for variable value
2) terraform apply -var "subnet_cidr_block=10.0.30.0/24"
3) Recommended: create variables file: terraform.tfvars. We define key value pairs for our variables.
subnet_cidr_block = "10.0.40.0/24"
Use Case for Input Variables:
If we want to use the same setup of infrastructure for dev, stage and Prod, we can create separate variables file for each environment (like terraform-dev.tfvars, terraform-stage.tfvars and terraform-prod.tfvars).
Pass custom variables file:
terraform apply -var-file terraform-dev.tfvars
The default value makes the variable optional.
Set variable using TF environment variable:
export TF_VAR_avail_zone="eu-west-3b"
We need to use the variable name as "avail_zone" in the terraform code.
Connect local project with Git Repository:
- Goto local project and do git initialization: git init
cd existing_repo
git remote add origin https://gitlab.com/kishoregit/terraform-learn-practice.git
git branch -M main
git push -uf origin main
In case of any issues, troubleshoot the issue. Used few of the below commands for troubleshooting:
git push --set-upstream origin main
git pull origin main (fatal: refusing to merge unrelated histories)
git pull --allow-unrelated-histories origin main
git push -f origin main
Using variable inside a string:
variable vpc_cidr_block {}
resource "aws_vpc" "myapp-vpc" {
cidr_block = var.vpc_cidr_block
tags = {
Name = "${var.env_prefix}-vpc"
}
}
Display content using Terraform show:
$ terraform state show aws_vpc.myapp-vpc
# aws_vpc.myapp-vpc:
resource "aws_vpc" "myapp-vpc" {
arn = "arn:aws:ec2:ap-south-1:581862344336:vpc/vpc-0ab3522121958aae4"
assign_generated_ipv6_cidr_block = false
cidr_block = "10.0.0.0/16"
default_network_acl_id = "acl-0ec0a793c67e9ce93"
default_route_table_id = "rtb-0a10c483a7b09fddc"
default_security_group_id = "sg-08cf9ff2e64795459"
dhcp_options_id = "dopt-0c5e1a40219bacdce"
enable_classiclink = false
enable_classiclink_dns_support = false
enable_dns_hostnames = false
enable_dns_support = true
enable_network_address_usage_metrics = false
id = "vpc-0ab3522121958aae4"
instance_tenancy = "default"
ipv6_netmask_length = 0
main_route_table_id = "rtb-0a10c483a7b09fddc"
owner_id = "581862344336"
tags = {
"Name" = "dev-vpc"
}
tags_all = {
"Name" = "dev-vpc"
}
}
user_data: Is an entry point script that gets executed on ec2 instance when the server is initiating.
Mention it while creating aws_instance.
This block will only get executed once (only on the initial run)
. user_data = <<EOF
#!/bin/bash
sudo yum update -y && sudo yum install -y docker
sudo systemctl start docker
sudo usermod -aG docker ec2-user
docker run -p 8080:80 nginx
EOF
Reference above code from a shell script:
Create a file "entry-script.sh" and add above code to the file. And reference it as follows
entry-script.sh
#!/bin/bash
sudo yum update -y && sudo yum install -y docker
sudo systemctl start docker
sudo usermod -aG docker ec2-user
docker run -p 8080:80 nginx
user_data = file("entry-script.sh")
user_data script (or) commands will be handed over to ec2-instance once it is created. Terraform won't be having any info that the commands are executed successfully or not. Terraform has its own concept to execute commands or scripts i.e., provisioners.
Terraform greatly helps us for setting up
- Initial Infrastructure setup
- Manage Infrastructure
- Initial Application Setup
But it can't help much for creating and managing applications.
For that purpose we need to rely on other configuration management tools, such as
- Chef
- Puppet
- Ansible
Provisioners:
"remote-exec": invokes script (or) commands on a remote resource after it is created.
- inline - list of commands
Whenever we are using "remote-exec" we need to tell to Terraform, how to connect to the remote server. We can also mention the connection block within the provisioner itself, in case if a single provisioner among other wants to connects to a different server.
Alternatives to remote-exec:
Use configuration management tools (like, Ansible, chef, puppet). Once server provisioned, hand over to those tools.
connection {
type = "ssh"
host = self.public_ip
user = "ec2-user"
private_key= file(var.private_key_location)
}
provisioner
"remote-exec" {
/*inline = [
"export ENV=dev",
"mkdir new"
]*/
#This flie should be copied first to remote server
script = file("entry-script-on-ec2.sh")
}
"file" provisioner: copy files or directories from local to newly created resource.
Source - Source file or folder
Destination - Absolute path
provisioner "file" {
source = "entry-script.sh"
destination = "/home/ec2-user/entry-script-on-ec2.sh"
}
"local-exec" provisioner: invokes a local executable after a resource is created. Locally, NOT on the created resource.
Alternatives to remote-exec:
-Use "local" provider
provisioner "local-exec" {
command = "echo ${self.public_ip} > output.txt"
}
- Provisioners are not recommended by Terraform.
- Breaks idempotency concept
- Terraform doesn't know what you execute
- If provisioner fails, although ec2 instance is up terraform will still consider it as a failure and ec2 instance marked for the deletion.
Modules:Without modules, we will end up with complex configurations, huge file, and no overview.
Modules = container for multiple resources, used together
Why Modules?
- organize and group configurations
- encapsulate into distinct logical components
- re-use
We can treat "Modules" like function definitions in the programming.
input variables = like function arguments
output variables - like function return values
We can create our own modules (or) use existing modules created by Terraform or other companies. With "terraform init" the module will be automatically installed.
The recommended way of writing terraform configuration files is writing the code to its specific files, like:
- main.tf
- variables.tf
- outputs.tf
- providers.tf
Project Structure:
- root module
- /modules = "child modules"
"child module" - a module that is called by another configuration.
Using Module:
module "myapp-subnet" {
source = "modules/subnet"
#Declaring module variables from main variables.tf file
avail_zone = var.avail_zone
subnet_cidr_block = var.subnet_cidr_block
env_prefix = var.env_prefix
vpc_id = aws_vpc.myapp-vpc.id
default_route_table_id = aws_vpc.myapp-vpc.default_route_table_id
}
Modules output:
-like return value of module
-to expose/export resource attributes to parent module.
step1:
#Define the output in module specific output.tf
output "subnet" {
value = aws_subnet.myapp-subnet-1
}
step2:
#Output the modules whereever it is used
#syntax:module.<module_name_defined>.<output_name_defined_in_module>.id
subnet_id = module.myapp-subnet.subnet.id
terraform init - whenever moduled is added/changed. We have to run terraform init.
Whenever we get an error in Terraform, crash.log will be created. We can refer this log to get more details about the error.
How do we set the variable values from CI/CD pipeline?
Using Terraform environment variable: TF_VAR_name
Also, we can use below method:
#In Jenkins file, access terraform output variable
EC2_PUBLIC_IP = sh(
script: "terraform output ec2_public_ip",
returnStdout: true
).trim()
Problem: Each user/CI server must make sure they always have the latest state data before running Terraform.
How to we share a same Terraform state file?
- TF writes the data to this remote data store
- Different remote storage options possible (data backup, can be shared,keep sensitive data off disk)
Configure remote storage (for Terraform state):
terraform {
required_version = ">= 0.12"
backend "s3" {
bucket = "myapp-bucket"
key = "myapp/state.tfstate"
}
}
This config stores the Terraform state remotely.
───────── Credits to: TechWorldwithNana & BestTechReads ─────────
DISCLAIMER
The purpose of sharing the content on this website is to Educate. The author/owner of the content does not warrant that the information provided on this website is fully complete and shall not be responsible for any errors or omissions. The author/owner shall have neither liability nor responsibility to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the contents of this website. So, use the content of this website at your own risk.
This content has been shared under Educational And Non-Profit Purposes Only. No Copyright Infringement Intended, All Rights Reserved to the Actual Owner.
For Copyright Content Removal Please Contact us by Email at besttechreads[at]gmail.com
