🤖 Week 8: Ansible Lab Guide

• Understand the concepts and usage of Ansible.

• Integrate Ansible to the existing CI/CD pipeline to improve the continuous delivery of our application. Thus, our CI/CD pipeline will be robust and smooth for continuous delivery even for the complex setup of multiple target systems.

So far, we have used Jenkins as a Continuous Integration and Continuous Deployment tool.

However, Jenkins works best as a Continuous Integration only as it has problems dealing with more complex set up of multiple target deployment environments of servers.

➜ So we need to introduce Ansible, which is a Configuration Management tool, which is the best practice of handling updates and changes of target system requirements.

So here is the setup for our CI/CD pipeline this week:

Note: To proceed with the lab, we should have already set up our Jenkins Server and Docker Server in the previous lab. If you do not have the Jenkins Server or Docker Server (because you have already deleted those EC2 instances) then please try to set them up again by following the tutorial instructions in previous weeks.

Let's create a new EC2 server to be our Ansible Server.

The outline for this section:

• Setup a Linux EC2 instance “Ansible_Server”.

• Setup hostname for convenience (as we have many servers to manage now).

• Create ansibleadmin user.

• Add the user to sudoers file.

• Generate SSH keys to a quick authenticate (for the Ansible server to communicate/control Docker Server).

• Enable password based login.

• Authorise the Jenkins server's public key for the ansibleadmin user (so Jenkins can reach this server with the SSH Agent plugin).

• Install ansible.

sudo su -

nano /etc/hostname

reboot

sudo su -

useradd ansibleadmin

passwd ansibleadmin

nano /etc/sudoers

ansibleadmin ALL=(ALL) NOPASSWD: ALL

nano /etc/ssh/sshd_config

service sshd reload

ssh-keygen

sudo su -

yum install ansible

ansible --version

The outline what you need to do:

Connection checkpoint: Ansible is agentless. What must still be true about SSH connectivity, the remote user, and privilege escalation before a playbook can manage the Docker server?

• On the docker server:

  • Create ansibleadmin

  • Add ansibleadmin to sudoers files

  • Enable password login

• On the ansible server:

  • Add to hosts file.

  • Copy ssh the public key from ansible server to docker server.

  • Test the connection from ansible server to docker server.

Let's go!

• Let's SSH into the Docker Server:

• If you have not changed the hostname for the docker server, then please change the hostname of this server to be “docker-server”, reboot and ssh again:

• Alright, let’s login as a root user and change our current directory to /root:

sudo su -

• Let's create a new user “ansibleadmin”:

useradd ansibleadmin

• Let's add a password for that new user “ansibleadmin”:

passwd ansibleadmin

• Let’s add the ansibleadmin to be in the docker group:

  • This is necessary to run Docker commands as the ansibleadmin user without typing sudo every time.

sudo usermod -aG docker ansibleadmin

• We can double check if the ansibleadmin is now in the docker group:

id ansibleadmin

• Alright, to keep things simple, I want the ansibleadmin user to be able to execute any command without password so let's modify Sudoers file and save the file:

nano /etc/sudoers

• By default, EC2 server will not allow users to login with the password so we need to enable that:

nano /etc/ssh/sshd_config

• Enable Password Authentication like so, then save and exit from nano:

• Alright, let’s restart the ssh service for our change to take effective:

service sshd reload

• Nice, let's check the IP address of our Docker Server so we can add the IP address of Docker Server to our Ansible Inventory in the Ansible Server:

  • Keep in mind that, the highlighted is the private IP address so only EC2 within the same network will be able to communicate with it.

ifconfig

• [Switch to Ansible Server Terminal] Now, let's switch to the terminal that controls Ansible server.

• The default location for the inventory file (/etc/ansible/hosts) may vary based on how Ansible was installed. To confirm the location of the default inventory file, check the Ansible configuration:

ansible-config dump | grep DEFAULT_HOST_LIST

The screenshot confirms that the default location of the ansible hosts is indeed at /etc/ansible/hosts

• Let's modify the Ansible host file (Ansible Inventory):

  • This file is the "address book" of Ansible. Every machine you want Ansible to manage needs to be listed here.

nano /etc/ansible/hosts

• That's good! Let's switch back to the ansibleadmin user on the Ansible Server:

sudo su - ansibleadmin

• Let's change the current working directory to .ssh to see our public and private keys:

cd .ssh
ls -la

• We can inspect further the public key of Ansible Server by printing it out:

cat id_rsa.pub

• Let's copy the public key of ansibleadmin on the Ansible Server to the Docker Server:

  • So the ansibleadmin user can connect without a password. This sets up passwordless SSH, allowing Ansible to run commands and playbooks on the Docker Server automatically without needing manual password input every time as this is the core of automation.

ssh-copy-id [the-ip-address-of-docker-server]

• [Switch back to Docker Server Terminal] Let's switch back to terminal of Docker Server and check if the public key has been copied over:

• First, let's change from root to ansibleadmin:

sudo su - ansibleadmin

• Then, need to change current working directory to .ssh:

cd .ssh

• Print out the authorized_keys:

  • You should see the same content of the public key of ansibleadmin on the Ansible Server that we copied over in Docker Server.

cat authorized_keys

Now, it's time to test the connection between Ansible Server and Docker Server:

• [Switch back to Ansible Server Terminal] In the Ansible Server, we can use Ansible to ping the target server in the inventory like so:

ansible all -m ping

• Similarly, we can use ansible from Ansible Server to run the bash command like “uptime” on Docker Server like so:

ansible all -m command -a uptime

This is like how Ansible Server (master) can control everything on the Docker Server (slave) with any credentials or password authentication. Image, you can have ten different target servers in the Ansible Inventory file so one Ansible server can control all target servers easily.

Extra Note: you might wonder that while the manual setup of each server (e.g., creating a user, enabling SSH password authentication, etc.) works for a small number of servers, it quickly becomes cumbersome and error-prone when scaling to manage 10, 100, or more target servers. However, in the challenge exercise, you can use Ansible itself to automate the onboarding of new target servers (slaves) to make the process repeatable and scalable.

• Let's SSH into the Jenkins Server:

• In case you have not changed the hostname of Jenkins server, let’s change the hostname of this server to be “jenkins-server”, reboot and ssh again:

• Alright, let’s login as a root user and change our current directory to /root:

sudo su -

• Let’s start Jenkins service if you need to:

service jenkins start

• [On the Jenkins server, as root] Authorise the existing Jenkins key on the Ansible server. You are reusing the key pair you generated in the Docker lab so do not generate a new one.

• First confirm the key is still there (it was created as /root/jenkins_deploy_key in the Docker lab):

  ls -l /root/jenkins_deploy_key /root/jenkins_deploy_key.pub

  • [Only missing] If those two files are missing (for example you rebuilt the Jenkins server), re-create the pair and re-authorise it on the Docker server exactly as you did in the Docker lab before continuing:

    ssh-keygen -t rsa -b 4096 -m PEM -f /root/jenkins_deploy_key -N ""
    ssh-copy-id -i /root/jenkins_deploy_key.pub dockeradmin@<Docker-Server-Private-IP>

• Now authorise the same public key for ansibleadmin on the Ansible server. Use the Ansible server's private IP (172.31.x.x) so both instances are in the same VPC, and unlike the public IP it will not change when you stop and start the instance. You will be prompted for the ansibleadmin password you set in section 1b; this works because you enabled password login there.

  ssh-copy-id -i /root/jenkins_deploy_key.pub ansibleadmin@<Ansible-Server-Private-IP>

  

  • Verify before touching Jenkins. This should log in with no password and print the hostname and the Ansible version:

    ssh -i /root/jenkins_deploy_key ansibleadmin@<Ansible-Server-Private-IP> 'hostname && ansible --version'

    

• Let's add the ansibleadmin authentication as a Jenkins credential. Go to Manage Jenkins → Credentials → System → Global credentials (unrestricted) → + Add Credentials:

  • Kind: SSH Username with private key

  • ID: ansibleadmin-key

  • Description: ansibleadmin on ansible-server

  • Username: ansibleadmin

  • Private Key: Enter directly → paste the entire contents of /root/jenkins_deploy_key, including the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- lines. This is the same private key as the dockeradmin-key credential — one key pair, two credentials, because the username differs.

  • Passphrase: leave empty

  • Click Create

• Let’s create a new Jenkins job:

  • To save time, you can copy the configuration settings from the “BuildAndDeployOnDockerContainer” job from last week.

• We keep everything the same (git, github, maven, …) apart from two things in the copied job.

• First, scroll up to Build Environment, and in the SSH Agent → Credentials dropdown change the selection from dockeradmin (dockeradmin on docker-server) to ansibleadmin (ansibleadmin on ansible-server).

• Second, scroll down to Post Steps → Execute shell and replace the script inherited from last week's job with just the copy step for now. We will add the Ansible playbook commands later in Section 6:

ANSIBLE_SERVER=ansibleadmin@<Ansible-Server-Private-IP>
SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

# Copy the war artifact to the Ansible server's home directory
scp $SSH_OPTS target/*.war $ANSIBLE_SERVER:/home/ansibleadmin/

Note the destination is /home/ansibleadmin, not /home/dockeradmin.

• Let’s save the job for now! And run the job to test if the job can run correctly and copy the artifacts over to the Ansible Server!

• The result looks good! It should show build success and successfully connect to the Ansible Server to copy the artifact WAR file over that server.

  • Note that the WAR file is sent to Ansible instead of directly to Docker because Ansible is becoming the "deployment manager" in the pipeline.

• [Switch to the Ansible Server Terminal] In the Ansible Server, you can check if the WAR artifact file has been transferred from Jenkins server to Ansible Server like so:

  • Keep in mind, you might need to cd ~ to change the working directory to the home directory of ansibleadmin user first before listing the files inside it.

Alright, so far so good! Now, let’s install docker on the Ansible server so we can push the docker image of our web application to Dockerhub!

We can sudo into the root user or in this case, we can stay as ansibleadmin user and add “sudo” at the beginning of every command to execute them as a root user like so:

sudo yum install docker -y

• Let’s add the ansibleadmin to be in the docker group:

  • This is necessary to run Docker commands as the ansibleadmin user without typing sudo every time.

sudo usermod -aG docker ansibleadmin

• For this to take effect, please logout and login in as ansibleadmin again:

exit
sudo su - ansibleadmin

• We can double check if the ansibleadmin is now in the docker group:

id ansibleadmin

• Let’s start the docker service:

sudo service docker start

• Let’s start the docker status:

sudo service docker status

• Let’s create a Dockerfile to build an image from war artifact file:

nano Dockerfile

FROM tomcat:latest
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps
COPY ./*.war /usr/local/tomcat/webapps

• Let’s test if you can build a docker image from this Dockerfile:

docker build -t myapp:v1 .

• Let’s list all docker images to check if our new image is there:

docker images

• We can even test further by creating a container from this image and access this image via the browser:

  • The website should be available on the browser via the Ansible Public IP address with the port 8081.

docker run -d --name myapp-container -p 8081:8080 myapp:v1

It means that our Docker image is healthy!

sudo nano /etc/ansible/hosts

[dockerserver]
private-ip-address-of-docker-server

[ansibleserver]
private-ip-address-of-ansible-server

sudo su - ansibleadmin

ssh-copy-id ansibleadmin@<Ansible-Server-Private-IP>

ansible all -a uptime

nano buildAndPush.yml

---
- hosts: ansibleserver

  tasks:
  - name: create docker image
    command: docker build -t myapp:latest .

ansible-playbook buildAndPush.yml

docker images

docker tag [image-id] [new-image-repo-name]

docker login

docker push tomhuynhsg/myapp:latest

nano buildAndPush.yml

---
- hosts: ansibleserver

  tasks:
  - name: create docker image
    command: docker build -t myapp:latest .

  - name: create tag to push the image to dockerhub
    command: docker tag myapp:latest tomhuynhsg/myapp:latest

  - name: push the docker image to dockerhub
    command: docker push tomhuynhsg/myapp:latest

ansible-playbook buildAndPush.yml

• Let’s create a new Ansible Playbooks to create a container by pulling it from DockerHub:

nano pullToCreateContainer.yml

---
- hosts: dockerserver

  tasks:
  - name: create docker container by using the image on Dockerhub
    command: docker run -d --name mywebapp-container -p 8082:8080 tomhuynhsg/myapp:latest

• [Switch to Docker Server Terminal] Before running this playbook on Ansible Server, make sure Docker service is running on Docker Server:

sudo service docker start

• [Switch to Ansible Server Terminal] Let’s run this Playbooks on Ansible Server:

  • The website should be available to access on the browser on the public IP address of the Docker server with the port 8082.

ansible-playbook pullToCreateContainer.yml

• It's pretty good! However, to keep running the same playbook over and over again! We need:

  Idempotency checkpoint: On a second playbook run, which tasks should report changed and which should report ok? What would repeated destructive recreation reveal about the playbook design?

  • Remove existing container

  • Remove existing image

  • Create new container

• Let’s modify the playbook:

nano pullToCreateContainer.yml

---
- hosts: dockerserver

  tasks:
  - name: stop the existing container
    command: docker stop mywebapp-container
    ignore_errors: yes

  - name: remove the existing container
    command: docker rm mywebapp-container
    ignore_errors: yes

  - name: remove the existing image
    command: docker rmi tomhuynhsg/myapp:latest
    ignore_errors: yes

  - name: pull the latest image from dockerhub
    command: docker pull tomhuynhsg/myapp:latest

  - name: create docker container by using the image on Dockerhub
    command: docker run -d --name mywebapp-container -p 8082:8080 tomhuynhsg/myapp:latest

[Explanation] ignore_errors: yes is needed on the stop/remove tasks in case there is no existing container or image yet as this keeps the playbook idempotent (safe to run multiple times). The docker pull task deliberately has no ignore_errors: it is the step that guarantees we are deploying the image that was just pushed. If the pull fails, we want the playbook and therefore the Jenkins build to fail loudly, rather than silently re-running yesterday's image.

• Let’s run this Playbooks on Ansible Server again:

  • You should see that the playbook runs on the ansible server, gathers system facts, stops and removes the existing container, removes the old image, and then creates a new Docker container from the image on Docker Hub, completing successfully with no errors.

ansible-playbook pullToCreateContainer.yml

Alright, it’s perfect! Now, both of our Ansible Playbooks are ready for Jenkins to use!

Note: you can try to run this playbook pullToCreateContainer.yml many times to ensure this playbook can run multiple times without any issue. Also, the first time you run it and the next time you run it, then you might see something interesting in those runs.

• Open the configuration page of the Jenkins job again and replace the contents of the Execute shell box (which currently holds only the scp line from section 3) with the full script below.

  • First playbook → Builds a new container image and pushes it to the Docker Hub repo registry.

  • Sleep → Pushing images to DockerHub isn’t instant, so this wait briefly (15 seconds) ensures DockerHub has the new image available before trying to pull it.

  • Second playbook → Pulls the image from the Docker Hub repo registry and creates (or updates) the running container. set -e → makes the shell abort at the first failing command. Without it, a failing buildAndPush.yml would not stop the pipeline: sleep and the second playbook would still run, ssh would return the exit code of the last command only, and Jenkins would mark a broken deployment as SUCCESS. The inner set -e protects the remote command list; the outer one protects the scp.

Note: this separation (build & push vs pull & run) mirrors real-world pipelines as it builds artifacts in one stage and deploys in another.

Release checkpoint: If deployment fails after an image is pushed, which immutable artifact lets you roll back reliably? Why is reusing only the latest tag risky?

ANSIBLE_SERVER=ansibleadmin@<Ansible-Server-Private-IP>
SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

set -e

# Copy the war artifact to the Ansible server's home directory
scp $SSH_OPTS target/*.war $ANSIBLE_SERVER:/home/ansibleadmin/

# Run the two Ansible playbooks remotely
ssh $SSH_OPTS $ANSIBLE_SERVER '
set -e
cd /home/ansibleadmin
ansible-playbook /home/ansibleadmin/buildAndPush.yml
sleep 15
ansible-playbook /home/ansibleadmin/pullToCreateContainer.yml
'

• Try to run the Jenkins job to test if our configuration is all good!

• Please confirm that the website is available to access on the browser via the public IP address of Docker Server via the port 8082 (as defined in the pullToCreateContainer.yml playbook so feel free to change the port if you want to)

• Also, if you want to do further testing, you could try to push some changes to trigger the Jenkins job to see if the Ansible playbooks execute correctly to show the website with the new changes or not!

Awesome, you have completed this CI/CD Pipeline for this week! Please continue to test this pipeline further by making a change to the website and push it to Github to see the full automation in action!

Congratulations! You have successfully created this CI/CD Pipeline with Ansible Playbooks! Now, by adding more IP addresses/domain names into the Ansible Inventory, you can easily manage to deploy this web application to hundreds of different target servers!

You are now ready to receive a DevOps champion award in automation 🏆!

Using AWS EC2 User Data, you can automate the initial setup of an Ansible Slaves EC2 instance when it is launched. This allows you to configure servers without manual intervention with the benefits:

• Repeatability: Every instance launched with the script has a consistent configuration.

• Scalability: The same script can initialize dozens or hundreds of servers.

• Automation: Reduces manual setup, saving time and minimizing errors.

Here's a step-by-step guide:

• Launch an Ansible Slaves EC2 Instance with User Data

  • OS: default Amazon Linux 2023 AMI (Free-tier)

  • Instance Type: t3.micro (Free-tier)

  • Key pair: select and reuse your old key in other labs (devops_project_key)

  • Storage: Default 8 GB is plenty enough!

  • Security Group:

    • Optional Name: Jenkins_Slave_Security_Group

    • Source Type: Anywhere

    • Open ports:

      • 22 for SSH

      • Open other necessary ports (e.g., 8080 for Docker).

  • Add User data

    • Click on Advanced details

• Upload or paste in the User data:

• Note: remember that the public key of the Ansible master server should be at /home/ansibleadmin/.ssh/id_rsa.pub

  • After setting up the User data, you can launch the Ansible slave instance.

  • Wait for the instance is ready:

• [Optional] For testing if your user data script working correctly, you can choose to ssh to the Ansible slave to run these commands:

  • Check the hostname is correct:

    

  • Test Docker: docker --version

  • Test Ansible user login: su - ansibleadmin

    

  • Confirm SSH keys to match the public key in User data: cat /home/ansibleadmin/.ssh/authorized_keys

• [Back to the Ansible Master] Add the Ansible Slave instance to the Ansible inventory in the Ansible master (/etc/ansible/hosts)

• Test if the master can connect to the slave

ansible ansibleslave -m ping

• Test if the master can run commands on the slave:

ansible ansibleslaves -a uptime

Success! Using EC2 User Data is very efficient to quickly set up multiple Ansible slave EC2 instances for Ansible master to manage them.

[Extra thoughts]

If you would like to be more efficient to launch EC2 instances without GUI, you might consider:

• You can launch an EC2 instance directly from the AWS CLI by passing the User Data script.

  • Save the User Data script to a file, e.g., user-data.sh.

  • Use the following CLI command to launch the instance:

This approach aligns with real-world industry practices, enabling efficient and scalable infrastructure management.

Now that you have a solid understanding of AWS CLI, EC2 User Data, and Ansible, here are some advanced tasks to further enhance your skills:

1. Scale Up with Multiple Ansible Groups:

   • Use the AWS CLI to quickly launch multiple EC2 instances, for example:

     • A group for test-servers (staging environment).

     • A group for production-servers.

   • Configure your /etc/ansible/hosts file to manage these groups separately:

     • Note: your groups can have more ip addresses for more EC2 instances if you want to.

   [test-servers]
   192.168.x.x
   
   [production-servers]
   192.168.y.y

   • This separation enables precise control and testing before deploying changes to production.

2. Deploy Applications with Jenkins and Ansible:

   • Use Jenkins to integrate your CI/CD pipeline with Ansible's inventory groups:

     • First, deploy your application to test-servers.

     • Run some basic front-end tests using tools like curl to ensure the deployment is functional.

     • Once tests pass, automate the deployment to production-servers.

3. Automate Testing:

   • Implement Ansible tasks or playbooks to:

     • Verify that the application is running on test servers.

     • Use simple health checks (curl or HTTP status checks) to confirm that the deployed app responds as expected.

4. Leverage AWS CLI for Instance Management:

   • Automate instance creation with the following AWS CLI command:

   aws ec2 run-instances --image-id ami-xxxxxx --count 3 --instance-type t3.micro --key-name devops_project_key --security-group-ids sg-xxxxxx --user-data file://user-data.sh

   • This ensures repeatable infrastructure setup and avoids reliance on manual GUI interactions.

5. End-to-End Pipeline Simulation:

   • Push a new version of your web app to GitHub.

   • Trigger a Jenkins pipeline that:

     • Builds and pushes a Docker image.

     • Deploys the image to test-servers via Ansible.

     • Runs tests on test-servers.

     • Upon successful tests, deploy to production-servers.

6. [Optional - Thoughts] Experiment with Advanced Tools to Simulate Real-World Scenarios:

   • Practice a blue-green deployment strategy:

     • Deploy the new version of the app to a group (e.g., test-servers or a subset of production-servers).

     • Perform tests before switching traffic over to the updated servers with AWS Elastic Load Balancer (ELB) which we will learn later in later weeks.

By practicing these advanced challenges, you'll gain deeper insights into DevOps workflows and be well-prepared for real-world scenarios involving scalable infrastructure and efficient automation. Keep pushing boundaries, and you'll be ready to tackle increasingly complex projects with confidence! 🏆