# Kubernetes End-to-End DevSecOps Project

![](https://miro.medium.com/v2/resize:fit:1050/1*7qnCPIh9Bg3yB96XtrlCkg.gif align="left")

# **Introduction:**

In today’s rapidly evolving tech landscape, deploying applications using Kubernetes has become a crucial aspect of modern software development. This guide provides a detailed walkthrough for setting up an end-to-end Kubernetes project, covering everything from infrastructure provisioning to application deployment and monitoring.

# **Prerequisites:**

Before diving into the implementation, ensure you have the following in place:

1. Basic understanding of Kubernetes concepts.
    
2. Access to AWS or any other cloud provider for server instances.
    
3. A TMDB API key for accessing movie databases in your Netflix Clone application.
    
4. DockerHub account for pushing and pulling Docker images.
    
5. Gmail account for email notifications.
    
6. Jenkins, Kubernetes, Docker, and necessary plugins installed.
    

# **High-Level Overview:**

1. **Infrastructure Setup**: Provisioned servers for Jenkins, Monitoring, and Kubernetes nodes.
    
2. **Toolchain Integration**: Integrated essential tools like Jenkins, SonarQube, Trivy, Prometheus, Grafana, and OWASP Dependency-Check.
    
3. **Continuous Integration/Continuous Deployment (CI/CD)**: Automated workflows with Jenkins pipelines for code analysis, building Docker images, and deploying applications on Kubernetes.
    
4. **Security Scanning**: Implemented Trivy and OWASP Dependency-Check to scan for vulnerabilities in code and Docker images.
    
5. **Monitoring and Visualization**: Set up Prometheus and Grafana for real-time monitoring and visualization of both hardware and application metrics.
    
6. **Email Notifications**: Configured Jenkins for email alerts based on pipeline results.
    

You will get the Jenkinsfile and Kubernetes Manifest files along with the Dockerfile. Feel free to modify it accordingly

**Project GitHub Repo**\- [https://github.com/SreeniDevSecOps/Netflix-Clone-K8S-End-to-End-Project](https://github.com/SreeniDevSecOps/Netflix-Clone-K8S-End-to-End-Project)

We need four servers for our today’s Project

**Jenkins Server-** On this Server, Jenkins will be installed with some other tools such as sonarqube(docker container), trivy, and kubectl.

**Monitoring Server-** This Server will be used for Monitoring where we will use Prometheus, Node Exporter, and Grafana.

**Kubernetes Master Server-** This Server will be used as the Kubernetes Master Cluster Node which will deploy the applications on worker nodes.

**Kubernetes Worker Server-** This Server will be used as the Kubernetes Worker Node on which the application will be deployed by the master node.

Let’s create the following instances.

**Jenkins Server**

Click on **Launch Instances.**

![](https://miro.medium.com/v2/resize:fit:1050/0*ljWXDG-bi_UTaVGT align="left")

Provide the name of your Jenkins instance, and select the Ubuntu OS 22.04 version.

![](https://miro.medium.com/v2/resize:fit:1050/0*IavAgPMx_E1n3V9r align="left")

We need to configure multiple things on the Jenkins instance. So, select the t2.large instance type, provide the key or you can create if you want.

Keep the networking things as it is. But make sure to open all inbound and outbound traffic in the selected security groups.

![](https://miro.medium.com/v2/resize:fit:1050/0*HiZ4e0xUJ2uz6XGH align="left")

Increase the storage capacity for Jenkins Instance from 8GB to 35GB and click on **Launch Instance.**

![](https://miro.medium.com/v2/resize:fit:1050/0*_GxPQ1Izbh4jaHI- align="left")

**Monitoring Server**

Provide the name of your Monitoring Instance, and select the Ubuntu 22.04 OS.

![](https://miro.medium.com/v2/resize:fit:1050/0*_lnD8Q7W66-gCT9x align="left")

We need to configure the monitoring tools on this instance which needs a minimum of 4GB RAM. So, select the t2.medium instance type, provide the key or you can create if you want.

Keep the networking things as it is. But make sure to open all inbound and outbound traffic in the selected security groups.

![](https://miro.medium.com/v2/resize:fit:1050/0*M3OzpolZj3NK7p3d align="left")

Increase the storage capacity for Jenkins Instance from 8GB to 15GB and click on **Launch Instance.**

![](https://miro.medium.com/v2/resize:fit:1050/0*irnhtKbVp3CNpnL8 align="left")

**Kubernetes Master & Worker Node**

We have to create two Kubernetes Nodes which need at least 2 CPUs.

Provide the name of your Kubernetes Master Instance, and select the Ubuntu 22.04 OS.

In the Number of Instances, replace 1 with 2 because we need two Kubernetes Nodes.

![](https://miro.medium.com/v2/resize:fit:1050/0*v7zIATGGQNcVjeCP align="left")

Select the t2.medium instance type, provide the key or you can create if you want.

Keep the networking things as it is. But make sure to open all inbound and outbound traffic in the selected security groups then keep the rest of the things as it is and click on **Launch Instance.**

![](https://miro.medium.com/v2/resize:fit:1050/0*4hIzgXdiIpQNgv_R align="left")

Rename the Kubernetes Servers and all four servers will look like the below snippet.

![](https://miro.medium.com/v2/resize:fit:1050/0*4xwdATSGQGbFKdPG align="left")

Log in to the Jenkins Server

![](https://miro.medium.com/v2/resize:fit:1050/0*VwTLoG1ilD_gGD4_ align="left")

Download Open JDK and Jenkins

```plaintext
# Intsalling Java
sudo apt update -y
sudo apt install openjdk-11-jre -y
java --version

# Installing Jenkins
curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y
```

![](https://miro.medium.com/v2/resize:fit:1050/0*dGEFk7WX3_GIExLm align="left")

Check the status of the Jenkins server

![](https://miro.medium.com/v2/resize:fit:1050/0*nAstVZU4HcVeA-_x align="left")

Copy your Jenkins Server Public IP and paste it into your favorite browser with port number 8080.

![](https://miro.medium.com/v2/resize:fit:1050/0*j6QmEgcN1RhXg9rj align="left")

Run the command on your Jenkins server

```plaintext
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
```

Copy the output and paste it into your above snippet text field and click on Continue.

![](https://miro.medium.com/v2/resize:fit:1050/0*bV1Ghc7aQbT4nG1X align="left")

Click on the **Install suggested plugins**

![](https://miro.medium.com/v2/resize:fit:1050/0*6O-pHg_VmajH7_dC align="left")

Click on the **Skip and continue as admin**

![](https://miro.medium.com/v2/resize:fit:1050/0*Y4FUkKf7uNfsWX62 align="left")

Click on **Save and Finish**

![](https://miro.medium.com/v2/resize:fit:1050/0*rSrf_55fSz3yi0II align="left")

Install Docker and configure on the **Jenkins Server**

```plaintext
sudo apt update
sudo apt install docker.io -y
sudo usermod -aG docker jenkins
sudo usermod -aG docker ubuntu
sudo systemctl restart docker
sudo chmod 777 /var/run/docker.sock
```

![](https://miro.medium.com/v2/resize:fit:1050/0*K0SyIwqh6r4twKcW align="left")

Install Sonarqube on your **Jenkins Server**

We will use a docker container for Sonarqube

```plaintext
docker run -d --name sonar -p 9000:9000 sonarqube:lts-community
```

![](https://miro.medium.com/v2/resize:fit:1050/0*zwjhEIeiGiuQqvPh align="left")

Now, copy your Public IP of Jenkins Server and add 9000 Port on your browser.

The username and password will be admin

![](https://miro.medium.com/v2/resize:fit:1050/0*PECNuwWP66nVh3EV align="left")

Reset the password and click on **Update**

![](https://miro.medium.com/v2/resize:fit:1050/0*UJzC-Dv1BZ8K21W4 align="left")

You will see your Sonarqube Server in the below snippet.

![](https://miro.medium.com/v2/resize:fit:1050/0*88ff1wNc5cASEnVA align="left")

Install the Trivy tool on the **Jenkins Server**

```plaintext
sudo apt-get install wget apt-transport-https gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy
```

![](https://miro.medium.com/v2/resize:fit:1050/0*5Ivm9VrzWGQVgpqo align="left")

Install and Configure the Prometheus, Node Exporter, and Grafana on the **Monitoring Server**

Login to the **Monitoring Server**

![](https://miro.medium.com/v2/resize:fit:1050/0*klrP1kVMp1ZAh8lo align="left")

Create Prometheus user

```plaintext
sudo useradd \
 - system \
 - no-create-home \
 - shell /bin/false prometheus
```

![](https://miro.medium.com/v2/resize:fit:1050/0*g61upxXz7KeKTaIE align="left")

Download the Prometheus file on the **Monitoring Server**

```plaintext
wget https://github.com/prometheus/prometheus/releases/download/v2.49.0-rc.1/prometheus-2.49.0-rc.1.linux-amd64.tar.gz
```

![](https://miro.medium.com/v2/resize:fit:1050/0*oTo6GfEJLuGCubOi align="left")

Untar the Prometheus downloaded package

```plaintext
tar -xvf prometheus-2.49.0-rc.1.linux-amd64.tar.gz
```

![](https://miro.medium.com/v2/resize:fit:1050/0*yLFJJxek1ZC09bpC align="left")

Create two directories /data and /etc/prometheus to configure the Prometheus

```plaintext
sudo mkdir -p /data /etc/prometheus
Now, enter into the prometheus package file that you have untar in the earlier step.
cd prometheus-2.49.0-rc.1.linux-amd64/
```

![](https://miro.medium.com/v2/resize:fit:1050/0*REkXffXTd-0GMcnZ align="left")

Move the prometheus and promtool files package in /usr/local/bin

```plaintext
sudo mv prometheus promtool /usr/local/bin/
```

![](https://miro.medium.com/v2/resize:fit:1050/0*HyNcnLSzBW6IBr5T align="left")

Move the console and console\_libraries and prometheus.yml in the /etc/prometheus

```plaintext
sudo mv consoles console_libraries/ prometheus.yml /etc/prometheus/
```

![](https://miro.medium.com/v2/resize:fit:1050/0*YdZwvmApIux38URG align="left")

Provide the permissions to prometheus user

```plaintext
sudo chown -R prometheus:prometheus /etc/prometheus/ /data/
```

![](https://miro.medium.com/v2/resize:fit:1050/0*WSPiLCoYK9lIBood align="left")

Check and validate the Prometheus

```plaintext
prometheus --version
```

![](https://miro.medium.com/v2/resize:fit:1050/0*PDVmVLxSsQ3i1beq align="left")

Create a systemd configuration file for prometheus

Edit the file /etc/systemd/system/prometheus.service

```plaintext
sudo vim /etc/systemd/system/prometheus.service
```

and paste the below configurations in your prometheus.service configuration file and save it

```plaintext
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5
[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
 - config.file=/etc/prometheus/prometheus.yml \
 - storage.tsdb.path=/data \
 - web.console.templates=/etc/prometheus/consoles \
 - web.console.libraries=/etc/prometheus/console_libraries \
 - web.listen-address=0.0.0.0:9090 \
 - web.enable-lifecycle
[Install]
WantedBy=multi-user.target
```

![](https://miro.medium.com/v2/resize:fit:1050/0*WW8OmHVHEZ9V7LsT align="left")

Once you write the systemd configuration file for Prometheus, then enable it and start the Prometheus service.

```plaintext
sudo systemctl enable prometheus.service
sudo systemctl start prometheus.service
systemctl status prometheus.service
```

![](https://miro.medium.com/v2/resize:fit:1050/0*69KdSuxIIWkvibTo align="left")

Once the Prometheus service is up and running then, copy the public IP of your **Monitoring Server** and paste it into your favorite browser with a 9090 port.

![](https://miro.medium.com/v2/resize:fit:1050/0*eLZlyQ3LGYr7ZHDJ align="left")

Now, we have to install a node exporter to visualize the machine or hardware level data such as CPU, RAM, etc on our Grafana dashboard.

To do that, we have to create a user for it.

```plaintext
sudo useradd \
 - system \
 - no-create-home \
 - shell /bin/false node_exporter
```

![](https://miro.medium.com/v2/resize:fit:1050/0*pmXquTcZgumiT3Gl align="left")

```plaintext
Download the node exporter package
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
```

![](https://miro.medium.com/v2/resize:fit:1050/0*QeCD46qE2xrHGR5C align="left")

Untar the node exporter package file and move the node\_exporter directory to the /usr/local/bin directory

```plaintext
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
```

![](https://miro.medium.com/v2/resize:fit:1050/0*Qw4fn9OCYOd_x70B align="left")

Validate the version of the node exporter

```plaintext
node_exporter --version
```

![](https://miro.medium.com/v2/resize:fit:1050/0*u4QE8y7YWG48uZCO align="left")

Create the systemd configuration file for node exporter.

Edit the file

```plaintext
sudo vim /etc/systemd/system/node_exporter.service
```

Copy the below configurations and paste them into the /etc/systemd/system/node\_exporter.service file.

```plaintext
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5
[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter \
 - collector.logind
[Install]
WantedBy=multi-user.target
```

![](https://miro.medium.com/v2/resize:fit:1050/0*ozHg5sTLyV0vAjB0 align="left")

Enable the node exporter systemd configuration file and start it.

```plaintext
sudo systemctl enable node_exporter
sudo systemctl enable node_exporter
systemctl status node_exporter.service
```

![](https://miro.medium.com/v2/resize:fit:1050/0*SGVdDCsTtP781KyR align="left")

Now, we have to add a node exporter to our Prometheus target section. So, we will be able to monitor our server.

edit the file

```plaintext
sudo vim /etc/prometheus/prometheus.yml
```

Copy the content in the file

```plaintext
  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]
```

![](https://miro.medium.com/v2/resize:fit:1050/0*qw5IEs9nRY-YgdZA align="left")

After saving the file, validate the changes that you have made using promtool.

```plaintext
promtool check config /etc/prometheus/prometheus.yml
```

![](https://miro.medium.com/v2/resize:fit:1050/0*wlqsk1xZTACbELM8 align="left")

If your changes have been validated then, push the changes to the Prometheus server.

```plaintext
curl -X POST http://localhost:9090/-/reload
```

![](https://miro.medium.com/v2/resize:fit:1050/0*LgmoZgKhvmOyomaL align="left")

Now, go to your Prometheus server and this time, you will see one more target section as node\_exporter which should be up and running.

![](https://miro.medium.com/v2/resize:fit:1050/0*iH2F5PQ_qEXNJnyj align="left")

Now, install the Grafana tool to visualize all the data that is coming with the help of Prometheus.

```plaintext
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg - dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
```

![](https://miro.medium.com/v2/resize:fit:1050/0*UdRBb9H2Pr1CYGsN align="left")

Install the Grafana

```plaintext
sudo apt-get install grafana
```

![](https://miro.medium.com/v2/resize:fit:1050/0*f-IGofHxCibltuz1 align="left")

Enable and start the Grafana Service

```plaintext
sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server.service
sudo systemctl status grafana-server.service
```

![](https://miro.medium.com/v2/resize:fit:1050/0*Cpdv9LTczYcBCfkD align="left")

To access the Grafana dashboard, copy the public IP address of the **Monitoring Server** and paste it into your favorite browser with port 3000

username and password will be admin

![](https://miro.medium.com/v2/resize:fit:1050/0*HSBKIR4nW7lhvHwq align="left")

Reset the password

![](https://miro.medium.com/v2/resize:fit:1050/0*4eYNpBKliwqjRZNI align="left")

Click on **Data sources**

![](https://miro.medium.com/v2/resize:fit:1050/0*c8U8aAzAPuEIM4q8 align="left")

Select the **Prometheus**

![](https://miro.medium.com/v2/resize:fit:1050/0*KnDl6Adh7l4ahT4o align="left")

Provide the **Monitoring Server Public IP** with port 9090 to monitor the **Monitoring Server.**

![](https://miro.medium.com/v2/resize:fit:1050/0*gLkEoegRTHSj5sRs align="left")

Click on **Save and test.**

![](https://miro.medium.com/v2/resize:fit:1050/0*zddIflHAA2Bujac_ align="left")

Go to the dashboard section of Grafana and click on the Import **dashboard.**

![](https://miro.medium.com/v2/resize:fit:1050/0*YNlLRjkk78bIFENI align="left")

Add 1860 for the node exporter dashboard and click on **Load**.

![](https://miro.medium.com/v2/resize:fit:1050/0*G_uSe_R27tj7US3o align="left")

Then, select the Prometheus from the drop down menu and click on **Import**

![](https://miro.medium.com/v2/resize:fit:1050/0*F3GL6AId71sdCJQn align="left")

The dashboard will look like this

![](https://miro.medium.com/v2/resize:fit:1050/0*PBeOJN2em9lGIesa align="left")

Now, we have to monitor our **Jenkins Server** as well.

For that, we need to install the Prometheus metric plugin on our Jenkins.

Go to **Manage Jenkins** -> Plugin search for Prometheus metrics install it and restart your Jenkins.

![](https://miro.medium.com/v2/resize:fit:1050/0*AEJ7jr3iQEVOxRWb align="left")

Edit the /etc/prometheus/prometheus.yml file

```plaintext
sudo vim /etc/prometheus/prometheus.yml
```

```plaintext
- job_name: "jenkins"
    static_configs:
      - targets: ["<jenkins-server-public-ip>:8080"]
```

![](https://miro.medium.com/v2/resize:fit:1050/0*A-XtsFObUNPkrsCs align="left")

Once you add the Jenkins job, validate the Prometheus config file whether it is correct or not by running the below command.

```plaintext
promtool check config /etc/prometheus/prometheus.yml
```

Now, push the new changes on the Prometheus server

```plaintext
curl -X POST http://localhost:9090/-/reload
```

![](https://miro.medium.com/v2/resize:fit:1050/0*ZsfdagAaNPIURz-T align="left")

Copy the public IP of your **Monitoring Server** and paste on your favorite browser with a 9090 port with /target. You will see the targets that you have added in the /etc/prometheus/prometheus.yml file.

![](https://miro.medium.com/v2/resize:fit:1050/0*HeQA4Z9kB8vSL2g4 align="left")

To add the Jenkins Dashboard on your Grafana server.

Click on **New -&gt; Import.**

![](https://miro.medium.com/v2/resize:fit:1050/0*ADwMUfrySoYIgKIr align="left")

Provide the 9964 to Load the dashboard.

![](https://miro.medium.com/v2/resize:fit:1050/0*JNvfDH2DcCQp756n align="left")

Select the default Prometheus from the drop-down menu and click on **Import.**

![](https://miro.medium.com/v2/resize:fit:1050/0*DNwQRwgaEXGYCLCW align="left")

You will see your Jenkins Monitoring dashboard in the below snippet.

![](https://miro.medium.com/v2/resize:fit:1050/0*vRZAd2wKmhjWjSiS align="left")

Now, we have to integrate Email Alert. So, if our Jenkins pipeline will succeed or fail we will get a notification alert on our email.

To do that, we need to install the Jenkins Plugin, whose name is **Email Extension Template**.

Manage Jenkins -&gt; Plugins and install the **Email Extension Template** plugin.

![](https://miro.medium.com/v2/resize:fit:1050/0*Gp3vTWgxIRgzujMP align="left")

After installing the plugin, go to your email ID and click on Manage account and you will see what looks like the below snippet.

![](https://miro.medium.com/v2/resize:fit:1050/0*0NaTJplQJqdKIEwg align="left")

In the **Security** section, search for **App passwords** and click on it.

![](https://miro.medium.com/v2/resize:fit:1050/0*2x-t1miJZjdnEh-N align="left")

Gmail will prompt you for the password. Provide the password then you have to provide the name of your app where you are integrating email service.

![](https://miro.medium.com/v2/resize:fit:1050/0*EgtCqmTZVlVatfuy align="left")

You will get your password below. Copy the password and keep it secure somewhere.

![](https://miro.medium.com/v2/resize:fit:1050/0*jWhQmNQb6S_adx90 align="left")

Add your email ID and the password that you have generated in the previous step.

Go to **Manage Jenkins -&gt; Credentials.**

Click on **(global)**.

![](https://miro.medium.com/v2/resize:fit:1050/0*qh0sFHqogLmz125G align="left")

Click on **Add credentials**

![](https://miro.medium.com/v2/resize:fit:1050/0*xdZRkZReBAzMHj0- align="left")

Select the Username with password in Kind.

Provide your mail ID and generated password then provide the ID as mail to call both credentials.

![](https://miro.medium.com/v2/resize:fit:1050/0*3PnlKa2kcXNVnQiv align="left")

You can see we have added the credentials for the mail.

![](https://miro.medium.com/v2/resize:fit:1050/0*5d4sgYf0wOuz91X7 align="left")

Now, we have to configure our mail for the alerts.

Go to Jenkins -&gt; **Manage Jenkins** \-&gt; **System**

Search for Extend E-mail Notification.

Provide the [smtp.gmail.com](https://github.com/AmanPathak-DevOps/Netflix-Clone-K8S-End-to-End-Project) in the **SMTP server** and 465 in the **SMTP port.**

![](https://miro.medium.com/v2/resize:fit:1050/0*MoAcJbNX2I1zvTvE align="left")

Then, On the same page Search for Extend E-mail Notification.

Provide the [smtp.gmail.com](https://github.com/AmanPathak-DevOps/Netflix-Clone-K8S-End-to-End-Project) in the **SMTP server** and 465 in the **SMTP port.**

Select **Use SMTP Authentication** and provide the Gmail ID and its password in the Username and password.

To validate whether Jenkins can send the emails to you or not, check the **Test configuration by sending a test e-mail.**

![](https://miro.medium.com/v2/resize:fit:1050/0*7Fs-u_sQolwzrowO align="left")

You can see below for the reference.

![](https://miro.medium.com/v2/resize:fit:1050/0*Uznr1jz70hTwsa35 align="left")

Now, we will set up our Jenkins Pipeline. But there are some plugins required to work with them.

Download the following plugins

Eclipse Temurin installer

SonarQube Scanner

NodeJS

![](https://miro.medium.com/v2/resize:fit:1050/0*1-haMBx4x14cluSv align="left")

Now, configure the plugins

Go to **Manage Jenkins -&gt; Tools**

Click on Add JDK and provide the following things below

![](https://miro.medium.com/v2/resize:fit:1050/0*NVi1IeffQS7C8uGK align="left")

Click on Add NodeJS and provide the following things below

![](https://miro.medium.com/v2/resize:fit:1050/0*8vNrWER04DFO1XBG align="left")

Now, we will configure Sonarqube

To access the sonarqube, copy the Jenkins Server public IP with port number 9000

Then, click **Security** and click on **Users.**

![](https://miro.medium.com/v2/resize:fit:1050/0*cTuCvMOuZb7njZaf align="left")

Click on the highlighted blue box on the right to generate the token.

![](https://miro.medium.com/v2/resize:fit:1050/0*vr-9GWm9bFRcFnd5 align="left")

Now provide the name of your token and click on **Generate.**

![](https://miro.medium.com/v2/resize:fit:1050/0*eQi2TbbHB6tLXNj- align="left")

Copy the generated token and keep it somewhere.

![](https://miro.medium.com/v2/resize:fit:1050/0*H5WQmhd0QbGZgYi9 align="left")

Now, add the token to your Jenkins credentials

Go to **Manage Jenkins -&gt; Credentials.**

Select the Secret text in Kind.

Provide your token then provide the ID as sonar-token to call the credentials.

![](https://miro.medium.com/v2/resize:fit:1050/0*Q1Fg4CgCyvdNZ04A align="left")

Go to **Manage Jenkins -&gt; System**

Click on **Add Sonarqube**

![](https://miro.medium.com/v2/resize:fit:1050/0*YNdYW-I4NmundFV6 align="left")

Provide the name sonar-server with the Server URL and select the credentials that we have added.

![](https://miro.medium.com/v2/resize:fit:1050/0*4_Bz3capP2tyRyCh align="left")

Go to **Manage Jenkins -&gt; Tools**

Find Sonarqube Scanner and click on **Add**

![](https://miro.medium.com/v2/resize:fit:1050/0*kQrYv1XUv3Fb9-Rn align="left")

Provide the name sonar-server and select the latest version of Sonarqube.

![](https://miro.medium.com/v2/resize:fit:1050/0*qAmjmAWSYebLSp17 align="left")

To create a webhook, click on **Configuration** and select **Webhooks.**

![](https://miro.medium.com/v2/resize:fit:1050/0*D-iWit4S02oMCEv- align="left")

Click on **Create.**

![](https://miro.medium.com/v2/resize:fit:1050/0*qhQszIC1dYywthxp align="left")

Provide the name and Jenkins URL like below and click on **Create.**

![](https://miro.medium.com/v2/resize:fit:1050/0*26idM6DCOddjA88G align="left")

The Webhook will be showing the below snippet.

![](https://miro.medium.com/v2/resize:fit:1050/0*AUit5gD_wYRq3GEX align="left")

To create a project, click on **Manually.**

![](https://miro.medium.com/v2/resize:fit:1050/0*Az20K6tn3elsPbjM align="left")

Provide the name of your project and click on **Set up.**

![](https://miro.medium.com/v2/resize:fit:1050/0*PZ9N3tC7WI-kAR7S align="left")

Select the existing token and click on **continue.**

![](https://miro.medium.com/v2/resize:fit:1050/0*RiN5RYOfQPGrjY8y align="left")

Select the Other as your build and Linux as OS.

![](https://miro.medium.com/v2/resize:fit:1050/0*4Xt2tPdkn9rRvAv0 align="left")

Now, we will create the Jenkins Pipeline

Click on **Create item.**

![](https://miro.medium.com/v2/resize:fit:1050/0*vRNS7zr8_ImDvphY align="left")

Provide the name of your Jenkins Pipeline and select **Pipeline.**

![](https://miro.medium.com/v2/resize:fit:1050/0*rKSLJ0R4B7IDkZmY align="left")

Currently, we are just creating a pipeline for Sonarqube analysis of the code, quality gate for Sonarqube, and installing the dependencies.

In the post-build, we have added email alerts for the success or failure of the pipeline.

```plaintext
pipeline{
    agent any
    tools{
        jdk 'jdk'
        nodejs 'nodejs'
    }
    environment {
        SCANNER_HOME=tool 'sonar-server'
    }
    stages {
        stage('Workspace Cleaning'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'master', url: 'https://github.com/SreeniDevSecOps/Netflix-Clone-K8S-End-to-End-Project.git'
            }
        }
        stage("Sonarqube Analysis"){
            steps{
                withSonarQubeEnv('sonar-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Netflix \
                    -Dsonar.projectKey=Netflix \
                    '''
                }
            }
        }
        stage("Quality Gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }  
    }
    post {
     always {
        emailext attachLog: true,
            subject: "'${currentBuild.result}'",
            body: "Project: ${env.JOB_NAME}<br/>" +
                "Build Number: ${env.BUILD_NUMBER}<br/>" +
                "URL: ${env.BUILD_URL}<br/>",
            to: 'devsecopssreeni27@gmail.com',
            attachmentsPattern: 'trivyfs.txt,trivyimage.txt'
        }
    }
}
```

![](https://miro.medium.com/v2/resize:fit:1050/0*eZ3v6M-uaDhZmfd_ align="left")

Click on build pipeline and after getting the success of the pipeline.

You will see the Sonarqube code quality analysis which will look like the below snippet.

![](https://miro.medium.com/v2/resize:fit:1050/0*KZCBDeDaxQuFUewf align="left")

Now, we have to add one more tool for our application named OWASP Dependency-check.

Go to **Manage Jenkins -&gt; Plugins**

Search for OWASP Dependency-Check and install it.

![](https://miro.medium.com/v2/resize:fit:1050/0*0HyBKRMe5ddRGWgv align="left")

After installing, make sure to configure the OWASP.

Provide the name select the latest version of OWASP and click on **Save.**

![](https://miro.medium.com/v2/resize:fit:1050/0*jrE3zjN_ROM04odj align="left")

Now, add the OWASP dependency check stage in the Jenkins pipeline and click on **Save.**

```plaintext
stage('OWASP DP SCAN') {
            steps {
                dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'owasp-dp-check'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }
stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.txt"
            }
        }
```

![](https://miro.medium.com/v2/resize:fit:1050/0*OSfIU7IsgUKhmsEF align="left")

Now, click on **Build Now.**

![](https://miro.medium.com/v2/resize:fit:1050/0*ooHv1_rHr-iQYmLD align="left")

Once your pipeline is successful. Then, scroll down and you will see a dependency check.  
Click on it and you will see the output like the below snippet.

![](https://miro.medium.com/v2/resize:fit:1050/0*S9XpaB-ZomllxygT align="left")

Now, we have to build our Docker Image and push it to DockerHub

To do that, we need to configure the following things.

Go to **Manage Jenkins -&gt; Credentials**

Add Docker Credentials to your Jenkins

![](https://miro.medium.com/v2/resize:fit:1050/0*9ZBFiKmjLp1P6ZSD align="left")

Add your credentials and click on **Create.**

![](https://miro.medium.com/v2/resize:fit:1050/0*R2s4LZzjKowLjzBO align="left")

Install the following Docker plugins on your Jenkins

```plaintext
Docker
Docker Commons
Docker Pipeline
Docker API
docker-build-step
```

![](https://miro.medium.com/v2/resize:fit:1050/0*lvIGsbJ-otOO716Y align="left")

Restart your Jenkins

![](https://miro.medium.com/v2/resize:fit:1050/0*0-F7VPuQNo17qC_Z align="left")

Configure the tool in Jenkins

Go to **Manage Jenkins** -&gt; Tools and provide the below details.

![](https://miro.medium.com/v2/resize:fit:1050/0*IhNaDlbUnGFnlRFv align="left")

Our application is Netflix Clone. So we need some movie databases on our application.

For that, we have one application that will provide the API. So, we can use the API to get the movies on our application.

TMDB is one of them

Go to this link [https://www.themoviedb.org/](https://github.com/AmanPathak-DevOps/Netflix-Clone-K8S-End-to-End-Project)

Click on **Join TMDB**

![](https://miro.medium.com/v2/resize:fit:1050/0*BFt1z7f1xqSirxrt align="left")

Enter the details and click on **SignUp**

![](https://miro.medium.com/v2/resize:fit:1050/0*51oBLh5fw71IMP0e align="left")

Once you sign up, you will get a confirmation email on your account. Confirm it.

Log in to your TMDB account and go to the settings.

![](https://miro.medium.com/v2/resize:fit:1050/0*kxWiK46wzrD83Gpu align="left")

Go to the **API** section.

![](https://miro.medium.com/v2/resize:fit:1050/0*FoaEAFvQYSL1QhVM align="left")

Click on **Create** to generate API

![](https://miro.medium.com/v2/resize:fit:1050/0*3afDadk8jHPaRvfK align="left")

Select **Developer.**

![](https://miro.medium.com/v2/resize:fit:1050/0*2iFIRMR4nP-GKUQs align="left")

**Accept** the Terms & Conditions.

![](https://miro.medium.com/v2/resize:fit:1050/0*aq5LZpapiAJzPIqg align="left")

Provide the basic details and click on **Submit.**

![](https://miro.medium.com/v2/resize:fit:1050/0*kQarBUstYYW4qxxa align="left")

After clicking on **Submit**. You will get your **API.** Copy the API and keep it somewhere.

![](https://miro.medium.com/v2/resize:fit:1050/0*-TvV99Fauzlq59Pu align="left")

Now, we have to configure our Docker images where we will build our image with the help of new code and then, push it to DockerHub.

After pushing the image, we will scan our DockerHub Image to find the vulnerabilities in the image.

Make sure to replace the API with your API and if you are pushing Dockerfile on your Dockerhub account then, replace my username of the Dockerhub with yours.

![](https://miro.medium.com/v2/resize:fit:1050/0*SsaDUI7-LJRh57MM align="left")

Click on **Build**

![](https://miro.medium.com/v2/resize:fit:1050/0*-mmKRz7NqHcaF4MS align="left")

As you can see Our Pipeline is successful.

![](https://miro.medium.com/v2/resize:fit:1050/0*F-Tu0hUEB12lqRSc align="left")

Now, validate whether the docker image has been pushed to DockerHub or not.

Log in to your Dockerhub account.

As you can see in the below screenshot, Our Docker image is present on Docker Hub.

![](https://miro.medium.com/v2/resize:fit:1050/0*WrHAUAILtBMyVhfN align="left")

Now, we have to deploy our application using Kubernetes.

To do that, we need to install kubectl on the Jenkins server.

```plaintext
sudo apt update
sudo apt install curl
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
```

![](https://miro.medium.com/v2/resize:fit:1050/0*nuW-uuhNitMOiIjz align="left")

As you know, we have two Kubernetes Nodes of which one is the Master and the other one is the Worker Node.

Login to your both Kubernetes Master and Worker Nodes

**Master Node**

![](https://miro.medium.com/v2/resize:fit:1050/0*fH_hBvXug0EZfgvk align="left")

**Worker Node**

![](https://miro.medium.com/v2/resize:fit:1050/0*nHRSnRD3MxWrNpgy align="left")

Add the hostname to your Kubernetes master node

```plaintext
sudo hostnamectl set-hostname K8s-Master
```

![](https://miro.medium.com/v2/resize:fit:1050/0*CMTpTH8i4U4gFlwv align="left")

Add the hostname to your Kubernetes worker node

```plaintext
sudo hostnamectl set-hostname K8s-Worker
```

![](https://miro.medium.com/v2/resize:fit:1050/0*v0klh_JPLYpieFNR align="left")

Run the below commands on the both Master and worker Nodes.

```plaintext
sudo su
swapoff -a; sed -i '/swap/d' /etc/fstab
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# Apply sysctl params without reboot
sudo sysctl - system
apt update
sudo apt-get install -y apt-transport-https ca-certificates curl
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg - dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt update
apt-get install -y kubelet kubeadm kubectl kubernetes-cni
apt install docker.io -y
sudo mkdir /etc/containerd
sudo sh -c "containerd config default > /etc/containerd/config.toml"
sudo sed -i 's/ SystemdCgroup = false/ SystemdCgroup = true/' /etc/containerd/config.toml
systemctl restart containerd.service
systemctl restart kubelet.service
systemctl enable kubelet.service
```

Now, run the following commands **Only on the Master Node,** and then you will get the command that is highlighted in the below snippet.

```plaintext
kubeadm config images pull
kubeadm init
```

![](https://miro.medium.com/v2/resize:fit:1050/0*lyuG8YKdfDBcF_Tq align="left")

Exit from the root user and run the below commands

```plaintext
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

![](https://miro.medium.com/v2/resize:fit:1050/0*4COYOQO-IF_YklM2 align="left")

**Run on the Worker Node**

Run the below command as a root user

```plaintext
kubeadm join 172.31.59.154:6443 - token deq9nl.y34go2ziii0fu8c1 \
 - discovery-token-ca-cert-hash sha256:e93c56bd59b175b81845a671a82ffd1839e42272d922f9c43ca8d8f6d145ce02
```

![](https://miro.medium.com/v2/resize:fit:1050/0*ju3cFb9QwsHsmoE8 align="left")

Both nodes are not ready because the network plugin is not installed on the master node

![](https://miro.medium.com/v2/resize:fit:1050/0*_RQaZI1Me4uDTdCK align="left")

**Only on the Master Node**

Run the below command to install the network plugin on the Master node

```plaintext
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml
```

![](https://miro.medium.com/v2/resize:fit:1050/0*E0fWlC-ga_PDo6ba align="left")

Both nodes are ready.

![](https://miro.medium.com/v2/resize:fit:1050/0*eZwFS_T4lGFvnfA5 align="left")

Install the following Kubernetes Plugins on your Jenkins

```plaintext
Kubernetes
Kubernetes Credentials
Kubernetes Client API
Kubernetes CLI
Kubernetes Credential Provider
```

![](https://miro.medium.com/v2/resize:fit:1050/0*M6NvdF-7vQ3glbDm align="left")

![](https://miro.medium.com/v2/resize:fit:1050/0*u3FP1rCrgL9YIbM2 align="left")

Now, we will set Kubernetes Monitoring for both Master and worker Nodes

Run the below command on both Kubernetes Nodes

```plaintext
sudo useradd \
--system \
--no-create-home \
--shell /bin/false prometheus
```

![](https://miro.medium.com/v2/resize:fit:1050/0*z3ThZHHLyOLVAx8x align="left")

Download the node exporter package on both Kubernetes Nodes and Untar the node exporter package file and move the node\_exporter directory to the /usr/local/bin directory

```plaintext
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
```

![](https://miro.medium.com/v2/resize:fit:1050/0*Z-4kw3EoojQj42sX align="left")

Create the systemd configuration file for node exporter.

Edit the file

```plaintext
sudo vim /etc/systemd/system/node_exporter.service
```

Copy the below configurations and paste them into the /etc/systemd/system/node\_exporter.service file.

```plaintext
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5
[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter \
 - collector.logind
[Install]
WantedBy=multi-user.target
```

![](https://miro.medium.com/v2/resize:fit:1050/0*O8Toy6Vxwqa00CKR align="left")

Enable the node exporter systemd configuration file and start it.

```plaintext
sudo systemctl enable node_exporter
sudo systemctl enable node_exporter
systemctl status node_exporter.service
```

![](https://miro.medium.com/v2/resize:fit:1050/0*aHgpiGpAfDhVamZT align="left")

Now, we have to add a node exporter to our Prometheus target section. So, we will be able to monitor both Kubernetes Servers.

edit the file

```plaintext
sudo vim /etc/prometheus/prometheus.yml
```

Add both job names(Master & Worker nodes) with their respective public.

![](https://miro.medium.com/v2/resize:fit:1050/0*2fModr9eBeB6wKRb align="left")

After saving the file, validate the changes that you have made using promtool.

```plaintext
promtool check config /etc/prometheus/prometheus.yml
```

If your changes have been validated then, push the changes to the Prometheus server.

```plaintext
curl -X POST http://localhost:9090/-/reload
```

![](https://miro.medium.com/v2/resize:fit:1050/0*YWksH4zQpiKLX9nD align="left")

As you know, Jenkins will deploy our application on the Kubernetes Cluster. To do that, Jenkins must have the access keys or something to connect with the master node.

To do that copy the content inside .kube/config on Kubernetes Master node.

```plaintext
cat .kube/config
```

![](https://miro.medium.com/v2/resize:fit:1050/0*DNQLDllJuOCgwTMp align="left")

Save the file with the .txt extension.

![](https://miro.medium.com/v2/resize:fit:1050/0*ULw7a3-EKh8a3R58 align="left")

Now, add the Secret file in Jenkins Credentials.

Click on **Add credentials.**

![](https://miro.medium.com/v2/resize:fit:1050/0*CGe2OmY53s_Ty0Xv align="left")

Select the **Secret file** and provide the Secret file that you have saved earlier enter the ID k8s then click on **Create.**

![](https://miro.medium.com/v2/resize:fit:1050/0*FTpwoqhqBN73piM0 align="left")

Now, Add the deploy to the Kubernetes stage in your Jenkins pipeline.

```plaintext
stage('Deploy to Kubernetes'){
            steps{
                script{
                    dir('Kubernetes') {
                        withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'k8s', namespace: '', restrictKubeConfigAccess: false, serverUrl: '') {
                                sh 'kubectl apply -f deployment.yml'
                                sh 'kubectl apply -f service.yml'
                                sh 'kubectl get svc'
                                sh 'kubectl get all'
                        }   
                    }
                }
            }
        }
```

![](https://miro.medium.com/v2/resize:fit:1050/0*tJG7sshf18OIjVgW align="left")

Click on **Build Now**

You will see that our Application has been deployed successfully on Kubernetes.

![](https://miro.medium.com/v2/resize:fit:1050/0*RTqnlCOQR85hRnxX align="left")

You can validate whether your pods are running or not from your Kubernetes master node.

![](https://miro.medium.com/v2/resize:fit:1050/0*p4S7U9pq3Tue9D1h align="left")

Also, you can check the Console logs for the earlier results.

![](https://miro.medium.com/v2/resize:fit:1050/0*cFSvEyFx2ZbTXW72 align="left")

We got the email that our pipeline was successful.

We get the trivyfs.txt file which contains the vulnerabilities.

![](https://miro.medium.com/v2/resize:fit:1050/0*Jon4Qzdt30mKzbJb align="left")

Also, we got the vulnerabilities for our Docker Image.

![](https://miro.medium.com/v2/resize:fit:1050/0*hkWist9x-CqjMoBP align="left")

Jenkins sent the console logs by email.

![](https://miro.medium.com/v2/resize:fit:1050/0*CmWg6jNMZdgwLpXl align="left")

If you want to access your Netflix **Clone** Application**.**

Copy the Public IP of Worker Node and paste it on your favorite browser with port 32000 and see the magic.

![](https://miro.medium.com/v2/resize:fit:1050/0*N-bjUNKRLSU_rI9V align="left")

Another Snippet of our Netflix **Clone** application.

![](https://miro.medium.com/v2/resize:fit:1050/0*HOidmEUFvu5xgLUN align="left")

Go to the Grafana Dashboard and select Node Exporter.

You will see the real-time hardware specs of your Kubernetes master node.

![](https://miro.medium.com/v2/resize:fit:1050/0*R4TF7d6iKEqiBrTp align="left")

You will see the real-time hardware specs of your Kubernetes worker node.

![](https://miro.medium.com/v2/resize:fit:1050/0*eosfKXo8tPUk67h9 align="left")

# **Conclusion:**

In conclusion, this guide has equipped you with the knowledge and steps needed to deploy a sophisticated application using Kubernetes. From setting up infrastructure on AWS to integrating monitoring with Grafana and Prometheus, and finally deploying a Netflix Clone application, you’ve covered a wide array of DevOps practices.

🚀 Culmination of Our Kubernetes Expedition! 🎉

This undertaking marked the grand finale, concluding our enriching #30DaysOfKubernetes Series on its 30th day.

We trust you garnered valuable insights throughout this journey. Whether you’re a seasoned explorer or just joining, explore the comprehensive topics covered in our GitHub Repo.