Containerizing a Django Application

Background

As a capstone project for the Meta Back-End Developer Professional Certificate, I developed a comprehensive restaurant management system. This Django-based web application provides essential features including:

  1. Table reservation
  2. Menu and reservation management
  3. API access for both customers and administrators

Since its initial development, I have implemented several modifications to enhance the customer experience. The current version of the project code is available here.

The Next Step: Kubernetes Migration

To enhance scalability and prepare for future improvements, I will migrate this application to a Kubernetes environment.

Key Challenges and Solutions

  1. Flexible Django Settings:
    • Challenge: Django settings need to be adaptable to different environments (development, staging, production) without changing the code.
    • Solution: Utilize Kubernetes ConfigMaps containing environment-specific variables (e.g., DEBUG, ALLOWED_HOSTS, DATABASE_URL).
  2. Web Server and Proxy:
    • Challenge: Need an robust and efficient way to serve the application and handle static files.
    • Solution: Implement Nginx as a reverse proxy and web server serving static files and Gunicorn as dynamic web content provider.
  3. Database Persistence:
    • Challenge: Ensuring data persistence across pod restarts.
    • Solution: Employ Kubernetes PersistentVolumes (PV) and PersistentVolumeClaims (PVC).
  4. Secret Management:
    • Challenge: Securely managing sensitive information like database credentials.
    • Solution: Leverage Kubernetes Secrets.

Process Diagram

When a web request is sent to the Nginx service, it first checks if the requested content is part of the static files. If so, Nginx serves these files directly, optimizing the delivery of static assets like images and stylesheets. For dynamic content, the request is forwarded to the Django service, which processes it using Django and Gunicorn. This service may interact with the MySQL service to perform data read or write operations, ensuring that dynamic content is generated based on current data.

Implementation Steps

The migration process involved the following key steps:

  1. Customization:
    • Modify Django settings to accommodate the Kubernetes environment.
  2. Containerization:
    • Containerize the Django application and its dependencies.
    • Push code to GitHub, triggering GitHub Actions for automatic code review, and create and push the container image to Docker Hub.
  3. Kubernetes Resources:
    • Create necessary YAML files for deployments, services, and persistent volumes.
    • Implement ConfigMaps and Secrets for managing environment variables.
    • As for easier continuous deployment, use ArgoCD to auto-sync YAML files.
    • Employ Keel to monitor Docker image changes and update deployments if an image change is detected.
  4. Deployment:
    • Apply all resource files to the local Kubernetes cluster.
  5. Testing:
    • Thoroughly test the functionality of the application in the Kubernetes environment.

All Kubernetes configuration files and updated application code are available in the project’s github repository.

Preparation of Django Application

Before creating an image for the Django application, we need to modify and add lines in settings.py so that we have control over Django when it’s running in a container::

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
DEBUG = os.getenv("DEBUG") 
...
ALLOWED_HOSTS = os.environ.get('ALLOWED_HOSTS').split(',')
CSRF_TRUSTED_ORIGINS = os.environ.get('CSRF_TRUSTED_ORIGINS').split(',')
...
SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY')
...
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': os.getenv("MYSQL_DATABASE"),
'USER': os.getenv("MYSQL_USER"),
'PASSWORD': os.getenv("MYSQL_PASSWORD"),
'HOST': os.getenv("DB_HOST"),
'PORT': os.getenv("DB_PORT"),
'OPTIONS': {
'init_command': "SET sql_mode='STRICT_TRANS_TABLES'"
}
}
}
...
STATIC_ROOT=os.getenv("STATIC_ROOT")
...
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
...
TEMPLATES = [
{
...
'DIRS': [os.path.join(BASE_DIR, 'templates')],
...
],
...
MIDDLEWARE = [
'restaurant.middleware.HealthCheckMiddleware',
'django.middleware.common.CommonMiddleware',
...
]
  1. DEBUG: Enable or Disable Debug mode on Django. It is recommended to disable this option unless troubleshooting is needed.
  2. ALLOWED_HOSTS: A list of strings representing the host/domain names that this Django site can serve.
  3. CSRF_TRUSTED_ORIGINS: A list of trusted origins for unsafe requests (e.g., POST). This setting is particularly important in production environments where requests might come from various domains or subdomains.
  4. DATABASES: We will use ConfigMap later to pass DB information to Django server.
  5. STATIC_ROOT: The absolute path to the directory where collectstatic will collect static files for deployment. If the staticfiles contrib app is enabled (as in the default project template), the collectstatic management command will collect static files into this directory. Nginx will serve static files directly from the directory.
  6. BASE_DIR: Define the base directory of the Django project.
  7. TEMPLATES: Specifies where templates are located; ensure Django has access to the correct path since templates are not served as static files.
  8. MIDDLEWARE: Add 'restaurant.middleware.HealthCheckMiddleware',, which is a custom middleware which handles health check requests without triggering CSRF warnings. We also need to add a middleware.py file under the restaurant folder.
1
2
3
4
5
6
7
8
9
10
# middleware.py 
from django.http import HttpResponse
class HealthCheckMiddleware:
def __init__(self, get_response):
self.get_response = get_response

def __call__(self, request):
if request.path == '/health':
return HttpResponse('ok')
return self.get_response(request)

Creating Docker Image for Django Application

Run pip freeze > requirements.txt to get a list of all packages needed for the local development environment.

Here is what I got:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
asgiref==3.7.2
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
cryptography==42.0.5
defusedxml==0.8.0rc2
Django==5.0.2
django-storages==1.14.4
django-templated-mail==1.1.1
djangorestframework==3.14.0
djangorestframework-simplejwt==5.3.1
djoser==2.2.2
gunicorn==23.0.0
idna==3.6
mysqlclient==2.2.4
oauthlib==3.2.2
packaging==24.1
pycparser==2.21
PyJWT==2.8.0
python3-openid==3.2.0
pytz==2024.1
requests==2.31.0
requests-oauthlib==1.3.1
social-auth-app-django==5.4.0
social-auth-core==4.5.3
sqlparse==0.4.4
tzdata==2024.1
urllib3==2.2.1

Create a Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
FROM python:3.11

# Copy the requirements file into the container
COPY requirements.txt .

# Install the Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the entire project directory into the container
COPY ./app .

# Set the working directory to where manage.py is located
WORKDIR /littlelemon

# Start the application
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--chdir", "littlelemon","littlelemon.wsgi:application"]

Once code is pushed to GitHub, a GitHub Action is triggered to review the code using SonarQube. It then builds a Docker image and pushes it to Docker Hub. Finally, the image is scanned for vulnerabilities using Trivy, and the results are posted under the Security tab in GitHub.

Django YAML Files

Create a new namespace for our application.

Namespace: django-app-ns.yml

1
2
3
4
apiVersion: v1
kind: Namespace
metadata:
name: django-app

In this ConfigMap, we provide necessary environment variables for Django settings.

ConfigMap: django-config.yml

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: ConfigMap
metadata:
name: app-cm
namespace: django-app
data:
DB_PORT: "3306"
DB_HOST: "mysql-svc"
STATIC_ROOT: "/data/static"
DEBUG: "False"
ALLOWED_HOSTS: "localhost"
CSRF_TRUSTED_ORIGINS: "http://localhost:30005"

Create PersistentVolume and PersistentVolumeClaim for data storage. Here, hostPath is used as the type of PersistentVolume; for Docker Desktop on Windows, ensure that /run/desktop/mnt/host/ prefixes the path.

The PersistentVolume is set to ReadWriteMany, as both Django and Nginx pods need access to the staticfiles.

PersistentVolume: staticfiles-pv.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: v1
kind: PersistentVolume
metadata:
name: staticfiles-pv
namespace: django-app
labels:
type: local
app: django-staticfiles
spec:
storageClassName: manual
capacity:
storage: 500M
accessModes:
- ReadWriteMany
hostPath:
path: "/run/desktop/mnt/host/c/temp/k8s/static"

PersistentVolumeClaim: staticfiles-pvc.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: staticfiles-pvc
namespace: django-app
spec:
storageClassName: manual
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500M
volumeName: staticfiles-pv

Now, we are ready to create a deployment for Django. In this deployment configuration, an init container waits for MySQL service to start before launching the Django pod. After Django pod starts, a lifecycle hook is set to run commands necessary for migration and collect static files to the designated location /data/static.

Deployment: django-deploy.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
apiVersion: apps/v1
kind: Deployment
metadata:
name: django-deploy
namespace: django-app
annotations:
keel.sh/policy: force
keel.sh/trigger: poll
keel.sh/pollSchedule: "@every 10m"
spec:
replicas: 1
selector:
matchLabels:
app: django-app
template:
metadata:
labels:
app: django-app
spec:
volumes:
- name: staticfiles
persistentVolumeClaim:
claimName: staticfiles-pvc
initContainers:
- name: wait-for-mysql
image: busybox:1.28
command: ['sh', '-c', 'until nc -z mysql-svc 3306; do echo waiting for mysql; sleep 2; done;']
containers:
- image: mhe5/django-k8s-custom-web:latest
imagePullPolicy: Always
name: django-app-container
envFrom:
- configMapRef:
name: app-cm
env:
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_DATABASE
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_USER
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_PASSWORD
ports:
- containerPort: 8000
volumeMounts:
- mountPath: "/data/static"
name: staticfiles
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 5
resources:
requests:
memory: 100Mi
cpu: 50m
limits:
memory: 500Mi
cpu: 1
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 15
periodSeconds: 10
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- |
python manage.py makemigrations
python manage.py migrate
python manage.py collectstatic --noinput

A service for Django pod then can be added. The NodePort is enabled for testing purpuse and should be disabled after testing.

Service: django-svc.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: Service
metadata:
name: django-svc
namespace: django-app
labels:
app: app-svc
spec:
type: NodePort
ports:
- port: 8000
targetPort: 8000
nodePort: 30000
selector:
app: django-app

MySQL Database Configuration

MySQL YAML Files

PersistentVolume: mysql-pv.yml

A PersistentVolume and a PersistentVolumeClaim need to be created for MySQL database. Ensure that the directory path /c/temp/k8s/db-data/ is empty on the host before the initial start; otherwise, MySQL may fail during initialization.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-pv
namespace: django-app
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 500M
accessModes:
- ReadWriteOnce
hostPath:
path: "/run/desktop/mnt/host/c/temp/k8s/db-data/"

PersistentVolumeClaim mysql-pvc.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
namespace: django-app
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500M
volumeName: mysql-pv

Secret: mysql-secret.yml

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: Secret
metadata:
name: mysql-secret
namespace: django-app
type: Opaque
data:
MYSQL_ROOT_PASSWORD: ZGF0YWJhc2U0TWU=
MYSQL_DATABASE: bGl0dGxlbGVtb24=
MYSQL_USER: YWRtaW4=
MYSQL_PASSWORD: ZW1wbG95ZWVAMTIz

Deployment: mysql-deploy.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
namespace: django-app
spec:
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql:8.3
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_ROOT_PASSWORD
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_DATABASE
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_USER
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_PASSWORD
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql/
name: db-data
volumes:
- name: db-data
persistentVolumeClaim:
claimName: mysql-pvc

Service: mysql-svc.yml

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: Service
metadata:
name: mysql-svc
namespace: django-app
spec:
ports:
- port: 3306
targetPort: 3306
selector:
app: mysql

Nginx Server Configuration

Nginx YAML Files

ConfigMap: nginx-config.yml

This ConfigMap provides necessary configurations for the Nginx server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-cm
namespace: django-app
data:
default.conf: |
server {
listen 80;
server_name localhost;

location /static/ {
alias /data/static/;
}

location / {
proxy_pass http://django-svc:8000; # Point to your Django application's host and port
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}

Deployment: nginx-deploy.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deploy
namespace: django-app
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
volumes:
- name: nginx
configMap:
name: nginx-cm
- name: staticfiles
persistentVolumeClaim:
claimName: staticfiles-pvc
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: /etc/nginx/conf.d/default.conf
subPath: default.conf
name: nginx
readOnly: true
- mountPath: "/data/static"
name: staticfiles

Service: nginx-svc.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
namespace: django-app
labels:
app: nginx-svc
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 30005
selector:
app: nginx

Deploying Kubernetes Resources

My local Kubernetes cluster is Docker Desktop’s one-node Kubernetes cluster in version 1.30.2.
Set django-app as the default namespace for convenience:

1
kubectl config set-context --current --namespace=django-app

Get MySQL Database ready

Run kubectl apply -f .\DB\ to create all resources related to MySQL.

Make sure that the MySQL pod stays in a running state. If pods are in a crashing state and their logs can’t be viewed with kubectl logs command, view container logs from Docker desktop.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
PS > kubectl apply -f .\DB\
deployment.apps/mysql configured
persistentvolume/mysql-pv configured
persistentvolumeclaim/mysql-pvc configured
secret/mysql-secret configured
service/mysql-svc configured

PS > kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-56c5c8d58c-t2ch5 1/1 Running 0 19s

PS > kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
mysql 1/1 1 1 22s

PS > kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mysql-svc ClusterIP 10.96.42.122 <none> 3306/TCP 31s

To verify the credentials were created for MySQL database, test the access to the database using a temparary MySQL pod as shown below:

1
2
3
4
PS > kubectl run -it --rm --image=mysql:8.3 --restart=Never mysql-client -- mysql -h mysql-svc -u admin -pemployee@123
If you don't see a command prompt, try pressing enter.

mysql>

Get Django ready

Run kubectl apply -f .\Django-app\ to create all resources related to Django. Then verify Django pods remain in a running state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
PS > kubectl apply -f .\Django-app\
configmap/app-cm configured
deployment.apps/django-deploy configured
service/django-svc configured
persistentvolume/staticfiles-pv configured
persistentvolumeclaim/staticfiles-pvc configured

PS > kubectl get pod
NAME READY STATUS RESTARTS AGE
django-deploy-68bb98fcff-5ch2p 1/1 Running 0 15s
mysql-56c5c8d58c-t2ch5 1/1 Running 0 1h

PS > kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
django-deploy 1/1 1 1 23s
mysql 1/1 1 1 1h

PS \> kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
django-svc NodePort 10.102.97.139 <none> 8000:30000/TCP 20s
mysql-svc ClusterIP 10.96.42.122 <none> 3306/TCP 1h

With the NodePort service set for Django, we can test it in the browser at **http://localhost:30000**.

Note: There may be issues with serving static files. Research suggests this is related to Gunicorn, which is not designed to serve static files efficiently. This is why we’ll use Nginx to handle static files more effectively.

Get Nginx ready

As the same, run kubectl apply -f .\Django-app\ to create all resources related to Nginx.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
PS > kubectl get pod
NAME READY STATUS RESTARTS AGE
django-deploy-68bb98fcff-5ch2p 1/1 Running 0 2h
mysql-56c5c8d58c-t2ch5 1/1 Running 0 3h
nginx-deploy-68df898b4d-kn7qd 1/1 Running 0 20s

PS > kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deploy 1/1 1 1 23s
django-deploy 1/1 1 1 2h
mysql 1/1 1 1 3h

PS \> kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-svc NodePort 10.105.230.108 <none> 80:30005/TCP 25s
django-svc NodePort 10.102.97.139 <none> 8000:30000/TCP 2h
mysql-svc ClusterIP 10.96.42.122 <none> 3306/TCP 3h

Test the connection in browser at http://localhost:30005.

Test the Deployed Application

Verify Website Functionality

Home page

About page

Reservation page

On Sept. 24, there are few time slots available except 11AM and 12PM.

We made a reservation for Henry on Sept. 24 at 1PM.

Verify the Django Admin Portal

We need to access the Django pod and run python manage.py createsuperuser to create a superuser first. Use the following credentials (for testing purposes only):

  • Username: admin
  • Password: Admin

Note: Using simple credentials like this is not recommended for production environments.

1
2
3
4
5
6
7
8
9
10
11
12
PS > kubectl exec -it django-deploy-68bb98fcff-5ch2p -- /bin/bash
Defaulted container "django-app-container" out of: django-app-container, wait-for-mysql (init)
root@django-deploy-68bb98fcff-5ch2p:/littlelemon# python manage.py createsuperuser
Username (leave blank to use 'root'): admin
Email address:
Password:
Password (again):
The password is too similar to the username.
This password is too short. It must contain at least 8 characters.
This password is too common.
Bypass password validation and create user anyway? [y/N]: y
Superuser created successfully.

The Django admin user should be able to edit the menu and manage reservations through the admin portal.

Verify API Functionalities

Use Insomnia or a similar API testing tool to verify the following endpoints:

Admin view:

GET: View all Reservations:

GET: View menu items

POST: make a reservation

POST: add a new menu item

PATCH: Modify a reservation

PATCH: Modify a menu item

DELETE: Delete a reservation

DELETE: Delete a menu item

User view:

Create a user without any admin priviledges

GET: View menus

GET: View reservations (only reservations under user’s name should be visible)

POST: Add a menu item (request should be denied)

POST: Make a reservation

PATCH: Modify a menu item (request should be denied)

PATCH: Modify a reservation

DELETE: Delete a reservation

Delete own reservation
Delete other’s reservation (request should be denied)

DELETE: Delete a menu item (request should be denied)

Summary

We have successfully migrated a local Django project to a Kubernetes cluster and confirmed its functionality within this environment. This project has been a valuable learning experience, enhancing my skills in both back-end development and DevOps practices, while also being an enjoyable journey.

Kubernetes was the highlight of this project. Its smooth and efficient handling of our complex application reaffirmed its crucial role in our infrastructure. Although I was already familiar with Kubernetes, this project allowed me to leverage its features more extensively, particularly in orchestrating our Django application with other services.

Reflecting on the development process, I’ve utilized multiple CI/CD tools such as GitHub Actions, ArgoCD, and Keel. These tools significantly saved time and improved the application. However, there are areas for improvement in the CI/CD pipeline. I might need to restructure it using a Jenkins pipeline. Additionally, I’m considering expanding the application’s functionality with new features like a delivery order system. Who knows, I might even open my own restaurant one day.

Overall, this project largely mimics the development stage of a real-world development cycle. The next step is to design a production environment, which will be set in an EKS cluster.

Next Step and Future considerations:

  1. Migrate to AWS EKS
    • Implementing proper networking and security measures
    • Set up Ingress for external traffic routing.
    • Utilize AWS RDS for better database management
    • Leverage AWS CodePipeline for CI/CD
    • Optimize resource allocation and limits
    • Set up a domain to serve the website publicly
  2. Delivery Order API Feature
    • Implement user order cart functionality
    • Add order status viewing for users
    • Create admin/manager order modification capabilities
    • Develop a system for assigning orders to delivery crew
    • Allow delivery crew to view and confirm assigned orders

Reference

Deploying Django Apps in Kubernetes | The PyCharm Blog (jetbrains.com)