Is the Server Down? Send Automated Alerts to Your Team Using Python

And create a website monitoring script to track downtimes

Photo by Ussama Azam on Unsplash

Over the years, I’ve faced many issues where the server is down and the client lets us know. I think many other developers like me have faced similar problems. It doesn’t look good on the software company and developers' part!

Being an iOS developer with no knowledge of scripting languages like Python, I’ve always felt tied down and I’ve never dared to try them. Yes, there are system monitoring tools available — both open source and paid and those can be used, but integrating another dependency in the system when I just want a simple Email and SMS alert to be sent to the team members when the server is down — I didn’t like that idea.

I have a Mac mini in my home which is on 24/7 with a stable internet connection, so I thought — why not use it to ping my project’s server every few minutes or an hour and check its status? I decided to sit down and learn a few chunks from here and there to just be able to write the script and run it in cron to alert me when the server goes down.

I decided to include three things:

• Send all the teammates (including the Project Manager) an email alert with details of when my script is unable to ping the server.
• Send the project manager and team lead an SMS alert.
• Alert me with sound when it is night time (for experimental purposes).

As I know very little about the language, I decided to take it one step at a time.

Step 1: Creating a New Project (Site Monitor)

This was easy. I opened terminal, created a new directory for the project and a virtual environment, and then activated it:

mkdir sitemonitor
cd sitemonitor/
python3 -m venv monitor_env
source monitor_env/bin/activate

Step 2: Creating the Project Information Structure in a JSON File

I need to decide on the basic structure of the project information which I will be needing for the monitor. I needed project name, server URL path to ping to find the status of the server, and then the project members' information: their name, email address (for email alert), and mobile number(for SMS alert). As I would be using a trial Twilio account and decided to make a way for an SMS alert on/off for a project, depending upon the severity of the project, I added another key: send_sms_alert.

[
 {
  "server" : "url here",
  "project_name": "project name here",
  "send_sms_alert" : false,
  "manager": {
   "name" : "manager name here",
   "email" : "manager email address here",
   "number" : "manager mobile number here"
  },
  "lead": {
   "name" : "lead name here",
   "email" : "lead email address here",
   "number" : "lead mobile number here"
  },
  "other_members" : [
   {
    "name" : "member 1 name here",
    "email" : "member 1 email address here",
    "number" : "member mobile number here"
   }
  ]
 }
]

Step 3: Loading the Projects From JSON File

import json
def check_server_down(project):
    # here we ping the server and check it's status
def load_projects():
    f = open('projects.json')
    projects = json.load(f)
    for project in projects:
         check_server_down(project)
    f.close()
load_projects()

Step 4: Pinging the Project Server URL to Check its HTTP Response Status Code

We assume that we have an endpoint which if the server is up responds with 200 OK always. For this, we need to install the requests library first. But we need to make sure we have the virtual environment activated.

(monitor_env) Ashwins-MacBook-Pro:sitemonitor myMac$ pip install requests

Now, we call the URL, so inside the method checkServerDown we wrote earlier:

import logging
import requests
import json
def check_server_down(project):
    r = requests.get(project['server'], timeout=timeOutTime)
 
    if r.status_code != 200:
        logging.error("Server down of project {} at   {}".format(project['project_name'], datetime.now()))
        send_email_alert(project)
        if project['send_sms_alert']:
           send_sms_alert(project)
        play_alert()
def load_projects():
    f = open('projects.json')
    projects = json.load(f)
    for project in projects:
        check_server_down(project)
    f.close()
load_projects()

Also for logging purposes, we use the logging module.

Now as shown above we have three methods: to send an email alert, SMS alert, and play sound alert. We will be writing their implementation next.

Step 5: Send Email Alert

To send an email alert we need to import the smtplib module and also an smtp service provider. For this example, we’ll be using Gmail as our smtp. For this, however, we need to provide access. So go to the google accounts link that you wish to use as your default email for sending email alerts:

https://myaccount.google.com/lesssecureapps

and enable as shown in the image below:

import smtplib
smtpEmail = "SMTP_EMAIL_ADDRESS"
smtpPassword = "SMTP_PASSWORD"
emailFrom = "EMAIL_FROM"
def send_email_alert(project):
    conn = smtplib.SMTP('smtp.gmail.com',587) # create a new connection
    conn.ehlo()
    conn.starttls()
    conn.login(smtpEmail, smtpPassword)
    projectName = project['project_name']
    subject = '{} server is down'.format(projectName)
    manager =  project['manager']
    lead = project['lead']
    projectMembers = project['other_members']
    membersNames = ', '.join(map(str,list(map(lambda x: x['name'], projectMembers))))
    message = """The {} project server seems to be down. Please contact your team mates and fix it
       Team members:
       Project Manager: {}
       Mobile Number: {}
       Team Lead: {}
       Mobile Number: {}
       Other Members: 
       {}
     """.format(projectName, manager['name'], manager['number'], lead['name'], lead['number'], membersNames)
    emailTo = list(map(lambda x: x['email'], projectMembers))
    emailTo.insert(0, lead['email'])
    emailTo.insert(0, manager['email'])
    emailMessage = 'Subject: {}\n\n{}'.format(subject, message)
 
    conn.sendmail(emailFrom,emailTo,emailMessage) #sends email
    conn.quit() # closes the connection

Above, we created an SMTP connection, then created a list of email address from team members, project manager, and project lead and then used those to send an email from emailFrom. In the email body, we added details of all the team members and the project detail.

After this, I didn’t like the idea of writing the email and password on the script itself so decided to load it from the env file.

Step 6: Creating env File for Credentials

For this we needed to create a new .env file and add all our credentials there:

SMTP_EMAIL_ADDRESS=Email_address_here
SMTP_PASSWORD=your_smtp_email_password_here
EMAIL_FROM=your_from_email_for_alert_here

And then install the python-dotenv library:

(monitor_env) Ashwins-MacBook-Pro:sitemonitor myMac$ pip install python-dotenv

Now load the credentials to be used from env file:

from dotenv import load_dotenv
from pathlib import Path
import os
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)
smtpEmail = os.getenv("SMTP_EMAIL_ADDRESS")
smtpPassword = os.getenv("SMTP_PASSWORD")
emailFrom = os.getenv("EMAIL_FROM")

Step 7: Send SMS Alert

I decided to add an SMS alert to the system, which will be for the project manager and team lead. For this, I created a new Twilio account and got my test Twilio phone number: https://www.twilio.com/login

I installed Twilio library in my project:

(monitor_env) Ashwins-MacBook-Pro:sitemonitor myMac$ pip install twilio

Next added the Twilio credentials in my env file:

TWILIO_SID=TWILIO_SID_HERE
TWILIO_AUTH_TOKEN=TWILIO_AUTH_TOKEN_HERE

Then added the necessary code to send the SMS alert:

from twilio.rest import Client
def send_sms_alert(project):
    account_sid = os.getenv("TWILIO_SID")
    auth_token  = os.getenv("TWILIO_AUTH_TOKEN")
    client = Client(account_sid, auth_token)
    messageBody = '{} server is down. Please have a look in the system. This auto generated message has been sent to other team members as well. Thank you.'.format(project['project_name'])
    manager =  project['manager']
    lead = project['lead']
    projectMembers = project['other_members']
    mobileNumbers = list(map(lambda x: x['number'], projectMembers))
    mobileNumbers.insert(0, lead['number'])
    mobileNumbers.insert(0, manager['number'])
    mobileNumbers = list(set(mobileNumbers)) # remove the duplicates
    for number in mobileNumbers:
      message = client.messages.create(
          to=number, 
          from_="+19177655746",
          body=messageBody)

Step 8: Play Sound Alert

Just learning purposes, I decided to also add sound-playing functionality when the server is down. For this, I added the simpleaudio library:

(monitor_env) Ashwins-MacBook-Pro:sitemonitor myMac$ pip install simpleaudio

Now to play a sound, I added an alert:

import simpleaudio as sa
filename = 'RedAlert.wav'
wave_obj = sa.WaveObject.from_wave_file(filename)
def play_alert():
    play_obj = wave_obj.play()
    play_obj.wait_done()

This completes our script, which now sends an email, SMS, and plays audio when the server is down for a project. Now we need a script which is executed every few minutes or hours using a cron job.

Step 9: Shell Script for cron Job

For this, we create a shell script on the same directory of the project with the name cronjob.sh:

#!/bin/bash
cd /Users/myMac/Documents/sitemonitor
source monitor_env/bin/activate
python monitor.py

This script needs to be made executable with the following command:

Ashwins-MacBook-Pro:sitemonitor myMac$ chmod 777 cronjob.sh

The above command makes the file cronjob.sh readable, writeable, and executable:

Ashwins-MacBook-Pro:sitemonitor myMac$ ls -l
-rwxrwxrwx@ 1 myMac  staff     134 Jan 19 18:56 cronjob.sh

For more details see: https://en.wikipedia.org/wiki/Chmod

Step 10: Adding a New cron Job

On Unix-like operating systems, the crontab command opens the cron table for editing. The cron table is the list of tasks scheduled to run at regular time intervals on the system.

Some of the basic crontab commands:

> crontab -l 
this lists all the cron job in the system
> crontab -r
this removes all the cron job in the system
> crontab -e
this creates a new cron job in the system

Now since we need to add a new cron job for our executable script, that gets executed every 10 minutes:

Ashwins-MacBook-Pro:sitemonitor myMac$ crontab -e

This opens your vim editor, where you write the following:

10 * * * * /Users/myMac/Documents/sitemonitor/cronjob.sh >> /Users/myMac/Documents/sitemonitor/cronError.txt 2>&1

Next, press Esc and then wq and hit enter, as shown in the image below:

Now if we go and list cron jobs, it will show this result:

Ashwins-MacBook-Pro:sitemonitor myMac$ crontab -l

That’s all we need to do — the cronjob is now executed every 10 minutes.

Step 11: Making cron Job Run in MacOS (Catalina and Above)

For mac users with Catalina or above, the job will not be executed because of Mac’s security features. For this we need a final step:

Open System Preferences > Security and Privacy > Privacy > Full Disk Access:

Now, we need to add cron to the list.

For this, in your terminal:

Ashwins-MacBook-Pro:sitemonitor myMac$ cd /usr
Ashwins-MacBook-Pro:sitemonitor myMac$ open .

Then drag the sbin folder to the Favorite section. This is so when we go to add the cron to the full disk access, it will be easy to navigate to the item.

Now the cron job should be running fine.

This has been a learning experience for me. If you have any questions for me, do let me know in the comments section below. Thank you.

Happy coding!