API connection “retry logic with a cooldown period” simulator ( Python exercise )

This is a very simple API call “circuit-breaker” style simulator I’ve written in Python. But since it’s a stateless code snippet, you should call it most likely a “retry logic with a cooldown period” simulator. But there are valid use case scenarios when stateless is the desired state type. For example when a validation of a dataset against a service can either pass or fail and throw an exception and cause the execution flow to halt. This is typical for data-flow styled apps. In such case the circuit-open state is not acceptable. Anyway the goal is to make sure, that whenever we have a connection (or timeout) error during the API call ,we retry after 10 seconds, after 20 seconds, after 30 seconds and then quit trying. The ConnectionError exception is simulated using a non-routable address 10.255.255.1.

However in the microservices world, if you want to implement the full-scale state-full circuit-breaker, please have a look at this article.

import datetime
import time
import logging
import requests

iterator = 1
attempt = 1

# while cycle to simulate connection error using non-routable IP address
while iterator < 40:
    try:
        # if iterator inside the range to simulate success
        if 1 < iterator < 8:
            r = requests.get('https://data.police.uk/api/crimes-at-location?date=2017-02&location_id=884227')
        # else iterator outside the range to simulate the error event
        else:
            r = requests.get('http://10.255.255.1')
        if r.status_code != requests.codes.ok:
            logging.error('Wrong request status code received, %s', r.status_code)
        r = r.json()
        print(r)
        attempt = 1
    except (requests.exceptions.ConnectionError, requests.exceptions.Timeout,
            requests.exceptions.ConnectTimeout, requests.exceptions.ReadTimeout) as conn_err:
        print(f'bingo, ConnectionError, now lets wait {attempt * 10} seconds before retrying', (datetime.datetime.now()))
        time.sleep(attempt * 10)
        attempt = attempt + 1
        if attempt > 3:
            logging.error('Circuit-breaker forced exit')
            raise conn_err
    iterator = iterator + 1

Also try avoiding Python time.sleep() method in AWS Lambdas as this would not be cost efficient, AWS Step Functions would be much more appropriate.

*Updated March 14th 2019: Considering the cost analysis in this article, it might be actually ok to have time.sleep() in the Lambda, depends on your use case though.

https://blog.scottlogic.com/2018/06/19/step-functions.html

Advertisements

Spinning up a Docker container with Flask and Python

Imagine you need to replicate an existing web API returning a JSON ( listing all feeds in some system ) on your local machine for further development purposes and possible extensions. Today I’ll demonstrate how to achieve this using Docker container , Python and Flask. Note that this tutorial requires some previous experience with Python and Docker. Have a look at Flask, it’s a powerful and easy to use Python web framework.

The source web API we’ll be replicating is returning a valid JSON structure listing all the feeds:

[{
  "feed_name": "feed1",
  "feed_type": "feed type 1",
  "filemasks": [
    "filemask11",
    "filemask12"
  ]
},
{
  "feed_name": "feed2",
  "feed_type": "feed type 2",
  "filemasks": [
    "filemask21",
    "filemask22"
  ]
}]

Let’s save this dummy JSON file as feeds.json on our local file system.

Next we’ll setup the environment and start with Docker:

mkdir docker-api

mkdir docker-api/app

mkdir docker-api/feeds

cd docker-api


#1) create Dockerfile:

FROM python:3.6-stretch

COPY . .

RUN pip install -r requirements.txt

WORKDIR /app/

ENTRYPOINT ["python3"]

CMD ["app.py"]


#2) create requirements.txt:

Flask==0.10.1


#3) download the feeds.json file from the website
to your local filesystem into docker-api/feeds/

 

Let’s move forward with the Python application, which is reading from the Docker image

the feeds.json file and exposing this JSON to the Flask web API. We won’t be stepping into

any actions like GET or PUT, just returning the complete JSON file listing all the feeds.

 

I prepared the Python app in the location docker-api/app/app.py and it is looking like this:

import os
from flask import Flask
from flask import Response

app = Flask(__name__)
@app.route('/api/v1/feeds')

def returner():
    os.chdir("..")
    path = os.path.abspath(os.curdir) + '/feeds/feeds.json'

    with open(path,"r") as f:
        data = f.read()
        resp = Response(response=data,
                        status=200,
                        mimetype="application/json")
        return(resp)

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

The next step is spinning up the docker container ( once we build the image of course )

cd docker-api/ 

docker build -t feeds . 

docker run -d -p 5000:5000 feeds 

docker container list 

*optionally docker container kill(or stop) container_id 
incase you need to "restart" the container
Btw. docker kill vs docker stop is an interesting topic and is discussed for example here
Let’s confirm that your Python project structure is looking like this:
docker

and voila, after running docker run -d -p 5000:5000 feeds , if you lookup the webpage

localhost:5000/api/v1/feeds

in your web browser, you should be getting the response with the desired JSON listing all the feeds.

You might want to check-out also curl.