Production grade dashboard

As an analyst or researcher, you ever wondered how you can present your findings/research to your clients, partners, manager in a clean and concise manner?

In this blog post I will go through the process of creating a production grade dashboard. This involves developing the dashboard, dockerizing it, using docker compose to attached a second container as a reverse proxy, and finally launching it on a cloud service provider.

For creating a dashboard, there are several providers, and many of them offer good functionality. PowerBI and Tableau are probably two of the dashboard providers that has the most publicity, and are both good choices as they are easy to work with and works out of the box.

However, in this blog I will focus on open source alternatives, and as I am quite familiar with Flask, going for Dash was very easy, as Dash use Flask under the hood.

If you have some experience in using R, Python or Julie, working with dash is very easy. If in addition you know a bit of CSS and HTML, you can more or less create what ever you like.

As Dash use Flask, and Flask utilize a development server, we need to use a production ready server. Gunicorn is very easy to work with and as you will see later, the setup is pretty straight forward. To serve static files, I have chosen Nginx, as it is the most popular and a relatively easy to setup reverse proxy.

The installation of the different components of this blog I assume you have already done.

Initialize project

As we want our dependecies to be consistent we use pipenv to setup a virtual environment

Create directory mkdir “YourDevelopmentDirectory”

make “YourDevelopmentDirectory” your current directory cd “YourDevelopmentDirectory”

To initilize the virtual environment pipenv shell

Project

As most people do not deploy Hello World apps, I wanted to show you how it works with a real app.

The data for this specific task is the volume of mortgages in Switzerland for the different kinds of banks. The data range is from 2002 to 2019, and it includs all the cantons in Switzerland. You can find the data here. Download the selection version of the file, as not all Cantons are represented before 2002

First let us import the different libraries needed.

import pandas as pd
import urllib.request

import dash
import dash_core_components as dcc
import dash_html_components as html
import plotly.graph_objects as go
import plotly.express as px
from jupyter_dash import JupyterDash
import dash_bootstrap_components as dbc

from dash.dependencies import Input, Output, State

import io
import os
import json
import geojson

#Read the CSV file (renamed to mor_loc.csv), no header and skip 3 first rows.
df = pd.read_csv("mor_loc.csv",header=None, skiprows=3)

#The data is read into only one column, so we split based on ";" and delete the qoutation marks. 
df = df[0].str.split(';', expand=True).replace('"', '', regex=True)

#We make the first row the name of the columns
df.columns = df.iloc[0]

#We drop the first row as this is now the column names
df = df.drop([0])

#Rename column INLANDAUSLAND to Cantons
df.rename(columns={'INLANDAUSLAND': 'Cantons'}, inplace=True)

#Replace the codeword for the different group of banks to the actual naming of the different banking groups.
df['BANKENGRUPPE'] = df['BANKENGRUPPE'].replace({'S10': 'AllBanks', 'G10': 'CantonalBanks', 'G15': 'BigBanks', 'G20': 'RegAndSavBanks', 'G25':'Raiffeisen'})
df

#We create a new dataframe (mortgage Cantons), where we group by the different banking groups
mor_Cant = df.groupby(["Date","Cantons", "BANKENGRUPPE"])['Value'].aggregate(lambda x: x).unstack().reset_index()

#Create a copy of the columns and drop the one which is not numeric
cols = mor_Cant.columns.drop('Cantons')

#Grab the columns which should be numeric and transform to numeric
mor_Cant[cols] = mor_Cant[cols].apply(pd.to_numeric, errors='coerce')

#Create a third dataframe only for "CH1" (All Switzerland)
mor_Nat = mor_Cant[mor_Cant["Cantons"] == 'CHE1']

#Drop the values in mor_Cant which are in mor_Nat dataframes
mor_Cant = mor_Cant[~mor_Cant.isin(mor_Nat)].dropna()

#Make all the float to int, as we do not really need decimals in tis case.
mor_Cant[cols] = mor_Cant[cols].astype(int)
mor_Cant

#Get the unique Cantons in the mor_Cant dataframe
available_indicators = mor_Cant['Cantons'].unique()

#Get the values for the different cantons in 2019
All_Bank2019 = mor_Cant.loc[mor_Cant["Date"] == 2019,:]
#Create an array of the values of 2019
All_Bank2019V = All_Bank2019.values
All_Bank2019V

array([[2019, 'AG', 89938, 14844, 25940, 19031, 24365],
       [2019, 'AI', 1992, 172, 1450, 318, 34],
       [2019, 'AR', 6520, 1290, 2317, 2200, 463],
       [2019, 'BE', 108226, 30760, 23373, 17087, 26340],
       [2019, 'BL', 39054, 9560, 18569, 4661, 1546],
       [2019, 'BS', 18809, 5247, 8208, 1454, 488],
       [2019, 'FR', 36291, 7567, 14794, 9548, 2069],
       [2019, 'GE', 53118, 26797, 11470, 4341, 649],
       [2019, 'GL', 4458, 490, 2453, 654, 540],
       [2019, 'GR', 34365, 8711, 16678, 5861, 516],
       [2019, 'JU', 7924, 1716, 2393, 2841, 682],
       [2019, 'LU', 49148, 7696, 24086, 10130, 5256],
       [2019, 'NE', 16579, 4520, 7614, 2128, 345],
       [2019, 'NW', 5492, 773, 3106, 1380, 82],
       [2019, 'OW', 4879, 508, 3151, 960, 185],
       [2019, 'SG', 59851, 7208, 22708, 19193, 6112],
       [2019, 'SH', 9278, 1461, 3936, 1072, 1929],
       [2019, 'SO', 33563, 6895, 5774, 9643, 8186],
       [2019, 'SZ', 24241, 4184, 13348, 3398, 1535],
       [2019, 'TG', 37159, 4563, 18155, 11588, 1001],
       [2019, 'TI', 50684, 14886, 11100, 13093, 443],
       [2019, 'UR', 4380, 628, 2314, 1265, 48],
       [2019, 'VD', 88451, 32625, 31015, 10969, 2974],
       [2019, 'VS', 40481, 12366, 11075, 13666, 455],
       [2019, 'ZG', 19606, 4855, 10204, 3358, 311],
       [2019, 'ZH', 197071, 60859, 91126, 15436, 13379]], dtype=object)

Visualization

To make the visualization more appealing I have included a map of Switzerland, where we visualize the different cantons and their contribution to the mortgage volume.

#Download link with geojson coordinates
swiss_url = 'https://raw.githubusercontent.com/empet/Datasets/master/swiss-cantons.geojson'

#Funtion to read the geojson file
def read_geojson(url):
    with urllib.request.urlopen(url) as url:
        jdata = json.loads(url.read().decode())
    return jdata 
jdata = read_geojson(swiss_url)

As for now we are working in jupyter notebook, we need to use JupyterDash.

We also import an external stylesheet which we can play with later on.

app = JupyterDash(__name__, external_stylesheets=[dbc.themes.LUMEN])

We set the title of first graph.

title = '2019 Cantonal mortgage volume'

Define the graph object and the figure we want to plot: * We use the Choropletmapbox as we want to display map of the cantons on the world map * Geojson; Get the longitude and latitude data * Locations: The cantons we just extracted from the mor_Cant dataframe * z; the mor_Cant dataframe for the year 2019 * customdata; array of the mor_Cant dataframe 2019 * Extract the data for the different banks to be on the map * Set title for the colorbar

swissmap = go.Figure(go.Choroplethmapbox(geojson=jdata, 
                                    locations=available_indicators, 
                                    z=All_Bank2019["AllBanks"],
                                    featureidkey='properties.id',
                                    coloraxis="coloraxis",
                                    customdata=All_Bank2019V,
                                    hovertemplate= 'Canton: %{customdata[1]}'+\
                                    '<br>All Banks: %{customdata[2]}CHF mill'+\
                                    '<br>Big Banks: %{customdata[3]}CHF mill'+\
                                    '<br>Cantonal Banks: %{customdata[4]}CHF mill'+\
                                    '<br>Raiffeisen Banks: %{customdata[5]}CHF mill'+\
                                    '<br>Reg & Sav Banks: %{customdata[6]}CHF mill<extra></extra>',
                                    colorbar_title="Millions CHF",
                                    marker_line_width=1))

We update the layout of the figure: * size of title * width and height of graph * what backgroundcolor and color of the different values in the map * mapbox: where the center of the map should be and what zoom we want to use.

swissmap.update_layout(title_text = title,
                  title_x=0.5,
                  #width=800,
                  #height=500,
                  #margin={"r":0,"t":30,"l":30,"b":30},
                  #paper_bgcolor="LightSteelBlue",
                  coloraxis_colorscale='Tealgrn',
                  mapbox=dict(style='carto-positron',
                              zoom=5.8, 
                              center = {"lat": 46.8181877 , "lon":8.2275124 },
                              ))

Let us create a barplot to stack the volume mortage of the different banks per canton (Excluding all banks as that is the total stack). * For this chart we use the plotly express function, which is more automated and therefore easier to define. * We create an automated slider defined by the different year * Set the max value of y axis to be the max of the stack bar * Set size of title and size of graph

bar_fig = px.bar(mor_Cant, x="Cantons", y=["BigBanks", "CantonalBanks", "Raiffeisen", "RegAndSavBanks"],
                 title="Mortgages per type of bank Switzerland",
                 animation_frame="Date",
                 labels={"value": "CHF mill", "variable": "Type of Bank"})
bar_fig.update_yaxes(range=[0, 183000])
bar_fig.update_layout(title_x=0.5, margin={"r":0,"t":30,"l":30,"b":30})

We can finally plot the graphs we have made so far, and create a dashboard, where we also add a line chart with an interactive callback function.

app.layout = html.Div(children=[
    #Set headline of the page.
    html.H1(children='Overview of the Swiss mortgage market', style = {'text-align':'center'}),
    
    #Set the dash core component graph, for the swissmap we created
    dcc.Graph(
        id='example-graph',
        figure=swissmap
    ),
    
    #Set the id for the line plot
    html.Div([
        dcc.Graph(id='line_graph')
    ]),
    
    #Set the details for dropdown of the line plot
    html.Div([

        html.Br(),
        html.Label(['Compare different bank categories in different canonts:'],style={'font-weight': 'bold', "text-align": "center"}),
        dcc.Dropdown(id='Cantons',
            options=[{'label':x, 'value':x} for x in df.Cantons.unique()],
            value='ZH',
            multi=False,
            disabled=False,
            clearable=True,
            searchable=True,
            placeholder='Choose Cuisine...',
            className='form-dropdown',
            style={'width':"90%"},
            persistence='string',
            persistence_type='memory'),

    ]),
    
    #Set the dash core component graph for the automated bar-graph we created
    dcc.Graph(
        id='bar-graph',
        figure=bar_fig
    )
    
])

#Set callback function. When the input of the dropdown we created above changes, 
#the output will aslo change accordingly. 
#In this case the input value is the canton and the values of that canton
@app.callback(
    Output('line_graph','figure'),
    [Input('Cantons','value')]
)

#The function defines the lineplot
def build_graph(DropDown):
    #Select the canton from the database which is defined in the dropdown
    dff=df[(df['Cantons']==DropDown)]

    linep = px.line(dff, x="Date", y="Value", color='BANKENGRUPPE', height=600)
    linep.update_layout(yaxis={'title':'Volume'},
                      title={'text':'Development of mortage volume',
                      'font':{'size':28},'x':0.5,'xanchor':'center'})
    return linep



app.run_server(mode="inline", port="8060")

Let us try to make the plot a bit more appealing, rearranging the plots, adding a nav and write a small explanation text.

Finishing Touches on the dashboard

#Add a font that is pleasant
fontawesome = 'https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css'

appBeautify = JupyterDash(__name__, assets_folder="static" ,external_stylesheets=[dbc.themes.LUMEN, fontawesome])

#Include a nav bar
navbar = dbc.Nav(className="nav nav-pills navbar-expand-lg navbar-light bg-ligth", children=[
    ## Add a logo to the navbar
    dbc.NavItem(html.Img(src=appBeautify.get_asset_url("mortgage.png"), height="40px")),
    ## create a button for readers to understand more about the app
    dbc.Col(width="auto"),
    dbc.NavItem(html.Div([
        dbc.NavLink("Info",href="/", id="info-popover", active=False),
        dbc.Popover(id="info", is_open=False, target="info-popover", children=[
            dbc.PopoverHeader("Try it out!"), 
            dbc.PopoverBody("Choose the different interactive tools and see how the mortgage market evolves")
        ])
    ])),
    ## Include link to the repository
    dbc.DropdownMenu(label="Code", nav=True, children=[
        dbc.DropdownMenuItem([html.I(className="fa fa-github"), "  Github"], 
                             href="https://www.google.com", target="_blank")
    ])
])



appBeautify.layout = html.Div(children=[
    #Set headline of the page.
    navbar,
    html.H1('Overview of the Swiss mortgage market',  id="nav-pills", 
            className="table-primary", style = {'text-align':'center', 'height':'50px'}),
    
    #Set the details for dropdown of the line plot
    html.Div([
        html.Br(),
        html.Label(['Compare different bank categories in different canonts:'],style={'font-weight': 'bold', "text-align": "center"}),
        dcc.Dropdown(id='Cantons',
            options=[{'label':x, 'value':x} for x in df.Cantons.unique()],
            value='ZH',
            multi=False,
            disabled=False,
            clearable=True,
            searchable=True,
            placeholder='Choose Cuisine...',
            className='form-dropdown',
            #style={'width':"90%"},
            persistence='string',
            persistence_type='memory'),

    ]),
    #Arrange the 
    dbc.Row([
        dbc.Col(
            children=[html.Br(),
                      html.P("On the right you can find the volume of mortgages in 2019. Under you can see the development over time. At the bottom you can interactively play with the chart and see how the mortages evolve as a total over time." 
                             ,className="table-dark",
                             style={'margin-left': '40px','color': 'white',
                                    'fontSize': 14,'margin-right': '40px'}),
                               html.Br(),dcc.Graph(id='line_graph'
                                                   ,style={'width': '100%', 'height':'80%'}),
    ]),dbc.Col(children=[html.Div([
            dcc.Graph(id='example-graph',
                      figure=swissmap,style={'width': '48%', 'align': 'right'})
        ])])]
    ),
    
    
    #Set the dash core component graph, for the swissmap we created
    
    
    #Set the id for the line plot
    
    
    
    
    #Set the dash core component graph for the automated bar-graph we created
    dcc.Graph(
        id='bar-graph',
        figure=bar_fig
    )
    
])

#Set callback function. When the input of the dropdown we created above changes, 
#the output will aslo change accordingly. 
#In this case the input value is the canton and the values of that canton
@appBeautify.callback(
    Output('line_graph','figure'),
    [Input('Cantons','value')]
)

def build_graph(DropDown):
    dff=df[(df['Cantons']==DropDown)]

    linep = px.line(dff, x="Date", y="Value", color='BANKENGRUPPE')
    linep.update_layout(yaxis={'title':'Volume'},
                      title={'text':'Development of mortage volume',
                      'font':{'size':17},'x':0.5,'xanchor':'center'})
    return linep

#Callback function for the header options
@appBeautify.callback(
    output=Output("info","is_open"), 
    inputs=[Input("info-popover","n_clicks")], 
    state=[State("info","is_open")])

def info_popover(n, is_open):
    if n:
        return not is_open
    return is_open

@appBeautify.callback(
    output=Output("info-popover","active"), 
    inputs=[Input("info-popover","n_clicks")], 
    state=[State("info-popover","active")])

def info_active(n, active):
    if n:
        return not active
    return active



appBeautify.run_server(mode="inline", port="8091")

The code tends to get very messy when working with the html tags of plotly, but our app is finally done. As this is an example app, I will not style it further, but feel free to work with the html tags and css to make it prettier. Also when you are developing a dashboard, I highly recommend to not work with the inline mode of dash, as it distorts the layout.

Deployment

First, let us wrap the app in a docker image. For this we have to do some minor tweaks to the code, and create a docker file. In this repository, you can find the setup as we do not deploy this jupyter file, but rather an app.py file, where practiacally the same code is found.

The adjustments we need to do:

• install flask and import it
• add: server = Flask(name)
• change: appBeautify = dash.Dash(name, assets_folder=“static” ,external_stylesheets=[dbc.themes.LUMEN, fontawesome])
  • to: appBeautify = dash.Dash(assets_folder=“static” ,external_stylesheets=[dbc.themes.LUMEN, fontawesome], server=server)
  • Here you can see that we have added the flask server to our dash app.
• change: appBeautify.run_server(debug=True, port=“8091”)
  • to: appBeautify.run_server(debug=True)
  • This is done since we will define in the dockerfile what port the container will run on, and since we are also running it with the wsgi gunicorn, which is a production ready http server.

The dockerfile

# Use slim python image with no fuzz
FROM python:3.8-slim-buster

ENV PYTHONUNBUFFERED 1

# Create a working directory.
RUN mkdir wd
WORKDIR wd

# Install Python dependencies.
COPY requirements.txt .
RUN pip3 install -r requirements.txt

# Copy the rest of the codebase into the image
COPY . ./

# Finally, run gunicorn.
CMD [ "gunicorn", "--workers=5", "--threads=1", "-b 0.0.0.0:9501", "app:server"]

To build the image, get the path of directory where the image is hosted and run: docker build “path/to/your/image”

When the build has finished, you want to run the image, but first you need to map the port of the image, in this case 9501 to a port on your computer(localhost).

docker run -d -p 9501:9501 nameOrIDofYourImage

You should now be able to access your container on http://127.0.0.1:9501/

Great, we are almost there. But we want to launch it with nginx as the reverse proxy that can handle static content. So we will need to do some changes to the docker file, create a nginx dockerfile and a docker-compose file to organize the communication between the two container instances.

#First we update the dockerfile to: 
# Use slim python image with no fuzz
FROM python:3.8-slim-buster

ENV PYTHONUNBUFFERED 1

# Create a working directory.
RUN mkdir wd
WORKDIR wd

# Install Python dependencies.
COPY requirements.txt .
RUN pip3 install -r requirements.txt

# Copy the rest of the codebase into the image
COPY . ./

We do this update since we will perform the python command in the docker-compose file.

Second we add get the nginx dockerfile file:

FROM nginx:1.13.3

RUN rm /etc/nginx/nginx.conf
COPY nginx.conf /etc/nginx/

RUN rm /etc/nginx/conf.d/default.conf
COPY project.conf /etc/nginx/conf.d/

COPY ./static ./static

I will not go in to detail about the nginx file, but have a look at the project.conf file, as it is from this file we set up the connection to the static files and decide what is going to be served.

As we want nginx to serve our static files, we add the static folder to the nginx folder.

Finally we have the docker-compose file

version: '3'

services:
  #The app container
  app:
    container_name: app
    restart: always
    build: ./plotly
    ports:
      - "8000:8000"
    command: gunicorn --workers=5 --threads=1 -b 0.0.0.0:8000 app:server

  #The nginx container
  nginx:
    container_name: nginx
    restart: always
    build: ./nginx
    ports:
      - "8085:80"
    depends_on:
      - app

We push these files and folders to our repository with git.

DigitalOcean

Create a digitalocean account, configure ssh and launch a droplet. The cheapest one will do for this project.

SSH in to the droplet and follow this link to setup a user. To check if your user works, open another terminal and try to ssh in again with: ssh youruser@ip-address

Then install docker, and docker-compose. Make sure to use Sudo. If not you will have problems in the intallation phase.

Finally install git.

From here on you need to download the git repository in the directory you may choose on the server: git clone https://github.com/vinwinter/Production-ready-dash

Make sure you download the https based link above and not the ssh based link(git), as you will need to set up ssh on your droplet.

An issue I encountered was that apache was running on port 80 on the server, so please use this command to stop it: sudo /etc/init.d/apache2 stop

Finally you can use the docker compose command: sudo docker-compose up –build -d

And voila, the server with your dashboard is up and running.

When you went to remove the containers, please use: sudo docker-compose rm -fs

Conclusion

We have finally managed to setup a dashboard, which can be accessed by everyone. However, an important thing to keep in mind is that if you add a domain your site will look more presentable, and also to add encryption for security. To attach a domain to your digitalocean account, you can have a look at this link. I have also added a link where the author is launching a node.js application with nginx as a reverse proxy and encryption through lets encrypt.

If you feel like you do not need nginx, feel free to move the static folder to the plotly folder and use the image under the dockerfile heading above. Then you only need to expose the container port to your local or remote host, as we did above.

Finally, there are easier ways to launch this dashboard, where you can use Heroku, which is brilliant for one of projects, but the benefit of containerizing is that it can easily be scaled, such as deploying it to Kubernetes. In fact, Docker-compose is an easy way to test if the containers are operating as they should in development, before you push it to Kubernetes.

• ← Previous Post
• Next Post →