Flask App For Data Processing & Visualization

Hey there! Ever found yourself drowning in data and wishing for a slick way to sift through it, visualize trends, and build interactive dashboards? Well, you're in luck! We're about to dive deep into creating a powerful Flask-based web application designed to do just that. This isn't your average data-crunching tool; it's a user-friendly portal that brings your GitHub data to life. We'll be working with four key JSONL files: pr_issue_single.jsonl, pr_detail.jsonl, issue_detail.jsonl, and merged_data.jsonl. Each plays a vital role in painting a comprehensive picture of pull requests and issues. Get ready to learn how to load, display, detail, and visualize this data, all wrapped up in a responsive, mobile-friendly interface.

Setting Up Your Project: The Foundation

Before we start coding, let's get our project structure and dependencies in order. A well-organized project is the first step to building a successful application. We'll need a requirements.txt file to list all the libraries our Flask app will depend on. This ensures that anyone can set up the project easily and run it without a hitch. Think of it as a recipe for your application's environment.

requirements.txt - The Essential Ingredient List

Here’s what you'll need in your requirements.txt file:

Flask
Flask-Bootstrap
Pandas
Jinja2
Plotly

Make sure to save this in the root directory of your project. To install these, simply navigate to your project's root directory in your terminal and run:

pip install -r requirements.txt

This command will download and install all the necessary packages. Now, let's talk about organizing our code. A clean structure makes development and maintenance a breeze. We'll adopt a standard Flask project layout:

• app.py: This will be our main Flask application file, housing all our routes and logic.
• templates/: This directory will contain all our HTML (Jinja2) templates for rendering web pages.
• static/: This directory will hold our static files like CSS, JavaScript, and images.
• data/: This is where we'll store our input JSONL files.

This organized approach ensures that everything is in its right place, making it easier to find and modify specific parts of the application as we build it out. It’s all about creating a solid foundation so we can focus on the exciting parts: the data and its presentation.

Data Loading: Bringing Your Information to Life

Our Flask application needs to access the data from the four JSONL files. For efficient access and manipulation, we'll load these files into memory when the application starts. Pandas DataFrames are excellent for this purpose, offering powerful data handling capabilities. We'll store these DataFrames in a way that allows for quick lookups, especially when we need to retrieve specific records or aggregate data for visualizations.

Loading the JSONL Files

In our app.py, the first thing we'll do is import the Flask library and other necessary modules. Then, we'll define a function to load our JSONL files. Each file will be loaded into a Pandas DataFrame, and we'll store these DataFrames in a dictionary. This makes it easy to access any dataset by its name.

import pandas as pd
from flask import Flask, render_template, request, jsonify

app = Flask(__name__)

data_store = {}

def load_data():
    try:
        data_store['pr_issue'] = pd.read_json('./data/pr_issue_single.jsonl', lines=True)
        data_store['pr_detail'] = pd.read_json('./data/pr_detail.jsonl', lines=True)
        data_store['issue_detail'] = pd.read_json('./data/issue_detail.jsonl', lines=True)
        data_store['merged_data'] = pd.read_json('./data/merged_data.jsonl', lines=True)
        print(