Chaitanya's Projects

Navigating the Rent-versus-Buy Decision with Deep Learning

Sat, 19 Apr 2025 00:00:00 +0000

This project presents a deep learning framework for forecasting monthly home prices and rental costs at the ZIP-code level across the United States. We integrate Redfin property sales data with American Community Survey rental figures to produce localized, forward-looking insights on the rent-versus-buy decision. If you’d like to explore the full code and dataset, you can find everything here.

Deciding whether to rent or buy a home is one of the most consequential financial choices individuals face, yet readily available tools often rely on broad averages, neglecting local variation and temporal trends. By forecasting both markets simultaneously at the ZIP-code scale, our approach empowers users with precise, neighborhood-level break-even estimates tailored to their financial assumptions.

Data Sources
Data Processing
Model Architecture
Visualization Dashboards
Evaluation Metrics and Results
Conclusion and Future Work
References

Data Sources

We draw upon two primary datasets:

Redfin Housing Data: Quarterly ZIP-code-level home sale prices and market metrics from 2013 to
American Community Survey (ACS) Rental Data: Annual median rent by ZIP code over the same period.

Data Processing

Alignment and Imputation: ZIP codes are aligned across both datasets, reshaped into full tensors via cartesian products, and missing values are filled by linear interpolation with endpoint extrapolation.
Zero-Aware Normalization: We mask zeros when computing means and standard deviations to ensure stable gradient flow and robustness to sparse entries.

For housing data $X_{\mathrm{housing}}\in\mathbb{R}^{B\times T\times F\times P}$, define an indicator $1_{x\neq0}$. For each feature $f$:

$$ \mu_f = \frac{1}{N_f + \epsilon}\sum_{b=1}^B\sum_{t=1}^T\sum_{p=1}^P 1_{x_{b,t,f,p}\neq0}\,x_{b,t,f,p},\qquad N_f = \sum_{b,t,p}1_{x_{b,t,f,p}\neq0}+\epsilon $$

\[\sigma_f = \sqrt{\frac{1}{N_f + \epsilon}\sum_{b,t,p}1_{x_{b,t,f,p}\neq0}\,x_{b,t,f,p}^2 - \mu_f^2}\]

$$ \mathrm{ZA\_Norm}(x_{f}) = \frac{1_{x_f\neq0}\,x_f - \mu_f}{\sigma_f + \epsilon},\quad x = \mathrm{ZA\_Norm}(x_{\mathrm{norm}})\,(\sigma_f + \epsilon) + \mu_f $$

Model Architecture

Our Transformer-based model jointly forecasts housing and rental prices 12 months ahead from a 36-month lookback. It consists of:

ZIP Code Embeddings: Compress a one-hot vector of length 24,488 into a dense embedding $Z_e\in\mathbb{R}^F$:

\[\mathbb{R}^{24{,}488}\;\xrightarrow{\mathrm{Embedding}}\;\mathbb{R}^F\]

MLP Projection Layers: Transform normalized housing tensors:

$$ X'_{\mathrm{housing}} = \mathrm{ReLU}\bigl(W_2\,\mathrm{ReLU}(W_1\,X_{\mathrm{housing}}+b_1)+b_2\bigr) $$

Transformer Encoder–Decoder:
- Encoder input $X_{\mathrm{enc}}\in\mathbb{R}^{B\times T\times d_{\mathrm{embed}}}$
- Decoder input $X_{\mathrm{dec}}\in\mathbb{R}^{B\times T\times(P+1)}$
We follow “Attention Is All You Need” with two modifications: we project encoder outputs down before feeding the decoder, and we initialize the decoder with the last observed ZA-normalized values.

Visualization Dashboards

We deployed two complementary interfaces:

Flask Dashboard: Interactive map and ZIP selector for side-by-side forecast charts, allowing users to input mortgage rate and down payment. This application can be run locally by following the steps in our Colab Notebook.
Tableau Dashboard: Choropleth map illustrating rent vs. buy cost-effectiveness, with adjustable parameters to explore regional patterns. You can view the full Tableau visualization here

These tools translate complex model outputs into intuitive visuals, enabling non-technical users to compare localized forecasts easily.

Evaluation Metrics and Results

We trained with Huber loss and monitored four metrics defined as:

$$ \mathrm{RMSE} = \frac{1}{n}\sum_{i=1}^n(y_i - \hat y_i)^2,\quad \mathrm{MAE} = \frac{1}{n}\sum_{i=1}^n|y_i - \hat y_i|, $$

$$ R^2 = 1 - \frac{\sum_i(y_i - \hat y_i)^2}{\sum_i(y_i - \bar y)^2},\quad \mathrm{MAPE} = \frac{100}{n}\sum_{i=1}^n\left|\frac{y_i - \hat y_i}{y_i}\right| $$

Under our final hyperparameters (36-month window, embedding = 32, feed-forward = 128, dropout = 0.3), we achieved:

Housing Forecasts: RMSE = $155,560.5, MAE = $84,031.0, MAPE = 42.4 %, R² = 0.73
Rental Forecasts: RMSE = $97.1, MAE = $57.7, MAPE = 5.98 %, R² = 0.95

Conclusion and Future Work

By integrating spatio-temporal data with a zero-aware Transformer framework, we deliver precise, neighborhood-level rent-versus-buy forecasts. Limitations include sparsely populated ZIP codes and sensitivity to market shocks. Future enhancements could incorporate macroeconomic indicators, refine spatial granularity to census tracts, and add confidence intervals for uncertainty quantification.

Despite these challenges, our system marks a significant step toward democratizing housing market forecasts, empowering individuals and investors with localized, forward-looking evidence.

References

Fan et al., “Determinants of House Price: A Decision Tree Approach,” Urban Studies (2006).
Vaswani et al., “Attention Is All You Need,” arXiv (2017).
Redfin Data Center: https://www.redfin.com/news/data-center
ACS Median Gross Rent: https://data.census.gov/table/ACSDT5Y2022.B25064

Advancing sEMG Hand Signal Classification

Sun, 08 Dec 2024 00:00:00 +0000

Surface Electromyography (sEMG) captures the tiny electrical signals generated by our muscles when they contract. These signals are at the heart of many prosthetic control systems and human–computer interfaces. In this report, we explore two main strategies:

Windowed FFT-based feature extraction: breaking the signal into fixed-length segments and analyzing frequency components
Non-windowed modeling: feeding raw signal samples directly into machine learning models

Our goal is to understand how each approach balances accuracy and processing speed when classifying hand gestures.

Accurate and fast gesture recognition is crucial for applications like real-time prosthetic devices, where even small delays or misclassifications can impact the user experience. By comparing a traditional signal-processing pipeline against newer, end-to-end learning methods, we aim to find the sweet spot between performance and complexity.

Problem Statement
Data Source
Methodology
Evaluation Technique
Results
Conclusion & Future Work
References

Problem Statement

Surface Electromyography measures the electrical activity produced by muscle contractions. Windowing segments signals into fixed intervals for robust feature extraction but introduces computational overhead and fixed temporal boundaries. Non-windowed methods model each timestep directly, potentially reducing latency. This study evaluates both approaches to determine their impact on classification accuracy and inference time.

Data Source

The dataset utilized in this research is the “EMG Data for Gestures,” available through the UCI Machine Learning Repository. It consists of raw electromyographic (EMG) signals recorded from 36 subjects using a MYO Thalmic bracelet worn on the forearm. The bracelet’s eight evenly distributed sensors capture myographic signals transmitted via Bluetooth. Each subject performed two series of six or seven static hand gestures, each held for 3 seconds followed by a 3-second rest period.

The data files contain ten columns: time (in milliseconds), eight sEMG channels, and a class label indicating the gesture performed. Class labels are defined as:

0 — unmarked data
1 — hand at rest
2 — hand clenched in a fist
3 — wrist flexion
4 — wrist extension
5 — radial deviation
6 — ulnar deviation
7 — extended palm (performed by a subset of subjects)

For simplicity, data from all subjects were concatenated across recordings, focusing on a generalized classification problem rather than individual differences.

Methodology

Below is a detailed methodology description that includes the precise mathematical formulations from the original report. We compare a windowing-based signal-processing pipeline against a non-windowing strategy that models raw samples directly.

1. Experimental Setup

All models are trained and evaluated on a MacBook M3 Pro using Metal Performance Shaders (MPS) for GPU acceleration. Data are standardized and random seeds fixed to ensure reproducibility.

2. Windowing Approach

a. Segmentation into Windows

Let the raw dataset be ${t_i, x_i, y_i}_{i=1}^n$, where

$t_i$ is the timestamp (ms),
$x_i\in\mathbb{R}^M$ is the $M$-channel sEMG vector, and
$y_i$ is the gesture label.

Applying a sliding window of length $W$ yields:

\[X^{(j)} = [\,x_i, x_{i+1}, \dots, x_{i+W-1}\], \quad Y^{(j)} = [\,y_i, y_{i+1}, \dots, y_{i+W-1}\],\]

for $j=1,\dots,N$, where $N$ is the number of windows.

[Image Source](https://www.danorlandoblog.com/use-the-sliding-window-pattern-to-solve-problems-in-javascript/)

b. Frequency-Domain Transformation (FFT)

For each channel $m=1,\dots,M$, perform a Discrete Fourier Transform on each windowed signal $x_m^{(j)}(n)$:

$$ X_m^{(j)}(k) = \sum_{n=0}^{W-1} x_m(n)\,e^{-2\pi i\,k\,n / W}, \quad k=0,\dots,\lfloor W/2\rfloor. $$

Record magnitude and phase:

$$ |X_m^{(j)}(k)| = \sqrt{\Re(X_m^{(j)}(k))^2 + \Im(X_m^{(j)}(k))^2}, \quad \angle X_m^{(j)}(k) = \arctan\!\frac{\Im(X_m^{(j)}(k))}{\Re(X_m^{(j)}(k))}. $$

Stacking across $M$ channels yields a feature vector $H^{(j)}\in\mathbb{R}^{2M(K+1)}$, where $K=\lfloor W/2\rfloor$.

[Image Source](https://www.researchgate.net/figure/Three-Components-of-a-Complex-Number-In-Phase-Quadrature-and-Phase-Incoming-radar-wave_fig2_332511933)

c. CNN for Frequency-Domain Feature Extraction

Treating frequency bins as a “spatial” axis and channels as input planes, a 1D convolution with filters $W^{(\ell)}\in\mathbb{R}^{F\times C_{\mathrm{in}}\times C_{\mathrm{out}}}$ and bias $b^{(\ell)}$ computes

$$ h^{(\ell)}(k,d) = f\Bigl( \sum_{c=1}^{C_{\mathrm{in}}} \sum_{\delta=0}^{F-1} H(k+\delta,c)\,W^{(\ell)}_{d,\delta,c} + b^{(\ell)}_d \Bigr), $$

where $f$ is LeakyReLU:
$f(x)=\max(0,x)+\alpha\,\min(0,x).$

Each block also includes BatchNorm, MaxPool and Dropout.

d. Deep Cross Network (DCN)

Given embedding $x^{(0)}$, each cross layer $\ell$ updates

$$ x^{(\ell+1)} = x^{(\ell)} \,\circ\,(x^{(\ell)}\,W^{(\ell)}) \;+\; b^{(\ell)} \;+\; x^{(\ell)}, \quad \ell=0,\dots,L-1, $$

where $\circ$ is the Hadamard (elementwise) product.

[Image Source](https://arxiv.org/pdf/2008.13535)

e. Multi-Layer Perceptron (MLP)

The final DCN output $x^{(L)}$ is passed through fully connected layers:

$$ z^{(1)} = f\bigl(W^{(1)}\,x^{(L)} + b^{(1)}\bigr), \quad z^{(K)} = W^{(K)}\,z^{(K-1)} + b^{(K)}, \quad \hat y = \mathrm{softmax}\bigl(z^{(K)}\bigr). $$

f. Model Architecture Overview

The complete architecture for the windowing approach is as follows:

3. Non-Windowing Approach

a. Raw Feature Input

Use each instantaneous vector $x\in\mathbb{R}^M$ without any segmentation.

b. Random Forest Classifier

As a baseline, average class probabilities from $T$ trees:

\[P(y=c\mid x) = \frac{1}{T} \sum_{t=1}^T P_t(y=c\mid x).\]

[Image Source](https://www.researchgate.net/figure/Architecture-of-the-Random-Forest-algorithm_fig1_337407116)

c. Outer-Product Neural Network (OPNN)

Compute the outer product $O = x\,x^\top\in\mathbb{R}^{M\times M}$, flatten to $z\in\mathbb{R}^{M^2}$, then

\[\hat y = \mathrm{softmax}\bigl(W_2\,f(W_1\,z + b_1) + b_2\bigr).\]

4. Loss & Optimization

All deep models use cross-entropy loss:

\[\mathcal{L}(\theta) = -\sum_{c} y_c \,\log \hat y_c,\]

optimized with Adam.

[Image Source](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html)

Evaluation Technique

Accuracy and inference time (ms/sample) are measured on an M3 Pro with Metal Performance Shaders. Accuracy is defined as:

\[\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}.\]

Results

| Approach | Accuracy (%) | Inference Time (ms/sample) | Parameters | |------------------------------|--------------|----------------------------|------------| | Windowing (CNN+DCN+MLP) | 97.86 | 0.0828 | 46.8 M | | Non-Windowing (RandomForest) | 98.01 | 0.0313 | — | | Non-Windowing (OPNN+MLP) | 98.43 | 0.0081 | 11.0 M |

Conclusion & Future Work

Non-windowed deep learning offers higher accuracy and lower latency, suggesting that time-frequency segmentation may not be necessary for static gesture recognition. Future work should explore dynamic gestures and cross-user generalization.

References

Olmo & Domingo (2020). EMG Characterization. Materials, 13(24), 5815.
Raez et al. (2006). EMG Signal Analysis. Biological Procedures Online, 8, 11–35.
Rani et al. (2023). sEMG & AI. IEEE Access, 11, 105140–105169.
Asogbon et al. (2018). Window Conditioning in EMG. IEEE CBS.
Krilova et al. (2018). EMG Data for Gestures. UCI Repository.

Clustering Subreddit Communities

Sat, 31 Aug 2024 00:00:00 +0000

This project models Reddit user interactions to understand behavior and patterns. This project utilizes Spectral Clustering from the Graph Laplacian (using normalized cuts) to uncover local communities where users have more concentrated interactions. We discuss the importance of understanding user behavior, the results of this project, and areas of improvement for further research. If you’re interested, then let’s dive right in!

Reddit is a vast platform with countless communities, and understanding how these communities connect can provide valuable insights. For instance, knowing which Subreddits are often visited by the same users can help in creating better content recommendations, improving user engagement, or even in moderating content more effectively.

Here are the main sections of this article:

Loading the Data
Constructing the Graph Laplacian
K-Means Algorithm
Results and Observations
References

Loading the Data

The dataset was found on Kaggle and is called Subreddit Interactions for 25,000 Users. The dataset contains three columns: “user”, “subreddit”, and “utc-stamp”. For our analysis, we are only interested in the first two columns.

Before we start, we need to set up some libraries for our analysis and load in our data (Note: throughout this article, I will not be showing the subreddit names too often as far too many of them are NSFW!).

import numpy as np
import pandas as pd
import networkx as nx
from wordcloud import WordCloud
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.cluster import KMeans
from tqdm.notebook import tqdm
import os

interactions = pd.read_csv('reddit_data.csv').iloc[:, :-1]  # remove utc times
interactions.tail()

Preprocessing the Data

Now that we have our data loaded in, we need to somehow standardize user interactions so that one user does not have greater influence than another user. The approach outlined in the code below groups the interactions by username and finds the proportion of visits made by a user on specific subreddits. Here’s the mathematical formulation of this problem:

Given a dataset of interactions where each interaction is defined by a ‘username’ and a ‘subreddit’, we want to compute the probability of a user interacting with a specific subreddit.

\[p(u, s) = \frac{\sum_{i=1}^{n} r(u_i = u, s_i = s)}{\sum_{i=1}^{n} r(u_i = u)}\]

Here, $n$ is the total number of interactions, $u_i$ is the $i$-th username, $s_i$ is the $i$-th subreddit in the dataset, and $r$ is the indicator function. Here is how we can implement this in Python efficiently:

subreddit_counts = interactions.groupby(['username', 'subreddit']).size().reset_index(name='subreddit_count')
total_counts = interactions.groupby('username').size().reset_index(name='total_count')
merged_df = pd.merge(subreddit_counts, total_counts, on='username')
merged_df['probabilities'] = merged_df['subreddit_count'] / merged_df['total_count']
interactions = pd.merge(interactions, merged_df[['username', 'subreddit', 'probabilities']], on=['username', 'subreddit'], how='left').values

del subreddit_counts, total_counts, merged_df

interactions = np.vstack(tuple(set(map(tuple,interactions))))  # removing duplicate interactions
display(interactions[-5:])

Now we have converted interactions into a numpy array and added a third columns with the interaction proportions by user. The data now only contains unique rows because we do not need to model redundant edges in our matrix. With this, we can create some maps that will help construct our adjacency matrix and reduce the computational complexity for algorithms later in this project.

nodes = np.unique(interactions[:, 1])
nodes_map = {node: i for i, node in enumerate(nodes)}
reverse_map = {i: node for i, node in enumerate(nodes)}
print("Number of Subreddit pages:", nodes.shape[0])

Constructing the Graph Laplacian

With our data set up and ready to go, we can begin constructing a weighted Adjacency Matrix. In our case, let $A$ be an adjacency matrix where $A_{i, j}$ represents the weighted connection between node $i$ and node $j$. Nodes are mapped from pages using a mapping function $\text{nodes_map}$ such that $i = \text{nodes_map}(\text{page1})$ and $j = \text{nodes_map}(\text{page2})$. If you wish to skip all the math, you can click here.

For each user $u$, let $P_u$ be the set of pages visited by $u$, and let $p_1$ and $p_2$ be any two pages in $P_u$. Define the class probabilities associated with $p_1$ and $p_2$ as $c(p_1)$ and $c(p_2)$, respectively.

The Adjacency Matrix $A$ is defined as follows [2]: $A_{i, j} = \sum_{u \in U} \sum_{\substack{p_1, p_2 \in P_u \ p_1 \neq p_2}} c(p_1) \cdot c(p_2)$

Where:

$U$ is the set of all unique users.
$P_u$ is the set of pages visited by user $u$.
$c(p_1)$ and $c(p_2)$ are the class probabilities for pages $p_1$ and $p_2$, respectively.
$i = \text{nodes_map}(p_1)$ and $j = \text{nodes_map}(p_2)$.
$A_{i, j}$ is incremented by the product of the class probabilities $c(p_1) \cdot c(p_2)$.

With $A$ defined, we must then remove any disconnected edges.

Define a mask vector $\mathbf{m}$ of length $n$ such that:

\[\mathbf{m}_i = \begin{cases} 1 & \text{if } \sum_{j=1}^{n} A_{i, j} \neq 0 \ 0 & \text{if } \sum_{j=1}^{n} A_{i, j} = 0 \end{cases}\]

Here, $\mathbf{m}_i$ is $1$ if node $i$ has at least one connection, and $0$ if it has no connections.

The reduced adjacency matrix $A{\prime}$ is obtained by selecting only the rows and columns where $\mathbf{m}[i] = 1$:

\[A’ = A[\mathbf{m} = 1, \mathbf{m} = 1]\]

The Degree Matrix $D$ is defined as follows [2]:

Given a reduced adjacency matrix $A{\prime}$ (obtained from the original adjacency matrix $A$ after removing isolated nodes), the normalized diagonal degree matrix $D$ is defined as:

\[D = \text{diag}\left(\frac{1}{\sqrt{\sum_{j=1}^{m} A’_{i,j}}}\right)\]

The Graph Laplacian Matrix $L$ is defined as follows [1]:

Via the normalized cuts method [1], the Graph Laplacian can be defined as:

\[L = D \times A \times D\]

To make the matrix symmetrical we can simply do:

\[L = L + L^T\]

Computing top $k$ eigenvectors:

Given the eigenvalue decomposition for $L$:

\[Lx_i = \lambda_i x_i\]

Where:

$L$ is symmetric.
$\lambda_i$ are the eigenvalues, ordered such that $ \lambda_1 \leq \lambda_2 \leq \dots \leq \lambda_n $.
$x_i$ are the corresponding eigenvectors.

The top $k$ eigenvectors then become:

\[X_k = [x_{(1)}, x_{(2)}, \dots, x_{(k)}]\]

Implementing the Algorithm

With all the math aside, we can now create and run our algorithm!

Click here to view the code

A = np.zeros(shape=(nodes.shape[0], nodes.shape[0]), dtype=np.float32)

for u in tqdm(np.unique(interactions[:, 0])):
    users = interactions[interactions[:, 0] == u, 1:]
    
    for ind, page1 in enumerate(users[:-1]):
        i = nodes_map[page1[0]]
        for page2 in users[ind:]:
            j = nodes_map[page2[0]]
            if page1[0] != page2[0]:
                A[i, j] += page1[1].astype(np.float32) * page2[1].astype(np.float32)

A = A + A.T

mask = np.sum(A, axis=1) != 0
A = A[mask][:, mask]

D = np.diag(1/np.sqrt(np.sum(A, axis=1)))

L = D @ A @ D  # Graph Laplacian via Normalized Cuts method [1]
v, x = np.linalg.eigh(L)

print(f"A.shape: {A.shape}")
print(f"D.shape: {D.shape}")
print(f"L.shape: {L.shape}")
print(f"x.shape: {x.shape}")

Since the computation of the eigenvalue decomposition is extremely time-consuming, we can save the eigenvalue decomposition matrix with the code below.

eigs = os.path.join(os.getcwd(), f'GL_Eigenvectors_Weighted')
os.makedirs(eigs, exist_ok=True)
np.save(os.path.join(eigs, 'eigenvectors_x.npy'), x)

K-Means Algorithm

With our eigenvectors matrix computed, we can apply the k-means algorithm on this new dataset. Because of the nature of the Graph Laplacian, the data will naturally separate at weak edges, which creates local communities in the data. If you wish to skip all the math, you can click here.

The k-means algorithm aims to solve [2]:

Given $m$ data points $x^i \in \mathbb{R}^n, i = 1, . . . , m$, K-means clustering algorithm groups them into $k$ clusters by minimizing the function over ${r^{ij},\mu^j}.$

\[J = \sum_{i=1}^m \sum_{j=1}^k r^{ij} \lVert x^i − \mu^j \lVert^2,\]

where $r^{ij} = 1$ if $x^i$ belongs to the $j$-th cluster and $r^{ij} = 0$ otherwise.

Once the minimization of the function over ${r^{ij},\mu^j}.$ is solved,

$\mu^j = \frac{\sum_{i=1}^m r^{ij} x^i}{\sum_{i=1}^m r^{ij}}$ $r^{ij} = \begin{cases} 1 & if \>\> j = \underset{j=1,...,k}{\operatorname{arg min}} \lVert x^i − c^j \lVert^2 \\ 0 & if \>\> j \neq \underset{j=1,...,k}{\operatorname{arg min}} \lVert x^i − c^j \lVert^2 \end{cases}$

Implementing K-Means for Spectral Clustering

Let’s now implement the algorithm to find $k$ communities.

Click here to view the code

k = 1000

X = x[:, -k:]
labels = KMeans(n_clusters=k, n_init='auto').fit(X.real).labels_  # kmeans centers

labs = np.unique(labels)

clusters = []
lengths = []
for c in tqdm(labs):
    inds = np.where(labels == c)[0]
    cluster = {}
    
    for i in inds:
        if reverse_map[i] in cluster:
            cluster[reverse_map[i]] += 1
        else:
            cluster[reverse_map[i]] = 1
            
    clusters.append(cluster)
    lengths.append(len(cluster))
    
groups = pd.DataFrame({"Cluster": labs, "Subreddits": clusters, "Cluster Size": lengths}).sort_values(by="Cluster Size", ascending=False)
del clusters, cluster

results = pd.DataFrame({"Cluster": labels, "Subreddit": [reverse_map[i] for i in range(len(labels))]}).sort_values(by="Subreddit", ascending=False)

display(groups.head(), results.head())

Results and Observations

To understand how the spectral clustering algorithm performed, we can visualize two of the clusters to understand their relatedness to eachother. I had to run the algorithm several times because there were way too many NSFW Subreddits showing up!

Click here to view the code

top_percent_index = int(len(groups) * 0.02)

random_indices = np.random.choice(np.arange(1, top_percent_index), size=2, replace=False)

for i in random_indices:
    subreddit_dict = groups.iloc[i]['Subreddits']
    cluster_nodes = list(subreddit_dict.keys())
    
    indices = [nodes_map[node] for node in cluster_nodes if node in nodes_map]
    subgraph = nx.from_numpy_array(A[np.ix_(indices, indices)])
    subgraph = nx.relabel_nodes(subgraph, {i: reverse_map[i] for i in indices})
    edge_weights = np.array([A[u, v] for u, v in subgraph.edges()])
    edge_weights = (edge_weights - edge_weights.min()) / (edge_weights.max() - edge_weights.min()) * 5
    
    wordcloud = WordCloud(width=800, height=400, background_color='black', colormap='OrRd').generate_from_frequencies(subreddit_dict)
    
    plt.figure(figsize=(10, 5))
    plt.imshow(wordcloud, interpolation='bilinear')
    plt.axis('off')
    plt.title(f"Cluster {groups.iloc[i]['Cluster']} (Size: {groups.iloc[i]['Cluster Size']})")
    plt.show()
    
    plt.figure(figsize=(12, 8))
    pos = nx.spring_layout(subgraph, seed=42)  # Positions for all nodes
    nx.draw(subgraph, pos, with_labels=False, node_color='red', edge_color='gray', width=np.log10(edge_weights+0.8), node_size=50, font_size=10)
    plt.title(f"Subcommunity Network for Cluster {groups.iloc[i]['Cluster']}")
    plt.show()
    
    print('\n\n')

While the Spectral Clustering approach has revealed clear community trends among subreddits, the presence of outlier subreddits suggests that there may be room for optimization, particularly in choosing the number of clusters ($k$). This could be due to either the complexity of user behavior or a suboptimal value of $k$. Additionally, the choice of using commenting activity as a measure of subreddit engagement, while useful, might not be the best indicator of true user interest. Future projects could explore other data features, like subreddit joins or upvotes, which might provide a more accurate representation of user behavior.

The computational expense of Spectral Clustering also raises the question of whether simpler algorithms could achieve similar results. While Spectral Clustering excels at capturing complex relationships, it might be worth exploring more straightforward models that could offer efficiency gains, especially for large-scale applications. Despite these considerations, the model’s ability to uncover nuanced community structures without relying on complex neural networks is a significant advantage, making it a promising tool for building unsupervised recommendation systems in the future.

References

[1] Jianbo Shi and J. Malik, “Normalized cuts and image segmentation,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000, doi: 10.1109/34.868688.

[2] Xie, Yao, “Spectral Clustering.” Class lecture, Computational Data Analytics, Georgia Institute of Technology, Atlanta, GA. August 21, 2024.

Classifying Emotions from Videos

Sun, 21 Jul 2024 00:00:00 +0000

Sentiment Analysis is widely used across industries to track how customers react to various stimuli, like in social media algorithms. With the rise of advanced machine learning algorithms, we can analyze many different types of data to make informed decisions on business strategy. One such recent breakthrough is the use of video and audio data to classify human emotions, and this project covers a detailed approach to doing so without any complex neural networks. No matter how experienced you are, I’m sure you will find this project very interesting!

This project addresses the complex challenge of emotion classification using audio and video data, leveraging the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). By developing a robust model to accurately classify emotions based on speech, this project has significant implications across multiple fields. In mental health, it can aid in early diagnosis and treatment of emotional disorders by providing therapists with deeper insights into patients’ emotional states. In customer service, automated emotion recognition can enhance user experience by enabling more empathetic and responsive interactions. Moreover, in human-computer interaction, it can lead to the development of more intuitive and emotionally aware systems, enhancing user engagement and satisfaction.

Here are the contents of this project. If you are here just for a casual read, then you can go ahead and skip over to the second section.

Loading the Data
Video Feature Extraction
Audio Feature Extraction
Emotion Classification
Conclusion
References

Loading the Data

To start this project, we download data from here [7]. For this project, we will only be focusing on the speech dataset. Here are the libraries you will need to start:

import requests
import os
from tqdm.notebook import tqdm
import zipfile
import cv2
import numpy as np
import gc
from io import BytesIO
import tempfile
import matplotlib.pyplot as plt
%matplotlib inline

Downloading the Data

If you choose to get an API key from their website, you can use the code below to download the dataset to your computer (you can download it manually as well). But, the dataset is large, so unless your computer is powerful, I would not recommend you do this locally.

Click here to view the code

record_id = '1188976'
response = requests.get(f'https://zenodo.org/api/records/{record_id}', params={'access_token': YOUR_API_KEY})
data = response.json()

all_files = data['files']
speech_files = []
song_files = []
for d in all_files:
    components = d['key'].split('_')
    if components[1] == 'Speech':
        speech_files.append(d)
    else:
        song_files.append(d)
os.makedirs('Speech', exist_ok=True)

for file in tqdm(speech_files):
    file_url = file['links']['self']
    file_name = os.path.join('Speech', file['key'])

    if not os.path.exists(file_name):
        response = requests.get(file_url, params={'access_token': YOUR_API_KEY})
        with open(file_name, 'wb') as f:
            f.write(response.content)
        print(f"Downloaded: {file_name}")

print("Download complete.")

Once you download the data, you should have 1440 videos in a folder called ‘Speech’. Here are some example frames from the videos:

Processing Each Video

Once we have the dataset downloaded and ready to go, we need to reduce the size of each frame and standardize the number of frames in each video to a desired count. This keeps our data compact and reduces redundancy. For example, we can standardize the frame count to 50 so that every video only has 50 frames. This makes our analysis easier because we can combine all the videos into training and testing tensors.

Click here to view the code

def resize_frame(frame, scale_factor):
    """Resizes a frame by a given scale factor."""

    height, width = frame.shape[:2]
    new_dimensions = (int(width * scale_factor), int(height * scale_factor))
    resized_frame = cv2.resize(frame, new_dimensions, interpolation=cv2.INTER_AREA)
    return resized_frame

def interpolate_frames(frames, target_frame_count):
    """Interpolates frames to a given frame count."""

    original_frame_count = len(frames)
    if original_frame_count == target_frame_count:
        return frames

    indices = np.linspace(0, original_frame_count - 1, target_frame_count)
    interpolated_frames = []
    for i in indices:
        lower_index = int(np.floor(i))
        upper_index = int(np.ceil(i))
        if lower_index == upper_index:
            interpolated_frames.append(frames[lower_index])
        else:
            lower_weight = upper_index - i
            upper_weight = i - lower_index
            interpolated_frame = (lower_weight * frames[lower_index] + upper_weight * frames[upper_index]).astype(np.uint8)
            interpolated_frames.append(interpolated_frame)

    return interpolated_frames

Processing Each Zip File

With the two functions defined earlier, we can now process each video, but we still need to create our training and testing tensors. To do this, we need every video for all actors in the speech dataset. Each actor contains about 60 videos that are relevant to this project. We randomly split the videos into training and testing (40 training and 20 testing) for each actor and merged the videos (and labels) into combined tensors. Once we have tensors for each zip file, we combine them into our full training and testing datasets. Keep in mind, that the training and testing tensors so far are only for the video data, so we will need other code for processing the audio data.

Click here to view the code

def process_zip_file(zip_path, target_frame_count, scale_factor, train_size):
    """Processes a zip file and returns training and testing tensors."""

    tensors_shape = (int(720*scale_factor), int(1280*scale_factor), 3, target_frame_count, 60)
    video_tensors = np.zeros(tensors_shape, dtype=np.uint8)
    identifier = np.zeros((60, 7), dtype=np.uint8)

    i = 0
    with zipfile.ZipFile(zip_path, 'r') as zip_ref:
        for file_info in tqdm(zip_ref.infolist()):
            file_name = file_info.filename.split('/')[1]
            if (file_name.startswith('02')) and (file_name.endswith('.mp4')):
                try:
                    with zip_ref.open(file_info) as video_file:
                        video_bytes = BytesIO(video_file.read())
                        video_tensor = video_to_array(video_bytes, target_frame_count, scale_factor)

                        if video_tensor is not None:
                            id = np.array(file_name[:-4].split('-'), dtype=np.uint8)
                            identifier[i, :] = id
                            video_tensors[:, :, :, :, i] = video_tensor

                            del file_name
                            del video_bytes
                            del video_tensor
                            del id
                            gc.collect()

                except Exception as e:
                    print(f"Failed to process {file_info.filename}: {e}")
                i += 1

    permutation = np.random.permutation(i)
    training_indices = permutation[:train_size]
    all_indices = np.arange(i)
    testing_indices = np.array([j for j in all_indices if j not in training_indices])

    training_tensor = video_tensors[:, :, :, :, training_indices]
    testing_tensor = video_tensors[:, :, :, :, testing_indices]
    del video_tensors
    gc.collect()

    training_identifier = identifier[training_indices, :]
    testing_identifier = identifier[testing_indices, :]
    del identifier
    gc.collect()

    return training_tensor, training_identifier, testing_tensor, testing_identifier

folder_path = 'Speech'
target_frame_count = 50
scale_factor = 1/3

train_size = 40  # out of 60 videos (per actor)
test_size = 60 - train_size

train_shape = (int(720*scale_factor), int(1280*scale_factor), 3, target_frame_count, 24*train_size)
test_shape = (int(720*scale_factor), int(1280*scale_factor), 3, target_frame_count, 24*test_size)

train_videos = np.zeros(train_shape, dtype=np.uint8)
test_videos = np.zeros(test_shape, dtype=np.uint8)

id_train = np.zeros((24*train_size, 7), dtype=np.uint8)
id_test = np.zeros((24*test_size, 7), dtype=np.uint8)

i = 0
for zip_filename in tqdm(os.listdir(folder_path), desc='Total Progress'):
    if zip_filename.endswith(".zip") and zip_filename.startswith("Video"):
        zip_path = os.path.join(folder_path, zip_filename)
        print(f"Processing {zip_filename}...")
        training_tensor, training_identifier, testing_tensor, testing_identifier = \
            process_zip_file(zip_path, target_frame_count, scale_factor, train_size)

        print('Training Tensor Shape:', training_tensor.shape)
        print('Testing Tensor Shape:', testing_tensor.shape)

        train_videos[:, :, :, :, i*train_size:(i+1)*train_size] = training_tensor
        id_train[i*train_size:(i+1)*train_size, :] = training_identifier
        test_videos[:, :, :, :, i*test_size:(i+1)*test_size] = testing_tensor
        id_test[i*test_size:(i+1)*test_size, :] = testing_identifier

        plt.imshow(training_tensor[:, :, :, 0, 0]) # Showing a random frame from each actor's video tensors
        plt.show()
        del training_tensor
        del training_identifier
        del testing_tensor
        del testing_identifier

        gc.collect()
        i += 1

y_train = id_train[:, 2].astype(np.int8) # Extracting the emotion labels from ids
y_test = id_test[:, 2].astype(np.int8)

Video Feature Extraction

Now that we have our video tensors, we need to extract relevant information from the videos. To do this, I implemented a customized function based on the popular Histogram of Oriented Gradients algorithm, but in 3D! The image below shows an example of how the 2D approach works.

Source: https://www.sciencedirect.com/topics/computer-science/histogram-of-oriented-gradient

Here are some libraries to get us started:

import os
import numpy as np
from scipy.signal import convolve2d
import tensorly as tl
import gc
import matplotlib.pyplot as plt
from tqdm.notebook import tqdm

Breakdown of Custom 3D Histogram of Oriented Gradients for Dimensionality Reduction

The formulation of this Histogram of Oriented Gradients algorithm is loosely based on recent research in the field of computer vision. However, this project translates this approach into 3 dimensions. Below is an overview of the methodology applied to the training and testing datasets. If you are not too fond of math, you can skip to the next section!

1. Iterate through every sample and convert the frames to grayscale.

2. We then compute the gradients with respect to the height, width [1], and frames ($\frac{\partial V}{\partial x}$, $\frac{\partial V}{\partial y}$, $\frac{\partial V}{\partial z}$).

3. Using these gradients, we can compute the three-dimensional gradient magnitude for each video.

\[G = \sqrt{\left( \frac{\partial V}{\partial x}\right)^2 + \left( \frac{\partial V}{\partial y} \right)^2 + \left( \frac{\partial V}{\partial z}\right)^2}\]

4. Generally for images ($I$), we compute the gradient direction by $\theta = \arctan \left( \frac{\frac{\partial I}{\partial y}}{\frac{\partial I}{\partial x}} \right)$ [2]. Since we have videos, we must compute the azimuthal angle and the polar angle to capture the 3D feature space [4].

\[\theta_{azimuth} = \arctan \left( \frac{\frac{\partial V}{\partial y}}{\frac{\partial V}{\partial x}} \right)\] \[\phi_{polar} = \arctan \left( \frac{\sqrt{\left( \frac{\partial V}{\partial x}\right)^2 + \left( \frac{\partial V}{\partial y} \right)^2}}{\frac{\partial V}{\partial z}} \right)\]

5. With our three sets of features, we can now partition the video into cells [3]. In our case, the cell size is ($5$, $6$, $5$), which will group sets of 180 pixels together.

6. With our grouped pixels, we cluster the gradient magnitudes of each pixel ($G_{(i, j, k)}$) into bins based on the azimuthal and polar angles. With $9$ bins, we sum the gradient magnitudes for all the pixels belonging to each bin for both types of angles, which reduces the dimensionality from $180$ points in each cell to $9 \times 2 = 18$ points per cell. We can then save these results to disk.

Implementing the Algorithm

Now that we have a foundational understanding of the HOG3D algorithm, we can create our custom implementation. I use two main functions–one for calculating the shape of the new data, and one for implementing the HOG3D algorithm itself.

Click here to view the code

def calculate_total_descriptor_shape(width, height, n_frames, cell_size, nbins):
    """Calculate the total shape of the descriptor."""

    cells_per_width = width // cell_size[0]
    cells_per_height = height // cell_size[1]
    cells_per_depth = n_frames // cell_size[2]

    return cells_per_width, cells_per_height, cells_per_depth, nbins, 2

def compute_hog3d_rgb(frames, cell_size, nbins, v, gaussian_filter):
    """Compute the HOG3D features for a video."""

    width, height, channels, n_frames = frames.shape
    gray_frames = tl.tenalg.mode_dot(frames, np.array([0.2989, 0.5870, 0.1140]), mode=2)

    if gaussian_filter:
        print('Applying Gaussian Filter')
        gaussian_filter = np.array([[1, 4, 7, 10, 7, 4, 1],
                                [4, 12, 26, 33, 26, 12, 4],
                                [7, 26, 55, 71, 55, 26, 7],
                                [10, 33, 71, 91, 71, 33, 10],
                                [7, 26, 55, 71, 55, 26, 7],
                                [4, 12, 26, 33, 26, 12, 4],
                                [1, 4, 7, 10, 7, 4, 1]]) / 1115



        data = np.zeros_like(gray_frames)
        for i in range(n_frames):
            data[:, :, i] = convolve2d(gray_frames[:, :, i], gaussian_filter, mode='same')
    else:
        data = gray_frames
        del gray_frames
        gc.collect()

    gx = np.gradient(data, axis=0)
    gy = np.gradient(data, axis=1)
    gz = np.gradient(data, axis=2)

    magnitude = np.sqrt(gx**2 + gy**2 + gz**2)
    azimuthal_angle = np.arctan2(gy, gx)
    polar_angle = np.arctan2(np.sqrt(gx**2, gy**2), gz)

    output_shape = calculate_total_descriptor_shape(width, height, n_frames, cell_size, nbins)
    hog3d_descriptors = np.zeros(output_shape, dtype=np.float32)

    for i in range(0, n_frames - cell_size[2], cell_size[2]):
        for y in range(0, height - cell_size[1], cell_size[1]):
            for x in range(0, width - cell_size[0], cell_size[0]):
                cell_magnitude = magnitude[x:x + cell_size[0], y:y + cell_size[1], i:i + cell_size[2]]
                cell_azimuthal = azimuthal_angle[x:x + cell_size[0], y:y + cell_size[1], i:i + cell_size[2]]
                cell_polar = polar_angle[x:x + cell_size[0], y:y + cell_size[1], i:i + cell_size[2]]

                hist_azimuthal, _ = np.histogram(cell_azimuthal, bins=nbins, range=(-np.pi, np.pi), weights=cell_magnitude)
                hist_polar, _ = np.histogram(cell_polar, bins=nbins, range=(0, np.pi), weights=cell_magnitude)

                hist_azimuthal = hist_azimuthal / (np.linalg.norm(hist_azimuthal) + 1e-6)
                hist_polar = hist_polar / (np.linalg.norm(hist_polar) + 1e-6)

                hog3d_descriptors[x//cell_size[0], y//cell_size[1], i//cell_size[2], :, 0] = hist_azimuthal
                hog3d_descriptors[x//cell_size[0], y//cell_size[1], i//cell_size[2], :, 1] = hist_polar

    del cell_magnitude, cell_azimuthal, cell_polar, hist_azimuthal, hist_polar, gx, gy, gz, magnitude, azimuthal_angle, polar_angle
    gc.collect()

    return hog3d_descriptors

Visualizing Gradient Magnitude, Azimuthal Angle, and Polar Angle

You might be wondering–what does all this crazy math look like if you were to visualize it? Well, this is what it would look like:

Creating our HOG3D Dataset

Now that we have our algorithms set up, we just need to process each video, apply our dimensionality reduction algorithm, and combine these new datasets.

Click here to view the code

def process_videos(videos, cell_size=(5, 6, 5), nbins=9, gaussian_filter=False):
    """Process a batch of videos."""

    width, height, channels, n_frames, video_count = videos.shape

    for v in tqdm(range(video_count)):
        hog3d_descriptors = compute_hog3d_rgb(videos[:, :, :, :, v].astype(np.float32), cell_size, nbins, v, gaussian_filter)
        if v == 0:
            descriptors_shape = hog3d_descriptors.shape
            all_descriptors = np.zeros((video_count, *descriptors_shape), dtype=np.float32)

        all_descriptors[v] += hog3d_descriptors

        del hog3d_descriptors
        gc.collect()

    return all_descriptors

H_train = process_videos(train_videos)
H_test = process_videos(test_videos)

Audio Feature Extraction

Once we have our video data set up, we need to extract some features for our audio data. Here are some libraries to get us started:

import os
import numpy as np
import librosa
import librosa.display
from IPython.display import Audio
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import zipfile
from skfda.representation.basis import BSplineBasis
from tqdm.notebook import tqdm
import gc

Breakdown of Feature Engineering and Extraction Process

The audio feature extraction process in the code below is a customized multi-step process that is formulated as shown below.

1. Process the audio files using Librosa and manipulate the audio by adding noise, changing the pitch, and changing the pitch of the audio to get multiple variations of the same audio sample.

Here are some examples of what this sounds like:

Original Audio

Noise Audio

Pitch Changed Audio (+4)

Pitch Changed Audio (-6)

2. For each of the transformed audio files, we extract the following features using Librosa:

Mel-Frequency Cepstral Coefficients (MFCCs): A representation of the short-term power spectrum of a sound, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency. Source
Mel Spectogram: A spectrogram where the frequencies are converted to the Mel scale, which approximates the human ear’s response more closely than the linear frequency scale. Source
Zero Crossing Rate (ZCR): The rate at which the audio signal changes sign from positive to negative or vice versa. It is a measure of the noisiness of the signal. Source
Root Mean Square Energy (RMSE): A measure of the energy in the audio signal, calculated as the square root of the mean squared values of an audio signal. Source
Chromagram: A representation of 12 different pitch classes, or semitones, of a musical octave. They are calculated by mapping the entire frequency spectrum onto these 12 bins. Source
Spectral Contrast: A measure of the difference in amplitude between peaks and valleys in a sound spectrum. Source

It might help to visualize what these features look like:

3. Once we collect this data, we combine all the extracted features and create a B-Spline feature space for this data. B-splines, or Basis Splines, are piece-wise polynomial approximations of a curve. They are defined recursively as such [5]:

\[B_{i, j}(x) = \frac{x - t_i}{t_{i+j} - t_i} B_{i, j-1}(x) \\ + \frac{t_{i+j+1} - x}{t_{i+j+1} - t_{i+1}} B_{i+1, j-1}(x)\]

for $j \ge 1$ with the initial condition:

\[B_i^0(x) = \begin{cases} 1 & \text{if } t_i \le x < t_{i+1} \\ 0 & \text{otherwise} \end{cases}\]

Here:

$B_{i, j}(x)$ is the B-Spline basis function of degree $k$.
$x$ is the parameter.
$t_i$ are the knots.

The B-Spline curve $C(x)$ of degree $j$ can be defined as a linear combination of these basis functions:

\[C(t) = \sum_{i=0}^{n} P_i B_{i, j}(x)\]

where $P_i$ are the control points.

Here’s what the basis functions look like for various orders:

This is a heatmap of a basis matrix with an order of 4:

4. Once we develop this feature space, we project the extracted feature space onto the B-Spline feature space, which transforms the data into a lower dimensional approximation. For example, if we have $12$ knots, then no matter how many columns our data has, we would have knots + $2$, or $14$, columns.

The B-Spline transformation reduces the dimensionality from the image earlier to this:

5. Now, we just have to combine the data and save it to disk.

Implementing the algorithm

With our understanding of the feature extraction process, we can now implement the Python code to extract these features and combine it into our audio data.

Click here to view the code

def add_noise(y, noise_factor=0.005):
    """Add noise to the audio signal"""

    noise = np.random.randn(len(y))
    augmented_data = y + noise_factor * noise
    return augmented_data

def change_pitch(y, sr, n_steps):
    """Change the pitch of the audio signal"""

    return librosa.effects.pitch_shift(y, sr=sr, n_steps=n_steps)

def change_speed(y, speed_factor):
    """Change the speed of the audio signal"""

    return librosa.effects.time_stretch(y, rate=speed_factor)

def lse_solver(X, y):
    """Least Squares Estimation Solver"""

    return np.linalg.inv(X.T @ X) @ X.T @ y

def transform_features(y, sr):
    """Transform the audio signal and merge the data"""

    mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13).T
    mels = librosa.feature.melspectrogram(y=y, sr=sr, hop_length=512, n_mels=128)
    zcr = librosa.feature.zero_crossing_rate(y, frame_length=2048, hop_length=512).T
    rmse = librosa.feature.rms(y=y, frame_length=2048, hop_length=512).T
    chroma = librosa.feature.chroma_stft(y=y, sr=sr, hop_length=512).T
    spectral_contrast = librosa.feature.spectral_contrast(y=y, sr=sr, hop_length=512).T

    features = np.concatenate((mfccs, zcr, rmse, chroma, spectral_contrast), axis=1)

    del mfccs
    del mels
    del zcr
    del rmse
    del chroma
    del spectral_contrast
    gc.collect()

    return features

def extract_features(y, sr, nknots=10, order=4):
    """Extract features from the audio signal using B-Splines"""

    features = transform_features(y, sr)
    m, n = features.shape

    knots = np.linspace(0, 1, nknots)
    xx = np.linspace(0, 1, m)

    bs = BSplineBasis(knots=knots, order=order)
    bs_basis = bs(xx).T[0]

    bs_features = lse_solver(bs_basis, features)

    del bs
    del bs_basis
    del features
    gc.collect()

    return bs_features

nknots = 12
Audio_data = np.zeros((60*24, nknots+2, 34*8), dtype=np.float32)

inds = np.zeros((60*24, 7), dtype=np.uint8)
i = 0
with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
    for folder in tqdm(zip_ref.infolist()):
        file_name = folder.filename
        if file_name.endswith('.wav'):
            with zip_ref.open(folder) as audio_file:
                idx = np.array(file_name.split('/')[1][:-4].split('-')).astype(np.uint8)
                inds[i, :] = idx

                y, sr = librosa.load(audio_file)

                y_noise = add_noise(y)
                y_pitch1 = change_pitch(y, sr, 4)
                y_pitch2 = change_pitch(y, sr, -6)
                y_pitch3 = change_pitch(y, sr, 3)
                y_speed1 = change_speed(y, 1.5)
                y_speed2 = change_speed(y, 2.0)
                y_speed3 = change_speed(y, 0.8)
                y_speed4 = change_speed(y, 0.5)

                features1 = extract_features(y_noise, sr, nknots, 4)
                features2 = extract_features(y_pitch1, sr, nknots, 4)
                features3 = extract_features(y_pitch2, sr, nknots, 4)
                features4 = extract_features(y_pitch3, sr, nknots, 4)
                features5 = extract_features(y_speed1, sr, nknots, 4)
                features6 = extract_features(y_speed2, sr, nknots, 4)
                features7 = extract_features(y_speed3, sr, nknots, 4)
                features8 = extract_features(y_speed4, sr, nknots, 4)

                features = np.concatenate((features1, features2, features3, features4, features5, features6, features7, features8), axis=1)
                m, n = features.shape

                Audio_data[i, ...] = features

                del features, features1, features2, features3, features4, \
                    features5, features6, features7, features8, y_noise, y_pitch1, y_pitch2, y_pitch3, \
                    y_speed1, y_speed2, y_speed3, y_speed4, y, sr

                gc.collect()
                i += 1

Emotion Classification

Now that we finished most of the tedious work, we just have a couple more steps till the finish line! Here’s the necessary imports for this section:

Click here to view the code

import os
import gc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.neural_network import MLPClassifier
from tensorly.tenalg import multi_mode_dot
from tensorly.decomposition import partial_tucker
from tqdm.notebook import tqdm

Partial Tucker Decomposition for Additional Dimensionality Reduction

Currently, the shapes of the data are as follows:

video_train –> ($960$, $48$, $71$, $10$, $9$, $2$)
video_test –> ($480$, $48$, $71$, $10$, $9$, $2$)
audio_train –> ($960$, $14$, $272$)
audio_test –> ($480$, $14$, $272$)

So, if we were to flatten out the data, we would have $48 * 71 * 10 * 9 * 2 = 613440$ columns for the videos, and $14 * 272 = 3808$ columns for the audio data, which is just too large. We need to reduce the dimensionality further using higher-order PCA, or Tucker Decomposition.

Below is a brief overview of how Tucker Decomposition works. Again, if you’re not the biggest fan of math, you can skip to the next section.

Given a tensor $\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$, Tucker decomposition approximates $\mathcal{X}$ as [6]:

\[\mathcal{X} \approx \mathcal{G} \times_1 A^{(1)} \times_2 A^{(2)} \times_3 \cdots \times_N A^{(N)}\]

where:

$\mathcal{G} \in \mathbb{R}^{J_1 \times J_2 \times \cdots \times J_N}$ is the core tensor.
$A^{(n)} \in \mathbb{R}^{I_n \times J_n}$ are the factor matrices for each mode $n$.

The operator $\times_n$ denotes the mode-$n$ product between a tensor and a matrix. Specifically, the mode-$n$ product of a tensor $\mathcal{G}$ with a matrix $A$ is defined as:

$(\mathcal{G} \times_n A)_{i_1 i_2 \cdots i_{n-1} j i_{n+1} \cdots i_N} = \sum_{i_n} \mathcal{G}_{i_1 i_2 \cdots i_N} A_{j i_n}$

This image more clearly demonstrates the math:

Source: https://www.researchgate.net/figure/Third-order-Tucker-decomposition_fig1_257482079

Implementing the algorithm

Tensorly makes it super easy for use to implement Partial Tucker Decomposition, which is a customized form of the full Tucker, but it maintains the integrity of dimensions that aren’t being reduced, like the number/order of samples. Once we reduce the video and audio data and flatten it, we can combine them together into our X_train and X_test for classification!

Click here to view the code

# Video Tucker Decomposition
video_ranks = [8, 8, 3, 1]
video_modes = [1, 2, 3, 5]

temp_data, _ = partial_tucker(video_train, video_ranks, modes=video_modes, verbose=True, tol=1e-4)

V_train = temp_data[0]
video_train_factors = temp_data[1]

del temp_data
gc.collect()

V_test = multi_mode_dot(video_test, [U.T for U in video_train_factors], modes=video_modes)

V_train = V_train.reshape(V_train.shape[0], -1)
V_test = V_test.reshape(V_test.shape[0], -1)


# Audio Tucker Decomposition
audio_ranks = [10, 150]
audio_modes = [1, 2]

temp_data, _ = partial_tucker(audio_train, audio_ranks, modes=audio_modes, verbose=True, tol=1e-4)

A_train = temp_data[0]
audio_train_factors = temp_data[1]

del data
gc.collect()

A_test = multi_mode_dot(audio_test, [U.T for U in audio_train_factors], modes=audio_modes)

A_train = A_train.reshape(A_train.shape[0], -1)
A_test = A_test.reshape(A_test.shape[0], -1)

# Combining the Data
X_train = np.concatenate((V_train, A_train), axis=1)
X_test = np.concatenate((V_test, A_test), axis=1)

print(X_train.shape)
print(X_test.shape)

Training a Multilayer Perceptron Model using Sklearn

We could just train a Support Vector Classifier or Random Forest (and I do on my Github), but I decided to use a simple neural network instead because a lot of the data is not linearly separable.

mlp = MLPClassifier(alpha=0.07, batch_size=200, epsilon=1e-8, n_iter_no_change=10, learning_rate='adaptive', hidden_layer_sizes=(250, 500, 100), max_iter=1000, random_state=42)
mlp.fit(X_train, y_train)

y_pred = mlp.predict(X_test)

Using this model, we get an accuracy of $85.625$%! Considering that we did not use any complex deep-learning architectures, this is pretty impressive! Here’s a more detailed overview of the performance:

As we can see, most of the labels performed extremely well, especially disgust. This is most likely because disgust is a very strong emotion, which can be easier to pick up with our algorithm. However, sad and surprised emotions did not perform as well. These errors can easily be fixed with a more robust model or more trial and error with the current model architecture.

Conclusion

This project showcases the potential of a custom approach to emotion classification using multimodal data. By combining 3D Histograms of Oriented Gradients for video features and advanced audio feature extraction with B-Spline transformations, we were able to create a robust and efficient feature extraction pipeline. The integration of these features through Partial Tucker Decomposition significantly reduced the dimensionality, making our models more efficient without sacrificing performance. The impressive accuracy of our ensemble method underscores the power of combining video and audio data, paving the way for more sophisticated and accurate emotion detection systems in real-world applications. This custom approach not only enhances the accuracy of emotion classification but also demonstrates the potential for broader applications in any field requiring nuanced analysis of multimedia data.

The importance of this project extends beyond its technical achievements, as it lays the groundwork for practical applications that can profoundly impact various domains. By improving emotion classification accuracy, we can develop tools that better understand and respond to human emotions, ultimately enhancing communication and interaction in both personal and professional settings. The integration of advanced machine learning techniques in this project demonstrates the potential to create more empathetic and intelligent systems, underscoring the transformative power of technology in understanding and interpreting human emotions.

If you’ve made it this far, thank you for giving this a read, and happy learning!

References:

[1] SpringerLink (Online service), Panigrahi, C. R., Pati, B., Mohapatra, P., Buyya, R., & Li, K. (2021). Progress in Advanced Computing and Intelligent Engineering: Proceedings of ICACIE 2019, Volume 1 (1st ed. 2021.). Springer Singapore : Imprint: Springer. https://doi.org/10.1007/978-981-15-6584-7

[2] Zoubir, Hajar & Rguig, Mustapha & Aroussi, Mohamed & Chehri, Abdellah & Rachid, Saadane. (2022). Concrete Bridge Crack Image Classification Using Histograms of Oriented Gradients, Uniform Local Binary Patterns, and Kernel Principal Component Analysis. Electronics. 11. 3357. 10.3390/electronics11203357. https://doi.org/10.3390/electronics11203357

[3] S V Shidlovskiy et al 2020 J. Phys.: Conf. Ser. 1611 012072 https://doi:10.1088/1742-6596/1611/1/012072

[4] Nykamp DQ, “Spherical coordinates.” From Math Insight. http://mathinsight.org/spherical_coordinates

[5] Hastie, T., Tibshirani, R., Friedman, J. (2009). Basis Expansions and Regularization. In: The Elements of Statistical Learning. Springer Series in Statistics. Springer, New York, NY. https://doi.org/10.1007/978-0-387-84858-7_5

[6] Kolda, Tamara G., and Brett W. Bader. “Tensor Decompositions and Applications.” SIAM Review, vol. 51, no. 3, 2009, pp. 455-500. Society for Industrial and Applied Mathematics. https://doi.org/10.1137/07070111X.

[7] Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) [Data set]. In PLoS ONE (1.0.0, Vol. 13, Number 5, p. e0196391). Zenodo. https://doi.org/10.5281/zenodo.1188976

Facial Recognition

Wed, 19 Jun 2024 00:00:00 +0000

This project leverages the power of Higher-Order Discriminant Analysis (HODA) and Support Vector Machines (SVM) to classify celebrity photos. And yes, this project does not use any neural networks! Facial recognition technology has become a crucial tool in modern society due to its wide-ranging applications in security, healthcare, social media, and entertainment. By enabling machines to identify and verify individuals based on their facial features, it enhances security systems, simplifies user authentication processes, and offers personalized user experiences.

The growing importance of facial recognition technology underscores the need for advanced algorithms that can accurately and efficiently process high-dimensional image data. HODA is a supervised feature extraction algorithm, and this notebook implements the algorithm as outlined in “Tensor Decompositions for Feature Extraction and Classification of High Dimensional Datasets” by Anh Huy Phan and Andrzej Cichocki [1].

Here are the main sections of this article:

Loading the Data
Feature Extraction using HODA
Fitting the Model
Conclusion
References

Loading the Data

The dataset was found on Kaggle and is called the Hollywood Celebrity Facial Recognition Dataset. There are 17 actors and 100 photos per actor that are relevant to us. Scarlett Johanson has 200 photos because she’s just that awesome! However, we will only be using 100 to standardize the model training.

Before we start, we need to set up some libraries for our analysis.

import numpy as np
import pandas as pd
import zipfile
import io
import matplotlib.pyplot as plt
from scipy.ndimage import zoom
import seaborn as sns
%matplotlib inline
import tensorly as tl
from tensorly.tenalg import multi_mode_dot
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from collections import defaultdict
from tqdm.notebook import tqdm

Once we have our libraries set up, we can run the code below to create our training and testing tensors with dimensions as height x width x samples, and labels for each sample. The data processing ensures that all of the photos are the same size and then converts them to grayscale. We could leave RGB on, but it doesn’t add much information for our analysis and it takes three times longer to compute.

Click here to view the code

target_shape = (240, 240)

X = []
y = []

zip_path = 'archive.zip'
i = 0
with zipfile.ZipFile(zip_path, 'r') as zip_ref:
    try:
        for file in zip_ref.infolist():
            folder_name = file.filename.split('/')[0]
            y.append(folder_name)
            with zip_ref.open(file.filename) as image_file:
                img = plt.imread(io.BytesIO(image_file.read()), format='jpeg')
                
                if img.shape[:2] != target_shape:
                    factors = (target_shape[0] / img.shape[0], target_shape[1] / img.shape[1], 1)
                    img = zoom(img, factors, order=3)
                    
                gray_img = (img@np.array([0.2989, 0.5870, 0.1140])).astype(np.float32)
                X.append(gray_img)
    
            i += 1
    except Exception as e:
        print(f'Error on iteration {i}')

X = np.array(X).transpose(1, 2, 0)
y = np.array(y)

label_indices = defaultdict(list)
for index, label in enumerate(y):
    label_indices[label].append(index)

train_indices = []
test_indices = []

for label, indices in label_indices.items():
    np.random.shuffle(indices)
    train_indices.extend(indices[:97])
    test_indices.extend(indices[97:100])

X_train, X_test = X[..., train_indices], X[..., test_indices]
y_train, y_test = y[train_indices], y[test_indices]

print("Training data shape:", X_train.shape, y_train.shape)
print("Testing data shape:", X_test.shape, y_test.shape)

Now that we have our data set up, let’s look at some examples of the photos.

Click here to view the code

indices = np.random.choice(X_train.shape[2], 25, replace=False)

fig, axes = plt.subplots(5, 5, sharex=True, sharey=True, figsize=(10, 10))
for i, ax in enumerate(axes.flat):
    ax.imshow(X_train[..., indices[i]], cmap='gray')
    ax.set_title(y_train[indices[i]])

fig.suptitle('25 Randomly Sampled Images from the Dataset')
plt.tight_layout()
plt.show()

We can see that not all photos are standardized, and there are many inconsistencies in age, facial accessories, angles, and more. Let’s find out how much this will affect our analysis!

Feature Extraction using HODA

As mentioned earlier, this feature extraction method is based off of the paper, “Tensor Decompositions for Feature Extraction and Classification of High Dimensional Datasets” by Anh Huy Phan and Andrzej Cichocki [1], and it is an iterative process. If you are not the biggest fan of math, you can skip over to the next section!

The HODA algorithm is formulated as such:

\[\underline{\tilde{\mathbf{X}}}^{(k)} = \underline{\mathbf{X}}^{(k)} - \underline{\tilde{\mathbf{X}}}^{(c_k)}\] \[\underline{\tilde{\mathbf{X}}}^{(c)} = \sqrt{K_c} \left( \underline{\tilde{\mathbf{X}}}^{(c)} - \underline{\tilde{\mathbf{X}}} \right)\]

input : $\underline{\mathbf{X}}$: Concatenated tensor of $K$ training samples $I_1 \times I_2 \times \cdots \times I_N \times K$
output : $\mathbf{U}^{(n)}$: $N$ orthogonal basis factors $I_n \times J_n \ (n = 1, 2, \ldots, N)$
output : $\mathbf{G}$: Training feature tensors $J_1 \times J_2 \times \cdots \times J_N \times K$

begin

Initialize $\mathbf{U}^{(n)}$
Calculate $\underline{\tilde{\mathbf{X}}}$ and $\underline{\tilde{\mathbf{X}}}^{(c)}$ as outlined above.

repeat for $n = 1$ to $N$ do

\[\tilde{\mathbf{Z}}_n = \underline{\tilde{\mathbf{X}}} \times_{\{(n,N+1)\}} \{ \mathbf{U}^{(T)} \}\] \[\mathbf{S}_w^n = \tilde{\mathbf{Z}}_n \tilde{\mathbf{Z}}_n^{(T)}\] \[\tilde{\mathbf{Z}}_n^b = \underline{\tilde{\mathbf{X}}}^{(c)} \times_{\{(n,N+1)\}} \{ \mathbf{U}^{(T)} \}\] \[\mathbf{S}_b^n = \tilde{\mathbf{Z}}_n^b \tilde{\mathbf{Z}}_n^{b(T)}\] \[\varphi = \frac{\text{trace}(\mathbf{U}^{(n)T} \mathbf{S}_b^n \mathbf{U}^{(n)})}{\text{trace}(\mathbf{U}^{(n)T} \mathbf{S}_w^n \mathbf{U}^{(n)})}\] \[\left[ \mathbf{U}^{(n)}, \Lambda \right] = \text{eigs}(\mathbf{S}_b^n - \varphi \mathbf{S}_w^n, J_n, 'LM')\] \[or /> [\mathbf{U}^{(n)}, \Lambda] = \text{eigs}(\mathbf{S}_b^n, \mathbf{S}_w^n, J_n, 'LM')\] \[\left[ \mathbf{U}^{(n)}, \Lambda \right] = \text{eigs}(\mathbf{U}^{(n)} \mathbf{U}^{(n)T} \mathbf{X}_{\underline{-n}} \mathbf{X}_{\underline{-n}}^T \mathbf{U}^{(n)} \mathbf{U}^{(n)T}, J_n, 'LM')\]

until a criterion is met
$\mathbf{G} = \mathbf{X}_{\underline{-}(N+1)} \left[ \mathbf{U} \right]^T$

end

The output of the model is similar to Tucker Decomposition, but it is a supervised approach to it. So, understanding Tucker Decomposition can help get an idea of what’s going on under the hood. The diagram below explains it more clearly.

Source: https://www.researchgate.net/figure/Third-order-Tucker-decomposition_fig1_257482079

In our case, the factor matrices, when multiplied by the original tensor across each axis, will generate a lower dimensional representation of the images. And so, our algorithm must return these factor matrices.

Implementing the HODA Algorithm

With all the math out of the way, we can implement our algorithm! The function in the code below returns the factor matrices that need to be multiplied across specific modes, as mentioned earlier. We compute the HODA algorithm on our training data, get the factor matrices, and multiply them across each axis for both training and testing tensors.

Click here to view the code

def HODA(X_train, y_train, new_dims, alpha=1.0):
    I, K = X_train.shape[:-1], X_train.shape[-1]
    training_samples = K // len(np.unique(y_train))

    X_c = np.zeros_like(X_train).astype(np.float32)
    for lab in np.unique(y_train)[1:]:
        X_c[..., y_train == lab] = np.mean(X_train[..., y_train == lab].astype(np.float32), axis=-1, keepdims=True)

    X_mean = np.mean(X_train.astype(np.float32), axis=-1, keepdims=True)

    U_n = [np.random.rand(I[i], new_dims[i]).astype(np.complex64) for i in range(len(new_dims))]

    X_v = np.sqrt(training_samples).astype(np.float32) * (X_c - X_mean)

    X_tilde = X_train.astype(np.float32) - X_c

    for _ in tqdm(range(10), desc='Total Iterations'):
        for n in range(len(new_dims)):
            Z_tilde = tl.unfold(multi_mode_dot(X_tilde.astype(np.complex64), [U_n[i].T for i in range(len(U_n)) if i != n], modes=[i for i in range(len(U_n)) if i != n]), mode=n)

            S_w = (Z_tilde @ Z_tilde.T).astype(np.complex64)

            Z_v = tl.unfold(multi_mode_dot(X_v.astype(np.complex64), [U_n[i].T for i in range(len(U_n)) if i != n], modes=[i for i in range(len(U_n)) if i != n]), mode=n)

            S_b = (Z_v @ Z_v.T).astype(np.complex64)

            phi = np.trace(U_n[n].T @ S_b @ U_n[n]) / np.trace(alpha * (U_n[n].T @ S_w @ U_n[n]) + (1-alpha) * np.eye(new_dims[n]).astype(np.complex64))
            print(phi)

            U_n[n] = np.linalg.eig(S_b - phi * S_w)[1][:, :new_dims[n]]
            
            X_train_unfolded = tl.unfold(X_train.astype(np.complex64), mode=n)

            U_n[n] = np.linalg.eig(U_n[n] @ U_n[n].T @ X_train_unfolded @ X_train_unfolded.T @ U_n[n] @ U_n[n].T)[1][:, :new_dims[n]]

    return U_n, phi

new_dims = (10, 10)
U_n, phi = HODA(X_train, y_train, new_dims, alpha=1e7)
print('Phi at Convergence:', phi)

G_train = multi_mode_dot(X_train, [np.real(U.T) for U in U_n], modes=[0, 1])
G_test = multi_mode_dot(X_test, [np.real(U.T) for U in U_n], modes=[0, 1])

With ‘new_dims’ set at (10, 10), the dimensionality of both ‘G_train’ and ‘G_test’ are reduced to 10 x 10 x samples. Now, you might be wondering: what do the example images from earlier look like now? Well, let’s find out!

Click here to view the code

G_train = multi_mode_dot(X_train, [np.real(U.T) for U in U_n], modes=[0, 1])
G_test = multi_mode_dot(X_test, [np.real(U.T) for U in U_n], modes=[0, 1])

fig, axes = plt.subplots(5, 5, sharex=True, sharey=True, figsize=(10, 10))
for i, ax in enumerate(axes.flat):
    ax.imshow(G_train[..., indices[i]], cmap='gray')
    ax.set_title(y_train[indices[i]])

fig.suptitle('25 Randomly Sampled Images from the Dataset')
plt.tight_layout()
plt.show()

The images look nothing like they did before, and they appear to be uninformative! But, let’s put it through a model and see how well it performs.

Fitting the Model

We’ll be using a Support Vector Classifier from sklearn’s library for classification. The first step is to flatten the columns of the data, and we can then fit the model with our new features.

G_train = tl.unfold(G_train, mode=-1)
G_test = tl.unfold(G_test, mode=-1)

svc_model = SVC(probability=True, C=100, kernel='rbf', gamma='scale')
svc_model.fit(G_train, y_train)

The model performed fairly well, with an accuracy of 70.59% with minimal tuning! Here’s a detailed overview of the performance:

Most of the predictions are falling across the diagonal, which shows us that the algorithm can pick up relevant features. To improve the model, we could use neural networks, but I wanted to focus on the ability to classify high-dimensional data with simpler models.

Conclusion

In this project, we’ve demonstrated the power of Higher-Order Discriminant Analysis (HODA) combined with Support Vector Machines (SVM) for classifying celebrity photos. Despite not using any neural networks, the model achieved impressive results, showcasing the effectiveness of advanced tensor decompositions for feature extraction. Facial recognition technology, as explored here, has vast applications in various fields including security, healthcare, social media, and entertainment, making it an indispensable tool in modern society. Looking ahead, there are several ways to enhance this project. Incorporating larger and more diverse datasets could further improve model robustness. Additionally, optimizing the HODA algorithm and exploring its integration with other machine-learning models could yield even better performance.

Thank you for following along on this journey into facial recognition technology. I hope you found it informative and fun!

References

[1] Anh Huy Phan, Andrzej Cichocki, Tensor decompositions for feature extraction and classification of high dimensional datasets, Nonlinear Theory and Its Applications, IEICE, 2010, Volume 1, Issue 1, Pages 37-68, Released on J-STAGE October 01, 2010, Online ISSN 2185-4106, https://doi.org/10.1587/nolta.1.37, https://www.jstage.jst.go.jp/article/nolta/1/1/1_1_37/_article/-char/en

Understanding Customer Churn

Wed, 24 Apr 2024 00:00:00 +0000

Customer churn is a core component of marketing analytics and marketing-focused data science. Analyzing customer churn is crucial for shaping business strategy as it provides insights into why customers leave, allowing businesses to identify and address underlying issues, improve customer retention, and enhance overall customer satisfaction. By understanding the factors that contribute to churn, companies can implement targeted interventions, optimize their products or services, and tailor marketing efforts to retain existing customers and attract new ones. Did this pique your interest? Then let’s dive right in!

This project showcases the efficacy of logistic regression models in predicting customer churn (1 for churned, and 0 for retained). The project initially covers Principal Component Analysis (PCA) for dimensionality reduction and logistic regression (using gradient ascent and Newton’s method).

Here are the main questions this project aims to answer:

How effective are the logistic regression models at predicting customer churn?
How can different approaches to logistic regression change the outcome of the results?
How well can we predict churn and what does that tell us about shaping business strategy?

Here are the contents of this project:

Understanding the Data
PCA for Dimensionality Reduction
Logistic Regression
Observations and Results
References

Understanding the Data

The dataset was found on Kaggle at this source and includes 14 columns:

RowNumber: A unique identifier for each row in the dataset.
CustomerId: Unique customer identification number.
Surname: The last name of the customer (for privacy reasons, consider anonymizing this data if not already done).
CreditScore: The customer’s credit score at the time of data collection.
Geography: The customer’s country or region, providing insights into location-based trends in churn.
Gender: The customer’s gender.
Age: The customer’s age, valuable for demographic analysis.
Tenure: The number of years the customer has been with the bank.
Balance: The customer’s account balance.
NumOfProducts: The number of products the customer has purchased or subscribed to.
HasCrCard: Indicates whether the customer has a credit card (1) or not (0).
IsActiveMember: Indicates whether the customer is an active member (1) or not (0).
EstimatedSalary: The customer’s estimated salary.
Exited: The target variable, indicating whether the customer has churned (1) or not (0).

Setting up our Analysis

Let’s start our analysis by importing the necessary libraries for this project:

import pandas as pd
import numpy as np
from numpy.linalg import svd
from numpy.linalg import solve
from scipy.special import expit as sigmoid
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, \
                            roc_curve, auc, precision_recall_curve, average_precision_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from math import e
import time

Now, let’s import the data and try to understand what the data looks like under the hood. Here’s a quick description

# Reading our data into a pandas dataframe
data = pd.read_csv('Churn_Modelling.csv', index_col=False).rename(columns={'Exited': 'Churned'})

# Creating another column based on 'NumOfProducts' that converts it to categorical labels
data['NumOfProducts_Categorical'] = data['NumOfProducts'].astype(str)

# Checking number of unique customers
n_customers = len(data['CustomerId'].unique())
print(f'\nNumber of unique customers: {n_customers}')

display(data.describe())

There are 10000 unique customers and 10000 data points, which means that each row of the dataset corresponds to exactly one customer. Let’s dig into the data a little bit more!

Exploratory Data Visualization

Since our dataset contains both categorical and continuous variables, we need to identify which ones are what for creating appropriate visualizations.

categorical_cols = ['Geography', 'Gender', 'NumOfProducts_Categorical', 'HasCrCard', 'IsActiveMember']
continuous_cols = ['CreditScore', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 'EstimatedSalary']

Now let’s create some cool visualizations, starting with a pairplot!

sns.pairplot(data[continuous_cols + categorical_cols + ['Churned']], hue='Churned', palette='Greens')
plt.show()

It is immediately apparent that there is a significant correlation between many of the independent variables. It should be noted that since the variables are plotted against each other, the matrix of plots has a diagonal of ones in a sense (because variables are perfectly correlated to themselves). If the plot was viewed as a triangular matrix, all of the relevant information would still be intact.

Here are some histograms for the continuous data:

Click here to view the code

fig, axs = plt.subplots(2, 3, figsize=(15, 10))
for i, col in enumerate(continuous_cols):
    ax = axs.flatten()[i]
    sns.histplot(data[col], kde=True, ax=ax, color='g')
    ax.set_title(f'Distribution of {col}')
plt.tight_layout()

The distributions of the CreditScore, Balance, and Age columns are relatively normal with right skew, outliers at 0, and left skewed, respectively. The distribution of NumOfProducts drops significantly in the two product ranges. The distributions of Tenure and EstimatedSalary are relatively constant throughout.

Here are some countplots for the categorical data:

Click here to view the code

fig, axs = plt.subplots(2, 3, figsize=(15, 10))
for i, col in enumerate(categorical_cols):
    ax = axs.flatten()[i]
    sns.countplot(x=data[col], ax=ax, palette='Greens')
    ax.set_title(f'Frequency of {col}')
plt.tight_layout()

The distributions of the categorical variables immediately show the differences between individual groups within the data. This is especially noticeable in the Geography, HasCrCard, and NumOfProduct_Categorical (same as the continuous version of this data) columns.

We can also gain better insights into the continuous columns when we split the data by churn rate. I created boxplots below to visualize this:

Click here to view the code

fig, axs = plt.subplots(2, 3, figsize=(15, 10))
for i, col in enumerate(continuous_cols):
    ax = axs.flatten()[i]
    sns.boxplot(x='Churned', y=col, data=data, ax=ax, palette='Greens')
    ax.set_title(f'Distribution of {col} by Churn Rate')
plt.tight_layout()

These plots showcase the distributions of the continuous variables across the Churned columns. Although some of the boxplots are nearly identical, there are slight variations in the distributions that the models should be able to learn from.

Preprocessing Data

Now that we have a better understanding of our dataset, we can start building our model, but the first step is to preprocess the data. This step includes creating dummy variables for the categorical columns and scaling the continuous columns to have a mean of 0 and a standard deviation of 1. This is because principal component analysis requires that the data is centered.

Click here to view the code

# Splitting the data between X and y (independent and dependent variables)
X = data.drop(columns=['RowNumber', 'CustomerId', 'Surname', 'Churned'], axis=1, inplace=False)
y = data[['Churned']]

print('\033[1mOriginal X data\033[0m:')
display(X.head())
print('\n\033[1my data\033[0m:')
display(y.head())

# Creating dummy columns for all of the non-binary categorical columns
X = pd.get_dummies(X, columns=['Geography', 'Gender', 'NumOfProducts_Categorical'])

# Scaling the data using sklearn's StandardScaler function (normalizes the data with mean of 0 and var of 1)
scaler = StandardScaler()
X[continuous_cols] = scaler.fit_transform(X[continuous_cols])

print('\033[1mNormalized X data\033[0m:')
display(X.head())

PCA for Dimensionality Reduction

Now that our data is fully ready, let’s go over the math to reduce the dimensionality of the data [1]! If you’re not a fan of math, feel free to skip to the next section.

1. Decompose matrix $\mathbf{X}$ :

\[\mathbf{X} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^T\]

2. Truncate SVD by keeping the top $k$ dimensions:

\[\mathbf{X} \approx \mathbf{U}_k \mathbf{\Sigma}_k \mathbf{V}_k^T\]

3. Compute compression error:

\[\text{compression_error}(\mathbf{\Sigma}, k) = \frac{\sqrt{\sum_{i=k+1}^r \sigma_i^2}}{\sqrt{\sum_{i=1}^r \sigma_i^2}}\]

4. Find $k$ such that the error is minimized:

\[\text{compression_error}(\mathbf{\Sigma}, k) \leq \text{error_rate}\]

Here’s a visualization of what principal components look like:

Source: https://towardsdatascience.com/principal-component-analysis-pca-explained-visually-with-zero-math-1cbf392b9e7d

Implementing the Algorithm

Now that the math is out of the way, we can create an algorithm to implement this. I’ll be splitting the algorithm into three main functions: finding the rank, decomposing/truncating the matrices, and computing the compression error [1]. Upon some trial and error, setting the error_rate to 0.1 was sufficient for the algorithm to find an optimal solution.

Click here to view the code

def svd_decompose (X, k):
    """Decomposes matrix with numpy.linalg.svd() and obtains top k dimensions for U, Sigma, and VT."""
    
    assert k > 0
    Uk, Sk, VTk = svd(X, full_matrices=False)
    
    return (Uk[:,:k], Sk[:k], VTk[:k,:])

def compression_error (Sigma, k):
    """Computes the compression error of the svd_decompose() function."""
    
    return Sigma[k:].dot(Sigma[k:])**0.5 / Sigma.dot(Sigma)**0.5

def find_rank (Sigma, error_rate=0.1):
    """Finds rank based on Sigma and desired error rate."""
    
    for i in range(len(Sigma) - 1):
        k = i + 1
        error = compression_error(Sigma, k)
        if error <= error_rate:
            break
            
    print(f'\n\033[1mCompression Error\033[0m: {error}')
    
    return k

# Getting the full SVD decomposition matrices without reduced dimensionality
U, Sigma, VT = svd_decompose (X, X.shape[1])
print('\n\033[1mShapes of SVD Matrices\033[0m:')
print(U.shape, Sigma.shape, VT.shape)

# Finding optimal rank (k) to use for our analysis
error_rate = 0.1
k = find_rank(Sigma, error_rate=error_rate)
print(f'There are \033[1m{k}\033[0m relevant features at an error rate of \033[1m{error_rate}\033[0m.')

The compression error is ~3.6% at an allowed error rate of 0.1. The optimal rank is 13, which means that we reduced the number of features by about 23%. This shows us that most of the data is independent and that it is most likely not heavily correlated. However, we were still able to reduce the dimensionality with minimal loss.

Now that we have our optimal rank, we can simply decompose the matrix and project the reduced feature space onto our data.

Uk, Sk, VTk = svd_decompose (X, k)
X_compressed = X.dot(VTk.T)
X_compressed.head()

Logistic Regression

With our reduced dataset, we can now start building models. Starting with logistic regression, I will be covering a gradient ascent approach with a learning decay function, and Newton’s Method [2].

Gradient Ascent Approach

The gradient ascent algorithm is based on the first derivative, which generally takes a while to compute with a constant learning rate. However, with a decay function, the model converges at an optimum much faster.

Let’s outline the algorithm we will be using for this method.

For each iteration $t$ from 0 to $T-1$:

1. Update the learning rate $\alpha_t$ using the decay rate:

\[\alpha_t = \alpha \times \beta^t\]

2. Calculate the gradient of the log-likelihood with respect to (\theta):

\[\nabla \log L(\theta^{(t)}) = \mathbf{X}^T (\mathbf{y} - \sigma(\mathbf{X} \theta^{(t)}))\]

3. Perform a gradient ascent step to update $\theta$:

\[\theta^{(t+1)} = \theta^{(t)} + \alpha_t \nabla \log L(\theta^{(t)})\]

4. Calculate the logistic loss (negative log-likelihood):

\[\text{log loss} = -\frac{1}{m} \sum_{i=1}^m \left[ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right]\]

where $ p_i = \sigma(\mathbf{X}_i \theta) $ and $ \sigma(z) $ is the sigmoid function:

\[\sigma(z) = \frac{1}{1 + e^{-z}}\]

The optimal parameter vector $\theta_{\text{optimal}}$ is obtained within $T$ iterations:

\[\theta_{\text{optimal}} = \theta^{(T)}\]

Training the Model

With the math out of the way, we can train the model.

Click here to view the code

start = time.time()

# Setting up our hyperparameters
ALPHA = 0.05        # learning rate
DECAY_RATE = 0.95   # incrementally reduces learning rate
MAX_STEP = 100      # total no. of iterations

# Get the data coordinate matrix, X, and label vector, y
X_train = np.array(X_train)
y_train = np.array(y_train)

# Store guesses of theta, for subsequent analysis
thetas = np.zeros((k, MAX_STEP+1))

# Incrementally going up the gradient function to converge at an optimal theta
losses_ga = []
for t in range(MAX_STEP):
    alpha_t = adjust_learning_rate(ALPHA, t, DECAY_RATE)
    gradient = grad_log_likelihood(thetas[:,t:t+1], y_train, X_train)
    s = alpha_t * gradient
    thetas[:,t+1:t+2] = thetas[:,t:t+1] + s
    losses_ga.append(compute_log_loss(thetas[:,t:t+1], X_train, y_train))

# Storing optimal theta
theta_ga = thetas[:, MAX_STEP:]

end = time.time()
training_time = end - start

# Visualizing training loss
plt.figure(figsize=(10, 6))
plt.plot(losses_ga, label='Log-Loss', color='g')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood Loss')
plt.title('Loss Over Iterations')
plt.legend()
plt.show()

print(f"Training Time: {training_time:.3f} seconds")

Here’s the training loss for this model:

As you can see, the model converges at an optimal answer and fast! This model only took 0.344 seconds to train, with an accuracy of 84.6% on the training data. Here’s the detailed performance of the model:

Newton’s Method

Unlike the gradient ascent approach, which is based on the first derivative, Newton’s Method involves the use of the Hessian matrix, which is the second derivative. In theory, this should be faster than a standard gradient ascent approach without a decay function, but will it be faster than the model from earlier with a decay? Let’s find out!

First, let’s break down the math of what’s going on.

For each iteration $t$ from 0 to $T-1$:

1. Calculate the Hessian matrix $ H(\theta^{(t)}) $:

\[H(\theta^{(t)}) = -\mathbf{X}^T \text{diag}(\sigma(\mathbf{X} \theta^{(t)}) (1 - \sigma(\mathbf{X} \theta^{(t)}))) \mathbf{X}\]

2. Calculate the gradient of the log-likelihood with respect to $\theta$:

\[\nabla \log L(\theta^{(t)}) = \mathbf{X}^T (\mathbf{y} - \sigma(\mathbf{X} \theta^{(t)}))\]

3. Solve for the Newton step $s$ by solving the linear system:

\[H(\theta^{(t)}) s = -\nabla \log L(\theta^{(t)})\]

4. Perform a Newton step to update $\theta$:

\[\theta^{(t+1)} = \theta^{(t)} + s\]

5. Calculate the logistic loss (negative log-likelihood):

\[\text{log loss} = -\frac{1}{m} \sum_{i=1}^m \left[ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right]\]

where $ p_i = \sigma(\mathbf{X}_i \theta) $ and $ \sigma(z) $ is the sigmoid function:

\[\sigma(z) = \frac{1}{1 + e^{-z}}\]

The optimal parameter vector $\theta_{\text{optimal}}$ is obtained after $T$ iterations:

\[\theta_{\text{optimal}} = \theta^{(T)}\]

Training the Model

Let’s assess how this new algorithm performs!

Click here to view the code

start = time.time()

# Setting up our hyperparameters
MAX_STEP = 55      # total no. of iterations

k = X_train.shape[1]
thetas_newt = np.zeros((k, MAX_STEP+1))
losses_newt = []
for t in range(MAX_STEP):
    hessian = hess_log_likelihood(thetas_newt[:,t:t+1], X_train)
    gradient = grad_log_likelihood(thetas_newt[:,t:t+1], y_train, X_train)
    s = solve(hessian, -gradient)
    thetas_newt[:,t+1:t+2] = thetas_newt[:,t:t+1] + s
    losses_newt.append(compute_log_loss(thetas[:,t:t+1], X_train, y_train))
theta_newt = thetas_newt[:, MAX_STEP:]

end = time.time()
training_time = end - start

plt.figure(figsize=(10, 6))
plt.plot(losses_newt, label='Log-Loss', color='g')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood Loss')
plt.title('Loss Over Iterations')
plt.legend()
plt.show()

print(f"Training Time: {training_time:.3f} seconds")

Here’s the training loss for this model:

The model trained in much fewer steps than the gradient ascent approach, but each step took significantly longer with the Hessian computation. Because of this, the model took 0.713 seconds to converge, which is nearly half the speed of the other model. The accuracy is also slightly lower at 84.16%, which is not a significant difference. Here’s a more detailed overview of the performance:

Observations and Results

In conclusion, comparing the logistic regression models trained with gradient ascent and Newton’s method provides useful insights into predicting customer churn. The gradient ascent model converges faster and performs slightly better, though both models have a solid ROC AUC of 0.83. There’s room to reduce false negatives, which is important for keeping customers. Using better features or combining these models with other techniques could help. For the business, these results show that improving predictive models can help identify customers at risk of churning, leading to better retention strategies.

References

[1] Vuduc, Richard, “Compression via the PCA and the SVD.” Class lecture, Computing for Data Analysis, Georgia Institute of Technology, Atlanta, GA. April 19, 2024.

[2] Vuduc, Richard, “Logistic Regression.” Class lecture, Computing for Data Analysis, Georgia Institute of Technology, Atlanta, GA. April 12, 2024.

Forecasting Bitcoin Prices using LSTM Neural Networks

Fri, 31 Mar 2023 00:00:00 +0000

This project explores the capabilities of Long Short-Term Memory (LSTM) neural networks in predicting Bitcoin prices. Having explored day trading (and failing), I aimed to develop an accurate and robust forecasting model that could provide valuable insights into short-term Bitcoin price movements, while having fun in the process. This project inspired my passion for data science, so I hope you enjoy the read!

The results of my project were encouraging. Using 5-fold cross-validation, the Mean Absolute Percentage Error (MAPE), which is a measure of error in a model, was reduced by approximately 18.34% compared to the initial model, reaching a value of 8.57%! This indicated that the LSTM model’s predictions deviated from the actual prices of Bitcoin by an average of 8.57%. Furthermore, the rolling origin cross-validation approach yielded even more impressive results, with the MAPE further decreasing to 3.08%! However, does this mean that the model is actually performing well? We’ll explore the process, results, and implications of this project. Don’t worry if you don’t understand what these models mean or do, I will explain everything in more detail throughout this article.

The article is split into five main sections:

Setting up the Data
Basic Stacked LSTM Model
5-fold Cross-Validation
Rolling Origin Cross-Validation
Conclusion

Setting up the Data

Like any coding project, let’s first import the necessary libraries:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Dropout
from tensorflow.keras.optimizers import Adam
from sklearn.model_selection import TimeSeriesSplit
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.metrics import mean_squared_error, mean_absolute_error
from math import sqrt
import talib
import requests
import requests
import csv
import json

Requesting Data from CryptoCompare

I retrieved Bitcoin price data from CryptoCompare’s API using the requests library. To sign up for a free API key for crypto pricing data, visit: https://www.cryptocompare.com/

The function below fetches data for the past 364 days using the limit parameter and the current time using the toTs parameter. The response is then converted to JSON format, and the relevant data is extracted using the Data key. A pandas DataFrame is then created from the extracted data, with the time column converted to datetime format. Finally, the function returns the DataFrame containing the historical data.

Click here to view the code

def fetch_historical_data():
    url = "https://min-api.cryptocompare.com/data/v2/histoday"
    params = {
        'fsym': 'BTC',
        'tsym': 'USD',
        'limit': 364,
        'toTs': int(pd.Timestamp.utcnow().timestamp()),
        'api_key': API_KEY
    }
    response = requests.get(url, params=params)
    json_data = response.json()
    print(json_data)  # Print json_data to debug

    historical_data = json_data['Data']['Data']
    df = pd.DataFrame(historical_data)
    df['time'] = pd.to_datetime(df['time'], unit='s')
    return df

# Test the function
df = fetch_historical_data()
df.head()

Feature Engineering

The code block below creates technical indicators for Bitcoin’s historical daily price data and then prepares the data for our LSTM model. Technical indicators such as SMA, EMA, RSI, MACD, Bollinger Bands, ATR, Stochastic Oscillator, CCI, OBV, ROC, PSAR, and VWAP (I will explain more about these indicators in the next section) are calculated using the TA-Lib library. Additional price-derived features such as daily return and log return and time-based features such as day of the week are also calculated. Missing values are then dropped, and only the desired features are selected for further analysis. The data is then scaled using the MinMaxScaler, and the scaled data is split into training and testing sets with a 75/25 ratio. The resulting train_data and test_data DataFrames are used for model training and evaluation, respectively.

Click here to view the code

# Technical Indicators
df['SMA'] = talib.SMA(df['close'], timeperiod=14)  # Simple Moving Average (14 days)
df['EMA'] = talib.EMA(df['close'], timeperiod=14)  # Exponential Moving Average (14 days)
df['RSI'] = talib.RSI(df['close'], timeperiod=14)  # Relative Strength Index (14 days)
df['MACD'], df['MACD_signal'], _ = talib.MACD(df['close'], fastperiod=12, slowperiod=26, signalperiod=9)  # Moving Average Convergence Divergence
df['BB_upper'], df['BB_middle'], df['BB_lower'] = talib.BBANDS(df['close'], timeperiod=20)  # Bollinger Bands (20 days)
df['ATR'] = talib.ATR(df['high'], df['low'], df['close'], timeperiod=14)  # Average True Range (14 days)
df['%K'], df['%D'] = talib.STOCH(df['high'], df['low'], df['close'], fastk_period=14, slowk_period=3, slowd_period=3)  # Stochastic Oscillator
df['CCI'] = talib.CCI(df['high'], df['low'], df['close'], timeperiod=14)  # Commodity Channel Index (14 days)
df['OBV'] = talib.OBV(df['close'], df['volumeto'])  # On Balance Volume
df['ROC'] = talib.ROC(df['close'], timeperiod=10)  # Rate of Change (10 days)
df['PSAR'] = talib.SAR(df['high'], df['low'], acceleration=0.02, maximum=0.2)  # Parabolic Stop and Reverse

# Calculate VWAP
typical_price = (df['high'] + df['low'] + df['close']) / 3
cumulative_typical_price_volume = (typical_price * df['volumeto']).cumsum()
cumulative_volume = df['volumeto'].cumsum()

df['VWAP'] = cumulative_typical_price_volume / cumulative_volume

# Price-derived features
df['daily_return'] = df['close'].pct_change()  # Daily return
df['log_return'] = np.log(df['close'] / df['close'].shift(1))  # Log return

# Time-based features
df['day_of_week'] = df['time'].dt.dayofweek  # Day of the week (0: Monday, 1: Tuesday, ..., 6: Sunday)

# Drop missing values
df.dropna(inplace=True)

# Select only the columns that you want to use as features
feature_columns = ['SMA', 'EMA', 'RSI', 'MACD', 'MACD_signal', 
                   'BB_upper', 'BB_middle', 'BB_lower', 'daily_return', 'log_return', 
                   'day_of_week', 'ATR', '%K', 'CCI', 'OBV', 'ROC', 'PSAR', 'VWAP']
numerical_data = df[feature_columns]

# Scale the data
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_numerical_data = scaler.fit_transform(numerical_data)

# Create a new DataFrame with scaled data
scaled_data = pd.DataFrame(scaled_numerical_data, columns=numerical_data.columns, index=numerical_data.index)

# Split the data into training and testing sets
train_data, test_data = scaled_data[:int(len(df) * 0.75)], scaled_data[int(len(df) * 0.75):]

Visualizing the Indicators

Technical indicators are used in investing to analyze past price and volume data of a financial asset and to identify patterns, trends, and potential future price movements. Technical indicators are mathematical calculations based on historical data and are often plotted on charts to provide visual representation. Traders and investors use technical indicators to make informed decisions about buying, selling, or holding a financial asset. Technical indicators can be used for different purposes such as identifying trend reversals, determining market momentum, detecting oversold or overbought conditions, and generating trading signals.

SMA (Simple Moving Average): Represents the average closing price over a specified period, smoothing out short-term price fluctuations and highlighting overall trends. It helps identify potential support or resistance levels and can signal trend reversals.

EMA (Exponential Moving Average): Similar to SMA, but it gives more weight to recent prices, making it more responsive to new information. It is useful for identifying the short-term trend direction and potential entry or exit points.

RSI (Relative Strength Index): Measures the speed and change of price movements, indicating overbought or oversold conditions. It can help in identifying potential trend reversals and price corrections.

MACD (Moving Average Convergence Divergence): Shows the relationship between two moving averages of a security’s price, highlighting potential trend reversals, and providing buy or sell signals.

Bollinger Bands: Consist of a middle band (simple moving average) with an upper and lower band, representing volatility. They can help identify potential breakouts and price targets, as well as overbought or oversold conditions.

ATR (Average True Range): Measures market volatility by considering the range of price movements. A higher ATR indicates higher volatility, while a lower ATR suggests lower volatility. It can help set stop-loss levels and manage risk.

Stochastic Oscillator (%K and %D): Identifies potential trend reversals and overbought or oversold conditions by comparing a security’s closing price to its price range over a specified period.

CCI (Commodity Channel Index): Measures the deviation of a security’s price from its average price, indicating overbought or oversold conditions and potential trend reversals.

OBV (On Balance Volume): Relates price and volume to show the flow of funds into or out of a security, helping identify potential price breakouts or reversals.

ROC (Rate of Change): Measures the percentage change in price over a specified period, indicating momentum and potential trend reversals.

PSAR (Parabolic Stop and Reverse): Provides potential entry and exit points by indicating trend direction and stop levels.

VWAP (Volume Weighted Average Price): Represents the average price at which a security has traded throughout the day, considering both price and volume. It can help identify intraday trends and potential entry or exit points.

Basic Stacked LSTM Model

A stacked LSTM (Long Short-Term Memory) model is a type of neural network that consists of multiple LSTM layers stacked on top of each other. Each LSTM layer learns a different level of abstraction from the input data, and the output of one layer is fed as input to the next layer. Stacked LSTM models are used in time series analysis and prediction — in this case, predicting Bitcoin prices — because they can capture the complex dependencies and patterns in the data over long periods.

The model can be trained on historical data, and once trained, it can be used to make predictions about future prices. The function below splits the training and testing data from earlier between X and y.

Click here to view the code

def create_sequences(df, window_size):
    X, y = [], []
    for i in range(window_size, len(df)):
        X.append(df.iloc[i - window_size:i].values)
        y.append(df.iloc[i, 0])
    return np.array(X), np.array(y)

window_size = 60
X_train, y_train = create_sequences(train_data, window_size)
X_test, y_test = create_sequences(test_data, window_size)

With the data set up, we can build the LSTM model.

The model below consists of three LSTM layers stacked on top of each other, with dropout layers in between to help prevent overfitting. Here’s a brief explanation of each layer in the model:

First LSTM layer: This layer has 50 LSTM units and is set to return sequences because the next layer is also an LSTM layer. It takes the input shape from the training data.
First dropout layer: This layer applies a dropout rate of 20% to the first LSTM layer’s output, randomly setting some of the activations to zero during training to reduce overfitting.
Second LSTM layer: This layer has 50 LSTM units and also returns sequences because the next layer is another LSTM layer.
Second dropout layer: This layer applies a dropout rate of 20% to the second LSTM layer’s output.
Third LSTM layer: This layer has 50 LSTM units and does not return sequences since the next layer is a Dense layer.
Third dropout layer: This layer applies a dropout rate of 20% to the third LSTM layer’s output.
Dense output layer: This layer produces the final output of the model, with a single output unit for the predicted value.

The model is compiled using the Adam optimizer with a learning rate of 0.001 and mean squared error as the loss function. The model is then trained for 50 epochs (or iterations of the LSTM model) with a batch size of 32 using both training and validation data.

Click here to view the code

# Define the LSTM model
model = Sequential()
model.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50))
model.add(Dropout(0.2))
model.add(Dense(units=1))

# Compile and train the model
model.compile(optimizer=Adam(learning_rate=0.001), loss='mean_squared_error')
history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_test, y_test), verbose=1)

Results

To evaluate the performance of the LSTM model, I first calculated the train and test loss by measuring the difference between the predicted and actual values using the model’s evaluate() method. I then used the model to predict Bitcoin prices and transformed the predicted and actual values back to their original scale using the scaler’s inverse_transform() method. To assess the accuracy of the predicted values, I calculated the root mean squared error (RMSE) and mean absolute percentage error (MAPE) compared to the actual values. The RMSE and MAPE are significant as they help to evaluate the model’s performance at predicting future Bitcoin prices, while visualizations aid in understanding the alignment between the predicted and actual values. The plots below show the predicted and actual values for both the training and testing datasets.

It is crucial to highlight the significance of the difference between the model’s fit on the training and testing data. This will help us to identify areas of improvement to enhance the model’s predictive capabilities for Bitcoin prices. The training MAPE is 26.91% and the testing MAPE is 2.74%.

In the following sections, we will explore ways to further improve the model’s performance and accuracy.

5-Fold Cross-Validation

Generally in 5-fold cross-validation, the dataset is divided into 5 equal parts or “folds”. The model is trained on 4 folds, using them as the training set, and the remaining fold is used as the validation set. This process is repeated 5 times, with each fold being used as the validation set once. The model’s performance is averaged over the 5 validation sets to obtain a single performance metric. This method helps assess the model’s ability to generalize to unseen data. This approach has many strengths, but it also comes with its fair share of limitations.

Strengths:

Provides a more robust estimate of model performance than a single train-test split.
Reduces the risk of overfitting by averaging performance over multiple validation sets.
Provides a good indication of how well the model will perform on new, unseen data.
Limitations:
May not be suitable for time series data with strong temporal dependencies, as shuffling the data may lead to leakage of information from the future into the training set.
Can be computationally expensive, especially for large datasets or complex models, as the model has to be trained multiple times.

5-fold cross-validation may provide a more robust estimate of model performance, but I did not get my hopes too high because the data might have significant temporal dependencies.

Developing the Model

This code implements a 5-fold Time Series Cross-Validation technique using the TimeSeriesSplit function from sklearn.model_selection. In this approach, the dataset is split into 5 non-overlapping time-ordered folds. For each fold, the model is trained on all previous folds and validated on the current fold. This process is repeated 5 times, with each fold serving as the validation set once.

The lstm_5fold_cross_validation function takes in the input data X, target variable y, number of splits n_splits, and a scaler object scaler. Inside the function, a TimeSeriesSplit object is created with the specified number of splits. The function then iterates through each train and test index pair generated by the TimeSeriesSplit object, trains the LSTM model, and calculates the RMSE and MAPE values for each fold.

Finally, the function returns the RMSE and MAPE values, as well as the unscaled predictions and true values for each fold. The code uses the 5-fold Time Series Cross-Validation technique, which is different from the rolling origin cross-validation technique mentioned in your previous question.

Click here to view the code

def create_lstm_model(input_shape):
    model = Sequential()
    model.add(LSTM(units=50, return_sequences=True, input_shape=input_shape))
    model.add(Dropout(0.2))
    model.add(LSTM(units=50, return_sequences=True))
    model.add(Dropout(0.2))
    model.add(LSTM(units=50))
    model.add(Dropout(0.2))
    model.add(Dense(units=1))

    model.compile(optimizer=Adam(learning_rate=0.001), loss='mean_squared_error')
    return model

def lstm_5fold_cross_validation(X, y, n_splits=5, scaler=None):
    tscv = TimeSeriesSplit(n_splits=n_splits)
    rmse_values, mape_values, accuracy_values = [], [], []
    predictions, true_values = [], []  # Add lists to store predictions and true_values

    # Define early stopping
    early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

    for train_index, test_index in tscv.split(X):
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]

        # Create and train the model
        model = create_lstm_model((X_train.shape[1], X_train.shape[2]))
        model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_test, y_test), verbose=0, callbacks=[early_stopping])

        # Make predictions
        y_test_pred = model.predict(X_test)

        # Unscale the predictions and true values
        y_test_pred_reshaped = y_test_pred.reshape(-1, 1)
        y_test_reshaped = y_test.reshape(-1, 1)
        dummy_test = np.zeros((y_test_pred_reshaped.shape[0], X_test.shape[2] - 1))

        y_test_pred_unscaled = scaler.inverse_transform(np.hstack((y_test_pred_reshaped, dummy_test)))[:, 0]
        y_test_unscaled = scaler.inverse_transform(np.hstack((y_test_reshaped, dummy_test)))[:, 0]

        predictions.append(y_test_pred _unscaled)  # Append unscaled predictions
        true_values.append(y_test_unscaled)  # Append unscaled true_values

        # Calculate the RMSE
        rmse = sqrt(mean_squared_error(y_test_unscaled, y_test_pred_unscaled))
        rmse_values.append(rmse)

        # Calculate the MAPE
        mape = np.mean(np.abs((y_test_unscaled - y_test_pred_unscaled) / y_test_unscaled)) * 100
        mape_values.append(mape)

    return rmse_values, mape_values, predictions, true_values  # Return predictions and true_values

# Prepare the data using the create_sequences function
window_size = 60
X, y = create_sequences(scaled_data, window_size)

# Call the lstm_cross_validation function
rmse_5f_values, mape_5f_values, predictions_5f, true_values_5f = lstm_5fold_cross_validation(X, y, n_splits=5, scaler=scaler)

Visualizing the Results

This visualization shows the results of the 5-fold cross-validation on the LSTM model. The plot displays both the predicted and actual values of Bitcoin prices along a time series. The purpose of this visualization is to compare the predicted values with the actual values to see how well the model performs in predicting Bitcoin prices. The plot also helps to identify any patterns or trends in the data that the model may have missed. Overall, the model seems to be doing a much better job at predicting Bitcoin prices than the basic stacked LSTM model.

The MAPE has been reduced by approximately 18.34%, which highlights the effectiveness of employing cross-validation techniques on regression models. A MAPE of 8.57% after cross-validation signifies that the LSTM model’s predictions deviate from the actual values by an average of 8.57%. This relatively low error rate implies that the LSTM model is proficient at capturing the underlying dynamics of the time series data.

The improved accuracy can offer valuable insights for investors, enabling them to make more informed decisions and better manage risk in the financial markets. By utilizing a cross-validated LSTM model, investors can gain a deeper understanding of the factors influencing price movements and identify potential trends or market anomalies. Consequently, this enhanced predictive capability can lead to more strategic investment planning and potentially higher returns.

However, it is crucial to recognize that the model’s performance is not flawless, and the inherent uncertainty in financial markets remains. The 8.57% error rate, while relatively low, indicates that predictions should be treated with caution and not solely relied upon for decision-making. Combining the model’s insights with other market indicators and fundamental analysis can help investors create a more comprehensive and balanced investment strategy.

Rolling Origin Cross-Validation

Now we’re getting into some serious business! Rolling origin cross-validation, also known as time series cross-validation, is specifically designed for time series data. In this method, the dataset is split into a training and validation set, with the validation set starting at a certain point in time. The model is trained on the initial training set and validated on the validation set. The process is then repeated by rolling the starting point of the validation set forward in time, expanding the training set to include the previous validation set, and using a new validation set. This ensures that the training set always precedes the validation set in time, preserving the temporal structure of the data.

Here’s a visual representation of the rolling origin cross-validation approach:

Each row represents an iteration of the model, with the green and orange dots being training and testing data points, respectively (the white dots are data points that are not included in the iteration). This incremental structure has strengths and weaknesses as outlined below.

Strengths:

Preserves the temporal structure of the data, making it suitable for time series forecasting.
Provides a realistic estimate of the model’s performance on new, unseen data by validating data points that come after the training data.
Reduces the risk of data leakage due to shuffling the dataset.

Limitations:

Can be computationally expensive, as the model needs to be retrained multiple times.
May not work well for datasets with strong seasonality or recurring patterns, as the model may not be exposed to the full range of these patterns during training.

Rolling origin cross-validation preserves the temporal structure, making it more suitable for time series forecasting but may not capture seasonal patterns effectively. Additionally, the predictions from this model will only capture the last 50 data points because of its computational extensivity.

Developing the Model

This code implements a variation of the rolling origin cross-validation technique for time series data. The lstm_cross_validation function iterates through the time series dataset, creating a rolling window for training and testing the LSTM model. At each step, the training set expands by one observation, and the test set consists of a single observation following the training set.

However, this implementation differs from a standard Time Series Cross-Validation in one aspect. In a typical rolling origin cross-validation, you would divide the data into K-folds and use the first K-1 folds for training and the Kth fold for testing, then move the window forward. The current implementation does not use a fixed number of K-folds but rather moves the window forward one step at a time until the end of the dataset (similar to the image in the previous section, but only using one data point for testing at a time instead of three). This leads to a much larger number of iterations, as the model is tested on every single observation after the initial training set.

The code does not use TimeSeriesSplit from sklearn.model_selection, which is commonly used to create K-folds in rolling origin cross-validation. Instead, it uses a custom for loop to create the rolling window.

Click here to view the code

def create_lstm_model(input_shape):
    model = Sequential()
    model.add(LSTM(units=50, return_sequences=True, input_shape=input_shape))
    model.add(Dropout(0.2))
    model.add(LSTM(units=50, return_sequences=True))
    model.add(Dropout(0.2))
    model.add(LSTM(units=50))
    model.add(Dropout(0.2))
    model.add(Dense(units=1))

    model.compile(optimizer=Adam(learning_rate=0.001), loss='mean_squared_error')
    return model

def lstm_cross_validation(X, y, test_size, scaler=None):
    rmse_values, mape_values, predictions, true_values = [], [], [], []

    # Define early stopping
    early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

    train_size = X.shape[0] - test_size
    for i in range(train_size, X.shape[0]):
        X_train, X_test = X[:i], X[i:i+1]
        y_train, y_test = y[:i], y[i:i+1]

        # Create and train the model
        model = create_lstm_model((X_train.shape[1], X_train.shape[2]))
        model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_test, y_test), verbose=0, callbacks=[early_stopping])

        # Make predictions
        y_test_pred = model.predict(X_test)

        # Unscale the predictions and true values
        y_test_pred_reshaped = y_test_pred.reshape(-1, 1)
        y_test_reshaped = y_test.reshape(-1, 1)
        dummy_test = np.zeros((y_test_pred_reshaped.shape[0], X_test.shape[2] - 1))

        y_test_pred_unscaled = scaler.inverse_transform(np.hstack((y_test_pred_reshaped, dummy_test)))[:, 0]
        y_test_unscaled = scaler.inverse_transform(np.hstack((y_test_reshaped, dummy_test)))[:, 0]

        predictions.append(y_test_pred_unscaled)
        true_values.append(y_test_unscaled)

        # Calculate the RMSE
        rmse = sqrt(mean_squared_error(y_test_unscaled, y_test_pred_unscaled))
        rmse_values.append(rmse)

        # Calculate the MAPE
        mape = np.mean(np.abs((y_test_unscaled - y_test_pred_unscaled) / y_test_unscaled)) * 100
        mape_values.append(mape)

    return rmse_values, mape_values, predictions, true_values

# Prepare the data using the create_sequences function
window_size = 60
X, y = create_sequences(scaled_data, window_size)

# Determine the test size (e.g., 20% of the dataset)
test_size = int(len(X) * 0.2)

# Call the lstm_cross_validation function
rmse_values, mape_values, predictions, true_values = lstm_cross_validation(X, y, test_size, scaler=scaler)

Visualizing the Results

The visualization below displays the results of the rolling origin cross-validation on the LSTM model. Just like the previous plot, this plot also displays both the predicted and actual values of Bitcoin prices along a time series. The purpose of this visualization is to compare the predicted values with the actual values to see if there was any improvement in the model’s performance/fitting. Since this cross-validation approach only covers the past 50 days, there are only predictions for this time frame. However, it appears that the model is performing extremely well!

The MAPE has been significantly reduced to 3.08% after implementing the rolling origin cross-validation technique, which emphasizes the effectiveness of this method in refining regression models. A MAPE of 3.08% signifies that the LSTM model’s predictions deviate from the actual values by an average of only 3.08%. This remarkably low error rate indicates that the LSTM model is highly adept at capturing the underlying dynamics of the time series data.

The enhanced accuracy can provide valuable insights for investors, allowing them to make more informed decisions and manage risk more effectively in the financial markets. By using a rolling origin cross-validated LSTM model, investors can gain a more profound understanding of the factors driving price movements and identify potential trends or market anomalies. As a result, this improved predictive capability can contribute to more strategic investment planning and potentially higher returns.

It is essential, however, to recognize that the model’s performance is not perfect, and the inherent uncertainty in financial markets still prevails. The 3.08% error rate, while impressively low, suggests that predictions should be approached with prudence and not solely relied upon for decision-making. Integrating the model’s insights with other market indicators and fundamental analysis can help investors develop a more comprehensive and well-rounded investment strategy, further mitigating potential risks.

Conclusion

In conclusion, the LSTM project we have explored throughout this article demonstrates the power and versatility of deep learning techniques for time series forecasting in financial markets. By leveraging a diverse set of technical indicators, price-derived features, and time-based features, the LSTM model — paired with cross-validation techniques — captures complex patterns and relationships in the historical data, offering improved predictive accuracy for future price movements.

This LSTM-based approach can be applied to various use-cases, including algorithmic trading, portfolio management, and risk management. For algorithmic trading, traders can use the model’s predictions to design and execute trading strategies, identifying potential entry and exit points in the market. Portfolio managers can benefit from the model’s insights to make informed decisions about asset allocation and rebalancing, optimizing their portfolio’s risk-adjusted returns. In the realm of risk management, financial institutions can utilize the LSTM model to forecast potential price fluctuations, allowing them to set appropriate risk limits and manage their exposure effectively.

Moreover, the LSTM model can be extended to include additional features or data sources, such as sentiment analysis from news articles or social media, improving its predictive capabilities further. It can also be adapted to forecast various financial instruments, including stocks, commodities, currencies, and cryptocurrencies, highlighting its versatility across different asset classes.

Overall, this LSTM project showcases the potential of deep learning in financial market forecasting, opening up new opportunities for investors, traders, and financial institutions to make data-driven decisions and stay ahead in the ever-evolving world of finance. As a reader, you are now better equipped to understand and explore the potential of LSTM models in finance, and hopefully you are inspired to take this project a step further with your own flair! By staying informed about these cutting-edge technologies, you can actively participate in the financial market’s ongoing transformation and seize new opportunities that arise from advancements in predictive analytics!

References

https://www.elearnmarkets.com/blog/best-25-technical-indicators/

https://machinelearningmastery.com/stacked-long-short-term-memory-networks/

https://stats.stackexchange.com/questions/564311/how-to-cross-validate-a-time-series-lstm-model

Chaitanya's Projects

Navigating the Rent-versus-Buy Decision with Deep Learning

Contents

Data Sources

Data Processing

Model Architecture

Visualization Dashboards

Evaluation Metrics and Results

Conclusion and Future Work

References

Advancing sEMG Hand Signal Classification

Contents

Problem Statement

Data Source

Methodology

1. Experimental Setup

2. Windowing Approach

a. Segmentation into Windows

b. Frequency-Domain Transformation (FFT)

c. CNN for Frequency-Domain Feature Extraction

d. Deep Cross Network (DCN)

e. Multi-Layer Perceptron (MLP)

f. Model Architecture Overview

3. Non-Windowing Approach

a. Raw Feature Input

b. Random Forest Classifier

c. Outer-Product Neural Network (OPNN)

4. Loss & Optimization

Evaluation Technique

Results

Conclusion & Future Work

References

Clustering Subreddit Communities

Contents

Loading the Data

Preprocessing the Data

Constructing the Graph Laplacian

Implementing the Algorithm

K-Means Algorithm

Implementing K-Means for Spectral Clustering

Results and Observations

References

Classifying Emotions from Videos

Contents

Loading the Data

Downloading the Data

Processing Each Video

Processing Each Zip File

Video Feature Extraction

Breakdown of Custom 3D Histogram of Oriented Gradients for Dimensionality Reduction

Implementing the Algorithm

Visualizing Gradient Magnitude, Azimuthal Angle, and Polar Angle

Creating our HOG3D Dataset

Audio Feature Extraction

Breakdown of Feature Engineering and Extraction Process

Implementing the algorithm

Emotion Classification

Partial Tucker Decomposition for Additional Dimensionality Reduction

Implementing the algorithm

Training a Multilayer Perceptron Model using Sklearn

Conclusion

References:

Facial Recognition

Contents

Loading the Data

Feature Extraction using HODA

Implementing the HODA Algorithm

Fitting the Model

Conclusion

References

Understanding Customer Churn

Contents

Understanding the Data

Setting up our Analysis

Exploratory Data Visualization

Preprocessing Data

PCA for Dimensionality Reduction

Implementing the Algorithm

Logistic Regression

Gradient Ascent Approach