Visualizing Deep Learning

I personally find Deep Learning and Neural Networks fascinating. The goal of this visualization is to demonstrate the actual operations behind a Neural Network.
As you adjust the iteration slider, the page will display every operation that goes into executing a Neural Network.

This network takes an input of 3 numbers. Each number is either 1 or 0. The goal of the network is to find a pattern between the first and second number being 1 and 0 in either order, but not both 0 or 1. (Exclusive OR)
For example, 0-0-0, 1-1-0, 0-0-1 should return 0, and 1-0-1, 0-1-1 should return 1.
The Final Error field and Error graph will display how far off the network is from its goal. The goal is to have an error close to 0.

The defaulted iteration count is 10 to prevent the page from executing too many calculations on first load. You should be able to get accurate results by setting the iteration count to 10000.

I've always found that interactive models like this help me learn the best! Hope you find this helpful as well.

Code for the network can be found here: MultiLayer.ts
For a simpler network see here: SingleLayer.ts
All of the network math is contained within the project as well and can be found here: MatrixUtil.ts

The network structure was inspired by the following blog post: Basic Python Network

Brief Function Guide:

Sigmoid:

s(x) = \frac{1}{1 + e^{-x}}

Sigmoid's First Derivative:

s'(x) = \frac{ds(x)}{dx} = s(x)(1 - s(x))

Matrix Element-Wise Operations

This applies to all basic arithmetic operations

x11	x12
x21	x22

y11	y12
y21	y22

x11 + y11	x12 + y12
x21 + y21	x22 + y22

Matrix Multiplication

x11	x12	x13
x21	x22	x23

y11	y12
y21	y22
y31	y32

x11y11 + x12y21 + x13*y31	x11y12 + x12y22 + x13*y32
x21y11 + x22y21 + x23*y31	x21y12 + x22y22 + x23*y32

Matrix Scalar

x11	x12
x21	x22

a*x11	a*x12
a*x21	a*x22

Matrix Transpose

x11	x12	x13
x21	x22	x23

x11	x21
x12	x22
x21	x23

Matrix Element-Wise Apply

x11	x12
x21	x22

f(x11)	f(x12)
f(x21)	f(x22)

Interactive Network:

Hidden Neuron Count:

Learning Rate:

Iteration Count:

Hidden Neuron Count: 4

Learning Rate: 0.1

Iteration Count: 10

Final Error: 0.5067021977843195

Iteration (of 10): 10

Forward Pass:

L1 Weighted Sum

-0.08756	0.819148	-0.91416	0.092082
0.486604	0.266100	-1.72065	0.046920
-0.34682	0.766748	-0.45994	0.965229
0.227347	0.213700	-1.26644	0.920067

Inputs

0	0	1
0	1	1
1	0	1
1	1	1

L1 Weights

-0.25925	-0.05240	0.454217	0.873146
0.574170	-0.55304	-0.80649	-0.04516
-0.08756	0.819148	-0.91416	0.092082

L1 Output

0.478122	0.694055	0.286148	0.523004
0.619306	0.566135	0.151786	0.511727
0.414153	0.682817	0.386998	0.724167
0.556593	0.553222	0.219866	0.715055

= S(

L1 Weighted Sum

-0.08756	0.819148	-0.91416	0.092082
0.486604	0.266100	-1.72065	0.046920
-0.34682	0.766748	-0.45994	0.965229
0.227347	0.213700	-1.26644	0.920067

)

L2 Weighted Sum

0.065925

0.196956

-0.25857

-0.10310

Inputs (L1 Output)

0.478122	0.694055	0.286148	0.523004
0.619306	0.566135	0.151786	0.511727
0.414153	0.682817	0.386998	0.724167
0.556593	0.553222	0.219866	0.715055

L2 Weights

0.685648

0.673746

-0.81662

-0.94806

L2 Output (Final Result)

0.516475

0.549080

0.435713

0.474246

= S(

L2 Weighted Sum

0.065925

0.196956

-0.25857

-0.10310

)

Back Propagation Pass:

L2 Error

-0.51647

0.450919

0.564286

-0.47424

Target Output

L2 Output

0.516475

0.549080

0.435713

0.474246

L2 Gradients

0.249728

0.247591

0.245867

0.249336

= S'(

L2 Weighted Sum

0.065925

0.196956

-0.25857

-0.10310

)

L2 Delta

-0.12897

0.111643

0.138739

-0.11824

L2 Gradients

0.249728

0.247591

0.245867

0.249336

L2 Error

-0.51647

0.450919

0.564286

-0.47424

L1 Error

-0.08843	-0.08689	0.105327	0.122279
0.076548	0.075219	-0.09117	-0.10584
0.095126	0.093475	-0.11329	-0.13153
-0.08107	-0.07966	0.096563	0.112105

L2 Delta

-0.12897

0.111643

0.138739

-0.11824

·T(

L2 Weights

0.685648

0.673746

-0.81662

-0.94806

)

L1 Gradients

0.249521	0.212342	0.204267	0.249470
0.235766	0.245626	0.128747	0.249862
0.242630	0.216577	0.237230	0.199748
0.246797	0.247167	0.171525	0.203750

= S'(

L1 Hidden Sum

-0.08756	0.819148	-0.91416	0.092082
0.486604	0.266100	-1.72065	0.046920
-0.34682	0.766748	-0.45994	0.965229
0.227347	0.213700	-1.26644	0.920067

)

L1 Delta

-0.02206	-0.01845	0.021514	0.030505
0.018047	0.018475	-0.01173	-0.02644
0.023080	0.020244	-0.02687	-0.02627
-0.02000	-0.01969	0.016563	0.022841

L1 Gradients

0.249521	0.212342	0.204267	0.249470
0.235766	0.245626	0.128747	0.249862
0.242630	0.216577	0.237230	0.199748
0.246797	0.247167	0.171525	0.203750

L1 Error

-0.08843	-0.08689	0.105327	0.122279
0.076548	0.075219	-0.09117	-0.10584
0.095126	0.093475	-0.11329	-0.13153
-0.08107	-0.07966	0.096563	0.112105

L1 Weight Updates

0.000307	0.000055	-0.00103	-0.00034
-0.00019	-0.00012	0.000482	-0.00036
-0.00009	0.000057	-0.00005	0.000062

= 0.1 · (T(

Inputs

0	0	1
0	1	1
1	0	1
1	1	1

)·

L1 Delta

-0.02206	-0.01845	0.021514	0.030505
0.018047	0.018475	-0.01173	-0.02644
0.023080	0.020244	-0.02687	-0.02627
-0.02000	-0.01969	0.016563	0.022841

)

L2 Weight Updates

-0.00008

0.000300

0.000773

0.000559

= 0.1 · (T(

L1 Output

0.478122	0.694055	0.286148	0.523004
0.619306	0.566135	0.151786	0.511727
0.414153	0.682817	0.386998	0.724167
0.556593	0.553222	0.219866	0.715055

)·

L2 Delta

-0.12897

0.111643

0.138739

-0.11824

)

New L1 Weights

-0.25894	-0.05234	0.453185	0.872803
0.573974	-0.55316	-0.80601	-0.04552
-0.08766	0.819205	-0.91421	0.092145

L1 Delta

-0.25925	-0.05240	0.454217	0.873146
0.574170	-0.55304	-0.80649	-0.04516
-0.08756	0.819148	-0.91416	0.092082

L1 Weight Updates

0.000307	0.000055	-0.00103	-0.00034
-0.00019	-0.00012	0.000482	-0.00036
-0.00009	0.000057	-0.00005	0.000062

New L2 Weights

0.685560

0.674046

-0.81585

-0.94750

L2 Delta

0.685648

0.673746

-0.81662

-0.94806

L2 Weight Updates

-0.00008

0.000300

0.000773

0.000559