**Demonstration Video:**

We will be partnering up with TechInDeep to bring you more Deep Learning Tutorials.

Before continuing with this post make sure to check out the ML.NET tutorial series:

Learn how to build neural networks from scratch:

**Source Code Download:**

**Programming Language:**

- C#

**What libraries we are going to use:**

- We are not going to use any third party library.

**Who is this article for:**

- This article is for developers who would like to build their own neural network from scratch.

**Additional Questions:**

- You should post all questions here.

**Workflow:**

- First we will implement basic linear algebra operations
- Then implement feed forward logic
- Finally implement backpropagation algorithm
- Now we will be ready to implement basic XOR example

So let’s get started…

## Matrix Math

Since we are going to implement the neural network algorithm from scratch, we would need basic linear algebra library. We can either download optimized one, or write it ourselves. In the download project I am including one very basic linear algebra library for you. All matrix operations are under the *MatrixLibrary* project.

In order to succeed at writing your first neural network, you will need to brush up on some basic linear algebra. Just make sure you are clear on matrix operations such as:

- Adding & subtracting matrices
- Multiplying matrices by scalars
- Multiplying matrices by matrices
- Transpose of a matrix

If you are familiar with these concepts then you are ready for the next section. Remember all matrix related code, and it’s implementation, can be found in *MatrixLibrary *project.

## The Neural Network

In the sample project I am providing the neural network consists of: Input Layer -> Hidden Layer -> Output Layer as presented in the image

What we see here is that we have 2 inputs (X1 and X2), a hidden layer with 2 neurons (a1, a2) and an output layer that consists of only one neuron. On the image above, we can see the weights. They are located between the input layer and hidden layer, and the other weights are located between the hidden layer and the output layer. So if you do the math and do a *FeedForward *pass, you can see that the chosen weights between the layers are working exactly like they should. Once our neural network is trained, the weights between the layers should have the same effect.

## **Initializing the network**

First we need to initialize our network weights to some random numbers. This is OK, because the backpropagation algorithm will update (change) the weights in order to “learn” the XOR operation. The weight initialization process requires it’s own article and explanation, but at this point, for this solution we just want to set up some random values. Let’s look at the code:

weights_0_1[i, j] = objRandom.NextDouble(); for the weights between the input layer and hidden layer and

weights_1_2[i, j] = objRandom.NextDouble(); for the weights between the hidden layer and output layer

## **Training the neural network**

So now we have our weights initialized. But if we calculate the feedforward math, we will see that the result is not even close to what an XOR network should output. So let’s fix this.

In order to fix it, we need to train the network. We will train the network using the backpropagation algorithm. It all starts by deciding the value for our first two variables:

*Epochs*which represents how many times we do: one forward pass and one backward pass of*all*training samples*Learning Rate*which is defined in the context of optimization, and minimizing the loss function of a neural network

*We will cover all parameters in detail in some future post…*

These variables are represented in the code as:

double *learning_rate* = 0.5;

int *epochs* = 2000;

Before updating the network weights, we first need to implement the so called *ForwardPass.*

## **FeedForward pass**

These calculations are fairly simple. All we need to do is multiply two matrices and pass the result through *activation function*. So let’s see how to do this:

Matrix Layer_0 = Matrix.CreateRowMatrix(input); – we create a *row matrix *from a double array.

Matrix Layer_1 = MatrixMath.Sigmoid(MatrixMath.Multiply(Layer_0, weights_0_1));

Here we can see that the first operation we execute is a matrix multiplication (Layer_0 with weights_0_1). The result of the operation (matrix) is passed down to an *activation function.* We do the same for the next layer.

Matrix Layer_2 = MatrixMath.Sigmoid(MatrixMath.Multiply(Layer_1, weights_1_2));

*Matrix.CreateRowMatrix(doub**le**[]** input)* – function to convert a vector (double) array to a row matrix.

*MatrixMath.Multiply(Matrix a, Matrix b)* – function to multiply two matrices

**Activation function**

This topic will be discussed in detail in some future post. For now all we need to know about activation function is that is needed to do a complex non-linear mappings between the inputs and response variable. For this example I chose to use the Sigmoid activation function.

## **Backpropagation algorithm**

In order to make the neural network learn the XOR operation we need to propagate the error back to the layer weights. In order to do that we simply calculate the error between the *predicted value* and the *desired value*.

*Predicted value* is the value that our neural network outputs

*Desired value *is the value we want the neural network to output

Let’s see how off our network is. Calculating the error is very easy. We use the following equation:

Or using C# code:

`Matrix error = MatrixMath.Pow(MatrixMath.Subtract(Layer_2, desired_output_matrix), 2);`

- Layer_2 is the predicted value that we get as a result of performing the Feedforward pass
- desired_output_matrix is the desired value, the value that we want our network to predict

```
Matrix Layer_2_delta = MatrixMath.ElementWiseMultiplication(MatrixMath.Subtract(Layer_2, desired_output_matrix),
MatrixMath.SigmoidDerivative(Layer_2));
```

```
Matrix Layer_1_delta = MatrixMath.ElementWiseMultiplication(
MatrixMath.Multiply(Layer_2_delta, MatrixMath.Transpose(weights_1_2)),
MatrixMath.SigmoidDerivative(Layer_1));
```

Calculating the Layer_1 and Layer_2 delta values. We basically calculate this for each layer except the output layer. This is where we use derivative of our activation function *Sigmoid,* to see how much we need to change the weights of the neural network. *This explanation is too simple but we will have a detailed post on this subject as well…*

```
Matrix weights_1_2_delta = MatrixMath.Multiply(MatrixMath.Transpose(Layer_1), Layer_2_delta);
Matrix weights_0_1_delta = MatrixMath.Multiply(MatrixMath.Transpose(Layer_0), Layer_1_delta);
```

Remember we need the change (delta) in the weights for each of our layer. So in this step we calculate the delta (change) for the weights.

Next we update our weights using the following code:

```
weights_0_1 = MatrixMath.ElementWiseSubtraction(weights_0_1, MatrixMath.Multiply(weights_0_1_delta, learning_rate));
weights_1_2 = MatrixMath.ElementWiseSubtraction(weights_1_2, MatrixMath.Multiply(weights_1_2_delta, learning_rate));
```

Instead of having a hard change in the weights, we control it using the learning rate we introduced earlier. So we multiply our learning rate with the weight delta matrix.

*The mathematics and the detailed description will be provided very soon…*

**Complete Source Code: **C# Neural Network

Learn how to build neural networks from scratch:

## Roman Golubin

February 4, 2020 at 12:05 pmHi!

Xor problem is not a problem. Try this:

public static Matrix Mod(Matrix input)

…

resultMatrix[i, j] = input[i, j] % 1;

…

public static Matrix ModDerivative(Matrix input)

…

resultMatrix[i, j] = 1;

…

and training with one neuron only ))

## vanco

February 8, 2020 at 8:03 amHi,

I am solving this problem with Neural Network and two neurons, because I like to show how Neural Networks can be used to solve non-linear problems. XOR is the perfect example for that. Other than that, great suggestion. Thank you for writing it up.

## Darren

November 6, 2020 at 12:08 amThanks for the explanation and the source code. I was really interested to see this working, but when I downloaded and ran the code, the result did not solve the xor problem… in all four scenarios it returns around .80 to .99 in the output. I notice that the downloaded code is not the same as the code in your video… is it perhaps not the final version?

## vanco

November 7, 2020 at 7:48 amHi,

I lost the original code so I had to re-do it. Anyways I am trying to put out a detailed tutorials on Neural Networks in C# that may help you.

Neural Networks Part 1: https://code-ai.mk/neural-networks-making-predictions/

Neural Networks Part 2: https://code-ai.mk/neural-network-making-predictions-part-2/

## Darren

November 9, 2020 at 3:45 amHi,

I left a message earlier saying that the code was not working…. It is actually working very well. I made a mistake copying and pasting code into my project.

Thankyou very much for this example. It has been facinating learning about this technique.

## andreas schroeder

January 10, 2021 at 11:47 amHi,

nice illustrative and well working example. Just what I need to learn and experiment. Logic Gates are a great topic, as they are easy to understand, evaluate the correctness etc. Sure no Neural Network is really required, but this is not a valid application of a NN but a learning sandbox.

I tried different c’# Neural Network examples, and this worked best. With Matrix calculation it is not too slow and still one can see what it does.

Thanks so much.

## vanco

January 10, 2021 at 7:02 pmHi,

So although XOR example is a good for starters, there is one other reason why I decided to solve it with Neural Network. And that is because XOR is not linearly separable. You can plot the following:

0 0 -> 0

1 0 -> 1

0 1 -> 1

1 1 -> 0

There is no way how you can linearly separate this plot. So even using a simple Neural Network with a single neuron will not be able to solve it. And that’s because you would get the following: y=ax+c which is an equation for a straight line. So what you want to do is introduce some more flexibility by adding additional neuron. If possible try to see how a Neural Network visually solves this problem. I think it will be even more interesting.

Thank you for the comment.