Neural networks are a type of machine learning algorithm inspired by the structure and function of the human brain. They are made up of interconnected nodes, or \u201cneurons,\u201d that can learn to perform specific tasks by analyzing large amounts of data.<\/p>\n\n\n\n
Neural networks have many applications, including image recognition, natural language processing, speech recognition, and predictive analytics. They excel at identifying patterns and making complex decisions, making them a valuable tool in many industries.<\/p>\n\n\n\n
Neural networks are highly flexible and can adapt to a variety of problems. They can learn from data and improve their performance over time, making them powerful tools for tackling complex, real-world challenges.<\/p>\n\n\n\n
2. Task, model, performance measurement, and experience<\/strong><\/h2>\n\n\n\nTo build an effective neural network, you need to consider several key components that define its development and success. These include clearly identifying the task the network will perform, selecting an appropriate model, establishing performance measurement criteria, and leveraging experience through training data. <\/p>\n\n\n\n
Each element plays a crucial role in shaping how well the neural network learns and generalizes to new data. Here\u2019s what each element does and why they\u2019re important:<\/p>\n\n\n\n
\n- Task – <\/strong>The first step in building a neural network is to define the task you want it to perform. This could be anything from image classification to natural language processing. Defining the task helps you design the appropriate network architecture and choose the correct training data.<\/li>\n\n\n\n
- Model – <\/strong>Model selection is an important aspect of building a neural network. Different types of neural network models (like feedforward neural networks, convolutional neural networks, and recurrent neural networks) are suited for different tasks. Choosing the right model improves the network’s performance and ability to solve the desired problem.<\/li>\n\n\n\n
- Performance measurement<\/strong> – Once you’ve defined the task, you need to determine how to measure your neural network\u2019s performance of your neural network. Metrics can include things like accuracy, precision, recall, or F1-score. Depending on the specific problem you’re trying to solve.<\/li>\n\n\n\n
- Experience<\/strong> – The final step is to provide the neural network with training data, allowing it to learn and improve its performance over time. This experience phase is crucial for building a high-performing model that can generalize well to new, unseen data.<\/li>\n<\/ul>\n\n\n\n
In this post, we\u2019re looking at the first artificial neuron model, so we\u2019ll define the task as a linear regression problem. <\/p>\n\n\n\n
The linear regression problem is when we use the artificial neuron to represent functions. Instead of using a function like y = mx + b, we present the data to the artificial neuron and let it discover the appropriate values for the coefficients m<\/em> and b <\/em>that represent the problem. We will return to this in Section 4.<\/p>\n\n\n\nIn the next section, we will describe artificial neuron modeling. Don\u2019t be scared of the math \u2014 we won\u2019t go too deep into it, promise!<\/p>\n\n\n\n
3. The First representation of an artificial neuron (Perceptron)<\/strong><\/h2>\n\n\n\nThe foundation of modern neural networks can be traced back to the earliest mathematical representation of an artificial neuron, known as the Perceptron<\/a>. First introduced by Warren McCulloch and Walter Pitts in 1943, this simple yet powerful model mimics how biological neurons process information. By taking weighted inputs, applying a bias, and passing the result through an activation function, the perceptron is the fundamental building block for more advanced neural network architectures. <\/p>\n\n\n\nThis section explores the key components of the perceptron and how they contribute to its ability to make decisions. <\/p>\n\n\n\n
Equation 1 is the mathematical model of the artificial neuron:<\/p>\n\n\n\n![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/03\/formula1.png\")
<\/figure>\n\n\n\nWhere: <\/p>\n\n\n\n
y<\/em> is the output,<\/p>\n\n\n\nphi is the activation function,<\/p>\n\n\n\n
X<\/em> is the vector containing all the inputs,<\/p>\n\n\n\nW<\/em> is the vector containing all the weights,<\/p>\n\n\n\nb<\/em> is the bias<\/p>\n\n\n\nFrom linear algebra, we can describe the vector product as:<\/p>\n\n\n\n![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/04\/Captura-de-Tela-2025-04-10-as-14.17.57.png\")
<\/figure>\n\n\n\nWhere: <\/p>\n\n\n\n
n<\/em> is the total number of inputs and weights.<\/p>\n\n\n\n\n- Inputs – <\/strong>The first representation of an artificial neuron, proposed by McCulloch and Pitts, takes a set of inputs (x<\/em>1<\/sub><\/em>, x<\/em>2<\/sub><\/em>, …, x<\/em>n<\/sub><\/em>) and assigns a weight (w<\/em>1<\/sub><\/em>, w<\/em>2<\/sub><\/em>, …, w<\/em>n<\/sub><\/em>) to each input.<\/li>\n\n\n\n
- Weighted sum – <\/strong>The neuron then computes the weighted sum of the inputs, which is the sum of the products of each input and its corresponding weight, as shown by Equation 2.<\/li>\n\n\n\n
- Bias – <\/strong>The bias, represented by the parameter b<\/em>, is added to the weighted sum before the activation function is applied. This allows the neuron to shift its activation threshold, enabling more complex decision boundaries.<\/li>\n\n\n\n
- Activation function – <\/strong>Finally, the neuron applies an activation function, such as a step function or a sigmoid function, to the weighted sum to determine the neuron’s output. We will talk about activation functions later.<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/03\/neuron.png\")
Depicts the neuron McCulloch and Pitts used as inspiration to develop the Perceptron model. <\/figcaption><\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/03\/equation-with-artificial-neuron.png\")
Visual representation of Equation 1, which makes it easy to compare with a real neuron. <\/figcaption><\/figure>\n<\/div>\n\n\nNext, we will describe how to use this model to solve a linear regression problem. Plus, we\u2019ll also show you how to measure the model error and train it to reduce that error.<\/p>\n\n\n\n
4. The regression problem<\/strong><\/h2>\n\n\n\nAs described in Section 2, the first task we will assign to our artificial neuron is a regression problem. To simplify the representation, we can remove the activation function and keep only one input.
<\/p>\n\n\n
\n![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/03\/simplification.png\")
Simplified representation of an artificial neuron<\/figcaption><\/figure>\n<\/div>\n\n\nBefore we dive deeper into linear regression, let\u2019s answer a few questions:<\/p>\n\n\n\n
What is linear regression?<\/strong><\/h3>\n\n\n\nLinear regression is a fundamental machine learning algorithm that can be used to model the relationship between a dependent variable and one or more independent variables. The goal is to find the best-fitting straight line that minimizes the distance between the data points and the line.<\/p>\n\n\n\n
What are the applications of linear regression?<\/strong><\/h3>\n\n\n\nLinear regression has many applications, including predicting sales, forecasting stock prices, and analyzing the relationship between various factors in social and economic studies. It’s a powerful tool for understanding and quantifying the relationships between variables.<\/p>\n\n\n\n
How to implement a linear regression?<\/strong><\/h3>\n\n\n\nTo implement linear regression, we need to define the model equation, which takes the form y = mx + b<\/em>, where y<\/em> is the dependent variable, x<\/em> is the independent variable, m<\/em> is the slope, and b<\/em> is the y-intercept. We then use optimization techniques to find the values of m<\/em> and b<\/em> that best fit the data.<\/p>\n\n\n\nWhat are the limitations of linear regression?<\/strong><\/h3>\n\n\n\nWhile linear regression is a practical algorithm, it has limitations. It assumes a linear relationship between the variables, which may not always be the case. It can also be sensitive to outliers and may not perform well when dealing with complex, non-linear relationships.<\/p>\n\n\n\n
Check out this post to learn more about using regression in AI development.<\/em><\/a><\/p>\n\n\n\nSo, we\u2019ve defined the task and model we will work on, but we need two more steps to make everything fit. Let\u2019s take a look at how we measure the model results and how we train it. <\/p>\n\n\n\n
There are different methods and algorithms to measure and train artificial neurons. They get more complex as the problems we tackle get more challenging. But, to understand how we connect everything we will use the Mean Square Error (MSE) and the backpropagation algorithms to solve our problem. <\/p>\n\n\n\n
4.1 Measuring the error<\/strong><\/h2>\n\n\n\nTo measure the error of our model, we use a loss or cost function that quantifies the difference between the predicted values and the actual values of the dependent variable. The goal is to minimize this function by adjusting the slope and y-intercept values. The most commonly used loss function in linear regression is the Mean Squared Error (MSE)<\/a>, which calculates the average squared difference between the predicted and actual values, as described in Equation 3. <\/p>\n\n\n\n![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/04\/Captura-de-Tela-2025-04-10-as-14.20.10.png\")
<\/figure>\n\n\n\nWhere: Y is the predicted value, and \u0176 is the reference value from the training data set.
Ok, now what do we do with the value of the MSE?<\/strong><\/p>\n\n\n\nWe use it to adjust the model\u2019s weight and bias, which makes it predict values closer to the reference value from the data set. For that, we use the backpropagation algorithm<\/a> and a bunch of hyperparameters<\/a> to help us train the model. <\/p>\n\n\n\n4.2 Training the model<\/strong><\/h2>\n\n\n\nArtificial neurons are trained by adjusting the weights and biases associated with them through a process called backpropagation. During training, the input data is fed through the neural network, and the output is compared to the desired output. The difference between the two is used to calculate the loss, and then the weights and biases are adjusted to minimize the loss using optimization algorithms like gradient descent. This process is repeated iteratively until the network reaches satisfactory accuracy.<\/p>\n\n\n\n
The Gradient Descent<\/a> algorithm in machine learning is used to minimize the cost function and find the optimal set of weights by following the steepest descent in the negative direction of the gradient. <\/p>\n\n\n\nIn each iteration, the parameters are updated by subtracting the gradient multiplied by a learning rate, which is a hyperparameter that determines the step size. The process is repeated until the convergence criteria are met. <\/p>\n\n\n\n
The gradient descent equation is shown in Equation 4, where \ud835\udefbF(y)<\/em> is the gradient of the Cost Function, is the learning rate and w<\/em> is the model weight.<\/p>\n\n\n\n![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/04\/Captura-de-Tela-2025-04-10-as-14.24.16.png\")
<\/figure>\n\n\n\nFigure 4 presents the process of converging the weight to a minimum value using the gradient descendent. It is important to note a few points:<\/p>\n\n\n\n
\n- Learning rate<\/strong>: The learning rate hyperparameter has no exact value. It must be chosen empirically by testing different values and observing which gives the best results. However, a hint is to use very small values, such as 0.0001 or 0.00001, as high values make the algorithm pass by the local minimum, skipping the optimum weight value and rapidly exploding. <\/li>\n<\/ul>\n\n\n\n
\n- Epochs<\/strong>: The number of epochs is the number of times we want the backpropagation algorithm to run through our model to adjust the weights and bias. The epochs are another hyperparameter that must be chosen by the people developing the model. The hint here is to test because if the training is short and the number of epochs is low, the model can\u2019t learn very well and won\u2019t generalize the outputs correctly (this phenomenon is known as model underfit). Meanwhile, if the number of epochs is too high, the model becomes very specialized on the training data and loses the ability to generalize (leading to a phenomenon known as model overfit).<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2025\/03\/Gradient-Descendent.png\")
Graphical representation of the Gradient Descendent algorithm applied to a model to find the minimum weight value that optimizes the cost function.<\/em><\/figcaption><\/figure>\n<\/div>\n\n\nToday, algorithms already exist that help to choose the best hyperparameter values for Perceptrons. However, here, we are presenting an example developed from scratch without the help of any framework or library. This process is essential to understanding how we train bigger neural networks. The algorithms are more sophisticated in reducing training time, but the process is the same. <\/p>\n\n\n\n
4.3 Show me the code<\/strong><\/h2>\n\n\n\nOkay, after all this theory and math, let\u2019s dive into the code to understand how to develop a simplified artificial neuron to solve a linear regression task. All the code presented here is available in this repository<\/a>.
This Python code implements a simple neural network with a single neuron, capable of learning a linear relationship between inputs and outputs. It includes methods for forward propagation, loss computation, and training using gradient descent. Let\u2019s break it down step by step.<\/p>\n\n\nimport<\/span> random\n\nclass<\/span> NeuralNetwork<\/span>:<\/span>\n\tdef<\/span> __init__<\/span>(self)<\/span>:<\/span>\n \t # Random initialize weight<\/span>\n \t self.weight = random.random()\n \t self.bias = random.random()\n\n\tdef<\/span> forward<\/span>(self, _input: float | int)<\/span>:<\/span>\n \t # Calculate the weighted sum<\/span>\n \t output = self.weight * _input + self.bias\n \t return<\/span> output\n\n\tdef<\/span> predict<\/span>(self, X: [float])<\/span> -> [float]:<\/span>\n \t Y_predicted = []\n \t for<\/span> x in<\/span> X:\n prediction = self.forward(x)\n Y_predicted.append(prediction)\n \t return<\/span> Y_predicted\n\n
In this post, we\u2019re looking at the first artificial neuron model, so we\u2019ll define the task as a linear regression problem. <\/p>\n\n\n\n
The linear regression problem is when we use the artificial neuron to represent functions. Instead of using a function like y = mx + b, we present the data to the artificial neuron and let it discover the appropriate values for the coefficients m<\/em> and b <\/em>that represent the problem. We will return to this in Section 4.<\/p>\n\n\n\n In the next section, we will describe artificial neuron modeling. Don\u2019t be scared of the math \u2014 we won\u2019t go too deep into it, promise!<\/p>\n\n\n\n The foundation of modern neural networks can be traced back to the earliest mathematical representation of an artificial neuron, known as the Perceptron<\/a>. First introduced by Warren McCulloch and Walter Pitts in 1943, this simple yet powerful model mimics how biological neurons process information. By taking weighted inputs, applying a bias, and passing the result through an activation function, the perceptron is the fundamental building block for more advanced neural network architectures. <\/p>\n\n\n\n This section explores the key components of the perceptron and how they contribute to its ability to make decisions. <\/p>\n\n\n\n Equation 1 is the mathematical model of the artificial neuron:<\/p>\n\n\n\n Where: <\/p>\n\n\n\n y<\/em> is the output,<\/p>\n\n\n\n phi is the activation function,<\/p>\n\n\n\n X<\/em> is the vector containing all the inputs,<\/p>\n\n\n\n W<\/em> is the vector containing all the weights,<\/p>\n\n\n\n b<\/em> is the bias<\/p>\n\n\n\n From linear algebra, we can describe the vector product as:<\/p>\n\n\n\n Where: <\/p>\n\n\n\n n<\/em> is the total number of inputs and weights.<\/p>\n\n\n\n Next, we will describe how to use this model to solve a linear regression problem. Plus, we\u2019ll also show you how to measure the model error and train it to reduce that error.<\/p>\n\n\n\n As described in Section 2, the first task we will assign to our artificial neuron is a regression problem. To simplify the representation, we can remove the activation function and keep only one input. Before we dive deeper into linear regression, let\u2019s answer a few questions:<\/p>\n\n\n\n Linear regression is a fundamental machine learning algorithm that can be used to model the relationship between a dependent variable and one or more independent variables. The goal is to find the best-fitting straight line that minimizes the distance between the data points and the line.<\/p>\n\n\n\n Linear regression has many applications, including predicting sales, forecasting stock prices, and analyzing the relationship between various factors in social and economic studies. It’s a powerful tool for understanding and quantifying the relationships between variables.<\/p>\n\n\n\n To implement linear regression, we need to define the model equation, which takes the form y = mx + b<\/em>, where y<\/em> is the dependent variable, x<\/em> is the independent variable, m<\/em> is the slope, and b<\/em> is the y-intercept. We then use optimization techniques to find the values of m<\/em> and b<\/em> that best fit the data.<\/p>\n\n\n\n While linear regression is a practical algorithm, it has limitations. It assumes a linear relationship between the variables, which may not always be the case. It can also be sensitive to outliers and may not perform well when dealing with complex, non-linear relationships.<\/p>\n\n\n\n Check out this post to learn more about using regression in AI development.<\/em><\/a><\/p>\n\n\n\n So, we\u2019ve defined the task and model we will work on, but we need two more steps to make everything fit. Let\u2019s take a look at how we measure the model results and how we train it. <\/p>\n\n\n\n There are different methods and algorithms to measure and train artificial neurons. They get more complex as the problems we tackle get more challenging. But, to understand how we connect everything we will use the Mean Square Error (MSE) and the backpropagation algorithms to solve our problem. <\/p>\n\n\n\n To measure the error of our model, we use a loss or cost function that quantifies the difference between the predicted values and the actual values of the dependent variable. The goal is to minimize this function by adjusting the slope and y-intercept values. The most commonly used loss function in linear regression is the Mean Squared Error (MSE)<\/a>, which calculates the average squared difference between the predicted and actual values, as described in Equation 3. <\/p>\n\n\n\n Where: Y is the predicted value, and \u0176 is the reference value from the training data set. We use it to adjust the model\u2019s weight and bias, which makes it predict values closer to the reference value from the data set. For that, we use the backpropagation algorithm<\/a> and a bunch of hyperparameters<\/a> to help us train the model. <\/p>\n\n\n\n Artificial neurons are trained by adjusting the weights and biases associated with them through a process called backpropagation. During training, the input data is fed through the neural network, and the output is compared to the desired output. The difference between the two is used to calculate the loss, and then the weights and biases are adjusted to minimize the loss using optimization algorithms like gradient descent. This process is repeated iteratively until the network reaches satisfactory accuracy.<\/p>\n\n\n\n The Gradient Descent<\/a> algorithm in machine learning is used to minimize the cost function and find the optimal set of weights by following the steepest descent in the negative direction of the gradient. <\/p>\n\n\n\n In each iteration, the parameters are updated by subtracting the gradient multiplied by a learning rate, which is a hyperparameter that determines the step size. The process is repeated until the convergence criteria are met. <\/p>\n\n\n\n The gradient descent equation is shown in Equation 4, where \ud835\udefbF(y)<\/em> is the gradient of the Cost Function, is the learning rate and w<\/em> is the model weight.<\/p>\n\n\n\n Figure 4 presents the process of converging the weight to a minimum value using the gradient descendent. It is important to note a few points:<\/p>\n\n\n\n Today, algorithms already exist that help to choose the best hyperparameter values for Perceptrons. However, here, we are presenting an example developed from scratch without the help of any framework or library. This process is essential to understanding how we train bigger neural networks. The algorithms are more sophisticated in reducing training time, but the process is the same. <\/p>\n\n\n\n Okay, after all this theory and math, let\u2019s dive into the code to understand how to develop a simplified artificial neuron to solve a linear regression task. All the code presented here is available in this repository<\/a>.3. The First representation of an artificial neuron (Perceptron)<\/strong><\/h2>\n\n\n\n
<\/figure>\n\n\n\n<\/figure>\n\n\n\n\n
4. The regression problem<\/strong><\/h2>\n\n\n\n
<\/p>\n\n\nWhat is linear regression?<\/strong><\/h3>\n\n\n\n
What are the applications of linear regression?<\/strong><\/h3>\n\n\n\n
How to implement a linear regression?<\/strong><\/h3>\n\n\n\n
What are the limitations of linear regression?<\/strong><\/h3>\n\n\n\n
4.1 Measuring the error<\/strong><\/h2>\n\n\n\n
<\/figure>\n\n\n\n
Ok, now what do we do with the value of the MSE?<\/strong><\/p>\n\n\n\n4.2 Training the model<\/strong><\/h2>\n\n\n\n
<\/figure>\n\n\n\n\n
\n
4.3 Show me the code<\/strong><\/h2>\n\n\n\n
This Python code implements a simple neural network with a single neuron, capable of learning a linear relationship between inputs and outputs. It includes methods for forward propagation, loss computation, and training using gradient descent. Let\u2019s break it down step by step.<\/p>\n\n\nimport<\/span> random\n\nclass<\/span> NeuralNetwork<\/span>:<\/span>\n\tdef<\/span> __init__<\/span>(self)<\/span>:<\/span>\n \t # Random initialize weight<\/span>\n \t self.weight = random.random()\n \t self.bias = random.random()\n\n\tdef<\/span> forward<\/span>(self, _input: float | int)<\/span>:<\/span>\n \t # Calculate the weighted sum<\/span>\n \t output = self.weight * _input + self.bias\n \t return<\/span> output\n\n\tdef<\/span> predict<\/span>(self, X: [float])<\/span> -> [float]:<\/span>\n \t Y_predicted = []\n \t for<\/span> x in<\/span> X:\n prediction = self.forward(x)\n Y_predicted.append(prediction)\n \t return<\/span> Y_predicted\n\n