Programming Assign.

Unit 7

University of the People

CS 4407-01 Data Mining and Machine Learning

Mylsami Thanigasalam, Instructor

May 24, 2023

Developing an effective neural network requires a meticulous process of experimentation and

refinement to achieve optimal results. The design and training of a neural network involve

careful consideration of various variables, such as the number of neurons, learning rate,

momentum, weight range, and the number of training steps. These variables are interdependent,

and any modifications can significantly impact the network's performance and accuracy.

In my project, I embarked on an iterative journey to develop a neural network, evaluating four

different network configurations. The initial design consisted of 7 input neurons, 5 middle-layer

neurons, and 7 output neurons, including biases. The training settings were defined with a

learning rate of 0.3, momentum of 0.8, 5000 learning steps, and a weight range of -1 to 1.

However, this configuration yielded an average error rate of 0.8, which fell short of meeting the

assignment requirements. It became apparent that significant modifications were necessary to

enhance the network's performance.

For the second iteration, I decided to modify the network design by removing the biases.

Although this adjustment resulted in a slightly improved error rate, it still did not meet the

desired level of accuracy. This prompted me to explore alternative approaches that could

potentially yield higher accuracy within the minimum number of training steps.

In the third iteration, I maintained the previous training settings but increased the number of

middle-layer neurons to 10. This adjustment led to a significant improvement, with an error rate

of 0.6. The error graph exhibited a notable decrease with each additional training step, indicating

the potential for further accuracy enhancements. This improvement provided the motivation to

continue refining the network architecture.

Moving on to the fourth iteration, I decided to make additional modifications to the training

settings in an effort to optimize accuracy while maintaining the decreasing trend in the error

graph. The revised network design featured 7 input neurons, 10 middle-layer neurons, and 7

output neurons. The training settings were adjusted to include a learning rate of 5.0, momentum

of 0.2, 5000 learning steps, and a weight range of -1 to 1. This configuration resulted in an error

rate of 0.059473879933095074, meeting the assignment requirements and exhibiting a better

error graph trend than the previous iteration. Although I attempted further experimentation, no

significant improvements were achieved, leading me to conclude that this configuration was the

most effective. With confidence in the design, I decided to continue training the network for an

additional 30000 steps.

The additional training yielded remarkable results, with an error rate of 0.017443030540617112.

This achievement further solidified the effectiveness of the network design. Throughout this

iterative process, I gained valuable insights and lessons about the design and training of neural

networks.

One of the key lessons learned from this experience is the significance of the number of neurons

in achieving higher accuracy. Increasing the number of neurons, particularly in the middle layer,

often leads to improved performance. The added complexity and capacity for capturing intricate

patterns and relationships can contribute to enhanced accuracy. However, it is important to strike

a balance and avoid overfitting, where the network becomes overly specialized to the training

data and fails to generalize well to unseen data.

Moreover, the training settings, including the learning rate, momentum, weight range, and

number of learning steps, play a critical role in the convergence and performance of the network.

The learning rate determines the step size in weight adjustments during training, impacting the
speed and stability of convergence. A higher learning rate may result in faster convergence but

can also lead to overshooting and instability. On the other hand, a lower learning rate can ensure

stability but may slow down convergence. Similarly, the momentum parameter affects the impact

of previous weight adjustments on the current update, helping the network navigate through local

optima and avoid getting stuck in suboptimal solutions.

The weight range defines the initial range of weights assigned to the connections between

neurons. Setting an appropriate weight range is crucial for ensuring a balanced initialization of

the network and avoiding issues such as vanishing or exploding gradients. Lastly, the number of

learning steps determines the duration of the training process. Insufficient learning steps may

result in an undertrained network, while excessive learning steps can lead to overtraining and

poor generalization.

This iterative process of designing and training the neural network provided valuable insights

and lessons. It emphasized the importance of careful experimentation, fine-tuning, and attention

to detail throughout the process. While advanced algorithms and software tools play a significant

role in network development, hands-on experience and a deep understanding of the underlying

principles are indispensable for achieving optimal results.

In conclusion, developing an effective neural network demands patience, persistence, and a

willingness to engage in extensive experimentation and refinement. Through the iterative process

of testing and evaluating four different network configurations, I gained valuable insights into

the significance of the number of neurons, training settings, and the iterative design process

itself. The final network configuration achieved an error rate close to zero, demonstrating the

immense potential and power of neural networks across various applications.

The process of developing neural networks is a continuous journey of exploration and discovery.

The lessons learned from this experience extend beyond the specific project and serve as a

foundation for future endeavors. As technology advances and new challenges emerge, the ability

to design and train neural networks effectively becomes increasingly vital. With the ever-

growing volume of data and the need for intelligent decision-making systems, the optimization

of neural networks remains an exciting and ever-evolving field of study.

References:

1. McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity.
Retrieved from https://link.springer.com/article/10.1007/bf02478259

2. Haykin, S. (2009). Neural networks and learning machines (3rd ed.). Retrieved from
https://cours.etsmtl.ca/sys843/REFS/Books/ebook_Haykin09.pdf