Introduction to ML Neural Networks with Pytorch

The world of data analysis and machine learning is growing rapidly. As a scientist, you may be wondering what deep learning and neural networks are all about. I provide a short introduction to these topics with an emphasis on their potential applications in the field of data analysis and astronomy.

Deep Learning refers to artificial neural networks (ANNs) that use multiple layers of neurons for more complex tasks such as pattern recognition or image classification. Deep Learning models can learn patterns from large amounts of data by adjusting connections between different layers until desired performance is achieved - making them ideal for complex problems such as computer vision or natural language processing (NLP).

Python libraries like TensorFlow, Keras, PyTorch, Theano, Jax, etc., allow scientists to easily create powerful deep learning models without having extensive knowledge on coding algorithms themselves. These libraries enable researchers and developers alike to quickly prototype new ideas using pre-existing architectures while still allowing flexibility when needed for custom implementations. These libraries may be too easy to use for research and development purposes, as often users do not need some understanding to obtain the results of their research leading to sometimes mistakes.

Setup project environement¶

First, we define a virtual environment for our project. This is a good practice to avoid conflicts between different projects. Details on how to create a virtual environment can be found on our FAQ here

Once done, we can activate the environment and install the required packages. We will use the following packages (for both days):

You can install the packages using the following command (after activating your virtual environment):

python3 -m pip install -q -r requirements.txt

For those using Mac M-chips you should be able to use the GPU with Pytorch.

Verify that we have MPS support in PyTorch

MPS is a feature of Apple’s Metal API that allows to use the M-series’ GPUs.

Introduction and reminders from Day 1¶

What is deep learning?¶

In its essence, Deep Learning (DL) is one of many techniques collectively known as machine learning. Machine learning (ML) refers to techniques where a computer “learns” by examples. Machine learning is part of the grand scheme of automated machine decisions. People often refer machine learning as being a form of artificial intelligence (AI). AI is broad and aims at tackling problems the way humans do but to first order designates systems that learn or adapt decision rules. In computer science, a rule-based system is a system that stores and manipulate knowledge to interpret information or take decisions in a useful way. Normally, the term rule-based system only applies to systems involving human-crafted or curated rule sets. For instance in a factory, if an item is not conforming to the size or shape rule, it will be discarded.

The place and role of AI, ML, and Deep Learning in automated machine decisions. Image adapted from Tukijaaliwa, CC BY-SA 4.0, via Wikimedia Commons, original source

Building blocks of neural networks¶

Neurons¶

A first unit of neural networks is the neuron. A neuron takes a set of inputs $(x_1, \ldots, x_n)$, does a weighted sum and apply some simple function to the latter to produce one output. The following is a schematics of a neuron with three inputs.

In this example, the output corresponds to $output=f(w_1 \cdot x_1 + w_2 \cdot x_2 + w_3 \cdot x_3)$. $f$ is called activation function. The activation function is used to turn an unbounded input into an output that has a predictable form.

What is a Neural Network Activation Function?¶

An Activation Function is a function applied on the output of a neuron and decides whether it should be “activated”. It does not have to output a binary decision. The role of the activation function is to rank the input importance in the process of prediction using simple mathematical operations.

The primary role of the Activation Function is to transform the summed weighted input from the node into an output value fed to the next network layer.

💡 In deep learning, the Activation Function is often referred to as a Transfer Function in Artificial Neural Network.

The following figure illustrates some of the most common activation functions. These do not have to be bounded to outputs in [0, 1] intervals. Note also that activations functions could be parametric and those parameters adjusted during the training phase similarly to the weights.

Simple Example of activation function¶

Let’s assume we have a three-input neuron with a ReLU activation function and the following weights:

What will be the output to an input vector $x = [2, 3, 1]$ and $x^\prime = [1, -3, 0]$?

🔑 This process of passing inputs forward to get an output is known as feedforward.

✍️ What value do we obtain with a sigmoid function? What about other activation functions?

How to choose the right Activation Function?¶

Activation functions could vary within a network. But any choice need to match activation functions to the outputs based on the type of problem to solve.

As a rule of thumb, ReLU is a good activation function to start with in most problems. But it won’t always provide the best performance. But the following list gives some guidelines:

• ReLU activation function should only be for the hidden layers.
• Sigmoid and Tanh functions are not for in hidden layers as they make the model more susceptible to training issues(vanishing gradients).
• Swish function is recommended in neural networks with 40 hidden layers and more.

The following provides some recommendation for the output layer based on the type of prediction problem that you are solving:

• Regression: Linear Activation Functions
• Binary Classification: Sigmoid/Tanh Activation Function
• Multiclass Classification: Softmax
• Multilabel Classification: Sigmoid

The activation function in hidden layers is typically dependent on the type of neural network architecture. As a first try, we will use the following activation function:

• Convolutional Neural Network (CNN): ReLU (or SeLU etc)
• Recurrent Neural Network: Tanh and/or Sigmoid.

💡 All hidden layers usually use the same activation function. However, the output layer can typically use a different activation function from the hidden layers. The choice depends on the goal or type of prediction made by the model.

Example with coding a Neuron¶

Let’s manually implement a neuron using NumPy.

1. first define a ReLU activation function as $f(x) = \max(0, x)$
2. define a class Neuron which takes a vector of weights at initialization
3. define the Neuron.forward method which takes a vector of inputs and returns the output of the neuron
4. evaluate the code with the previous values.

class Neuron:
  def __init__(self, weights: np.ndarray, activation: Callable):
    pass

  def forward(self, inputs: np.ndarray) -> np.ndarray:
    pass

✍️ Code the neuron class

Combining Neurons to create a network¶

With a single neuron, we can create a scaled weighted sum operator. With a linear activation function, a single neuron corresponds to a linear regression model.

But with many connected neurons we can generate more complex operations.

The following example network has 3 inputs, a single hidden layer with 3 neurons an one output layer with 1 neuron.

A hidden layer corresponds to any layer between the input (first) layer and output (last) layer. There can be many hidden layers (deep network)

A neural network can have any number of layers with any number of neurons in those layers. The basic idea stays the same: feed the input(s) forward through the neurons in the network to get the output(s) at the end. For simplicity, we’ll keep using the network pictured above for the rest of this post.

We will skip the pen and paper calculations and implement this network in Python.

class NeuralNetwork:
    
    def __init__(self, 
        input_size: int, 
        hidden_size: int, 
        output_size: int, 
        weights: List[float] = None, 
        activation: Callable = sigmoid):
        pass

    def forward(self, X):
        pass

model = NeuralNetwork(3, 3, 1, activation=sigmoid)
model.forward([2, 3, 1])

💡 This implementation is freeform. You can use the previous Neuron class, make a Dense layer or work with arrays of weights.

✍️ Code the NeuralNetwork class

The weights are initialized at random and therefore without imposing the weights and doing the calculations by hand, it will be difficult to verify the model. Let’s train our model instead.

Training a Neural Network¶

Let’s use the common iris dataset, a multivariate data set used and made famous by the British statistician and biologist Ronald Fisher in his 1936 paper.

The data set consists of 50 samples from each of three species of Iris (setosa, virginica, and versicolor) and four measurements from each sample: the length and the width of the sepals and petals, in centimeters.

Using this dataset, let’s create a neural network model to predict the species given the measurements.

✍️ define the model variable and call .forward() on the model with the first row of the dataset.

The Loss function, objective function¶

Before we train our network, we first need to define a performance metric which the training procedure will optimize.

For simplicity here, we will use the Mean Squared Error (MSE) as the loss function:

where $N$ is the number of samples, $y_i$ is the true label, and $\hat{y}_i$ the output prediction of the network. The better our predictions, the lower MSE value, i.e. lower loss.

💡 The training procedure minimizes the loss function given a training set

✍️ define the loss function

Now we have to optimize this function, i.e. the loss of the neural network. We can adjust the network’s weights to change the predictions.

Optimization and backpropagation¶

⚠️ This part is a bit mathematical.

The loss function $L$ is function of all the weights $W$ and we want to minimize $L$ w.r.t $W$.

Let’s look at optimizing a single weight $w_{11}$ corresponding to the connection of the $x_{1}$ with the first neuron in the hidden layer $h_1$. The derivative of $L$ w.r.t. $w_{11}$ is

The other term is a bit more complicated. But since $w_{11}$ is between $x_1$ and $h_1$:

Finally, most activation functions $f$ have relatively simple derivative $f^\prime$. For instance, the sigmoid function has a simple derivative

We managed to express all the necessary terms in the derivative of $L$ w.r.t. $w_{11}$:

This system of calculating partial derivatives by working backwards is known as backpropagation. Backpropagation is the process of finding the derivative of the loss function w.r.t. all the weights $W$.

Training through Stochastic Gradient Descent¶

The final step of our model is to minimize the loss function w.r.t. all the weights using our training data.

A traditional approach uses an optimization algorithm essentially based on stochastic gradient descent (SGD). It is a variant of Gradient Descent (GD) but it uses only a random subset of the training data (also known as a mini-batch) rather than the entire training data set to calculate the gradient of the loss function. This random selection of data provides faster and more frequent updates to the model parameters, making it more efficient than GD when the data set is large. SGD repeats this process for a fixed number of iterations or until the convergence of the loss function.

The learning rate $\eta$ is an hyperparameter that controls the step size at each iteration, e.g.:

If we do this for every weight in the network, the loss will slowly decrease and our network will improve.

The SGD training process will look like this:

1. Choose one random sample/subset from our dataset.
2. Calculate all the partial derivatives of loss with respect to weights
3. Update the weights by subtracting the learning rate times the partial derivatives. and repeat until convergence.

To avoid having to code everything in the following we use sklearn.neural_network.MLPClassifier which implements an SGD optimizer (as well as others).

✍️ call the MLPClassifier with the iris dataset and fit the model. Use the solver='sgd' and set the learning rate alpha, hidden_layer_sizes.

We can also visualize the loss values as function of the training iterations, aka epoch.

⚠️ The above example is also an example of bad practice. We should use a validation set to monitor the performance of the model and avoid overfitting.

Why are deep neural networks hard to train?¶

Training NNs may sometimes lead to catastrophic failures (e.g., wrong activation function, forgetting to normalize the input data). There are two major challenges you might encounter when training your deep networks.

Vanishing Gradients¶

Like the sigmoid function, certain activation functions compress an infinite interval into [0, 1]. Hence, their derivative becomes small. For shallow networks with only a few layers that use these activations, this isn’t a big problem. But, when networks contain more layers, the product of those gradients becomes too small for training to work effectively.

Exploding Gradients¶

Exploding gradients occurs when significant error gradients accumulate and result in very large updates to neural network model weights during training with SGD based optimizers. With these exploding gradients, you can obtain unstable networks and the learning cannot be completed.

The values of the weights can also become so large as to overflow.

Key vocabulary¶

• Artificial Neural Network (ANN): A computational system inspired by the way biological neural networks in the human brain process information.
• Neuron: A building block of ANN. It is responsible for accepting input data, performing calculations, and producing output.
• Deep Neural Network: An ANN with many layers placed between the input layer and the output layer.
• Input data: Information or data provided to the neurons.
• Weights: The strength of the connection between two neurons. Weights determine what impact the input will have on the output.
• Bias: An additional parameter used along with the sum of the product of weights and inputs to produce an output.
• Activation Function: Determines the output of a neural network.
• Feed forward (forward propagation): The process of feeding the input data to the neurons.
• Back propagation (back propagation): The process of adjusting the weights and biases of the neurons through the gradient of the loss function.

Advantages & Disadvantages of Neural Networks¶

Advantages¶

• Fault tolerance: even if a few neurons are not “working” properly, that would not prevent the neural networks from generating outputs.

• Real-time Operations: neural networks can learn synchronously and easily adapt to their changing environments.

• Adaptive Learning: neural networks can learn how to work on different tasks, or sub-tasks with proper training.

• Parallel processing capacity: neural networks can perform multiple tasks simultaneously.

Disadvantages¶

• Unexplained behavior of the network: neural networks provide a solution to a problem but rarely a why and how it made the decisions it made. Understanding the behavior of a model represents additional tasks to the architect (i.e., the user).

• Determination of appropriate network structure: there is no generic rule (or rule of thumb) to design a neural network architecture. A proper network structure is achieved by trying the best network, in a trial and error approach. It is a process that involves refinement.

• Hardware dependence: To be efficient in computational speed (training or evaluation), neural networks require (or are highly dependent on) processors (CPU, GPU, TPU) with adequate processing capacity.

What sort of problems can Deep Learning not solve?¶

DL is not always capable to solve problems. It will not work in

• any case where only a small amount of training data is available,
• tasks requiring an explanation of how the answer was arrived at (interpretability not guaranteed),
• Classifying things which are nothing like their training data (not good at genaralization).

What sort of problems can Deep Learning solve but should not be used for?¶

Deep Learning needs a lot of computational power, it often relies on specialised hardware like graphical processing units (GPUs). Many computational problems can be solved using less intensive techniques, but could still technically be solved with Deep Learning. It is tempting to use the fashion techniques to solve old problems, but we should remain critical.

Deep learning can technically solve the following but it would probably be a wasteful way to do it:

• Logic operations, such as sums, averages, ranges etc.
• Modelling well defined systems, where the equations governing them are known and understood.
• Basic computer vision tasks such as edge detection, decreasing colour depth or blurring an image.

Deep Learning Libraries in Python¶

There are many software libraries available for Deep Learning. Python includes:

PyTorch¶

PyTorch was developed by Facebook in 2016 and it is a popular choice for Deep Learning applications. It was developed for Python from the start and feels a lot more “pythonic” than TensorFlow. Like TensorFlow it was designed to do more than just Deep Learning and offers some very low level interfaces. PyTorch Lightning offers a higher level interface to PyTorch to set up experiments. Like TensorFlow it is also very easy to integrate PyTorch with a GPU.

Jax + Flax¶

Jax is a Python library developed by Google for high-performance numerical computing and machine learning. It provides an interface that is similar to NumPy but that can run on GPUs and TPUs transparently for the users offering significantly improved performance. JAX includes automatic differentiation, which makes it easy to train machine learning models using techniques like stochastic gradient descent.

Equinox is a neural network library that is built on top of JAX (alternative to flax) with easy-to-use PyTorch-like syntax. It provides at most wrappers around JAX, which makes it more flexible than other libraries.

Flax is a neural network library that is built on top of JAX and is designed for flexibility. It provides a high-level API for defining and training neural networks and has built-in support for advanced training techniques like mixed precision and gradient checkpointing.

Keras¶

Keras is designed to be easy to use and usually requires fewer lines of code than other libraries. We have chosen it for this workshop for that reason. Keras can actually work on top of TensorFlow (and several other libraries), hiding away the complexities of TensorFlow while still allowing you to make use of their features.

The performance of Keras is sometimes not as good as other libraries and if you are going to move on to create very large networks using very large datasets then you might want to consider one of the other libraries. But for many applications the performance difference will not be enough to worry about and the time you will save with simpler code will exceed what you will save by having the code run a little faster.

TensorFlow¶

TensorFlow was developed by Google and is one of the older Deep Learning libraries, ported across many languages since it was first released to the public in 2015. It is very versatile and capable of much more than Deep Learning but as a result it often takes a lot more lines of code to write Deep Learning operations in TensorFlow than in other libraries. It offers (almost) seamless integration with GPU accelerators and Google’s own TPU (Tensor Processing Unit) chips that are built specially for machine learning.

⚠️ Tensorflow is about to be deprecated by Google (not further dev.) in favor of JAX.

Which library should I use/learn?¶

There are no standard answer to which library should I use or learn, even when having a specific problem.

TensorFlow and PyTorch are currently the most widely used deep learning libraries, while JAX is growing in popularity due to its efficient GPU and TPU acceleration, automatic differentiation, and high flexibility.

It is important to note that Google currently slows down the development of Tensorflow in favor of JAX. But Tensorflow remains important for insdustry applications. It provides many services and pre-trained models.

Jax is the new kid in the playground. Its development is extremely fast, and it gets a lot of traction in the scientific community. (A few developpers are former-astronomers.)

Exercise: Logical XOR network¶

In this exercise, you will implement a simple XOR network, i.e. implement a network that exactly matches the behavior of a simple logical XOR operator.

For this exercise, the network needs at least one hidden layer to solve the problem.

Logical XOR¶

XOR stands for “exclusive or”. The output of the XOR function has only a true value if the two inputs are different. If the two inputs are identical, the XOR function returns a false value. The following table shows the inputs and outputs of the XOR function.

The following figure illustrates the logical operations OR, AND, and XOR. In contrast to OR and AND problems, the XOR one cannot be linearly separated by a single boundaryline. To solve this problem, we will need two boundary lines, hence requires at least one hidden layer in our neural network.

The next step is to define a Neural Network that is able to solve the XOR problem. With the correct choice of functions and weight parameters, a Neural Network with one hidden layer is able to solve the XOR problem. Because we have logical operations, a threshold activation function should be sufficient.

Use

from sklearn.neural_network import MLPClassifier

Non-linear regression with Pytorch¶

In this tutorial, we will explore coding neural networks to do a non-linear regression with a multi-layer perceptron.

Generate mock data¶

In this exercise, we generate 100 pairs of values from a non-linear function

and we add noise to a random uniform sample:

✍️ Implement the function f(x) and generate a 100 data points (x, y) with noise. Plot your data to make sure it looks ok.

MLP in PyTorch¶

In this section, we code a simple network using PyTorch torch.nn library.

To define a model, PyTorch requires a class definition deriving nn.Module. This class pre-defines neural networks properties and important methods.

Below, we define our neural network class. The main aspects are

1. the initialization/constructor function __init__, which defines the properties of our model and parameters
2. the forward method which takes the model’s input and computes the corresponding prediction given the current model.

PyTorch allows flexibility in how to define neural network modules, we show below a concise implementation. Regardless, any implementation should work the same way eventually.

Typically, our network need to take x, apply a series of Linear layer followed by an activation function until the last output layer. This corresponds to a feed foward function looking something like

z = nn.Linear(1, 10)(x)
z = nn.Linear(10, 10)(z)
z = nn.ReLU()(z)
z = nn.Linear(10, 10)(z)
z = nn.ReLU()(z)
z = nn.Linear(10, 1)(z)

All together, a common implementation would be the following:

✍️ Implement an MLP module using PyTorch.

if you want to use a list of layers (aka pipeline in sklearn) you can use nn.Sequential

We can access the current parameter values of this model instance like so:

Exercise

1. What are these parameters? (Hint: they are in oder of the sequence and computations in the forward method.)

2. Guess how the weights are initialized.

Preparing our training data, Dataset¶

In PyTorch, there are multiple ways to prepare a dataset for the model training. The main aspect is to convert the data into tensors which are transparently moved to GPUs if needed.

To make a standard API that works with small and large datasets, PyTorch provides two data primitives:

• torch.utils.data.Dataset to store the data in a reusable format,
• torch.utils.data.DataLoader takes a Dataset object as input and returns an iterable to enable easy access to the training data (especially useful for batch training).

Dataset¶

To define a Dataset object, we have to write a class deriving the latter and specify two key methods:

1. __len__, which tells subsequent applications how many data points there are in total (useful if you load the data by chunk); and
2. the __getitem__ function, which takes an index as input and outputs a corresponding (input, output) pair.

DataLoader¶

Let’s instanciate also the data loader (or data feeder). For this example, we will provide the data in batches of 25 random examples (out of 100).

traindata is an iterator:

💡 The training set in our current example is small and training by batch is not necessary

Training the model¶

In many libraries, writing the training code is the most important part and varies from library to library. They also have common bricks:

• the loss function: let’s take MSE here.
• the optimizer: let’s take Adam (a flavor of stochastic gradient descent) with a learning rate 10^-4
• the training loop: which for many epoch, feed the data to the model, calculate the loss function, the gradient of it (backpropagation), and update the optimizer for the next iteration.

In PyTorch, this looks like the following:

Tuning Optimizers and learning curves¶

Because Adam is very popular and it is a good choice for training neural networks, we focus on building understanding of tuning its parameters: learning rate, exponential decay rate for the momentum term, and learning rate decay.

A small learning rate corresponds to updates and therefore a slower model training. i.e. it would require many updates to the parameters to reach the point of minima. On the other hand, a large learning rate would mean large steps to the parameters thus often tends to divergence instead of convergence.

An optimal learning rate value (default value 0.001 for Adam) means that the optimizer would update the parameters just right to reach the local minima in a given number of epochs. Varying learning rate between 0.0001 and 0.01 is the typical range in most cases, but this depends on the actual optimizer.

momentum decay $(\beta_1, \beta_2)$¶

$\beta_1$ is the exponential decay rate for the momentum term. It’s default value in PyTorch is 0.9 (from the original paper).

$\beta_2$ is the exponential decay rate for velocity term. The default value is 0.999 in the PyTorch implementation. This value should be set as close to 1.0 as possible to help with sparse gradient.

Learning Rate Decay¶

Learning rate decay decreases as the parameter values reach closer to the global optimal solution. This avoids overshooting the minima often resulting in faster convergence of the loss function. The PyTorch implementation refers to this as weight_decay with default value being zero.

Epsilon¶

Although not a tuning hyperparameter, it is a very tiny number to avoid any division by zero error in the implementation. The default value is 10^-8.

Exercise Explore the model’s behavior

1. Make sure you understand the various lines of code.
2. Change the model layout, activations, optimizer, etc.
3. Change the mini-batch configuration. e.g. All data at once

Run the model when you change something and discuss notable differences. Explain what your observations. (If you do not see anything, that could happen. Explain why)

Revisiting the XOR problem with pytorch.¶

✍️ train an XOR network with PyTorch: make the XOR class and train it on xor data.

Revisiting the Photometric Redshift problem with pytorch.¶

We revisit the problem we had in Day 1 of photometric redshift estimation using a neural network. We will use the same data as in the previous notebook.

Reminder, Predicting galaxy redshift via regression on 3D-HST photometry¶

We will explore a typical machine learning problem: predicting galaxy redshifts from photometry. We will use the 3D-HST survey photometry to predict the redshift of galaxies.

The 3D-HST survey is a Hubble Space Telescope Treasury program that has used 248 orbits of HST time to image four fields in the sky with grism spectroscopy (AEGIS, COSMOS, GOODS-S, UKIDSS-UDS). See Brammer et al., 2012 and the 3D-HST website for more information.

The data we will use is a subset of the 3D-HST photometry and redshifts. We will use the photometry in the F125W, F140W, F160W, F606W, F814W filters, and the redshifts from the 3D-HST catalog from Skelton et al (2014).

Reminder Redshift 101

The redshift of a galaxy is a measure of how much the wavelength of light from the galaxy has been stretched by the expansion of the Universe. The redshift $z$ is defined as

where $\lambda_{obs}$ is the observed wavelength of light from the galaxy and $\lambda_{em}$ is the wavelength of the light emitted by the galaxy.

Data preparation¶

As we did in the previous session, we clean the data.

Here we will go further than last time. We will extract only the f_* columns as input features and the zspec column as the target. We will keep only positive fluxes, star_flag = 0. We could debate on the choice of the features, but we will keep it simple for now.

Let’s look at the “spectra”

Finally, we can also convert the fluxes to log10(fluxes) to have a more normal distribution of the data.

Divide the data into train, test and validation sets¶

Divide the data into train, validation and test sets. We will use the following definitions:

• Training: The data used to update model parameters (e.g., coefficients or matrix element values).

• Validation: The data used to update model selection (for instance, we might change hyperparameters of a model based on the validation metrics).

• Testing: The data used to make final predictions, and possibly evaluate the final model score.

✍️ define the training, validation sets. (30% for validation, 70% for training)

Model definition: single layer regression¶

✍️ define a DNNRegressor model with mutliple layers and activation functions.

✍️ train the model, plot the learning curves. Be critical of the results. Maybe you need to tune the architecture, the optimizer, the learning rate, etc. Feel free to discuss with the group.

✍️ Make some plots to understand what your model does

As before, we will use the SHAP values to understand the model’s behavior.

The results are not as good as the tree based methods we explored last week. What can we do to improve the model?