PyTorch Learning Series: Part 1

Hey Everyone! 👋 Thanks for stopping by on this blog series on learning PyTorch.

I have been a great admirer of TensorFlow from a long time as it was the first Deep learning framework I learnt. And first ones are always special!

But if I am already well-versed in TensorFlow, why take the headache of learning a completely new framework?

To be honest that's a valid question and I have the answer to that: curiosity and necessity.

• Curiosity to learn PyTorch was there for a long time and hence I finally decided to take it up and dive into it.
• Necessity to understand, comprehend and implement the latest advancements happening in GenAI space. The super fast paced advancements in LLM and GenAI space is like crazy and to understand the work done through research, PyTorch is the number one choice as most of them are implemented using this framework.

Now that I have justified why I am diving into PyTorch, it's time to begin with the very first part of this series.

I will be following some combination of resources throughout and will be listing all of them at the end in Resources section for anyone to refer.

Let's get right into PyTorch!! 🥳

The Why?

In order to understand about anything, questioning the why of it is really crucial. You should be absolutely clear why you are studying this and why even this thing exists in first place!

Answering the question in context of PyTorch, we also have a very valid reason why PyTorch came into existence.

The beginning of its journey began in 2002 (well that's the year I was born too) with the development of Torch.

Torch is a framework that allowed handling Tensor based mathematical operations seamlessly on Graphical Processing Units (GPUs) leading to efficiency and enhanced performance. It was written in Lua programming language which is not so popular.

So Torch and PyTorch are similar?

Well not really! I also had this confusion initially but this is not correct. Meta AI researchers and scientists realised that Torch is a really helpful framework and can be heavily used for Deep Learning use cases that involved heavy tensor based mathematical operations. But the only friction holding it back was the language it was written in, Lua.

Not everyone was familiar with it and Deep Learning applications required Python as most of the scientists and researchers were accustomed to it. This made to the birth of PyTorch (Python + Torch) in September 2016, which is built on top of Torch and provides complete compatibility in Python.

PyTorch became useful because: it combines Python's ease of use with the efficiency of the Torch scientific computing framework known for high-performance tensor-based operations, especially on GPUs.

So we have successfully conquered the first question: The Why?

Next let's jump into next part which glides us through what advantages did PyTorch bring on the table.

The What?

Initial Release: PyTorch 0.1 (2017)

The very first release of PyTorch brought quite a few advantages over the other existing Deep Learning libraries like TensorFlow.

1. PyTorch introduced the concept of Dynamic Computation Graphs

   A computation graph is a crucial concept in deep learning frameworks, representing the structure of computations performed by a neural network. It is a directed graph where nodes correspond to mathematical operations or variables,

   A computation graph is defined as a directed acyclic graph (DAG), where:

   • Nodes represent operations (e.g., addition, multiplication) or variables (e.g., inputs, weights).
   • Edges indicate dependencies between these operations, showing how data flows from one node to another.

   

   Source: Computational Graphs

   Although all the existing deep learning frameworks created computation graphs to process the mathematical operations in the neural network during forward and backward propagation, they were all static in nature.

   Now to understand static and dynamic computation graph, let's take a very simple real-life example.

   Static Graph Example

   Imagine you are baking a cake for your friend's birthday party and he is very particular to have the Black Forest cake for his birthday. So you are now having a pre-defined set of requirements and values and have to make the cake accordingly. There is very little or no scope to improvise on your own as Black Forest cakes can be made only in a particular static way only.

   Similarly in Deep learning, if you are training an Image Classification model (CNN) on a set of images, you already have a predefined set of inputs (having some shape, dimensions) and the neural network architecture is fixed. Midway during training you cannot change anything or modify the input layer. Hence it is static in nature.

   Dynamic Graph Example

   Consider the cake baking scenario again. Now instead of a pre-defined cake demanded by your friend, he wants you to make a cake of your choice for his birthday. Now the entire process is quite flexible as you can make a cake of your own choice. Also based on the availability of ingredients modify it to suit the cake making process. If you have chocolate chips, you can go ahead with a Chocolate Cake or even try a Vanilla Cake with choco chip toppings. Hence the process is dynamic.

   If you are training the image classification model using dynamic computation graphs, you will have the flexibility on how each image is handled and processed. Basically if the dataset has different images (different dimensions, properties) you can decide to handle them differently and modify the neural network architecture accordingly as the graph will be created on runtime for each image processed allowing flexibility.

   Hence the usefulness of a graph being dynamic can be clearly observed.

2. PyTorch provided seamless integration with other Python libraries

   PyTorch was one of the first deep learning libraries that was compatible with all other Python libraries like NumPy, SciPy primarily because PyTorch itself was Python based. Unlike TensorFlow which although was compatible, didn't provide a seamless experience. PyTorch overcame that issue and made this efficient, mostly because the code was really intuitive.

   Note: Here I am talking about TensorFlow not TensorFlow using Keras API.

The initial release had a lot of impact in a very short span of time:

• It gained popularity very soon and was adapted by researchers due to its Pythonic and intuitive nature
• It was widely adopted in numerous research papers

PyTorch 1.0 (2018)

The next release witnessed some more improvements in the framework. Some of the notable improvements are:

• Bridged the gap between research and production environments.
• This release introduced TorchScript that was helpful for model serialization and optimization. This makes it easier to deploy models in production environments without needing to rewrite them in a different framework.
• Integration of Caffe2 in PyTorch significantly improved performance and expanded the capabilities of the framework. Caffe2 was a deep learning framework developed by Facebook that focused on production deployment and performance optimization. Some of the benefits of this integration include:
  • Caffe2's memory management techniques allowed for better utilization of GPU memory, enabling the training of larger models and handling of bigger datasets without running into memory issues.
  • With Caffe2's C++ runtime, models could be deployed in C++ applications, making it easier to integrate deep learning models into existing software systems, especially in mobile and embedded systems.

The main impact that this release brought was strengthening the PyTorch library so that it can be also used in deployment. The transition from research to deployment was achieved in this release

PyTorch 1.x Series

The next set of releases from 1.0.1 to 1.13 saw many changes but the major improvements were:

• Support for distributed training: Often at scale, deep learning applications process millions of data and hence training on a single instance is not practically possible. Distributed training is a need and it was supported in the releases in this series.
• ONNX compatibility: Open Neural Network Exchange is an open format designed to facilitate the interoperability of machine learning models across different frameworks. The model saved in PyTorch should be reusable in other frameworks, which is enabled by ONNX.
• Model Quantization: Introduced model quantization for compressing models and increase efficiency. This is highly application in today's LLM landscape where we need to work with billion's of parameter models and in order to accomodate it in our memory arrangements, we have to quantize it.
• Expanded ecosystem: The PyTorch ecosystem was expanded with domain specific sub-frameworks like torchtext (for NLP applications), torchaudio (for audio applications), torchvision (for Computer Vision applications)

These major releases were really impactful and increase the adoption of PyTorch in the research as well as industrial community.

PyTorch 2.0

The latest version of PyTorch is the 2.x series. It provided some major improvements in the performance and deployment perspective of PyTorch. Also there was added support for making PyTorch compatible in modern hardware like Tensor Processing Units (TPUs), AI Chips.

Due to this fact, PyTorch started to be increasingle adopted for deployment environments as well apart from the widespread research popularity.

Core Features: What Makes it Special?

Now that we have successfully answered the What and Why of PyTorch, it's time to summarise what core features does PyTorch bring on the table making it a favourable framework for developing deep learning applications.

• Tensor Computations: Easily handle tensor operations making computations efficient.

• GPU Acceleration: Seamless conversion between GPU and CPU. We can convert operations to be performed on GPU to improve efficiency.

  # Moving tensor to GPU
  x = x.cuda()  # That's it! 
  x = x.to('cuda' if torch.cuda.is_available() else 'cpu')
  

• Dynamic Computation Graphs: We have already covered about the advantages of these above.

• Automatic Differentiation: It uses autograd for automatic differentiation which is a crucial mathematical operation for training neural networks using backpropagation. PyTorch's autograd system is like having a mathematical genius doing all your calculus homework!

• Distributed training

• Interoperability with Other libaries using ONNX

PyTorch vs TensorFlow

Now comes the million dollar question, which of these deep learning frameworks is better. Well I have still not used PyTorch a lot and infact this is my first time learning it. So I got this beautiful comparison chart that can help you decide between PyTorch and TensorFlow.

Reference: CampusX YouTube Video

Another interesting way to compare is to observe the trend of these two frameworks. If you head over to Google Trends and try to compare the trend in both these frameworks, you will see the worldwide trend.

It can be clearly seen that TensorFlow was quite popular till 2021. But from then there has been a steep upsurge in the deman of PyTorch and currently it has surpassed TensorFlow in worldwide usage.

Now that you have the idea to decide why you should PyTorch, let's push a little bit more and get a brief overview of the core PyTorch libraries that we will be using throughout in the series

Core PyTorch Libraries

PyTorch as any other framework comes with a ton of features and libraries. But during our journey, we will be using a few of them that are super important for building neural network applications and coding those complex architectures!

---

torch: This is the core module that provides multidimensional arrays (tensors) and mathematical operations on them.

torch.autograd: this is the automatic differentiation engine that records operations on tensors to compute gradients for optimization.

torch.nn: it provides a library for neural networks, including layers, activations, loss functions, and utilities to build deep learning models.

torch.optim: this optimization algorithms (optimizers) such as SGD, Adam, and RMSprop, which are used for training neural networks. Similar to tf.keras.optimizers

torch.utils.data: contains utilities for data handling, including the Dataset and DataLoader classes for managing and loading datasets efficiently. Similar to ImageDataGenerators in TensorFlow.

torch.jit: supports Just-In-Time (JIT) compilation and TorchScript for optimizing models and enabling deployment without Python dependencies.

torch.distributed: contains tools for distributed training across multiple GPUs and machines, facilitating parallel computation.

torch.cuda: provides interfaces with NVIDIA CUDA to enable GPU acceleration for tensor computations and model training.

torch.backends: contains settings and allows control over backend libraries like cuDNN, MKL, and others for performance tuning.

torch.multiprocessing: contains utilities for parallelism using multiprocessing, similar to Python's multiprocessing module but with support for CUDA tensors.

torch.quantization: includes tools for model quantization to reduce model size and improve inference speed, especially on edge devices.

torch.onnx: module that supports exporting PyTorch models to the ONNX (Open Neural Network Exchange) format for interoperability with other frameworks and deployment.

---

You are now quite faimilar with PyTorch and has come to the very last section of the blog.

PyTorch Ecosystem Libraries

We can simply build each and every neural network from scratch using PyTorch. But if we want to use pre-built models and architectures, just like keras for Tf, we have some ecosystems that enable utilising PyTorch and run, build and develop models with utmost ease.

• Hugging Face Transformers: This library provides state-of-the-art pre-trained models for various natural language processing (NLP) tasks, including text classification, translation, and question answering. It is built on PyTorch, making it a go-to resource for NLP practitioners.
• Fastai: Fastai is a high-level library that simplifies the process of training fast and accurate neural networks using modern best practices. It is built on top of PyTorch, allowing users to leverage PyTorch's capabilities while benefiting from a more user-friendly interface.
• PyTorch Geometric: This extension library focuses on geometric deep learning, including graph neural networks and 3D data processing. It provides tools and functionalities specifically designed for working with non-Euclidean data structures.
• TorchMetrics: A modular metrics API for PyTorch, TorchMetrics is compatible with PyTorch Lightning and offers standardized implementations of many common metrics, making it easier to evaluate model performance.
• TorchElastic: This library enables dynamic scaling of PyTorch distributed training jobs, allowing for elasticity in resource management. It helps optimize resource usage during training, especially in cloud environments.
• Optuna: An automatic hyperparameter optimization framework, Optuna integrates well with PyTorch for tuning models. It helps automate the process of finding the best hyperparameters for machine learning models.
• Catalyst: Catalyst provides high-level features for training neural networks, focusing on reproducibility and fast experimentation. It streamlines the training process and enhances productivity for researchers and developers.
• Ignite: This high-level library assists in training neural networks in PyTorch, offering a lightweight engine for training and evaluating models. It simplifies the training loop and provides useful utilities.
• AllenNLP: Built on PyTorch, AllenNLP is an NLP research library designed to support researchers in deep learning for NLP tasks. It provides tools and models specifically tailored for NLP research.
• Skorch: Skorch is a scikit-learn compatible wrapper for PyTorch, allowing users to utilize PyTorch models with scikit-learn utilities and APIs. This integration makes it easier to combine PyTorch with the scikit-learn ecosystem.
• PyTorch Forecasting: This high-level library is designed for time series forecasting, making it easy to build, train, and evaluate complex forecasting models.
• TensorBoard for PyTorch: This tool allows for the visualization of training metrics, model graphs, and other useful data within TensorBoard, enhancing the ability to monitor and debug PyTorch models.

We will be utilising HuggingFace a lot in this journey. You will then get an idea of how these ecosystems make building with PyTorch super simple and easy.

Resources for Part 1

• https://www.youtube.com/watch?v=QZsguRbcOBM → BEST ONE
• https://pytorch.org/docs/stable/index.html → They have some really good docs

---

Ending Note:

This is the first blog in the PyTorch Learning Series. We have covered a very detailed high-level overview of the PyTorch framework and understood the why and what of this technology. We have also covered some parameters that can help anyone starting out with Deep Learning decide whether to choose TensorFlow or PyTorch.

I am also very excited to learn and share my learnings on the go on this robust and growing Deep Learning framework.Thanks for reading till here and I hope you have got something out of this journey!

If you have any questions, feedback, or would like to share your experiences, feel free to reach out. Let's learn, grow and innovate together!

Email : sakalyamitra@gmail.com

Twitter/X: https://x.com/sakalya_mitra

LinkedIn: https://www.linkedin.com/in/sakalya-mitra/

Take care, See you soon 😇