Basics of PyTorch tensor computations#
This is a tutorial on basic tensor computations that can be performed using the PyTorch framework. This will be the main framework used to construct surrogate models within this Jupyter Book. It is important to know the basics of PyTorch tensor computations to understand the surrogate modeling methods.
A PyTorch tensor is the analogue of a numpy array for the PyTorch framework, which is a very popular library for machine learning and creating surrogate models. Tensors are multi-dimensional arrays that are the basic unit of computation for the PyTorch framework. These will be used extensively for creating models and other computations. Tensors provide the user with the capability of performing computations on a Graphical Processing Unit (GPU) which can significantly accelerate computations. Here, we will cover some basic computations using PyTorch tensors. Further details are given within comments in the code blocks.
The first step is to import the PyTorch library which is imported through the name torch.
# Importing torch for tensor computations
import torch
# Also importing numpy as it is needed to show numpy to torch conversions and vice versa
import numpy as np
# Creating a new 1D tensor array
a_tensor = torch.tensor([1.0, 2.0, 3.0])
print("1D tensor:", a_tensor)
# Creating a new 2D tensor array
b_tensor = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
print("2D tensor:", b_tensor)
# It is also possible to create a 3D tensor
c_tensor = torch.tensor([[[1.0, 2.0], [3.0, 4.0]], [[5.0,6.0], [7.0,8.0]]])
print("3D tensor:", c_tensor)
1D tensor: tensor([1., 2., 3.])
2D tensor: tensor([[1., 2.],
[3., 4.]])
3D tensor: tensor([[[1., 2.],
[3., 4.]],
[[5., 6.],
[7., 8.]]])
# Acessing the dimension of the tensor
# Here, we are printing the values using a f string which is useful for printing strings and variables
print(f"Dimension of a_tensor: {a_tensor.ndim}")
print(f"Dimension of b_tensor: {b_tensor.ndim}")
print(f"Dimension of c_tensor: {c_tensor.ndim}")
Dimension of a_tensor: 1
Dimension of b_tensor: 2
Dimension of c_tensor: 3
# Accessing the shape of the tensor. Think of this as matrix dimensions if 2D or vector length if 1D
print(f"Shape of a_tensor: {a_tensor.shape}")
print(f"Shape of b_tensor: {b_tensor.shape}")
print(f"Shape of c_tensor: {c_tensor.shape}")
# The above returns a torch.Size array which can be indexed like a regular array
# Indexing shape of b_tensor
print(f"First index of b_tensor shape: {b_tensor.shape[0]}")
print(f"Second index of b_tensor shape: {b_tensor.shape[1]}")
Shape of a_tensor: torch.Size([3])
Shape of b_tensor: torch.Size([2, 2])
Shape of c_tensor: torch.Size([2, 2, 2])
First index of b_tensor shape: 2
Second index of b_tensor shape: 2
# The size method associated with a tensor can also be called to obtain the shape of the tensor array
print(f"Size of a_tensor: {a_tensor.size()}")
print(f"Size of b_tensor: {b_tensor.size()}")
print(f"Size of c_tensor: {c_tensor.size()}")
Size of a_tensor: torch.Size([3])
Size of b_tensor: torch.Size([2, 2])
Size of c_tensor: torch.Size([2, 2, 2])
# Some standard tensors such as a tensor with all zeros or all ones can also be defined as follows
# The numbers in the parentheses define the shape of the array, similar to the Numpy library
zero_tensor = torch.zeros(5) # 1D array of zeros
print(f"1D array of zeros: {zero_tensor}")
ones_tensor = torch.ones((5,2)) # 2D array of ones
print(f"2D array of ones: {ones_tensor}")
1D array of zeros: tensor([0., 0., 0., 0., 0.])
2D array of ones: tensor([[1., 1.],
[1., 1.],
[1., 1.],
[1., 1.],
[1., 1.]])
# It is also possible to create an evenly spaced tensor array using torch.linspace similar to MATLAB and Numpy
linear_tensor = torch.linspace(0,1,20)
print(f"Evenly spaced tensor: {linear_tensor}")
Evenly spaced tensor: tensor([0.0000, 0.0526, 0.1053, 0.1579, 0.2105, 0.2632, 0.3158, 0.3684, 0.4211,
0.4737, 0.5263, 0.5789, 0.6316, 0.6842, 0.7368, 0.7895, 0.8421, 0.8947,
0.9474, 1.0000])
We can index tensor arrays in a manner that is similar to indexing Numpy arrays. The next code block demonstrates examples of indexing different types of tensor arrays.
# Indexing a 1D tensor array
print(f"First element of a_tensor: {a_tensor[0]}")
print(f"Second element of a_tensor: {a_tensor[1]}")
First element of a_tensor: 1.0
Second element of a_tensor: 2.0
# Indexing a 2D array
print(f"Element of b_tensor in first row and first column: {b_tensor[0,0]}")
print(f"Element of b_tensor in second row and first column: {b_tensor[1][0]}")
# It is important to note that the above will return a 0-dim tensor which is essentially the representation of a single value within PyTorch
# To recover the single value from the tensor it is necessary to use the item method
print(f"Value of element of b_tensor in first row and first column: {b_tensor[0,0].item()}")
print(f"Value of element of b_tensor in second row and first column: {b_tensor[1][0].item()}")
Element of b_tensor in first row and first column: 1.0
Element of b_tensor in second row and first column: 3.0
Value of element of b_tensor in first row and first column: 1.0
Value of element of b_tensor in second row and first column: 3.0
# Indexing a 3D array and extracting the value using the item method
# The term depth in this case is used to indicate the third dimension of the array
print(f"Element of c_tensor in first row, first column and first position along depth: {c_tensor[0,0,0].item()}")
print(f"Element of c_tensor in second row, first column and second along the depth: {c_tensor[1][0][1].item()}")
Element of c_tensor in first row, first column and first position along depth: 1.0
Element of c_tensor in second row, first column and second along the depth: 6.0
Just like the Numpy library, it is also important to know how to reshape tensor arrays in case it is needed for building models or using tensors with other libraries.
# Reshaping aTensor from shape torch.Size([3]) to torch.Size([1,3])
print(f"Shape of a_tensor: {a_tensor.shape}")
a_tensor_reshaped_1 = a_tensor.reshape(1,3)
print(f"New shape of a_tensor: {a_tensor_reshaped_1.shape}")
# This can also be done by using -1 for one of the dimensions in the shape definition.
# If -1 is used then the dimension will match the remaining number of total elements in the array.
a_tensor_reshaped_2 = a_tensor.reshape(1,-1)
print(f"New shape of a_tensor: {a_tensor_reshaped_2.shape}")
Shape of a_tensor: torch.Size([3])
New shape of a_tensor: torch.Size([1, 3])
New shape of a_tensor: torch.Size([1, 3])
# Reshaping b_tensor from shape torch.Size([2,2]) to torch.Size([1,4])
print(f"Shape of b_tensor: {b_tensor.shape}")
b_tensor_reshaped = b_tensor.reshape(1,4)
print(f"New shape of b_tensor: {b_tensor_reshaped.shape}")
Shape of b_tensor: torch.Size([2, 2])
New shape of b_tensor: torch.Size([1, 4])
# Reshaping c_tensor from shape torch.Size([2,2,2]) to torch.Size([2,4])
# This can be done using -1 or using 4 as the dimension size
print(f"Shape of c_tensor: {c_tensor.shape}")
c_tensor_reshaped_1 = c_tensor.reshape(2,4)
print(f"New shape of c_tensor: {c_tensor_reshaped_1.shape}")
c_tensor_reshaped_2 = c_tensor.reshape(2,-1)
print(f"New shape of c_tensor: {c_tensor_reshaped_2.shape}")
Shape of c_tensor: torch.Size([2, 2, 2])
New shape of c_tensor: torch.Size([2, 4])
New shape of c_tensor: torch.Size([2, 4])
c_tensor_reshaped_3 = c_tensor.reshape(2,6) # This will give an error as there not enough elements in c_tensor to support this reshape
# c_tensor has 6 elements but the above reshape requires 8 elements
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[77], line 1
----> 1 c_tensor_reshaped_3 = c_tensor.reshape(2,6) # This will give an error as there not enough elements in c_tensor to support this reshape
2 # c_tensor has 6 elements but the above reshape requires 8 elements
RuntimeError: shape '[2, 6]' is invalid for input of size 8
On occasion, it is necessary to convert between Numpy arrays and PyTorch tensors to use data stored within arrays with particular libraries or for the purposes of plotting. The following blocks of code provide examples of coverting between Numpy arrays and PyTorch tensors.
# Converting a numpy array to a torch tensor array
array_numpy = np.array([[1.0,2.0,3.0],[4.0,6.0,8.0]])
print(f"Numpy array: {array_numpy}")
# Two examples of methods to convert numpy array to tensor
array_tensor_1 = torch.from_numpy(array_numpy)
print(f"Tensor conversion 1: {array_tensor_1}")
array_tensor_2 = torch.tensor(array_numpy)
print(f"Tensor conversion 2: {array_tensor_2}")
Numpy array: [[1. 2. 3.]
[4. 6. 8.]]
Tensor conversion 1: tensor([[1., 2., 3.],
[4., 6., 8.]], dtype=torch.float64)
Tensor conversion 2: tensor([[1., 2., 3.],
[4., 6., 8.]], dtype=torch.float64)
# Converting torch tensor array to numpy array
array_tensor = torch.tensor([[1.0,2.0,3.0],[4.0,6.0,8.0]])
print(f"Tensor array: {array_tensor}")
array_numpy = array_tensor.numpy()
print(f"Numpy array: {array_numpy}")
Tensor array: tensor([[1., 2., 3.],
[4., 6., 8.]])
Numpy array: [[1. 2. 3.]
[4. 6. 8.]]
Tensor computations can be performed both on a CPU as well as a GPU. However, using a GPU can be beneficial due to the acceleration that GPU tensor computations can provide. To indicate whether a GPU or CPU should be used for the computations, it is necessary to specify the device for a tensor array. It is also sometimes necessary to transfer a tensor hosted on the GPU to the CPU or vice versa for other computations or plotting purposes. This transfer is also necessary before converting the tensor array to a numpy array.
Additionally, you may have noticed the dtype being printed for a tensor. The dtype indicates the floating point precision for the tensor array. torch.float32 indicates 32-bit or single precision and torch.float64 indicates 64-bit or double precision. The following code blocks provide examples on specifying the device and dtype for a tensor array. They also show examples of transferring tensors between GPUs and CPUs on a machine.
The first code block demonstrates how to check if a GPU is available for tensor computations. If a dedicated GPU with CUDA is not available on your machine, then this will show False. Otherwise, it should show true.
# Checking to see if GPU is available
print(torch.cuda.is_available())
True
# Defining a tensor with explicitly mentioned device and dtype
d_tensor = torch.tensor([[1.0,2.0], [3.0,4.0]], dtype=torch.float32, device="cpu") # "cpu" indicates that the CPU is being used for computations
print(f"d_tensor dtype: {d_tensor.dtype}")
print(f"d_tensor device: {d_tensor.device}")
e_tensor = torch.tensor([[1.0,2.0], [3.0,4.0]], dtype=torch.float64, device="cuda") # "cuda" indicates that the GPU is being used for computations
print(f"e_tensor dtype: {e_tensor.dtype}")
print(f"e_tensor device: {e_tensor.device}")
# Another method for changing dtype and device
f_tensor = torch.tensor([[1.0,2.0], [3.0,4.0]], dtype=torch.float64)
f_tensor = f_tensor.to(dtype=torch.float32, device="cuda")
print(f"f_tensor dtype: {f_tensor.dtype}")
print(f"f_tensor device: {f_tensor.device}")
# It is also possible to use the to() method to set the dtype and device of one tensor the same as another tensor
g_tensor = torch.tensor([[1.0,2.0], [3.0,4.0]], dtype=torch.float32, device="cpu")
g_tensor = f_tensor.to(e_tensor)
print(f"g_tensor dtype: {g_tensor.dtype}")
print(f"g_tensor device: {g_tensor.device}")
# g_tensor has same dtype and device as e_tensor even though the original definition is different
d_tensor dtype: torch.float32
d_tensor device: cpu
e_tensor dtype: torch.float64
e_tensor device: cuda:0
f_tensor dtype: torch.float32
f_tensor device: cuda:0
g_tensor dtype: torch.float64
g_tensor device: cuda:0
# Transferring a GPU tensor to the CPU
e_tensor_cpu = e_tensor.cpu()
print(f"Tensor device: {e_tensor_cpu.device}") # Device has been converted to cpu
Tensor device: cpu
# Converting to numpy can only be done after the tensor is transferred to the CPU
e_tensor_numpy = e_tensor.cpu().numpy()
print(f"e_tensor data type: {type(e_tensor)}") # tensor has been converted to numpy
print(f"e_tensor_numpy data type: {type(e_tensor_numpy)}") # tensor has been converted to numpy
# The below code line will give an error since the tensor has not been transferred to the cpu first
e_tensor_numpy = e_tensor.numpy()
e_tensor data type: <class 'torch.Tensor'>
e_tensor_numpy data type: <class 'numpy.ndarray'>
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[83], line 7
4 print(f"e_tensor_numpy data type: {type(e_tensor_numpy)}") # tensor has been converted to numpy
6 # The below code line will give an error since the tensor has not been transferred to the cpu first
----> 7 e_tensor_numpy = e_tensor.numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.