In this PyTorch tutorial, we’ll discuss PyTorch Tensor, which are the building blocks of this Deep Learning Framework.
Let’s get started!
Have you worked with Python numpy before? If yes, then this section is going to be very simple for you! Even if you don’t have experience with numpy, you can seamlessly transition between PyTorch and NumPy!
A Tensor in PyTorch is similar to
numpy arrays, with the additional flexibility of using a GPU for calculations.
1. 2D Pytorch Tensor
Imagine a tensor as an array of numbers, with a potentially arbitrary number of dimensions. The only difference between a Tensor and a multidimensional array in C/C++/Java is that the size of all the columns in a dimension is the same.
For example, the below can be a valid representation of a 2 Dimensional Tensor.
[[1 2 3 4], [5 6 7 8]]
Note, however, that the below example is NOT a valid example, since Tensors are not jagged arrays.
[[1 2 3 4], [5 6 7]]
PyTorch Tensors are really convenient for programmers, since they are almost the same as
There are a couple of differences to
numpy methods, though, so it is advised that you also refer the official Documentation for further information.
2. Initializing an Empty PyTorch Tensor
Let’s consider the below example, which initializes an empty Tensor.
import torch # Creates a 3 x 2 matrix which is empty a = torch.empty(3, 2)
An empty tensor does NOT mean that it does not contain anything. It’s just that there is memory allocated for it.
import torch # Creates a 3 x 2 matrix which is empty a = torch.empty(3, 2) print(a) # Create a zero initialized float tensor b = torch.zeros(3, 2, dtype=torch.float32) print(b)
tensor([[3.4655e-37, 0.0000e+00], [4.4842e-44, 0.0000e+00], [ nan, 6.1657e-44]]) tensor([[0., 0.], [0., 0.], [0., 0.]])
The first tensor is a result of PyTorch simply allocating memory for the tensor. Whatever previous content in the memory is not erased.
The second tensor is filled with zeros, since PyTorch allocates memory and zero-initializes the tensor elements.
Notice the similarity to
numpy.zeros(). This is because PyTorch is designed to replace
numpy, since the GPU is available.
3. Finding PyTorch Tensor Size
Let’s create a basic tensor and determine its size.
import torch # Create a tensor from data c = torch.tensor([[3.2 , 1.6, 2], [1.3, 2.5 , 6.9]]) print(c)
tensor([[3.2000, 1.6000, 2.0000], [1.3000, 2.5000, 6.9000]])
To get the size of the tensor, we can use tensor.size()
PyTorch Tensor Operations
numpy, PyTorch supports similar tensor operations.
The summary is given in the below code block.
1. Basic Mathematical Operations on Tensors
import torch # Tensor Operations x = torch.tensor([[2, 3, 4], [5, 6, 7]]) y = torch.tensor([[2, 3, 4], [1.3, 2.6, 3.9]]) # Addition print(x + y) # We can also use torch.add() print(x + y == torch.add(x, y)) # Subtraction print(x - y) # We can also use torch.sub() print(x-y == torch.sub(x, y))
tensor([[ 4.0000, 6.0000, 8.0000], [ 6.3000, 8.6000, 10.9000]]) tensor([[True, True, True], [True, True, True]]) tensor([[0.0000, 0.0000, 0.0000], [3.7000, 3.4000, 3.1000]]) tensor([[True, True, True], [True, True, True]])
We can also assign the result to a tensor. Add the following code snippet to the code above.
# We can assign the output to a tensor z = torch.zeros(x.shape) torch.add(x, y, out=z) print(z)
tensor([[ 4.0000, 6.0000, 8.0000], [ 6.3000, 8.6000, 10.9000]])
2. Inline Addition and Subtraction with PyTorch Tensor
PyTorch also supports in-place operations like addition and subtraction, when suffixed with an underscore (_). Let’s continue on with the same variables from the operations summary code above.
# In-place addition print('Before In-Place Addition:', y) y.add_(x) print('After addition:', y)
Before In-Place Addition: tensor([[2.0000, 3.0000, 4.0000], [1.3000, 2.6000, 3.9000]]) After addition: tensor([[ 4.0000, 6.0000, 8.0000], [ 6.3000, 8.6000, 10.9000]])
3. Accessing Tensor Index
We can also use numpy based indexing in PyTorch
# Use numpy slices for indexing print(y[:, 1]
Reshape a PyTorch Tensor
numpy, we can use
torch.reshape() to reshape a tensor. We can also use
tensor.view() to achieve the same functionality.
import torch x = torch.randn(5, 3) # Return a view of the x, but only having # one dimension y = x.view(5 * 3) print('Size of x:', x.size()) print('Size of y:', y.size()) print(x) print(y) # Get back the original tensor with reshape() z = y.reshape(5, 3) print(z)
Size of x: torch.Size([5, 3]) Size of y: torch.Size() tensor([[ 0.3224, 0.1021, -1.4290], [-0.3559, 0.2912, -0.1044], [ 0.3652, 2.3112, 1.4784], [-0.9630, -0.2499, -1.3288], [-0.0667, -0.2910, -0.6420]]) tensor([ 0.3224, 0.1021, -1.4290, -0.3559, 0.2912, -0.1044, 0.3652, 2.3112, 1.4784, -0.9630, -0.2499, -1.3288, -0.0667, -0.2910, -0.6420]) tensor([[ 0.3224, 0.1021, -1.4290], [-0.3559, 0.2912, -0.1044], [ 0.3652, 2.3112, 1.4784], [-0.9630, -0.2499, -1.3288], [-0.0667, -0.2910, -0.6420]])
The list of all Tensor Operations is available in PyTorch’s Documentation.
PyTorch – NumPy Bridge
We can convert PyTorch tensors to numpy arrays and vice-versa pretty easily.
PyTorch is designed in such a way that a Torch Tensor on the CPU and the corresponding
numpy array will have the same memory location. So if you change one of them, the other one will automatically be changed.
To prove this, let’s test it using the
torch.numpy() and the
torch.numpy() is used to convert a Tensor to a numpy array, and torch.from_numpy() will do the reverse.
import torch # We also need to import numpy to declare numpy arrays import numpy as np a = torch.tensor([[1, 2, 3], [4, 5, 6]]) print('Original Tensor:', a) b = a.numpy() print('Tensor to a numpy array:', b) # In-Place addition (add 2 to every element) a.add_(2) print('Tensor after addition:', a) print('Numpy Array after addition:', b)
Original Tensor: tensor([[1, 2, 3], [4, 5, 6]]) Tensor to a numpy array: [[1 2 3] [4 5 6]] Tensor after addition: tensor([[3, 4, 5], [6, 7, 8]]) Numpy Array after addition: [[3 4 5] [6 7 8]]
Indeed, the numpy array has also changed it’s value!
Let’s do the reverse as well
import torch import numpy as np c = np.array([[4, 5, 6], [7, 8, 9]]) print('Numpy array:', c) # Convert to a tensor d = torch.from_numpy(c) print('Tensor from the array:', d) # Add 3 to each element in the numpy array np.add(c, 3, out=c) print('Numpy array after addition:', c) print('Tensor after addition:', d)
Numpy array: [[4 5 6] [7 8 9]] Tensor from the array: tensor([[4, 5, 6], [7, 8, 9]]) Numpy array after addition: [[ 7 8 9] [10 11 12]] Tensor after addition: tensor([[ 7, 8, 9], [10, 11, 12]])
NOTE: If you do not use the
numpy in-place addition using
a += 3 or
np.add(out=a), then the Tensor will not reflect the changes in the numpy array.
For example, if you try this:
c = np.add(c, 3)
Since you’re using
=, this means that Python will create a new object and assign that new object to the name called
c. So the original memory location is still unchanged.
Use the CUDA GPU with a PyTorch Tensor
We can make the NVIDIA CUDA GPU perform the computations and have a speedup, by moving the tensor to the GPU.
NOTE: This applies only if you have an NVIDIA GPU with CUDA enabled. If you’re not sure of what these terms are, I would advise you to search online.
We can check if we have the GPU available for PyTorch using
import torch if torch.cuda.is_available(): print('Your device is supported. We can use the GPU for PyTorch!') else: print('Your GPU is either not supported by PyTorch or you haven't installed the GPU version')
For me, it is available, so just make sure you install CUDA before proceeding further if your laptop supports it.
We can move a tensor from the CPU to the GPU using
device is a device object.
This can be
torch.device("cuda"), or simply
import torch x = torch.tensor([1, 2, 3], dtype=torch.long) if torch.cuda.is_available(): print('CUDA is available') # Create a CUDA Device object device = torch.device("cuda") # Create a tensor from x and store on the GPU y = torch.ones_like(x, device=device) # Move the tensor from CPU to GPU x = x.to(device) # This is done on the GPU z = x + y print(z) # Move back to CPU and also change dtype print(z.to("cpu", torch.double)) print(z) else: print('CUDA is not available')
CUDA is available tensor([2, 3, 4], device='cuda:0') tensor([2., 3., 4.], dtype=torch.float64) tensor([2, 3, 4], device='cuda:0')
As you can see, the output does show that our program is now being run on the GPU instead!
In this article, we learned about using Tensors in PyTorch. Feel free to ask any doubts or even suggestions/corrections in the comment section below!
We’ll be covering more in our upcoming PyTorch tutorials. Stay tuned!