PyTorch Tensor – A Detailed Overview

Filed Under: PyTorch
Pytorch Tensor

In this PyTorch tutorial, we’ll discuss PyTorch Tensor, which are the building blocks of this Deep Learning Framework.

Let’s get started!


PyTorch Tensor

Have you worked with Python numpy before? If yes, then this section is going to be very simple for you! Even if you don’t have experience with numpy, you can seamlessly transition between PyTorch and NumPy!

A Tensor in PyTorch is similar to numpy arrays, with the additional flexibility of using a GPU for calculations.

1. 2D Pytorch Tensor

Imagine a tensor as an array of numbers, with a potentially arbitrary number of dimensions. The only difference between a Tensor and a multidimensional array in C/C++/Java is that the size of all the columns in a dimension is the same.

For example, the below can be a valid representation of a 2 Dimensional Tensor.

[[1 2 3 4],
 [5 6 7 8]]

Note, however, that the below example is NOT a valid example, since Tensors are not jagged arrays.

[[1 2 3 4],
 [5 6 7]]

PyTorch Tensors are really convenient for programmers, since they are almost the same as numpy arrays.

There are a couple of differences to numpy methods, though, so it is advised that you also refer the official Documentation for further information.

2. Initializing an Empty PyTorch Tensor

Let’s consider the below example, which initializes an empty Tensor.

import torch 
# Creates a 3 x 2 matrix which is empty
a = torch.empty(3, 2)

An empty tensor does NOT mean that it does not contain anything. It’s just that there is memory allocated for it.

import torch 
# Creates a 3 x 2 matrix which is empty
a = torch.empty(3, 2)
print(a)

# Create a zero initialized float tensor
b = torch.zeros(3, 2, dtype=torch.float32)
print(b)

Output

tensor([[3.4655e-37, 0.0000e+00],
        [4.4842e-44, 0.0000e+00],
        [       nan, 6.1657e-44]])
tensor([[0., 0.],
        [0., 0.],
        [0., 0.]])

The first tensor is a result of PyTorch simply allocating memory for the tensor. Whatever previous content in the memory is not erased.

The second tensor is filled with zeros, since PyTorch allocates memory and zero-initializes the tensor elements.

Notice the similarity to numpy.empty() and numpy.zeros(). This is because PyTorch is designed to replace numpy, since the GPU is available.

3. Finding PyTorch Tensor Size

Let’s create a basic tensor and determine its size.

import torch 
# Create a tensor from data
c = torch.tensor([[3.2 , 1.6, 2], [1.3, 2.5 , 6.9]])
print(c)

Output

tensor([[3.2000, 1.6000, 2.0000],
        [1.3000, 2.5000, 6.9000]])

To get the size of the tensor, we can use tensor.size()

print(c.size())

Output

torch.Size([2, 3])

PyTorch Tensor Operations

Like numpy, PyTorch supports similar tensor operations.

The summary is given in the below code block.

1. Basic Mathematical Operations on Tensors

import torch 
# Tensor Operations
x = torch.tensor([[2, 3, 4], [5, 6, 7]])
y = torch.tensor([[2, 3, 4], [1.3, 2.6, 3.9]])

# Addition
print(x + y)
# We can also use torch.add()
print(x + y == torch.add(x, y))

# Subtraction
print(x - y)
# We can also use torch.sub()
print(x-y == torch.sub(x, y))

Output

tensor([[ 4.0000,  6.0000,  8.0000],
        [ 6.3000,  8.6000, 10.9000]])
tensor([[True, True, True],
        [True, True, True]])
tensor([[0.0000, 0.0000, 0.0000],
        [3.7000, 3.4000, 3.1000]])
tensor([[True, True, True],
        [True, True, True]])

We can also assign the result to a tensor. Add the following code snippet to the code above.

# We can assign the output to a tensor
z = torch.zeros(x.shape)
torch.add(x, y, out=z)
print(z)

Output

tensor([[ 4.0000,  6.0000,  8.0000],
        [ 6.3000,  8.6000, 10.9000]])

2. Inline Addition and Subtraction with PyTorch Tensor

PyTorch also supports in-place operations like addition and subtraction, when suffixed with an underscore (_). Let’s continue on with the same variables from the operations summary code above.

# In-place addition
print('Before In-Place Addition:', y)
y.add_(x)
print('After addition:', y)

Output

Before In-Place Addition: tensor([[2.0000, 3.0000, 4.0000],
        [1.3000, 2.6000, 3.9000]])
After addition: tensor([[ 4.0000,  6.0000,  8.0000],
        [ 6.3000,  8.6000, 10.9000]])

3. Accessing Tensor Index

We can also use numpy based indexing in PyTorch

# Use numpy slices for indexing
print(y[:, 1]

Output

tensor([6.0000, 8.6000])

Reshape a PyTorch Tensor

Similar to numpy, we can use torch.reshape() to reshape a tensor. We can also use tensor.view() to achieve the same functionality.

import torch 
x = torch.randn(5, 3)
# Return a view of the x, but only having 
# one dimension
y = x.view(5 * 3)

print('Size of x:', x.size())
print('Size of y:', y.size())

print(x)
print(y)

# Get back the original tensor with reshape()
z = y.reshape(5, 3)
print(z)

Output

Size of x: torch.Size([5, 3])
Size of y: torch.Size([15])

tensor([[ 0.3224,  0.1021, -1.4290],
        [-0.3559,  0.2912, -0.1044],
        [ 0.3652,  2.3112,  1.4784],
        [-0.9630, -0.2499, -1.3288],
        [-0.0667, -0.2910, -0.6420]])

tensor([ 0.3224,  0.1021, -1.4290, -0.3559,  0.2912, -0.1044,  0.3652,  2.3112,
         1.4784, -0.9630, -0.2499, -1.3288, -0.0667, -0.2910, -0.6420])

tensor([[ 0.3224,  0.1021, -1.4290],
        [-0.3559,  0.2912, -0.1044],
        [ 0.3652,  2.3112,  1.4784],
        [-0.9630, -0.2499, -1.3288],
        [-0.0667, -0.2910, -0.6420]])

The list of all Tensor Operations is available in PyTorch’s Documentation.

PyTorch – NumPy Bridge

We can convert PyTorch tensors to numpy arrays and vice-versa pretty easily.

PyTorch is designed in such a way that a Torch Tensor on the CPU and the corresponding numpy array will have the same memory location. So if you change one of them, the other one will automatically be changed.

To prove this, let’s test it using the torch.numpy() and the torch.from_numpy() methods.

torch.numpy() is used to convert a Tensor to a numpy array, and torch.from_numpy() will do the reverse.

import torch 
# We also need to import numpy to declare numpy arrays
import numpy as np

a = torch.tensor([[1, 2, 3], [4, 5, 6]])
print('Original Tensor:', a)

b = a.numpy()
print('Tensor to a numpy array:', b)

# In-Place addition (add 2 to every element)
a.add_(2)

print('Tensor after addition:', a)

print('Numpy Array after addition:', b)

Output

Original Tensor: tensor([[1, 2, 3],
        [4, 5, 6]])
Tensor to a numpy array: [[1 2 3]
 [4 5 6]]
Tensor after addition: tensor([[3, 4, 5],
        [6, 7, 8]])
Numpy Array after addition: [[3 4 5]
 [6 7 8]]

Indeed, the numpy array has also changed it’s value!

Let’s do the reverse as well

import torch
import numpy as np

c = np.array([[4, 5, 6], [7, 8, 9]])
print('Numpy array:', c)

# Convert to a tensor
d = torch.from_numpy(c)
print('Tensor from the array:', d)

# Add 3 to each element in the numpy array
np.add(c, 3, out=c)

print('Numpy array after addition:', c)

print('Tensor after addition:', d)

Output

Numpy array: [[4 5 6]
 [7 8 9]]
Tensor from the array: tensor([[4, 5, 6],
        [7, 8, 9]])
Numpy array after addition: [[ 7  8  9]
 [10 11 12]]
Tensor after addition: tensor([[ 7,  8,  9],
        [10, 11, 12]])

NOTE: If you do not use the numpy in-place addition using a += 3 or np.add(out=a), then the Tensor will not reflect the changes in the numpy array.

For example, if you try this:

c = np.add(c, 3)

Since you’re using =, this means that Python will create a new object and assign that new object to the name called c. So the original memory location is still unchanged.

Use the CUDA GPU with a PyTorch Tensor

We can make the NVIDIA CUDA GPU perform the computations and have a speedup, by moving the tensor to the GPU.

NOTE: This applies only if you have an NVIDIA GPU with CUDA enabled. If you’re not sure of what these terms are, I would advise you to search online.

We can check if we have the GPU available for PyTorch using torch.cuda.is_available()

import torch 
if torch.cuda.is_available():
    print('Your device is supported. We can use the GPU for PyTorch!')
else:
    print('Your GPU is either not supported by PyTorch or you haven't installed the GPU version')

For me, it is available, so just make sure you install CUDA before proceeding further if your laptop supports it.

We can move a tensor from the CPU to the GPU using tensor.to(device), where device is a device object.

This can be torch.device("cuda"), or simply cpu.

import torch 
x = torch.tensor([1, 2, 3], dtype=torch.long)

if torch.cuda.is_available():
    print('CUDA is available')
    # Create a CUDA Device object
    device = torch.device("cuda")

    # Create a tensor from x and store on the GPU
    y = torch.ones_like(x, device=device)
    
    # Move the tensor from CPU to GPU
    x = x.to(device)

    # This is done on the GPU
    z = x + y
    print(z)

    # Move back to CPU and also change dtype
    print(z.to("cpu", torch.double))
    print(z)
else:
    print('CUDA is not available')

Output

CUDA is available
tensor([2, 3, 4], device='cuda:0')
tensor([2., 3., 4.], dtype=torch.float64)
tensor([2, 3, 4], device='cuda:0')

As you can see, the output does show that our program is now being run on the GPU instead!


Conclusion

In this article, we learned about using Tensors in PyTorch. Feel free to ask any doubts or even suggestions/corrections in the comment section below!

We’ll be covering more in our upcoming PyTorch tutorials. Stay tuned!


References


Leave a Reply

Your email address will not be published. Required fields are marked *

close
Generic selectors
Exact matches only
Search in title
Search in content
Search in posts
Search in pages