Python lists or tuples of numbers are collections of Python objects that are individually allocated in memory, as shown on the left in figure. PyTorch tensors or NumPy arrays, on the other hand, are views over (typically) contiguous memory blocks containing unboxed C numeric types rather than Python objects. Each element is a 32-bit (4-byte) float in this case, as we can see on the right side of figure . This means storing a 1D tensor of 1,000,000 float numbers will require exactly 4,000,000 contiguous bytes, plus a small overhead for the metadata (such as dimensions and numeric type).

Making everything a python object is called boxing

Tensors are multidimensional arrays, and Pytorch provides a high level API for the purpose of providing efficient computations if needed. Numpy arrays cannot be operated upon by GPU's

Understanding How Tensors are stored

The contiguous chunks of memory managed by torch.Storageinstances. A storage is a one-dimensional array of numerical data: that is, a contiguous block of memory containing numbers of a given type, such as float (32 bits representing a floating-point number) or int64 (64 bits representing an integer). A PyTorch Tensor instance is a view of such a Storage instance that is capable of indexing into that storage using an offset and per-dimension strides.

⭐Multiple tensors can Index into the same storage, even if they index into the data differently. Therefore changing the storage through one tensor will change all those tensors who have the same offset.

This indirection between Tensor and Storage makes some operations inexpensive, like transposing a tensor or extracting a subtensor, because they do not lead to memory reallocations. Instead, they consist of allocating a new Tensor object with a different value for size, storage offset, or stride.

• Offset - The starting position/address of the tensor(address where the first element of tensor is stored)

• Stride - No. of elements to be skipped when the index is increased by 1 in each dimension.

• size - size of the tensor to be stored

#to access the offset
name_of_tensor.storage_offset()
#to access the stride of the storage
name_of_tensor.stride()

Accessing an element i, j in a 2D tensor results in accessing the storage_offset + stride[0] * i + stride[1] * j element in the storage.

See this practically 👇👇

This example illustrates how a tensor and it's transpose have the same underlying memory and that they differ only in shape and stride:

# In:
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
points
# Out:
tensor([[4., 1.],
				[5., 3.],
				[2., 1.]])
# In:
points_t = points.t()
points_t
# Out:
tensor([[4., 5., 2.],
				[1., 3., 1.]])
# In:
id(points.storage()) == id(points_t.storage())
# Out:
True

# In:
points.stride()
# Out:
(2, 1)
# In[34]:
points_t.stride()
# Out:
(1, 2)

Avoid the same underlying storage

points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
second_point = points[1].clone()
second_point[0] = 10.0
points
# Out[29]:
tensor([[4., 1.],
				[5., 3.],
				[2., 1.]])

Contiguous Tensors

A tensor whose values are laid out in the storage starting from the rightmost dimension onward (that is, moving along rows for a 2D tensor) is defined as contiguous. Contiguous tensors are convenient because we can visit them efficiently in order without jumping around in the storage (improving data locality improves performance because of the way memory access works on modern CPUs). This advantage of course depends on the way algorithms visit.

check if contiguous or not - name_of_tensor.is_contigouos()

convert to contigous - name_of_tensor_ctg = name_of_tensor.contiguous() .⭐This only reshuffles the storage and not allot a new storage

Moving tensors to GPU

points_gpu = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]], device='cuda')
or
points_gpu = points.to(device='cuda')

points_gpu = points.cuda() #defaults GPUindex 0
points_gpu = points.cuda(0)
points_cpu = points_gpu.cpu()

Note that the points_gpu tensor is not brought back to the CPU once the result has been computed. Here’s what happened in this line: 1 The points tensor is copied to the GPU. 2 A new tensor is allocated on the GPU and used to store the result of the multiplication. 3 A handle to that GPU tensor is returned. Therefore, if we also add a constant to the result

Serializing Tensors

Learn How to save our files in Tensor files

PyTorch uses pickle under the hood to serialize the tensor object, plus dedicated serialization code for the storage. Here’s how we can save our points tensor to an ourpoints.t file

torch.save(points, '../data/p1ch3/ourpoints.t')
#OR
with open('../data/p1ch3/ourpoints.t','wb') as f:
torch.save(points, f)

#Loading our points back is similarly a one-liner
# In:
points = torch.load('../data/p1ch3/ourpoints.t')
#or, equivalently,
# In:
with open('../data/p1ch3/ourpoints.t','rb') as f:
points = torch.load(f)

WE CANT READ TENSORS WITH A SOFTWARE OTHER THAN PYTORCH