1. Torch.utils.data.Dataset [read Dataset]
Datasets – PyTorch datasets is a collection of source code for datasets defined by PyTorch. Here is a basic framework for custom Datasets, initialization in __init__(), where __getitem__() and __len__() methods must be overridden. __getitem__() returns training data, such as images and labels, while __len__() returns data length.
class CustomDataset(data.Dataset) :Dataset = data.dataset
def __init__(self) :
# TODO
# 1. Initialize file path or list of file names.
Initialize the file path or give a list of file names (train, test, val, etc.)
pass
def __getitem__(self, index) :
# TODO
# 1. Read one data from file (e.g. using numpy.fromfile, PIL.Image.open).
# 2. Preprocess the data (e.g. torchvision.Transform).
# 3. Return a data pair (e.g. image and label).
Train and test (train and test)
The first step: read one data is a data
pass
def __len__(self) :
# You should change 0 to the total size of your dataset.
# returns data_len
return 0
Copy the code2. The torch. Utils. Data. DataLoader [] produce batch data
The DataLoader(object) parameters are as follows:
Parameters:
- Dataset: incoming dataset
- Batch_size (int, optional): Indicates the number of samples in each batch
- Shuffle (bool, optional): Reordering the data at the beginning of each epoch
- Sampler (sampler, optional): Customizes the sampling policy from the dataset. If this parameter is specified, shuffle must be False
- batch_sampler(Sampler, optional): Sampler is similar to sampler, but only returns one batch of indices at a time. Note that once this parameter is specified, So batch_size, shuffle, sampler, drop_last can no longer made (mutual exclusion, Mutually exclusive)
- Num_workers (int, optional): This parameter determines how many processes to handle data loading. 0 means all data will be loaded into the main process. (Default: 0)
- Collate_fn (callable, optional): Combines a list of samples into a mini-batch function
- Pin_memory (bool, optional) : If set to True, the data loader will copy the Tensors to CUDA pinned memory before returning them.
- **
drop_last(bool, optional):** If set to True: this is for the last batch of unfinished, such as your BATCH_size set to 64, and an epoch with only 100 samples, then the last 36 samples will be thrown away during training. If False (the default), execution continues normally, except that the final batch_size is smaller.- **
timeout(Numeric, optional):** If it is a positive number, it indicates the waiting time for a batch wait to be collected from the worker process. If it has not been collected beyond the specified time, the content will not be collected. This numeric should always be greater than or equal to 0. The default is 0- worker_init_fn (callable, optional): If not None, this will be called on the eachworker subprocess with the worker id (an int in [0, num_workers – 1]) as input, after seeding and before data loading. (default: None)
3. Use Dataset and DataLoader to generate custom training data
Suppose the TXT file holds the image and label of the data in the following format: the first column is the name of the image, and the second column is label
0.jpg 0
1.jpg 1
2.jpg 2
3.jpg 3
4.jpg 4
5.jpg 5
6.jpg 6
7.jpg 7
8.jpg 8
9.jpg 9
Copy the codeIt can also be multi-label data, for example:
0.jpg 0 10
1.jpg 1 11
2.jpg 2 12
3.jpg 3 13
4.jpg 4 14
5.jpg 5 15
6.jpg 6 16
7.jpg 7 17
8.jpg 8 18
9.jpg 9 19
Copy the codeThen we can customize a dataset to parse these data and read the images. Then we can use the DataLoader class to generate batch training data
3.1 User-defined Dataset
First, define a TorchDataset class to read image data and generate labels:
Note the initialization function:
import torch
from torch.autograd import Variable
from torchvision import transforms
from torch.utils.data import Dataset, DataLoader
import numpy as np
from utils import image_processing
import os
class TorchDataset(Dataset) :
def __init__(self, filename, image_dir, resize_height=256, resize_width=256, repeat=1) :
":param filename: data file TXT: format: imge_name. JPG label1_id labe2_id :param image_dir: Image path: image_dir+imge_name. JPG Complete path: Param resize_height: None, no scaling :param resize_width: None, no scaling, PS: When either resize_height or resize_width is None, equal scale scaling can be achieved :param repeat: the number of times all sample data are repeated. The default cycle is once. When repeat is None, it means infinite cycle.
self.image_label_list = self.read_file(filename)
self.image_dir = image_dir
self.len = len(self.image_label_list)
self.repeat = repeat
self.resize_height = resize_height
self.resize_width = resize_width
Initialization of correlation preprocessing
'''class torchvision.transforms.ToTensor'''
Shape =(H,W,C) in the pixel range [0, 255] pil. Image or numpy.ndarray data
# converts to shape=(C,H,W) pixel data and is normalized to [0.0, 1.0] torch.FloatTensor type.
self.toTensor = transforms.ToTensor()
"' class torchvision. Transforms. The Normalize (mean, STD) the transformation on the torch. * Tensor, given the mean (R, G, and B) and standard deviation (R, G, B), Normalized by the formula channel = (channel-mean)/STD. ' ' '
# self.normalize=transforms.Normalize()
def __getitem__(self, i) :
index = i % self.len
# print("i={},index={}".format(i, index))
image_name, label = self.image_label_list[index]
image_path = os.path.join(self.image_dir, image_name)
img = self.load_data(image_path, self.resize_height, self.resize_width, normalization=False)
img = self.data_preproccess(img)
label=np.array(label)
return img, label
def __len__(self) :
if self.repeat == None:
data_len = 10000000
else:
data_len = len(self.image_label_list) * self.repeat
return data_len
def read_file(self, filename) :
image_label_list = []
with open(filename, 'r') as f:
lines = f.readlines()
for line in lines:
# rstrip: Used to remove trailing characters, whitespace (including \n, \r, \t, ' ', i.e. newline, carriage return, TAB, space)
content = line.rstrip().split(' ')
name = content[0]
labels = []
for value in content[1:]:
labels.append(int(value))
image_label_list.append((name, labels))
return image_label_list
def load_data(self, path, resize_height, resize_width, normalization) :
There is much more normalization in the process of recovery as follows: Param Path: : Param resize_height: : Param resize_width: : Param normalization: return: ""
image = image_processing.read_image(path, resize_height, resize_width, normalization)
return image
def data_preproccess(self, data) :
Param data: :return: ""
data = self.toTensor(data)
return data
Copy the code3.2 DataLoader generates batch training data
The following two ways, TorchDataset set repeat=None, exit loop by max_iterate ""
train_data = TorchDataset(filename=train_filename, image_dir=image_dir,repeat=None)
train_loader = DataLoader(dataset=train_data, batch_size=batch_size, shuffle=False)
# [2] Second iteration method
for step, (batch_image, batch_label) in enumerate(train_loader):
image=batch_image[0,:]
image=image.numpy()#image=np.array(image)
image = image.transpose(1.2.0) [c,h,w]->[h,w,c]
image_processing.cv_show_image("image",image)
print("step:{},batch_image.shape:{},batch_label:{}".format(step,batch_image.shape,batch_label))
# batch_x, batch_y = Variable(batch_x), Variable(batch_y)
if step>=max_iterate:
break
# [3] Third iteration method
# for step in range(max_iterate):
# batch_image, batch_label=train_loader.__iter__().__next__()
# image=batch_image[0,:]
# image=image.numpy()#image=np.array(image)
# image = image transpose (1, 2, 0) # channel by [c, h, w] - > [h, w, c]
# image_processing.cv_show_image("image",image)
# print("batch_image.shape:{},batch_label:{}".format(batch_image.shape,batch_label))
# # batch_x, batch_y = Variable(batch_x), Variable(batch_y)
Copy the code