Summary of ItemReader

1.ItemReader: An interface that provides data

2. In this interface, there is only one method read(), which reads one piece of data and moves on to the next. At the end of the read, a null must be returned, otherwise the data has not been read.

The interface is defined as follows:

public interface ItemReader<T> { @Nullable T read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException; }Copy the code

As many as 33 default implementations are provided, covering almost all data source read types.

Pics7.baidu.com/feed/43a7d9…

2. Create the user table in the database

The database data is as follows:

3. Use JdbcPagingItemReader to read data from the database

@Configuration
public class DBJdbcDemoJobConfiguration {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
 
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
 
    @Autowired
    @Qualifier("dbJdbcDemoWriter")
    private ItemWriter<? super Customer> dbJdbcDemoWriter;
 
    @Autowired
    private DataSource dataSource;
 
    @Bean
    public Job DBJdbcDemoJob() {return jobBuilderFactory.get("DBJdbcDemoJob")
                .start(dbJdbcDemoStep())
                .build();
 
    }
 
    @Bean
    public Step dbJdbcDemoStep() {
        return stepBuilderFactory.get("dbJdbcDemoStep")
                .<Customer,Customer>chunk(100)
                .reader(dbJdbcDemoReader())
                .writer(dbJdbcDemoWriter)
                .build();
    }
 
    @Bean
    @StepScope
    public JdbcPagingItemReader<Customer> dbJdbcDemoReader() { JdbcPagingItemReader<Customer> reader = new JdbcPagingItemReader<>(); reader.setDataSource(this.dataSource); reader.setFetchSize(100); Reader. setRowMapper((rs,rowNum)->return Customer.builder().id(rs.getLong("id"))
                    .firstName(rs.getString("firstName"))
                    .lastName(rs.getString("lastName"))
                    .birthdate(rs.getString("birthdate")) .build(); }); MySqlPagingQueryProvider queryProvider = new MySqlPagingQueryProvider(); queryProvider.setSelectClause("id, firstName, lastName, birthdate");
        queryProvider.setFromClause("from Customer"); Map<String, Order> sortKeys = new HashMap<>(1); sortKeys.put("id", Order.ASCENDING);
        queryProvider.setSortKeys(sortKeys);
 
        reader.setQueryProvider(queryProvider);
 
        returnreader; }}Copy the code

}
Copy the code

}

<br/><br/><br/>
Read data from CVS/ TXT file<br/> Put a CSV file in the resources of the project. Take reading customer.csv as an example! [file](https://graph.baidu.com/resource/222620df58b12c167892e01583251119.png) FlatFileItemReaderCopy the code

@Configuration public class FlatFileDemoJobConfiguration { @Autowired private JobBuilderFactory jobBuilderFactory;

@Autowired private StepBuilderFactory stepBuilderFactory; @Autowired @Qualifier("flatFileDemoWriter") private ItemWriter<? super Customer> flatFileDemoWriter; @Bean public Job flatFileDemoJob(){ return jobBuilderFactory.get("flatFileDemoJob") .start(flatFileDemoStep()) .build();  } @Bean public Step flatFileDemoStep() { return stepBuilderFactory.get("flatFileDemoStep") .<Customer,Customer>chunk(100) .reader(flatFileDemoReader()) .writer(flatFileDemoWriter) .build(); } @Bean @StepScope public FlatFileItemReader<Customer> flatFileDemoReader() { FlatFileItemReader<Customer> reader = new FlatFileItemReader<>(); Reader.setresource (new ClassPathResource("customer.csv")); // Skip the first line of reader.setlinestoskip (1); DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer(); tokenizer.setNames(new String[]{"id","firstName","lastName","birthdate"}); // The parsed data is mapped to the object DefaultLineMapper<Customer> lineMapper = new DefaultLineMapper<>(); lineMapper.setLineTokenizer(tokenizer); lineMapper.setFieldSetMapper((fieldSet -> { return Customer.builder().id(fieldSet.readLong("id")) .firstName(fieldSet.readString("firstName")) .lastName(fieldSet.readString("lastName")) .birthdate(fieldSet.readString("birthdate")) .build(); })); lineMapper.afterPropertiesSet(); reader.setLineMapper(lineMapper); return reader; }Copy the code

}

<br/> Output methodCopy the code

@Component(“flatFileDemoWriter”) public class FlatFileDemoWriter implements ItemWriter { @Override public void write(List<? extends Customer> items) throws Exception { for (Customer customer:items) System.out.println(customer);

}
Copy the code

}

<br/> Print as follows:! [file](https://graph.baidu.com/resource/222b0a0a173d9936b8ec601583252983.png) <br/><br/><br/>## The file reads and writes FlatFileItem< br / > use FlatFileItemReader FlatFileItemWriter help us to do? FlatFileItem reads and writes to a fixed length (especially important for large files), so developers don't have to worry about file read/write flows.# # #, rounding FlatFileItemReader<br/> FlatFileItemReader <br/> FlatFileItemReader is a class that reads files, usually tabular or text file data. The following two properties of this class are requiredset*setResource Specifies the location of the file Resource: specify the file to be read using the ClassPathResource or FileSystemResourcesetLineMapper Line mapping: Specifies the mapping between lines and entity objects. The sample code uses DefaultLineMapper * seEncoding to read the encoding, which defaults to 'ISO-8859-1' *setStrict In Strict mode, if the input file does not exist, an exception is thrown and the current job is blocked. The default istrue<br/> Sample code:Copy the code

@Bean public FlatFileItemReader csvItemReader() { FlatFileItemReader csvItemReader = new FlatFileItemReader<>(); csvItemReader.setResource(new ClassPathResource(“data/sample-data.csv”)); csvItemReader.setLineMapper(new DefaultLineMapper() {{ setLineTokenizer(new DelimitedLineTokenizer() {{ setNames(new String[]{“name”, “age”}); }}); setFieldSetMapper(new BeanWrapperFieldSetMapper() {{ setTargetType(Person.class); }}); }}); return csvItemReader; }

<br/>

# # #, rounding FlatFileItemWriter<br/> FlatFileItemWriter <br/> FlatFileItemWriter <br/> FlatFileItemWriter <br/> FlatFileItemWriter <br/> FlatFileItemWriter <br/> FlatFileItemWritersetLineAggregator and FlatFileItemReadersetThe LineMapper method has similarities,setThe LineAggregator method aggregates object properties into strings, with delimiters as needed (setDelimiter), and the corresponding character names of the object attributes (setFieldExtractor) - LineAggregator interface is to create an aggregate string of object properties - ExtractorLineAggregator is an abstract class that implements the LineAggregator interface. The FieldExtractor is used to convert object properties to arrays. The extension class is responsible for converting arrays to strings (doAggregate) - DelimitedLineAggregator Inherits ExtractorLineAggregator. A more commonly used aggregator that splits arrays with specified symbols and inherits the ExtractorLineAggregator by default using comma-FormatterlineAggregator. The maximum length of an array of strings, the minimum length check, and formatting - PassThroughLineAggregator achieve LineAggregator interface, using the object is a kind of simple polymerization. The toString () returns a value, As polymerization string - RecursiveCollectionLineAggregator LineAggregator interface, the Collection < T > Collection traversal, Collection of polymerization by division system line separator, Aggregation of object fields is optional using the aggregation method corresponding to the LineAggregator interface. -setResource specifies the location of the output file, which is also required. The example code uses new ClassPathResource("/data/sample-data.txt"The actual development is more new FilePathResource() -setEncoding Sets the Encoding, default is ISO-8859-1 <br/>! [file] (https://graph.baidu.com/resource/222784a7966e50e2edb1101583297793.png) < br / > the sample code:Copy the code

@Bean public FlatFileItemWriter txtItemWriter() { FlatFileItemWriter txtItemWriter = new FlatFileItemWriter<>(); txtItemWriter.setAppendAllowed(true); txtItemWriter.setEncoding(“UTF-8”); txtItemWriter.setResource(new ClassPathResource(“/data/sample-data.txt”)); txtItemWriter.setLineAggregator(new DelimitedLineAggregator() {{ setDelimiter(“,”); setFieldExtractor(new BeanWrapperFieldExtractor() {{ setNames(new String[]{“name”, “age”}); }}); }}); return txtItemWriter; }

<br/>
<br/>
<br/>


Read data from an XML file1. Use StaxEventItemReader<T> to read XML data 2. Example: Add a customer. XML file to your project. Use reading this file as an example ** XML file to read **! [file] (https://graph.baidu.com/resource/22266ba2119d2ce23921a01583296313.png) < br / > pom. * * * * XML configurationCopy the code

StaxEventItemReader

@Configuration
public class XmlFileDemoJobConfiguration {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
 
    @Autowired
    private StepBuilderFactory stepBuilderFactory;
 
    @Autowired
    @Qualifier("xmlFileDemoWriter")
    private ItemWriter<? super Customer> xmlFileDemoWriter;
 
    @Bean
    public Job xmlFileDemoJob() {return jobBuilderFactory.get("xmlFileDemoJob")
                .start(xmlFileDemoStep())
                .build();
 
    }
 
    @Bean
    public Step xmlFileDemoStep() {
        return stepBuilderFactory.get("xmlFileDemoStep")
                .<Customer,Customer>chunk(10)
                .reader(xmlFileDemoReader())
                .writer(xmlFileDemoWriter)
                .build();
    }
 
    @Bean
    @StepScope
    public StaxEventItemReader<Customer> xmlFileDemoReader() {
        StaxEventItemReader<Customer> reader = new StaxEventItemReader<>();
 
        reader.setResource(new ClassPathResource("customer.xml")); / / specified need to deal with the root tag of the reader. SetFragmentRootElementName ("customer"); Map<String,Class> Map = new HashMap<>(); map.put("customer",Customer.class); XStreamMarshaller unMarshaller = new XStreamMarshaller(); unMarshaller.setAliases(map); reader.setUnmarshaller(unMarshaller);returnreader; }}Copy the code

}
Copy the code

}

<br/> Print as follows:! [file](https://graph.baidu.com/resource/222ac917e105e5648fe1c01583297500.png) <br/><br/><br/>## XML file processing<br/> XML file processing needs to introduce the Spring-OXM package, only the OUTPUT of XML is detailed, XML reading similar to XML writing object StaxEventItemWriter, similar to FlatFileItemWriter use. Both StaxEventItemWriter and FlatFileItemWriter existsetThe StaxEventItemWriter is encoded utF-8 * by defaultsetRootTagName Sets the root node label name *setMarshaller <br/>Copy the code

@Bean public StaxEventItemWriter xmlItemWriter() { StaxEventItemWriter xmlItemWriter = new StaxEventItemWriter<>(); xmlItemWriter.setRootTagName(“root”) xmlItemWriter.setEncoding(“UTF-8”); xmlItemWriter.setResource(new ClassPathResource(“/data/sample-data.xml”)); xmlItemWriter.setMarshaller(new XStreamMarshaller() {{ Map<String, Class> map = new HashMap<>(); map.put(“person”,Person.class); setAliases(map); }}); return xmlItemWriter; }

<br/>
<br/>
<br/>


Read data from multiple files<br/> 1. A very common 2 when multiple files are read from a given directory at a time. We can use MultiResourceItemReader to register an input file and set up the proxy ItemReader to handle each source file. Example: We store three CSV files starting with file in the project classpath as follows: [file](https://graph.baidu.com/resource/222cdef6af54e1131e13b01583300700.png) <br/> MultiResourceItemReaderCopy the code

@Configuration public class MultipleFileDemoJobConfiguration { @Autowired private JobBuilderFactory jobBuilderFactory;

@Autowired
private StepBuilderFactory stepBuilderFactory;

@Autowired
@Qualifier("multiFileDeWriter")
private ItemWriter<? super Customer> multiFileDeWriter;

@Value("classpath*:/file*.csv")
private Resource[] inputFiles;

@Bean
public Job multipleFileDemoJob(){
    return jobBuilderFactory.get("multipleFileDemoJob")
            .start(multipleFileDemoStep())
            .build();

}

@Bean
public Step multipleFileDemoStep() {
    return stepBuilderFactory.get("multipleFileDemoStep")
            .<Customer,Customer>chunk(50)
            .reader(multipleResourceItemReader())
            .writer(multiFileDeWriter)
            .build();
}

private MultiResourceItemReader<Customer> multipleResourceItemReader() {

    MultiResourceItemReader<Customer> reader = new MultiResourceItemReader<>();

    reader.setDelegate(flatFileReader());
    reader.setResources(inputFiles);

    return reader;
}

@Bean
public FlatFileItemReader<Customer> flatFileReader() {
    FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
    reader.setResource(new ClassPathResource("customer.csv"));
   // reader.setLinesToSkip(1);

    DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
    tokenizer.setNames(new String[]{"id","firstName","lastName","birthdate"});

    DefaultLineMapper<Customer> lineMapper = new DefaultLineMapper<>();
    lineMapper.setLineTokenizer(tokenizer);
    lineMapper.setFieldSetMapper((fieldSet -> {
        return Customer.builder().id(fieldSet.readLong("id"))
                .firstName(fieldSet.readString("firstName"))
                .lastName(fieldSet.readString("lastName"))
                .birthdate(fieldSet.readString("birthdate"))
                .build();
    }));
    lineMapper.afterPropertiesSet();

    reader.setLineMapper(lineMapper);

    return reader;

}
Copy the code

}

<br/> Output methodCopy the code

@Component(“multiFileDeWriter”) public class MultiFileDeWriter implements ItemWriter { @Override public void write(List<? extends Customer> items) throws Exception { for (Customer customer:items) System.out.println(customer);

}
Copy the code

}

<br/> Print as follows:! [file] (https://graph.baidu.com/resource/22239a27970444932658a01583300958.png) < br / > < br / > the reference:  https://blog.csdn.net/wuzhiwei549/article/details/88592509 https://blog.51cto.com/13501268/2298081 https://www.jianshu.com/p/9b7088471371Copy the code