Cockroach crawler: Another Java crawler implementation

• Java8 applications use some of the new features of java8
• maven

Here is a point-by-point introduction:

Public static void main(String[] args){CockroachConfig config = new CockroachConfig().setAppName(" CockroachConfig ") .setThread(2); CockroachContext context = new CockroachContext(config); TaskQueue queue = TaskQueue.of(); context.start(queue); New Thread(() -> {int I = 0; int I = 0; int I = 0; while(true){ i++; try { Thread.sleep(1000); String url = "http://www.xicidaili.com/wt/"+i; System.out.println(url); queue.push(new Task(url)); } catch (InterruptedException e) { e.printStackTrace(); } if (i > 1000) { break; } } }).start(); }Copy the code

First let’s try a custom client

Public class SelfHttpClient implements HttpClient {public HttpClient setProxy(HttpProxy proxy){// Set the proxy implementation method} public TaskResponse doGet(Task Task) throws Exception{// Get request implementation method} public HttpClient proxy(){// Apply the proxy to the HTTP client method} public TaskResponse doPost(Task Task) throws Exception{// Post request implementation method} public HttpClient setCookie(String cookie){// Set Public HttpClient setHttpHeader(Map<String, String> httpHeader){// Set header implementation method}}Copy the code

Apply custom HTTP clients to crawlers

CockroachConfig config = new CockroachConfig().setAppName(" I am a cockroach ").setThread(2) // Number of crawler threads .setHttpClient(SelfHttpClient.class)Copy the code

When an error occurs in an HTTP request page, the system will locate the error handling class. If there is no custom error handling class, the system will use DefaultTaskErrorHandler by default, which will print the error information. The specific implementation code is as follows.

public class DefaultTaskErrorHandler implements ITaskErrorHandler { private Logger logger = Logger.getLogger(DefaultTaskErrorHandler.class); @Override public void error(Task task,String message) { logger.info("task error: "+message); }}Copy the code

If you need to customize your error handling class, you can copy the code above and implement the ITaskErrorHandler interface to implement your own processing logic in the error method.

After we define our custom error handling class, we need to apply the custom class to the crawler.

CockroachConfig config = new CockroachConfig().setAppName(" I am a cockroach ").setThread(2) // Number of crawler threads .setHttpClient(SelfHttpClient.class) .setStore(SelfStore.class) .setTaskErrorHandler(SelfTaskErrorHandler.class);Copy the code

CockroachConfig config = new CockroachConfig().setAppName(" I am a cockroach ").setThread(2) // Number of crawler threads SetHttpClient (SelfHttpClient. Class). SetProxys (" 100.100.100.100:8888101101 101.101:8888)"Copy the code

As shown above, we can set several proxy IP addresses and eventually generate a proxy pool for all proxy IP addresses. Before crawler request, we will randomly select an IP address from the proxy pool to act as proxy.

Address user-Agent issues in HTTP requests

A user-Agent pool is implemented in the program, and each request will randomly take out a User-Agent to use. At present, there are 17 kinds of User-Agents integrated in the program, and we will consider opening this part into the configuration and customize the configuration (is it meaningful?). .

Exception handling problems in the program

At present, I am not very good at the exception processing, and I have tried my best to control the exception in a controllable range. There are many custom exceptions defined in the program. I will not go into details because I have no right to say here.

Recently took a break to add annotation support, so what does a crawler look like with annotations?

@EnableAutoConfiguration @AppName("hello spider") @Store(PrintStore.class) @AutoClose(true) @ThreadConfig(num = 1) @CookieConfig("asdfasdfasdfasdfasfasdfa") @HttpHeaderConfig({ "key1=value1", "Key2 =value2"}) @proxyConfig ("1.1.1.1,2.2.2.2") public class CockroachApplicationTest {public static void main(String[] args) throws Exception { TaskQueue queue = TaskQueue.of(); queue.push(new Task("http://blog.zhangyingwei.com")); CockroachApplication.run(CockroachApplicationTest.class,queue); }}Copy the code

Above is the demo of all the annotations, so forget the demo part, if it is really just a demo, how to write?

@EnableAutoConfiguration
public class CockroachApplicationTest {
    public static void main(String[] args) throws Exception {
        TaskQueue queue = TaskQueue.of();
        queue.push(new Task("http://blog.zhangyingwei.com"));
        CockroachApplication.run(CockroachApplicationTest.class,queue);
    }
}Copy the code

Yes, it’s that simple. The contents of this crawler is to crawl the page http://blog.zhangyingwei.com and print the results. In the process of crawler results, the program uses the PringStore class by default to print all results.

@CookieConfig(cookieGenerator = CookieGeneratorTest.class)Copy the code

/** * Created by zhangyw on 2017/12/19. */ public class CookieGeneratorTest implements StringGenerator { @Override public String get(Task task) { String cookie = "v="+ UUID.randomUUID().toString(); System.out.println(cookie); return cookie; }}Copy the code

Before each HTTP request occurs, the program calls the Generator’s GET method. Gets the cookie value of this time and appends it to the HTTP request header.

Now the web crawler has to do a little bit of distribution, and it bothers me.

In fact, I have seen several so-called distributed crawler source code, their so-called distributed, even pseudo distributed are not!! Using redis as message middleware is distributed? This is called distributed? This is not distributed at all, I was going to use Redis as message middleware to install a distributed B, but halfway through writing it, I suddenly felt a little sick, so I deleted the code, and the program was quiet, but also my own peace of mind.

Distributed this pit must be dug!!

So, my distribution will include:

• Distributed messaging middleware (perhaps using Redis or implementing one yourself; For the sake of quietness, you will most likely implement one yourself.
• Distributed task scheduling
• Distributed fault tolerance mechanism
• Distributed transaction
• Condition monitoring

So, is the hole getting bigger? Damn, a little scared!! As for the pit when to fill, can fill, depends on the mood…

In fact, I’m not in the mood to fill this distributed pit yet…