One, foreword

The project of the company has made a cluster to realize the request diversion. Since there are more or less online request failures or system anomalies, in order to check the log information of failed requests, we have to open the log files of all services to locate and analyze the problems, which is very troublesome to operate. Therefore, our development team decided to design a log viewing system to solve the above problems.

Second, implementation ideas

By default, application service logs are stored in a directory on the local server. To view the logs, you should output the logs of multiple servers to one log file.

Due to the Logback logging framework and RabbitMQ message queues used by the project, the two can be integrated.

So we can output the logs from the project code to the RabbitMQ queue, read the queue data through Logstash, and finally to a log file.

Three, prepare the environment

Test environment: CentOS 7 whose IP address is 192.168.2.13

# 3.1 RabbitMQ configuration

To set up the RabbitMQ environment, see CentOS 7.2 Installing RabbitMQ in this website.

After the RabbitMQ configuration is complete, log in to the RabbitMQ management page and perform the following steps:

  • Create a queue named log_queue
  • Create a switch named Rabbit.log (direct type)
  • Bind the log_queue queue to the Rabbit.log switch

Operation diagram:

# 3.2 Logstash configuration file

Modify the Logstash configuration file. There are also blog posts about Logstash on this site. You can check them out in The Basics of Logstash.

input { rabbitmq { type =>"all" durable => true exchange => "rabbit.log" exchange_type => "direct" key => "info" host => "192.168.2.13" port => 5672 user => "light" password => "light" queue => log_queue" auto_delete => false}} output { file { path => "/usr/test-log/test-%{+YYYY-MM-dd}.log" codec => multiline { pattern => "^\d" negate => true what => "previous" } } }Copy the code

Note: Multiline is a Logstash plug-in that needs to be installed manually.

The configuration indicates that the Logstash service reads logs from RabbitMQ and outputs them to a specified directory file.

Four, coding

# 4.1 rely on

List the main dependencies:

<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-amqp</artifactId> </dependency>  <dependency> <groupId>org.springframework.amqp</groupId> <artifactId>spring-rabbit</artifactId> </dependency>Copy the code

Log files

Called logback – spring. XML

<? The XML version = "1.0" encoding = "utf-8"? > <configuration debug="false"> <! <property name="LOG_HOME" value="d:/" /> <! - the console output - > < appender name = "STDOUT" class = "ch. Qos. Logback. Core. ConsoleAppender" > < encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <! -- Format output,%d: date; %thread: specifies the thread name. %-5level: level, 5 characters wide from the left; % MSG: log message; %n: newline -> <pattern>%d{YYYY-MM-DD HH: MM :ss.SSS} [%thread] %-5level %logger{50} - % MSG %n</pattern> </appender> <appender name="RABBITMQ" class="org.springframework.amqp.rabbit.logback.AmqpAppender"> <layout> <pattern><! [CDATA[ %d %p %t [%c] - <%m>%n ]]></pattern> </layout> <! -- rabbitMQ address --> <addresses>192.168.2.13:5672</addresses> <username>light</username> <password>light</password> <declareExchange>true</declareExchange> <exchangeType>direct</exchangeType> <exchangeName>rabbit.log</exchangeName> <routingKeyPattern>info</routingKeyPattern> <generateId>true</generateId> <charset>UTF-8</charset> <durable>true</durable> <deliveryMode>NON_PERSISTENT</deliveryMode> <autoDelete>false</autoDelete> </appender> <logger name="com.light.rabbitmq" level="info" additivity="false"> <appender-ref ref="STDOUT"/> <appender-ref ref="RABBITMQ"/> </logger> <! -- Log output level, level default DEBUG, root is logger, It is logger root --> <root level="INFO"> <appender-ref ref="STDOUT" /> <appender-ref ref="RABBITMQ" /> </root> </configuration>Copy the code

ExchangeType and exchangeName in the configuration are the type and name of the switch we created above.

# 4.3 Test classes

Custom exceptions:

public class CustomException extends RuntimeException{ private static final long serialVersionUID = 1L; private int code; private String msg; public CustomException(int code, String msg) { super(msg); this.code = code; this.msg = msg; } public int getCode() { return code; } public void setCode(int code) { this.code = code; } public String getMsg() { return msg; } public void setMsg(String msg) { this.msg = msg; }}Copy the code

Simulated log printing:

@Component public class DemoTask { private static Logger logger = LoggerFactory.getLogger(DemoTask.class); private int num = 1; @Scheduled(fixedRate = 3000) public void writeLog() { try { if (num % 5 == 0) { throw new CustomException(500, "Custom exception error "); } logger.info("==={}===={}","hello rabbitmq", System.currentTimeMillis()); num++; } catch (CustomException e) { e.printStackTrace(); logger.error("=={}==", e); }}}Copy the code

5. Code testing

Execution startup class:

@EnableScheduling

@SpringBootApplication

public class RabbitmqTestApplication {



	public static void main(String[] args) {

		SpringApplication.run(RabbitmqTestApplication.class, args);

	}



}
Copy the code

The execution result is shown as follows:

Log information about code execution has been printed to the specified log file.

supplement

Since the logs of multiple servers are printed to the same file, you need to print the HOST IP address corresponding to the log information to distinguish the log source. The specific implementation steps are as follows:

  • Custom log converter

You need to inherit from the ClassicConverter class

public class CustomLogConverter extends ClassicConverter { public String convert(ILoggingEvent event) { try { return InetAddress.getLocalHost().getHostAddress(); } catch (UnknownHostException e) { e.printStackTrace(); } return null; }}Copy the code
  • Modify the logback-spring. XML file

Only key configuration information is displayed

<conversionRule conversionWord="ip" converterClass="com.light.rabbitmq.log.CustomLogConverter" /> <appender name="RABBITMQ" class="org.springframework.amqp.rabbit.logback.AmqpAppender"> <layout> <pattern><! [CDATA[%ip %date{yyyy-MM-dd HH:mm:ss} | %highlight(%-5level) | %yellow(%thread) | %green(%logger) | %msg%n ]]></pattern>  </layout> </appender>Copy the code