Recently I was asked to compare the major open source performance testing frameworks. I searched a few and listed JMeter, K6, Gatling, Siege, NGrinder, Locust, and FunTester.

Here are some qualitative indicators collected:

tool language use In the form of a case distributed Ease of use expanding Traffic arrangements link community readability
JMeter Java The Client/command line JMX file In the optimal low poor poor 11800000 poor
k6 JavaScript The command line JS script no In the optimal In the optimal 1840000 optimal
Gatling Scala The command line Scala scripts no poor optimal poor In the 333000 optimal
siege C The command line The command line no optimal poor no no 882000 In the
ngrinder Groovy The Web page Groovy scripts optimal optimal optimal poor poor 219000 optimal
locust Python Command line/web Python scripts In the In the optimal poor optimal 930000 optimal
FunTester Java&Groovy Command line/service interface Parameters/Scripts is In the optimal optimal optimal 342000 optimal

Due to some performance test comparison, some of the performance test frameworks are not suitable for my current needs, so I give up a few. Here’s the framework for quitting and why.

Gatling

Introduction to the

Gatling is an open source performance testing tool. The tool allows developers to build and execute tests and easily manage their tests locally or in the cloud. To write tests using Gatling, we need to use Scala, and Gatling allows you to define Scala classes that provide similar functionality, but are much more readable.

Give up the reason

Gatling execution steps are as follows:

  • Write or record a script (ScalaLanguage script)
  • Compile the script (runshCommand)
  • Select the script in interactive mode
  • Waiting for run results
  1. First of all, this process is very difficult to automate, especially manuallyshellCommand, and then visually select the script to execute on the interfaceID.
  2. languageScalaNon – mainstream nature, the use of the way is not quite in line with the present habit
  3. It is difficult to customize test cases, including result validation and tandem testing

Boast two sentences

Its excellent recording function, can quickly generate test scripts, through simple configuration (modify script call API) to complete the writing of use cases.

siege

Introduction to the

Siege is a stress test and measurement tool designed for WEB development to assess the application’s ability to withstand pressure: it can configure multiple concurrent access to a WEB site, log the time of all requests for each user, and repeat the process for a certain number of concurrent access.

This is written in C language, which is similar to curl and AB testing framework.

Give up the reason

  1. Pure command execution is hard to like
  2. The test report is also command line output and lacks recording and summary capabilities
  3. I don’t think it’s updated

Boast two sentences

Simple to use and good for improvising an interface performance test.

locust

Introduction to the

Locust is an easy-to-use distributed user load testing tool. It is used to load test web sites (or other systems) and calculate how many concurrent users a system can handle.

Give up the reason

  1. The technology stack is Java, and Python is relatively new
  2. Every time you needshellUse cases cannot be switched arbitrarily after command startup
  3. The distributed scheme is ineffective and lacks synchronization and coordination functions

Boast two sentences

  1. Use cases are simple and readable
  2. Script form use cases, extensible
  3. Powerful and significantly better than JMeter

Of course, due to the general understanding of LOCUST, I am not familiar with many aspects, especially the quantification of performance indicators. I will also test the performance of LocUST in the next performance test framework.

nGrinder

Introduction to the

The nGrinder is an application that executes Groovy or Jython test scripts on a range of machines, and the internal engine is based on the Grinder. The nGrinder wraps the Console and Agent of the Grinder in controller and Agent, respectively, and extends a number of features to support concurrent testing.

Give up the reason

I have to say I really liked this framework at first, nothing else, just simplicity. Deploying and building from the start, and writing the first script, is simple. But:

  1. Pure Web operation interface
  2. Execution and results are difficult to expand

I gave up. Of course, you can choose to rewrite this part of the project to address these shortcomings, which I did.

Boast two sentences

If you are a test engineer for the Java technology stack, nGrinder is a very good Web performance testing framework, in addition to the JMeter client-side testing framework. If you want to script, monitor, and execute in one step, the nGrinder is the way to go.

FunTester.Tencent Cloud Author of the Year,Boss direct hire contract author.Official GDevOps media partner, non-famous test development, welcome to follow.
  • FunTester test framework architecture diagram
  • Soft start of performance test
  • How to become a Full stack automation engineer
  • FunTester Moco Server framework architecture diagram
  • Use Case Scheme of Distributed Performance Testing Framework (PART 1)
  • Use Case Scheme of Distributed Performance Testing Framework (PART II)
  • Appium 2.0 quick reference
  • Probe into branch problems in link pressure measurement
  • Agile testing is two or three things
  • A complete guide to automated Testing Frameworks
  • Moco framework interface hit ratio statistics practice
  • Distributed performance testing framework single-node internal test