What problem is chaos engineering used to solve?

• Simulate a data center failure
• Forced system clock synchronization
• Simulate I/O exceptions in driver code
• Simulate delays between services
• Random throw function throws an exception

• Do chaos experiments on what?
• What is the scope of chaos experiment?
• What experiments are carried out?
• What are the valid matching conditions for the experiment?

• Target: Refers to the component where the experiment takes place, such as the container, application framework (Dubbo, Redis, Zookeeper), etc.
• Scope: Scope of experiment implementation, which refers to the machine or cluster that triggers the experiment.
• Matcher: experimental rule Matcher. It defines the relevant experimental matching rules according to the configured Target. Because each Target may have its own special matching conditions, such as HSF and Dubbo in RPC domain, it can match services provided by service providers and services invoked by service consumers. Redis in the cache domain can be matched according to set and GET operations.
• Action: indicates the simulated scenario. For example, you can test scenarios such as full disk, high DISK I/O reading and writing, and disk hardware failure. In the case of applications, you can abstract out experimental scenarios such as delays, exceptions, returning specified values (error codes, large objects, etc.), parameter tampering, and repeated calls.

Toolkit. // Experimental target dubbo. // scope, here is the host("1.0.0.1"). // Consumer (). // Consumer ()."com.example.HelloService"). // Component matcher, 1.0.0 interface version version"1.0.0"). // Experiment scenario, 3s delay(3000);Copy the code

Blade create dubbo delay --time 3000 --consumer --service com.example.HelloService --version 1.0.0Copy the code

• Delay: Action in the model, executing the delay drill scenario.
• –time: the action parameter in the model, indicating the delay time.
• — Consumer, –service, –version: Matchers in the model, experimental rule matchers.

• Definition of a component chaos experimental model, containing the component name and a list of supported experimental scenarios.

typeExpModelCommandSpec Interface {// Component Name Name() String // List of supported scenarios Actions() []ExpActionCommandSpec //... Slightly}Copy the code

• A definition of an experimental scenario action, containing the scenario name, parameters required for the scenario, and some experimental rule matchers.

typeExpActionCommandSpec Interface {Name() string Matchers() []ExpFlagSpec // Action parameter Flags() []ExpFlagSpec // Action Executor(channel channel) Executor //... Slightly}Copy the code

• The definition of an experimental matcher, including parameter names, parameter descriptions, and so on.

typeExpFlagSpec Interface {// Parameter name FlagName() String // Parameter Description FlagDesc() string // Whether parameter value FlagNoArgs() bool // Whether parameter is required FlagRequired() bool }Copy the code

type NetworkCommandSpec struct {
}

func (*NetworkCommandSpec) Name() string {
    return "network"
}

func (*NetworkCommandSpec) Actions() []exec.ExpActionCommandSpec {
    return []exec.ExpActionCommandSpec{
        &DelayActionSpec{},
        &DropActionSpec{},
        &DnsActionSpec{},
        &LossActionSpec{},
    }
}Copy the code

type DelayActionSpec struct {
}

func (*DelayActionSpec) Name() string {
    return "delay"
}

func (*DelayActionSpec) Matchers() []exec.ExpFlagSpec {
    return []exec.ExpFlagSpec{
        &exec.ExpFlag{
            Name: "local-port",
            Desc: "Port for external service",
        },
        &exec.ExpFlag{
            Name: "remote-port",
            Desc: "Port for invoking",
        },
        &exec.ExpFlag{
            Name: "exclude-port",
            Desc: "Exclude one local port, for example 22 port. This flag is invalid when --local-port or remote-port is specified",
        },
        &exec.ExpFlag{
            Name:     "device",
            Desc:     "Network device",
            Required: true,
        },
    }
}

func (*DelayActionSpec) Flags() []exec.ExpFlagSpec {
    return []exec.ExpFlagSpec{
        &exec.ExpFlag{
            Name:     "time",
            Desc:     "Delay time, ms",
            Required: true,
        },
        &exec.ExpFlag{
            Name: "offset",
            Desc: "Delay offset time, ms",
        },
    }
}

func (*DelayActionSpec) Executor(channel exec.Channel) exec.Executor {
    return &NetworkDelayExecutor{channel}
}Copy the code