sequence

This article focuses on the Cortex Backoff

Backoff

github.com/cortexproject/cortex/pkg/util/backoff.go

// Backoff implements exponential backoff with randomized wait times
type Backoff struct {
	cfg          BackoffConfig
	ctx          context.Context
	numRetries   int
	nextDelayMin time.Duration
	nextDelayMax time.Duration
}

// NewBackoff creates a Backoff object. Pass a Context that can also terminate the operation.
func NewBackoff(ctx context.Context, cfg BackoffConfig) *Backoff {
	return &Backoff{
		cfg:          cfg,
		ctx:          ctx,
		nextDelayMin: cfg.MinBackoff,
		nextDelayMax: doubleDuration(cfg.MinBackoff, cfg.MaxBackoff),
	}
}
Copy the code

Backoff defines the CFG, CTX, numRetries, nextDelayMin, and nextDelayMax attributes. NewBackoff provides a factory method based on BackoffConfig, with the default nextDelayMin being cfg.minbackoff

BackoffConfig

github.com/cortexproject/cortex/pkg/util/backoff.go

// BackoffConfig configures a Backoff
type BackoffConfig struct {
	MinBackoff time.Duration `yaml:"min_period"`  // start backoff at this level
	MaxBackoff time.Duration `yaml:"max_period"`  // increase exponentially to this level
	MaxRetries int           `yaml:"max_retries"` // give up after this many; zero means infinite retries
}
Copy the code

BackoffConfig defines the MinBackoff, MaxBackoff, and MaxRetries attributes

Ongoing

github.com/cortexproject/cortex/pkg/util/backoff.go

// Reset the Backoff back to its initial condition func (b *Backoff) Reset() { b.numRetries = 0 b.nextDelayMin = b.cfg.MinBackoff b.nextDelayMax = doubleDuration(b.cfg.MinBackoff, b.cfg.MaxBackoff) } // Ongoing returns true if caller should keep going func (b *Backoff) Ongoing() bool { // Stop if Context has errored or max retry count is exceeded return b.ctx.Err() == nil && (b.cfg.MaxRetries == 0 || b.numRetries <  b.cfg.MaxRetries) } // Err returns the reason for terminating the backoff, or nil if it didn't terminate func (b *Backoff) Err() error { if b.ctx.Err() ! = nil { return b.ctx.Err() } if b.cfg.MaxRetries ! = 0 && b.numRetries >= b.cfg.MaxRetries { return fmt.Errorf("terminated after %d retries", b.numRetries) } return nil } // NumRetries returns the number of retries so far func (b *Backoff) NumRetries() int { return b.numRetries } // Wait sleeps for the backoff time then increases the retry count and backoff time // Returns immediately if Context is terminated func (b *Backoff) Wait() { // Increase the number of retries and get the next delay  sleepTime := b.NextDelay() if b.Ongoing() { select { case <-b.ctx.Done(): case <-time.After(sleepTime): } } } func (b *Backoff) NextDelay() time.Duration { b.numRetries++ // Handle the edge case the min and max have the same  value // (or due to some misconfig max is < min) if b.nextDelayMin >= b.nextDelayMax { return b.nextDelayMin } // Add a  jitter within the next exponential backoff range sleepTime := b.nextDelayMin + time.Duration(rand.Int63n(int64(b.nextDelayMax-b.nextDelayMin))) // Apply the exponential backoff to calculate the next jitter // range, unless we've already reached the max if b.nextDelayMax < b.cfg.MaxBackoff { b.nextDelayMin = doubleDuration(b.nextDelayMin, b.cfg.MaxBackoff) b.nextDelayMax = doubleDuration(b.nextDelayMax, b.cfg.MaxBackoff) } return sleepTime } func doubleDuration(value time.Duration, max time.Duration) time.Duration { value = value * 2 if value <= max { return value } return max }Copy the code

Backoff mainly provides Ongoing and Wait methods; An Ongoing Return bool indicates whether an operation can continue. If Err is nil and B.fg. MaxRetries or B.Num retries < B.fg. MaxRetries returns true. The Wait method waits for execution to complete or for b.extdelay (). The NextDelay method increments numRetries and calculates sleepTime; The Err method returns CTX Err or an error indicating that the number of retries exceeds the threshold

The instance

// NewBackoffRetry gRPC middleware. func NewBackoffRetry(cfg util.BackoffConfig) grpc.UnaryClientInterceptor { return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ... grpc.CallOption) error { backoff := util.NewBackoff(ctx, cfg) for backoff.Ongoing() { err := invoker(ctx, method, req, reply, cc, opts...) if err == nil { return nil } if status.Code(err) ! = codes.ResourceExhausted { return err } backoff.Wait() } return backoff.Err() } }Copy the code

NewBackoffRetry shows how to use backoff, through a for loop, with backoff.retry () Ongoing operations, backoff.wait () at the end, and backoff.err () if not returned early.

summary

Cortex provides Backoff, which can be retried based on MinBackoff, MaxBackoff, and MaxRetries.

doc

  • cortex