Sentry Enterprise level data security solution - Relay PII and data cleansing

This document describes a configuration format that we want to eventually hide from the user. The only reason this page still exists is because Relay currently accepts this format in place of regular data cleansing Settings.

The following documents explore the syntax and semantics of the advanced data cleansing configuration that Relay uses and performs. This is sometimes called PII cleanup.

Github.com/getsentry/r…
Docs. Sentry. IO/product/dat…
Scrub personally identifiable information (PII)
- Docs. Sentry. IO/product/rel…

A basic example

Suppose you have an exception message that unfortunately contains an IP address that should not exist. Would you write:

{
  "applications": {
    "$string": ["@ip:replace"]}}Copy the code

It reads as “replace all IP addresses in all strings”, or “apply @ip: REPLACE to all $string fields”.

@ip:replace is called a rule, and $string is called a selector.

Develop. Sentry. Dev/pii/selecto…

The built-in rules

The following rules exist by default:

@ip:replace 和 @ip:hashUsed to replaceIPAddress.
@imei:replace 和 @imei:hashUsed to replaceIMEI.
@mac:replace,@mac:mask 和 @mac:hashUsed to matchMACAddress.
@email:mask,@email:replace 和 @email:hashUsed to matchemailAddress.
@creditcard:mask,@creditcard:replace 和 @creditcard:hashUsed to match credit card numbers.
@userpath:replace 和 @userpath:hashUsed to match local paths (for exampleC:/Users/foo/).
@password:removeUsed to delete a password. In this case, we are on the fieldkeyforpatternMatches, whether it contains or notpassword,credentialsOr a similar string.
@anything:remove,@anything:replace 和 @anything:hashUsed to delete, replace, orhashAny value. It’s essentially equivalent to a wildcard regular expression, but it also matches a lot better than strings.

Write your own rules

Rules generally consist of two parts:

Rule typeDescribes what to match. For an exhaustive list, seePII rule type.
- develop.sentry.dev/pii/types/
Rule editing methodDescribes how to handle matches. For a list, seePII editing method.
- Develop. Sentry. Dev/pii/the methods…

Each page comes with examples. Try these examples by pasting them into Piinguin’s PII Configuration column and clicking on the field to get suggestions.

getsentry.github.io/piinguin/

Interactive editing

The easiest way to solve this problem is if you already have the raw JSON payload from an SDK. Go to our PII configuration editor Piinguin and then:

Paste into the original event
Click on the data you want to erase
Paste the other payloads and see if they are healthy, if necessary, go to Step 2.

After iterating through the configuration, paste it back into the project configuration located at.relay/projects/ .json

Such as:

{
  "publicKeys": [{"publicKey": "___PUBLIC_KEY___"."isEnabled": true}]."config": {
    "allowedDomains": ["*"]."piiConfig": {
      "rules": {
        "device_id": {
          "type": "pattern"."pattern": "d/[a-f0-9]{12}"."redaction": {
            "method": "hash"}}},"applications": {
        "freeform": ["device_id"]}}}}Copy the code

PII rule type

pattern

: Custom Perl-style regular expressions (PCRE).

{
  "rules": {
    "hash_device_id": {
      "type": "pattern"."pattern": "d/[a-f0-9]{12}"."redaction": {
        "method": "hash"}}},"applications": {
    "$string": ["hash_device_id"]}}Copy the code

imei

: Matches the IMEI or IMEISV.

{
  "rules": {
    "hash_imei": {
      "type": "imei"."redaction": {
        "method": "hash"}}},"applications": {
    "$string": ["hash_imei"]}}Copy the code

mac

: Matches a MAC address.

{
  "rules": {
    "hash_mac": {
      "type": "mac"."redaction": {
        "method": "hash"}}},"applications": {
    "$string": ["hash_mac"]}}Copy the code

ip

: Matches any IP address.

{
  "rules": {
    "hash_ip": {
      "type": "ip"."redaction": {
        "method": "hash"}}},"applications": {
    "$string": ["hash_ip"]}}Copy the code

creditcard

: Matches the credit card number.

{
  "rules": {
    "hash_cc": {
      "type": "creditcard"."redaction": {
        "method": "hash"}}},"applications": {
    "$string": ["hash_cc"]}}Copy the code

userpath

: Matches the local path (for example, C:/Users/foo/).

{
  "rules": {
    "hash_userpath": {
      "type": "userpath"."redaction": {
        "method": "hash"}}},"applications": {
    "$string": ["hash_userpath"]}}Copy the code

anything

: Matches any value. This is basically equivalent to a wildcard regular expression.

For example, to delete all strings:

{
  "rules": {
    "remove_everything": {
      "type": "anything"."redaction": {
        "method": "remove"}}},"applications": {
    "$string": ["remove_everything"]}}Copy the code

multiple

: Combines multiple rules into one. This is a disjunction (OR) : The field in question must match only one rule to match the combination rule, not all of them.

{
  "rules": {
    "remove_ips_and_macs": {
      "type": "multiple"."rules": [
        "@ip"."@mac"]."hide_rule": false.// Hide the inner rules when showing which rules have been applied. Defaults to false.
      "redaction": {
        "method": "remove"}}},"applications": {
    "$string": ["remove_ips_and_macs"]}}Copy the code

alias

: aliases one rule to another. This is the same as multiple, except that you can wrap only one rule.

{
  "rules": {
    "remove_ips": {
      "type": "multiple"."rule": "@ip"."hide_rule": false.// Hide the inner rule when showing which rules have been applied. Defaults to false.
      "redaction": {
        "method": "remove"}}},"applications": {
    "$string": ["remove_ips"]}}Copy the code

PII editing method

remove

: Deletes the entire field. Relay can choose to set it to NULL or remove it completely.

{
  "rules": {
    "remove_ip": {
      "type": "ip"."redaction": {
        "method": "remove"}}},"applications": {
    "$string": ["remove_ip"]}}Copy the code

replace

: Replace key with static String.

{
  "rules": {
    "replace_ip": {
      "type": "ip"."redaction": {
        "method": "replace"."text": [censored]"}}},"applications": {"$string": ["replace_ip"]}}Copy the code

`mask`

: Replaces each character of the matching string with the masking character *. In contrast to replace, it preserves the length of the original string.

{
  "rules": {
    "mask_ip": {
      "type": "ip"."redaction": {
        "method": "mask"}}},"applications": {
    "$string": ["mask_ip"]}}Copy the code

`hash`

: replaces the string with its own hash version. Equal strings will produce the same hash value, so, for example, if you decide to hash the user ID rather than replace or delete it, you’ll still get an exact count of the affected users.

{
  "rules": {
    "hash_ip": {
      "type": "ip"."redaction": {
        "method": "hash"}}}"applications": {
    "$string": ["mask_ip"]}}Copy the code

PII selector

Selectors allow you to restrict rules to certain parts of the event. This is useful for unconditionally removing some data from events by variable/field name, but can also be used for conservative testing rules for real data.

Data cleansing always applies to the raw event payload. Keep in mind that some fields in the UI may be called differently in the JSON Schema. When viewing events, there should always be a link called “JSON” that lets you see what the data sweeper sees.

For example, what is called “Additional Data” in the UI is called extra in the event payload. To remove a specific key named foo, you could write:

[Remove] [Anything] from [extra.foo]
Copy the code

Another example. Sentry knows of two types of error messages: exception messages and top-level log messages. Here is an example of such an event payload sent by the SDK (which can be downloaded from the UI) :

{
  "logentry": {
    "formatted": "Failed to roll out the dinglebop"
  },
  "exceptions": {
    "values": [{"type": "ZeroDivisionError"."value": "integer division or modulo by zero"}}}]Copy the code

Since “error message” comes from the value of exception and “message” comes from logentry, we must write the following to remove both from the event:

[Remove] [Anything] from [exception.value]
[Remove] [Anything] from [logentry.formatted]
Copy the code

Boolean logic

You can use Boolean logic to combine selectors.

In order to!Reverses the selector for the prefix.fooMatch the JSON keyfooAnd the! foomatchesfooEverything else.
use&&Construct conjunctions (AND), for example:foo && ! extra.fooTo match the keyfooUnless inextraInternal.
use||Build disjunction (OR), for example:foo || barmatchingfoo 或 bar.

The wildcard

**Matches all subpaths, sofoo.**matchingfooAll of theJSONThe key.
*Matches a single path item, sofoo.*Match thanfooAll of the lower levelsJSONThe key.

Value types

Select the subsection according to json-type using the following:

$stringMatches any string value
$numberMatches any integer or floating point value
$datetimeMatches any field in the event that represents a timestamp
$arrayMatches any JSON array value
$objectMatches any JSON object

Select the known parts of the schema using the following methods:

$exceptionmatching{"exception": {"values": [...] }}A single exception instance in
$stacktraceMatches a stack trace instance
$frameMatch a frame
$requestEvent-matchedHTTPRequest context
$userMatches the user context of the event
$logentry(Also appliesmessageAttributes)
$threadmatching{"threads": {"values": [...] }}A single thread instance in
$breadcrumbmatching{"breadcrumbs": [...] }A single breadcrumb in
$spanMatch atrace span
- Docs. Sentry. IO/product/sen…
$sdkmatching{"sdk": ... }SDK context in

The sample

Remove the event. The user:

[Remove] [Anything] from [$user]
Copy the code

Delete all frame local variables:

[Remove] [Anything] from [$frame.vars]
Copy the code

Escape special characters

If the object key to match contains Spaces or special characters, it can be escaped using quotes:

[Remove] [Anything] from [extra.'my special value']
Copy the code

This matches the key my Special value in the attached data.

To escape ‘(single quote) inside quotes, replace it with’ (two quotes) :

[Remove] [Anything] from [extra.'my special '' value']
Copy the code

This matches the key my Special ‘value value in the attached data.

More and more

Sentry Enterprise level Data security solution – Relay getting started
Sentry Enterprise level data security solution – Relay operation mode
Sentry Enterprise level data security solution – Relay configuration option
Sentry Enterprise level data security solution – Relay monitoring & Metrics Collection
Sentry Enterprise level data security solution – Relay project configuration
Sentry Developer Contribution Guide – SDK Development (Performance Monitoring: Sentry SDK API Evolution)

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Sentry Enterprise level data security solution – Relay PII and data cleansing

A basic example

The built-in rules

Write your own rules

Interactive editing

PII rule type

PII editing method

`mask`

`hash`

PII selector

Boolean logic

The wildcard

Value types

The sample

Escape special characters

More and more

Sentry Enterprise level data security solution – Relay PII and data cleansing

A basic example

The built-in rules

Write your own rules

Interactive editing

PII rule type

PII editing method

mask

hash

PII selector

Boolean logic

The wildcard

Value types

The sample

Escape special characters

More and more

Related Posts

Elasticsearch (

MySQL >InnoDB MySQL >InnoDB

Best Node.js framework to use in 2021

`mask`

`hash`