preface

A long time ago, I received an email from a white guest, saying that the domain we send email to lacks security check, so we’d better make up the corresponding security check:

I'm an independent cyber security researcher i have found multiple issues in your website. Vulnerability : Missing SPF I am just looking at your SPF records then found following. SPF Records missing safe check which can allow me to send mail and phish easily any victim. PoC: <? php $to = "[email protected]"; $subject = "Password Change"; $txt = "Change your password by visiting here - [VIRUS LINK HERE]l"; $headers = "From: https://karve.io"; mail($to,$subject,$txt,$headers); ? > SPF record lookup and validation for: xxx.comCopy the code

After checking, I found that it was true. Our email was not accompanied by the corresponding security check, so it was easy for others to send it, which may cause serious email fraud to our users.

The dangers of sending fraudulent e-mails

For example, a hacker could impersonate our official support email address to send phishing emails to users via the site emkei.cz. If the user believes it is true, then it will cause loss to the user’s interests, which is E-mail fraud

At this time our users will receive a fake email, if the user believes, it is easy to be used by the fraud.

So the dangers of email fraud are still high. Security problems in this regard must be solved as soon as possible.

Flaws in email

While solving this problem, we need to ask why email has this problem.

The E-mail system is based on the Simple Mail Transfer Protocol (SMTP), an Internet standard for sending and relaying E-mail. However, as originally envisioned in 1981, SMTP does not support message encryption, integrity verification, or authentication of senders.

Due to these defects, the sender’s email information may be intercepted by the listener in network transmission to read the message content, resulting in privacy leakage, or may be subject to man-in-the-middle attack (MitM), resulting in email message tampering, resulting in phishing attacks. In order to solve these security problems and cope with the increasingly complex network environment, the mail community has developed many E-mail extension protocols, such as STARTTLS, S/MIME, SPF, DKIM and DMARC. Most of the current mail service manufacturers also use one or several combinations of the above extended protocols, supplemented by the application of firewall, Bayesian spam filter and other technologies to make up for the security defects of E-mail.

SMTP has some problems with its own security and confidentiality, so there are several plugins to help it:

  1. Improve transmission confidentiality, including end-to-end encryption –>STARTTLS.S/MIME
  2. Authentication of messages –>SPF.DKIM.DMARC

The essence of E-mail fraud is that the recipient of an E-mail cannot determine whether the source of the E-mail is legitimate. So this is where the authentication of the message comes in. Although STARTTLS and SMTP-STS ensure the encryption of emails during transmission to prevent eavesdropping, they still cannot solve the problems of sender identity forgery and message tampering. So the authentication part of the message is particularly important.

Therefore, this section mainly focuses on the setting of SPF, DKIM and DMARC protocols and practices them in combination with our site.

SPF

SPF concept

Sender Policy Framework (SPF) is a kind of email fraud detection technology that authenticates the identity of email Sender by IP address. It is a very efficient solution to spam. The SPF allows an organization to authorize a list of hosts to send emails for its domain, while the SPF record stored in the DNS is a TXT resource record that identifies which mail servers are allowed to send emails on behalf of the domain. SPF prevents spammers from sending emails with fake sender addresses. Recipients check the SPF record of domain names to determine whether messages purportedly from this domain are from an authorized mail server. If it is, it will be considered a normal email, otherwise it will be returned as a forged email. SPF also allows an organization to delegate some or all of its SPF policies to another organization, typically by delegating SPF Settings to a cloud provider.

In simple terms, this protocol is to verify the source IP address of the mailbox and match the SPF information recorded in its DNS. If found, then it is valid. If it doesn’t, there’s a problem with where the email came from. At this time, the general mailbox receiver will have three processing methods:

  1. Reject or delete
  2. Think of it as trash
  3. Normal inbox, but mail will have? Warning that the recipient cannot verify the true identity of the message

As for the policy, different mailbox recipients have different policies. The picture below (the Internet picture) makes it very clear:

SPF set

The SPF configuration is very simple. You can directly configure the SPF on the public DNS. Because the domain name DNS of our site is set in AWS Router 53. And because we have two types of email addresses for sending mail:

  1. Send a reminder emailno-replyMailbox: This type of mail is automatically sent by the AWS SES service
  2. Support and sale mailboxes for sending marketing emails. This type of mailbox is gmail and is sent manually

Therefore, the two domain names ses and Google need to be configured, so we add this item to the original TXT record in the secondary domain name xxx.com on router 53:

"v=spf1 include:_spf.google.com include:amazonses.com -all"
Copy the code

In this case, the SPF is configured. Isn’t it very simple?

Test whether SPF takes effect

Emkei.cz: SPF: SPF: SPF: SPF: SPF: SPF: SPF: SPF: SPF: SPF: SPF: SPF: SPF: SPF

Email received, but it seems that there is no difference with no configuration?? It’s not. Let’s open up the original message and take a look

The IP address of the email sender (emkei.cz) is 93.99.104.21. Then the Gmail receiver uses SPF to look up the SPF configuration of our site domain name. This IP address does not exist in all IP addresses under _spf.google.com and Amazonses.com. Therefore, SPF verification fails. If SPF authentication fails on the Receiving end of Gmail, a question mark is displayed indicating that Gmail cannot verify the source of the email. Users should be cautious.

Understand the setting of “All” in SPF

There are different SPF Settings. 98% domain names use ~all (softfail), which means that if the SPF entry does not correspond to the source mail server, the mail is considered as a soft failure. The following table shows the differences of ALL in SPF:

parameter The results of meaning
+all pass Permitting all the email, like have nothing configured.
-all fail Will only mark the email like pass if the source Email Server fits exactly, IP, MX, Etc. With the SPF entry. (Only by matching the CORRECT IP, MX, etc.)
~all softfail Allows to send the email, and if something is wrong will mark it like softfail.
? all neutral Without policy

We set it to fail -all

Check whether SPF is correct

After configuration, we can also through some external tools site to test whether our SPF configuration: normal tools.wordtothewise.com/spf this site shows all can use the domain name to send an overview of the IP

That is, if the IP from which the sender is coming is not in the list of these IP addresses. Then the SPF test won’t pass. It becomes FAIL.

DKIM set

Although we set SPF, we also saw the result. Although Fail was marked in the details of the email, the user still received it, and it was in the normal inbox with a question mark beside it. (In fact, different email receivers have different processing methods for SPF verification Fail. Some are put in the trash), but this can still trick unwary users. So we need to strengthen the authentication part. So you have a DKIM setup.

DKIM concept

DKIM Domain Keys Identified Mail (DKIM) is an email fraud detection technology that examines the message headers from a Domain signature to determine whether the message is spoofing or tampering.

DKIM uses the principle of encrypted signature and verification. When the sender sends an email, the signature field encrypted by the private key related to the domain name is inserted into the message header. After receiving the email, the recipient retrieves the sender’s public key (DNS TXT record) through DNS to verify the signature and determine the authenticity of the sender’s address and message. It should be noted that DKIM can only verify the authenticity of the sender’s address source, but cannot identify the authenticity of the email content. At the same time, DKIM cannot specify what the recipient should do if a message is received that is invalid or lacks an encrypted signature. In addition, the private key signature is the universal signature of all outgoing mails in the local domain. It is the mail relay server that signs the mails instead of the real mail sender, which means that DKIM does not provide the real end-to-end electronic signature authentication.

Email messages sent using DKIM include a Dkim-signature header field, which contains the encrypted Signature representation of the message (with the private key). The provider receiving the message can decode the signature using the public key, which is found in the sender’s DNS record. The E-mail provider then uses this information to determine if the message is genuine.

For example, when normal Gmail or SES sends an email, it encrypts the email with the private key automatically generated by the domain association, generates the Dkim-signature Signature, inserts the Dkim-signature Signature and related information into the email header, and sends the dkim-signature Signature to the recipient. When the recipient receives the email, You can obtain the public key through DNS query to verify the validity of the DKIM signature. In this way, you can confirm the email sending process, prevent the email from being tampered, and ensure the integrity of the email content. So as long as we can find the public key, to decrypt the private key information in the header field, it shows that it is real, in fact, asymmetric end-to-end encryption, but we expose the public key in the DNS TXT record of the domain name, the provider of the receiving email can look up on the public network. The following diagram illustrates the process:

DKIM set

Set the public key in DNS TXT. As we use aws SES and Gmail, we set the public key in DNS TXT.

SES DKIM set

By using DKIM in Amazon SES for email authentication, we can use the simplest way, which is Easy DKIM provided by SES. Simply speaking, in the configuration side of the verified domain name, click to generate the relevant configuration of DKIM. This will generate three Cname records, and then put the three Cname records into Router 53

In this way, the DKIM of SES is configured.

Gmail DKIM configuration

Next we configure Gmail, Gmail will have default DKIM setting by default, but the official recommendation is to use your own domain name, Gmail is also verified by our own domain name, a direct line of TXT record:

The p field is followed by a large section of the public key. This big string is also automatically generated, right in the background of Gmail:

  1. Go to theadmin.google.comUse yourGoogle AppLog in to the background and selectGoogle App–Gmail–Authenticate email. Select your domain name to generate a public key. The following figure

  1. Based on the generated information, go to your DNS and add a TXT record.
  2. Go back toadmin.google.comClick “Start Authentication” on the previous page.

This generates the public key and takes effect.

Test whether DKIM takes effect

Similarly, send a normal email to my mailbox. If DKIM is set and successful, PASS will appear in the mailbox details

The above screenshot shows that we have set up DKIM, and the verification of DKIM sent by the correct source is normal.

If it’s a fake, it’s not a PASS, it’s a FAIL.

One detail to note here is that all DKIM keys are stored in a subdomain named “_domainkey”.

In the preceding screenshot, given the DKIM signature field d=xxx.com, s= Google, the DNS query is: Google.domainkey.xxx.com (or the domainkey corresponding to ses), the TXT record of google.domainkey.xxx.com will be parsed on route 53. Check the public key information through DNS. If it matches the private key that sends the email, the email may not be spam. If it does not, the email may be spam or its content may be tampered with.

DMARC set

In general, SPF and DKIM, through setting can effectively block fake mail, however, because of different mail manufacturers, processing spam policy is different, there is a certain risk, for instance, a fake E-mail, netease and tencent the means of processing, can be directly refused, and into the box, gmail will usually normal Only it will pass? Prompt because the source of the message could not be determined. In this way, the risk prompt overlooked, there will be a great hidden trouble, lead to some users being deceived, considering our use of gmail for most users, this problem is to solve thoroughly, through further understand the security setting of mail service, found DMARC which can solve our problems, we expect the final effect is, Forged mail for verification after entering the bin or directly rejected.

That is, even if we set SPF and DKIM before, the mail that does not pass may be left in the trash bin or in the inbox with a question mark because of the relationship between different mail manufacturers. There’s some risk involved. So we need a policy that tells the receiver, if they get a fake message, to follow the policy I’ve specified. Don’t make decisions based on your own preferences. This is the DMARC setting.

DMARC concept

The common problem in SPF and DKIM is the lack of effective policies and feedback mechanisms. These two protocols do not define how to handle unauthenticated email from a domain claimed to be a domain, or unauthenticated email from a domain claimed to be hosted by a third party. How to feedback and count emails claiming to be a domain authentication success or failure.

DMARC’s domain-based Message Authentication, Reporting, and Conformance standards address these problems in SPF and DKIM, The main purpose of DMARC is to set “policies” that include what to do when you receive unauthenticated messages from a domain and what to do when unauthenticated messages are sent by third-party providers authorized by that domain. DMARC will also have the ISP send reports on the success or failure of authentication for a domain, which will be sent to the addresses defined in “RUA” (summary report) and “RUF” (forensics report). Meanwhile, DMARC relies on SPF records and DKIM keys configured for outbound mail flows to ensure the integrity of mail source and signature. DMARC policies are triggered when emails fail to pass SPF or DKIM checks.

The following diagram illustrates the process:

DMARC Common parameters

Common DMARC parameters are as follows:

  • Adkim :(plain text; Optional; The default value is r.) indicates that the domain name owner requires a strict or lax DKIM authentication mode. Valid values are as follows:
r: relaxed mode
s: strict mode
Copy the code
  • Aspf :(plain text; Optional; The default value is R.) Indicates that the domain name owner requires a strict or loose SPF authentication mode. Valid values are as follows:
r: relaxed mode
s: strict mode
Copy the code
  • fo: Fault reporting option (plain text; Optional; Default is 0), a colon-separated list whose contents are ignored if “ruf” is not specified.
    • 0: If all authentication mechanisms fail to produce a “pass” result, a DMARC failure report is generated;
    • 1: If either authentication mechanism produces results other than “pass”, then a DMARC failure report is generated;
    • D: If the signature verification of the message fails, a DKIM fault report is generated;
    • S: If the SPF verification fails, an SPF fault report is generated.
  • p: Required mail recipient policy (plain text; Required) indicates the policy that the receiver has set based on the domain name owner’s requirements.
    • None: The domain owner requests that no specific action be taken
    • Quarantine: Domain owner wants email recipients to flag messages that fail DMARC authentication as suspicious.
    • Reject: The domain owner expects the recipient to reject a message that fails DMARC authentication.
  • PCT :(plain text integer 0-100; Optional, default 100) The percentage of messages in the domain owner’s mail flow that apply DMARC policies.
  • Rf: Format for message-specific fault reporting (colon-separated plain text list; Optional; Default is “AFRF”)
  • Ri: Required interval between summary reports (plain text 32-bit unsigned integer; Optional; Default: 86400). Indicates that the interval between asking the receiver to generate a summary report does not exceed the required number of seconds.
  • Rua: Email address to send comprehensive feedback (comma-separated plain text list of DMARC URIs; Optional)
  • Ruf: Email address for sending detailed fault information (comma-separated DMARC URI plain text list; Optional)
  • Sp: policy that requires mail recipients to use for all subdomains (plain text; Optional). By default, the policy specified by P is applied to the domain name and subdomain.
  • V: version (plain text; The required) value is “DMARC1” and must be the first label.

DMARC configuration

The specific configuration of our site is the same on Router 53 (Google and SES have the same configuration, so only one configuration is required). Ses provides specific configuration to use Amazon SES to comply with DMARC, such as this one:

"v=DMARC1; p=quarantine; pct=25; rua=mailto:[email protected]"Copy the code

In simple terms, this policy tells the email provider to do the following:

  • Find all E-mail messages whose “From” field is example.com and is not authenticated by SPF or DKIM.
  • Send 25% of unauthenticated E-mail messages to the spam folder for quarantine (you can also use p= None to take no action; Or p=reject).
  • Send a report in the digest on all emails that are not authenticated (that is, a report that summarizes the data over a specific time period, rather than a separate report for each event). Email providers typically send such summary reports once a day, but specific policies vary from provider to provider.

Therefore, our specific configuration is as follows:

_dmarc.xxx.com.
Copy the code
"v=DMARC1; p=reject; rua=mailto:[email protected]; adkim=s; aspf=s"Copy the code

The following operations are performed:

  • SPF and DKIM authentication are strictly required
  • If validation fails, reception will be rejected
  • When rejected, feedback is sent to the specified email address

To check if your SPF and DKIM comply with DMARC, you can use the following commands:

nslookup -type=TXT _dmarc.xxx.com
Copy the code

And you can see that’s fine. Of course, if you can’t configure DMARC, here is a configuration assistant to help you configure:

DMARC test effect

Put the bin

Now that this is configured, we’re ready to test, plus we’ve changed the policy to put in the bin (p=quarantine), where emails sent through Emke.cz are put in the bin when SPF doesn’t pass

Refused to accept

When a hacker sends a fake email to a user and it is rejected, his email address (support) will receive a reject report:

normal

If it’s normal, all three policies will PASS, because we have two sending sources, so we have to try both

  1. This is a series of no-reply emails from SES

  1. This is gmail’s Support series

You can see that the mail ID is different, one is SES, one is Gmail

DMARC Tests whether SPF and DKIM are normal

We know that SPF and DKIM must be well matched when configuring DMARC. Although you can do it through NSLookup. You can also check your DMARC configuration by typing your domain name in the DMARC Record Checker.

conclusion

With SPF + DKIM + DMARC, our site can solve the problem of email fraud.

Finally, thanks for the test and corresponding test data of Shuli. For more good articles, see my blog: kebingzao.com/

The resources

  • Use SPF for E-mail authentication in Amazon SES
  • Use DKIM for E-mail authentication in Amazon SES
  • Comply with DMARC using Amazon SES
  • How To use an SPF Record to Prevent Spoofing & Improve E-mail Reliability
  • DANE based Email security research
  • DMARC guide
  • Enable the DMARC
  • rfc6376
  • www.dkim.org/

SPF test tool

  • Tools.wordtothewise.com/spf (shows all can use the domain name send overview of IP)
  • www.kitterman.com/spf/validat… (Concise and effective, showing DNS entries and results: Pass, softfail, Fail, Neutral, etc.)
  • Mxtoolbox.com/ (Classic)

DKIM test tools

  • Dkimvalidator.com/ (you need to send an email, if the test passes, when you have DKIM configuration, you will be able to view the results of the DKIM section on the website)
  • Dkimcore.org/tools/keych… (Fill in the selector and domain name to check, you can check whether the DKIM process is set correctly)
  • www.hhlink.com/ Check DKIM/1 (he can help you verify SPF and DKIM by sending him an email)

DMARC test tools

  • www.kitterman.com/dmarc/assis… (To help you better configure DMARC)
  • Dmarcian.com/dmarc-inspe… (Display all your DMARC information fields)

Fake mail delivery test site

  • Emkee.cz/(sending all kinds of fake emails)
  • Tool.chacuo.net/mailanonymo… (Forged mail, send Email from any sender, send Email from forged Email address, send Email from any Email address)