This blog post will teach you how to visually verify the emails sent by a 3rd party service. Such verification is necessary to ensure the emails have the expected style, and include the necessary information. The post teaches you how to deal with the difficult email markup generated from dynamic templates when doing image comparison.
Note: you can find the source code for this blog post in the repository cypress-sendgrid-visual-testing-example, and see the visual snapshots at its Percy page.
Dynamic templates
If your application is sending HTML emails using a 3rd party service like SendGrid, you might be using "dynamic templates" - meaning your email design is stored on the service, and your application triggers the send by providing the template data.
For example, I have designed the following "Confirmation code" email in SendGrid
To send an email, the application needs to provide the confirmation code, the application URL to open, and the user name. We can see the email preview while designing the template:
Because the template lives in a 3rd party system, it is simple for someone to modify it, and accidentally break the data to markup mapping, leading to bugs such as the missing user name in the greeting bug below:
In order to guarantee our users are going to get a nice-looking and correct confirmation email, we need to test the entire confirmation flow, and we need to verify the received email using visual testing.
Sending emails
Our application no longer needs to have HTML markup for every email. Instead it makes a call to the SendGrid API and provides the template ID and the template data object.
// pages/api/register.js
const initEmailer = require('../../src/emailer')
const emailer = await initEmailer()
await emailer.sendTemplateEmail({
to: email,
// the ID of the dynamic template we have designed
template_id: 'd-9b1e07a7d2994b14ae394026a6ccc997',
dynamic_template_data: {
code: confirmationCode,
username: name,
confirm_url: 'http://localhost:3000/confirm',
},
})
console.log('sent a confirmation email to %s', email)The emailer function uses the NPM package @sendgrid/client to contact SendGrid
// src/emailer.js
const sgClient = require('@sendgrid/client')
sgClient.setApiKey(process.env.SENDGRID_API_KEY)
/**
* Sends an email by using SendGrid dynamic design template
* @see Docs https://sendgrid.api-docs.io/v3.0/mail-send/v3-mail-send
*/
async sendTemplateEmail({ from, template_id, dynamic_template_data, to }) {
const body = {
from: {
email: from || process.env.SENDGRID_FROM,
name: 'Confirmation system',
},
personalizations: [
{
to: [{ email: to }],
dynamic_template_data,
},
],
template_id,
}
const request = {
method: 'POST',
url: 'v3/mail/send',
body,
}
const [response] = await sgClient.request(request)
return response
}We can print the email before posting it by digging into the @sendgrid/client code and adding a console command:
'{\n' +
" url: 'v3/mail/send',\n" +
" method: 'POST',\n" +
' headers: {\n' +
" Accept: 'application/json',\n" +
" 'Content-Type': 'application/json',\n" +
" 'User-Agent': 'sendgrid/7.4.3;nodejs',\n" +
" Authorization: 'Bearer <SENDGRID_API_KEY>'\n" +
' },\n' +
' maxContentLength: Infinity,\n' +
' maxBodyLength: Infinity,\n' +
' data: {\n' +
' from: {\n' +
" email: '<SENDGRID_FROM>',\n" +
" name: 'Confirmation system'\n" +
' },\n' +
' personalizations: [\n' +
' {\n' +
" to: [ { email: 'gleb@acme.co' } ],\n" +
' dynamic_template_data: {\n' +
" code: '8a82b408',\n" +
" username: 'Gleb',\n" +
" confirm_url: 'http://localhost:3000/confirm'\n" +
' }\n' +
' }\n' +
' ],\n' +
" template_id: 'd-9b1e07a7d2994b14ae394026a6ccc997'\n" +
' },\n' +
" baseURL: 'https://api.sendgrid.com/'\n" +
'}'Note that to run our application we need to provide the SENDGRID_API_KEY and SENDGRID_FROM environment variables.
Receiving emails
Every time we call the SendGrid API it sends an email to the to recipient. Our test needs to "catch" these emails. Luckily there is a free Ethereal email service with a great programmatic API we can use.
When the Cypress project starts, we can create a new email user.
// cypress/plugins/index.js
const makeEmailAccount = require('./email-account')
/**
* @type {Cypress.PluginConfig}
*/
module.exports = async (on) => {
const emailAccount = await makeEmailAccount()
on('task', {
getUserEmail() {
return emailAccount.email
},
getLastEmail() {
return emailAccount.getLastEmail()
},
})
}The email-account.js uses the nodemailer to create the user and retrieve the last email received.
// cypress/plugins/email-account.js
// use Nodemailer to get an Ethereal email inbox
// https://nodemailer.com/about/
const nodemailer = require('nodemailer')
// used to check the email inbox
const imaps = require('imap-simple')
// used to parse emails from the inbox
const simpleParser = require('mailparser').simpleParser
...
// Generate a new Ethereal email inbox account
const testAccount = await nodemailer.createTestAccount()
console.log('created new email account %s', testAccount.user)
console.log('for debugging, the password is %s', testAccount.pass)In the spec file we use the testAccount.email address when registering the new user.
// cypress/integration/spec.js
describe('Email confirmation', () => {
let userEmail
before(() => {
// get and check the test email only once before the tests
cy.task('getUserEmail').then((email) => {
expect(email).to.be.a('string')
userEmail = email
})
})
it('sends confirmation code', () => {
const userName = 'Joe Bravo'
cy.visit('/')
cy.get('#name').type(userName)
cy.get('#email').type(userEmail)
cy.get('#company_size').select('3')
cy.get('button[type=submit]').click()
})
})The test types the new Ethereal user into the form and clicks the "Submit" button
The plugin file prints the created test account email and password in the terminal. Using these credentials we can log into the Ethereal inbox to see the received email.
In the inbox we should see the email from SendGrid service with the random confirmation code generated by the application.
Tip: notice the "From" field above - I am using another random Ethereal email as a verified sender in the SendGrid settings. Even though the Ethereal email cannot send any emails, it can receive the sender verification email from SendGrid, and I can click the "confirm" link after logging into the Ethereal Web interface. This is exactly the same confirmation flow we are trying to verify in our application.
Cypress email test
Let's complete our test - we want to access the Ethereal inbox, find the last email, load its HTML inside the Cypress' browser (it is an HTML email after all!), click on the "Enter the confirmation code ..." button and enter the code in our application's page. Here is our complete test:
// cypress/integration/spec.js
const { recurse } = require('cypress-recurse')
describe('Email confirmation', () => {
let userEmail
before(() => {
// get and check the test email only once before the tests
cy.task('getUserEmail').then((email) => {
expect(email).to.be.a('string')
userEmail = email
})
})
it('sends confirmation code', () => {
const userName = 'Joe Bravo'
cy.visit('/')
cy.get('#name').type(userName)
cy.get('#email').type(userEmail)
cy.get('#company_size').select('3')
cy.get('button[type=submit]').click()
cy.log('**shows message to check emails**')
cy.get('[data-cy=sent-email-to]')
.should('be.visible')
.and('have.text', userEmail)
// retry fetching the email
recurse(
() => cy.task('getLastEmail'), // Cypress commands to retry
Cypress._.isObject, // keep retrying until the task returns an object
{
timeout: 60000, // retry up to 1 minute
delay: 5000, // wait 5 seconds between attempts
},
)
.its('html')
.then((html) => {
cy.document({ log: false }).invoke({ log: false }, 'write', html)
})
cy.log('**email has the user name**')
cy.contains(`Dear ${userName},`).should('be.visible')
cy.log('**email has the confirmation code**')
cy.contains('a', 'Enter the confirmation code')
.should('be.visible')
.invoke('text')
.then((text) => Cypress._.last(text.split(' ')))
.then((code) => {
cy.log(`**confirm the code ${code} works**`)
expect(code, 'confirmation code')
.to.be.a('string')
.and.have.length.gt(5)
// add synthetic delay, otherwise the email
// flashes very quickly
cy.wait(2000)
// unfortunately we cannot confirm the destination URL
// via <a href="..."> attribute, because SendGrid changes
// the href to its proxy URL
// before we click on the link, let's make sure it
// does not open a new browser window
// https://glebbahmutov.com/blog/cypress-tips-and-tricks/#deal-with-target_blank
cy.contains('a', 'Enter the confirmation code')
// by default the link wants to open a new window
.should('have.attr', 'target', '_blank')
// but the test can point the open back at itself
// so the click opens it in the current browser window
.invoke('attr', 'target', '_self')
.click()
cy.get('#confirmation_code', { timeout: 10000 }).should('be.visible')
// confirm the URL changed back to our web app
cy.location('pathname').should('equal', '/confirm')
cy.get('#confirmation_code').type(code)
cy.get('button[type=submit]').click()
// first positive assertion, then negative
// https://glebbahmutov.com/blog/negative-assertions/
cy.get('[data-cy=confirmed-code]').should('be.visible')
cy.get('[data-cy=incorrect-code]').should('not.exist')
})
})
})The test runs through the entire flow
Visual test for the registration form
Let's start with a visual assertion for the very first registration form page. We have an array of choices for visual testing - you can do visual testing yourself, or use a 3rd-party commercial service. In this blog post I will use Percy visual testing service - it is simple to set up and has a generous free tier. For setup steps, see the official Percy Cypress docs. In the spec itself, we can use cy.percySnapshot(<title>) command.
// cypress/integration/spec.js
// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
it('sends confirmation code', () => {
const userName = 'Joe Bravo'
cy.visit('/')
cy.get('#name').type(userName)
cy.get('#email').type(userEmail)
cy.get('#company_size').select('3')
cy.percySnapshot('1 - registration screen')
cy.get('button[type=submit]').click()
...
})
})Percy visual service runs asynchronously - it sends the DOM and CSS snapshots to its image rendering and comparing cloud, and then posts the status to GitHub. Thus we can have commits and pull requests where functional Cypress tests finish successfully, while visual Percy tests fail.
As soon as we run Percy for the very first time in this project, it notices a new visual snapshot - we need to verify the image to accept it (which turns the failing Percy status check into a passing one).
By default, Percy renders the DOM + CSS snapshot in both Chrome and Firefox browsers, and in two resolutions.
As you can see, Firefox renders the form almost identically, thus we do not need the extra images. But we should keep the two rendered resolutions, since the form's layout is responsive and should be tested in both viewports. We can control the browsers from the project's settings page in Percy.
Random data
We feel good - our registration form is from now on going to stay the same until we explicitly allow it to change. We run the test again ... and it fails!
In order to have exactly the same rendered images, the application page must have exactly the same data each time. We generated a random test email, thus the input field has different value. Ughh, what should we do?
We could try hiding the email input field from Percy by providing custom CSS like this:
// we need to ignore the userEmail region when doing
// the visual diff, since the text is dynamic
cy.percySnapshot('1 - registration screen', {
percyCSS: '#email { display: none; }',
})This does not solve the problem - it removes the element completely, creating an image that looks wrong.
Instead I will do the following trick:
cy.visit('/')
cy.get('#name').type(userName)
cy.get('#company_size').select('3')
// avoiding visual difference due to new email
cy.get('#email').type('gleb@acme.io')
cy.percySnapshot('1 - registration screen')
// type the real email
cy.get('#email').clear().type(userEmail)
cy.get('button[type=submit]').click()Now every build will have the same form data producing the same image.
Visual test for HTML email
Let's add a visual test for the HTML we retrieve from the server and write into the browser.
cy.contains('a', 'Enter the confirmation code')
.should('be.visible')
.as('@codeLink') // save under an alias
.invoke('text')
.then((text) => Cypress._.last(text.split(' ')))
.then((code) => {
cy.log(`**confirm the code ${code} works**`)
expect(code, 'confirmation code')
.to.be.a('string')
.and.have.length.gt(5)
// add synthetic delay, otherwise the email
// flashes very quickly
cy.wait(2000)
cy.percySnapshot('2 - email')
...The two second delay is very useful for making sure the rendered HTML appears in the video captured by Cypress during cypress run. Percy build shows the new snapshot for approval. We immediately can see the problem - we are showing the confirmation code. Because the code is dynamic data, even if we approve the current snapshot, every new snapshot will fail.
Luckily, we can play the same data trick - by now we grabbed the confirmation code, and can safely change it in the page to something constant. Unfortunately, SendGrid dynamic templates are strict with HTML markup, and we cannot add data attributes following our best practices for selecting elements. We can still use cy.contains to find and replace element's text.
// add synthetic delay, otherwise the email
// flashes very quickly
cy.wait(2000)
// replace the dynamic confirmation code with constant text
// since we already validated the code
cy.get('@codeLink').invoke(
'text',
`Enter the confirmation code abc1234`,
)
cy.contains('strong', new RegExp('^' + code + '$')).invoke(
'text',
'abc1234',
)
cy.percySnapshot('2 - email')Time-traveling debugger helps us confirm the test changes the dynamic text in two elements correctly.
With these changes, the visual snapshot is going to be consistent.
The email still works the same after our text changes, because the functional Cypress test confirms it. Let's make sure the confirmation screen is tested visually too.
Visual test for confirmation page
The localhost:3000/confirm is the page where the user enters the confirmation code. Our test verifies that the code is correct. We can add a visual assertion to the test, but after changing the dynamic data in the input field to a constant.
// confirm the URL changed back to our web app
cy.location('pathname').should('equal', '/confirm')
cy.get('#confirmation_code').type(code)
cy.get('button[type=submit]').click()
// first positive assertion, then negative
// https://glebbahmutov.com/blog/negative-assertions/
cy.get('[data-cy=confirmed-code]').should('be.visible')
cy.get('[data-cy=incorrect-code]').should('not.exist')
cy.get('#confirmation_code').clear().type('correct code')
cy.percySnapshot('3 - correct code')
We also need to confirm the rejection flow. If the user enters the wrong confirmation code, the application should show an error. We can create a separate test for this:
it('rejects wrong code', () => {
cy.visit('/confirm')
cy.get('#confirmation_code').type('wrong code')
cy.get('button[type=submit]').click()
// first positive assertion, then negative
// https://glebbahmutov.com/blog/negative-assertions/
cy.get('[data-cy=incorrect-code]').should('be.visible')
cy.get('[data-cy=confirmed-code]').should('not.exist')
cy.percySnapshot('incorrect code')
})There is no need for data manipulation in this case - the form has no dynamic data.
Recap
- Visual testing is a powerful mechanism for quickly confirming the entire browser page looks the same as before.
- Cypress can interact with remote services to fetch delivered emails and render them in its browser. Then we can use visual testing to confirm the email looks the same as expected.
- Dynamic data can pose challenges for visual testing, and we should ensure the page's DOM has the same data before taking a snapshot.
Happy email testing!
See also
We have shown alternative email testing approaches in the following blog posts:
- "Testing HTML Emails using Cypress" shows how to run a mock SMTP server inside Cypress during testing
- "Full Testing of HTML Emails using SendGrid and Ethereal Accounts" formats the email HTML text and then sends it using a 3rd party service to a temporary test account