How to switch to Sparkpost
I recently helped Amnesty International Flandres migrate to Sparkpost for outbound email and wanted to share a few tips for others considering doing the same. This post is based on experiences switching to Sparkpost, but should be relevant for pretty much all email service providers (ESPs) and there are a lot of ESPs to choose from.
If you're not using an ESP to send email from CiviCRM, then chances are you are sending email directly from the web server that hosts your CiviCRM. And you are not alone: sending email directly from the server is the default for CiviCRM, and many organisations have been happily sending CiviMail like this for years.
So why should you consider changing? In a word: deliverability. In a few more words: getting into the inbox is hard and there are lots of mistakes you can make along the way. Working with an ESP ensures you are following best practice and prevents you from making quite a few common mistakes. If you don't want to worry about things like setting up DKIM, getting your SPF records right, worrying about rate limiting, and so on (and why should you?) then an ESP is for you.
Another benefit of using an ESP is the detailed reporting that they make available. Sparkpost's dashboard gives you insights into % of emails that are delivered, the different bounce reasons, and lots more. Although a lot of this information find its way back into CiviCRM, it's often easier to do the analysis on the ESP dashboard.
Presuming that you are ready to make the switch and that you have read all the appropriate installation and configuration instructions) here are a few more things you can do to make sure the transition is as smooth as possible:
Audit your current setup
Before you switch, audit your current set up. Although a lot of these settings might change or become irrelevant once you have switched, knowing how things were working (or not working) before you started is invaluable when it comes to debugging any issues. Also, if you plan to use your web server as a back up server in the event that your ESP is unavailable, you will want to have it set up correctly. Things to check:
- What do current ISPs think about the emails you are sending? Find out by sending emails to the service providers you care about. Outlook and Gmail are a good place to start. Send one via CiviMail and one via Contacts > New Email since these different routes produce slightly different email. View the source of the email and look for the 'Authentication-Results' header. Make a note of any passes and failures with SPF, DKIM, etc. and consider fixing any misconfiguration that you see.
- Is your bounce processing set up correctly? Have you been receiving and processing bounce emails? Or have you been repeatedly sending email to invalid addresses? You can quickly find this out by checking for recent bounces in the Mail Bounces report. If no recent bounces are recorded, you either have a very clean email list or (more likely!) have been repeatedly sending email to invalid addresses. Email service providers look down upon this and may well have been sending your emails into the Spam folder as a punishment!
Diagnostic tools like MX toolbox and Mail Tester are another good source of information on the current state of affairs but take the these results with a pinch of salt - they may be oversensitive, report false positives. Stay focused on what real world email servers (like gmail and outlook) say about your emails.
Lets say you've set up your Sparkpost account, have configured the extension, and are ready to make it live. You might be feeling a bit apprehensive about switching it on, and with good reason. It's important to test as much as possible before making changes to such a mission critical system. Here are a couple of strategies you can use to test things out.
Try sending some emails through Sparkpost outside of CiviCRM. Sparkpost makes this easy to do using the application dashboard. Create a new template (https://app.sparkpost.com/templates) and hit the preview and send button. Or if you prefer, you can use Sparkpost's Postman integration to call the transmission API. Try and make the content of these test emails as authentic as possible - copying and pasting from a past mail sent via CiviCRM is a good way to do this. Have a look at the Authentication Results in received emails (as described above) to make sure things are working as you expect. If not, make the necessary changes.
If you're lucky, your emails will go straight to the inbox. If you're unlucky, you may find that some or all of them end up filtered into the Spam folder. There are a few reasons why email from Sparkpost might end up in the Spam folder. Avoid the temptation to 'blame microsoft' and ensure that you are following your ESPs best practice.
At Amnesty Flandres, we found it particularly hard to avoid Microsoft's spam filters in the first few days and weeks. Deliverability did improve in the end, and in retrospect, we think our struggles were at least in part to do with poor list hygiene before the switch.
Once you're happy with the the 'independent' testing, try turning on the extension and sending some test emails from CiviCRM. If you're lucky, you won't experience any issues, consider switching off the extension while you investigate.
Rather than risking sending an entire email with your new ESP, you may want to send out a smaller test. It is quite easy to set CiviCRM up to do this. Here is how:
- Disable the Send Scheduled Mailings job
- Adjust the Mailer settings and set the Mailer Batch Limit to 50 (or whatever size you want the test batches to be).
- Ensure that CiviCRM is set up to send using the method you want to test (the ESP or the web server)
- Manually trigger a Scheduled Mailings job with the Execute Now button which will cause CiviCRM to send 50 mails to Sparkpost for sending.
Repeat steps 3 + 4 as many times as you like. Once you are happy with the results, reset the Mailer Batch Limit, enable the job and the rest of the email will be sent via Sparkpost.
Avoid the temptation to think that when you switch to an ESP, you had over responsibility for deliverability to the ESP. If it were that simple, all the spammers would need to do is sign up for accounts with ESPs. Also avoid the temptation to think that because you are a non-profit, you couldn't possibly be seen as a spammer by receiving mail servers or the individuals that receive your emails. Now is the time to take a good look in the mirror. Does everyone in your org practice good email hygiene? Are you using your email lists responsibly and honouring unsubscribes?
All ESPs provide their own guidance on deliverability, and there is no getting around the fact: if you want to get into the inbox, you need to familiarize yourself with this stuff. Here, for example, is Sparkpost's deliverability guide. And remember that the advice on deliverability is always changing. You might want to consider subscribing to some independent sources of information - the Return Path Blog is a good way to keep up with the latest developments.
Warming up IPs
Sparkpost have their own thoughts on whether you will benefit from a dedicated IP or not, but having said that, they'll give one to anyone that applies. Opinions abound on whether it makes sense for you and I won't get into the debate here. My unscientific poll of CiviCRM providers suggests that most of them do use a dedicated IP, but I suspect a lot of Sparkpost customers are getting on fine on shared IPs.
If you do choose a dedicated IP, you will need to warm it up according to Sparkpost's warm up schedule. So how does one warm up IPS when you have hundreds of thousands of emails to send? One approach is to adjusting the mailer settings as described in the testing. Either set the batch limit to the daily limit and run one job via Sparkpost (reverting to the webserver once this has been sent). Or set the limit much lower than the daily limit and allow your email to trickle out slowly. A Mailer Batch Limit of 25 emails every 15 minutes will send out 100 emails an hour, or 2,400 a day.
Postmaster tools and DMARC
The big online email providers have their own set of postmaster tools that give you some further insight into what happens when your emails arrive at their servers. For example, Outlook.com's Smart Network Data Services (IP based - can be used with a Sparkpost dedicated IP) and Google's Postmaster tools (domain based) both provide valuable information on how your email is performing.
DMARC is another useful tool you should consider adding to your toolkit. Amongst other things, it facilitates creation of reports on whether emails sent from your domain are properly implementing DKIM and SPF. It's not the most user friendly thing in the world to set up, and requires a bit if investment of time to understand fully but if you're looking for more data on which to base your decision making, it is certainly worth a look. There are a few third party services like dmarcian that help you to make sense of the reports.
Need more help?
I've tried to give as much information as possible in the post above to help you get set up with Sparkpost and other ESPs. With a bit of luck, things will go smoothly for you, and you won't need to use most of it. If you do encounter any issues that can't be solved using the approaches above, I would be happy to help. Feel free to get in contact and I will do my best to answer your questions.