Evaluating Marketing Automation - System Performance and Usability

Evaluating the various claims in the marketing automation space is an interesting challenge. There are a variety of players, and many of the claims overlap quite a lot. It can be daunting to tell truth from fiction. I often get asked how one should independantly verify many of the claims, and I'm a big believer in the idea that the more educated a marketing automation buyer is, the better they do when they purchase a platform.


In light of that, I wanted to provide a series of actual tests that you can do when evaluating marketing automation systems. Real data (like this CSV file of sample contacts for this exercise), things to look for, and expected results are provided.


One of the basic elements to look at is system performance. It can be tempted to think of system performance as a "hygiene" factor that is common across the industry, but it's worth taking a deeper look at. The reason that this is the case is that marketing is, somewhat surprisingly, a very high scale activity. Think about the numbers for a second:

- An email campaign is sent out to your database of 50,000 people
- 30% of those people open the email, many of them multiple times, leading to 20,000 unique opens
- 3% of the email recipients visit your website, along with another 2,000 anonymous visitors, each seeing on average just less than 3 pages, for a total of 10,000 page views
- Each person in your database is scored against their title, their industry, their email opens and clicks, and their web activity (5 criteria x 50,000 people)
- The score for each person is updated in your CRM system

Very quickly, we have generated 380,000 transactions on a single day, and that is with a medium-sized database of 50,000 contacts only doing a very simple task. As soon as you begin to use your marketing automation platform more heavily, these numbers can grow quickly. If you don’t adequately assess the performance of the marketing automation platform you select, you may end up with slow performance affecting the experience of your marketing users

But how can this be assessed without a deep technical dive into each marketing automation platform? Quite simply, in fact.

In order to assess system performance, you need to see the system actually operating under conditions that you are interested in. Claims, quotes, and architecture diagrams will never compare with actually seeing the system do what you need it to do.

Here is a simple test that you can ask any of the marketing automation software vendors you are interested in evaluating to try during your demo with them.

Here is a link to a sample contact data file of 50,000 randomly generated contacts. Each one has basic information you would expect; name, address, company, email address. There are some duplicates in here, leaving just under 45,000 unique names. It’s not a huge file (only 5MB), so it should be easily accessible by anyone with Internet access.

When you are reviewing a demo of a marketing automation provider, give them the file (or the link) and ask them to do the following:

1. Upload the file to the marketing automation database (should take less than a few minutes, and you should be able to continue with the demo while it is being uploaded)
2. Ensure that the duplicates in the data are automatically handled (look for Claudia.Patterson@RainGolfEquipment.co.com as an example) to see if her data is clean (there are two records for her, which should be merged seamlessly)
3. Run a report that shows the breakdown in Job Titles within that set of data (to give you a sense of the platform’s ability to run rules against the data without impacting performance)
4. Export the full list again as a .csv file to test the system performance when doing a bulk export.
5. Delete the data that had been uploaded to ensure good performance when deleting in bulk


These 5 steps will tell you a lot about the performance of the marketing automation platform you are about to select. Things to look for are the following:
- How long does it take to upload the data?
- Is the platform usable while the upload is taking place?
- Are duplicates handled smoothly and accurately?
- Can the data be explored and reported on while in the marketing automation platform?
- Can data and reports be exported quickly and without causing platform performance issues?
- Can unneeded data be deleted in bulk, allowing you to keep your marketing database clean?

System performance is one of the most critical aspects of usability, and is very much worth evaluating. If you are considering an investment, this quick test is an easy one to perform and will give you very rich insights into whether the platform you are about to select will meet your needs.