Evaluating Marketing Automation - Data Management

Continuing on a theme that received great feedback, I wanted to provide another real, down in the details, way to evaluate the various claims in the marketing automation field. Last time we looked a way to ensure that a provider could have the performance needed for your marketing goals - a quick and simple upload that will test actual marketing automation system performance.

In this post, it's worth taking it one step further. Getting marketing data into a platform is one thing, but if the data is messy (and what marketing data isn't), it will not be of much use. If, for example, your marketing database has 100,000 names in it, and the titles are just as they were written, such as:

• VP Marketing
• V.P. Mktg
• Vice Pres Marketing
• Marketing Vice President
• Mktg VP

and you are asked to build a list of Vice Presidents of Marketing to target, how many will you find? 300? 800? We've seen many situations where dirty data returned 300 names, but the same query against cleansed data returned 17,000 names. Proper management of data makes a huge difference in your marketing results.

So, how do you test for this when considering a marketing automation software investment?

Quite simply - ask, in a demo, for each vendor you are considering to run a quick test. Here is a sample CSV file with typical marketing data. Titles, states, and countries are as they would be in a normal marketing or CRM database. The data is kept simple, and the titles are mostly in sales, marketing, and finance, while the addresses are in Canada, US, and UK.

Have each vendor run the following test for you:

• Upload the sample file
• Clean up the country fields so that US, USA, U.S.A, as well as the variations of Canada, and England/UK are normalized
• Clean up the "raw" job title fields to two new fields for "level" (VP, Director, etc), and "role" (marketing, finance, etc) so you can properly segment
• As a bonus, see if they can correct the missing leading "0" on New England zip codes - removed by Excel in many marketers' data files

When it is uploaded and cleansed check the data to see the following:

• The only countries in the file are "USA", "GBR" and "CAN" or however you chose to normalize the country data
• The people can easily be filtered by role into "Marketing", "Sales", or "Finance"
• The people can easily be filtered by level into "SVP", "VP", "Director", or "Manager"

Many marketing challenges come from bad data. An inability to do proper segmentation, personalization, lead scoring, or analytics can quickly result if you are not able to standardize and normalize the data in your marketing database. To avoid getting into this situation, it's worth having the marketing automation provider you are thinking of choosing run through this quick test with real sample data.

Evaluating Marketing Automation - System Performance and Usability

Evaluating the various claims in the marketing automation space is an interesting challenge. There are a variety of players, and many of the claims overlap quite a lot. It can be daunting to tell truth from fiction. I often get asked how one should independantly verify many of the claims, and I'm a big believer in the idea that the more educated a marketing automation buyer is, the better they do when they purchase a platform.


In light of that, I wanted to provide a series of actual tests that you can do when evaluating marketing automation systems. Real data (like this CSV file of sample contacts for this exercise), things to look for, and expected results are provided.


One of the basic elements to look at is system performance. It can be tempted to think of system performance as a "hygiene" factor that is common across the industry, but it's worth taking a deeper look at. The reason that this is the case is that marketing is, somewhat surprisingly, a very high scale activity. Think about the numbers for a second:

- An email campaign is sent out to your database of 50,000 people
- 30% of those people open the email, many of them multiple times, leading to 20,000 unique opens
- 3% of the email recipients visit your website, along with another 2,000 anonymous visitors, each seeing on average just less than 3 pages, for a total of 10,000 page views
- Each person in your database is scored against their title, their industry, their email opens and clicks, and their web activity (5 criteria x 50,000 people)
- The score for each person is updated in your CRM system

Very quickly, we have generated 380,000 transactions on a single day, and that is with a medium-sized database of 50,000 contacts only doing a very simple task. As soon as you begin to use your marketing automation platform more heavily, these numbers can grow quickly. If you don’t adequately assess the performance of the marketing automation platform you select, you may end up with slow performance affecting the experience of your marketing users

But how can this be assessed without a deep technical dive into each marketing automation platform? Quite simply, in fact.

In order to assess system performance, you need to see the system actually operating under conditions that you are interested in. Claims, quotes, and architecture diagrams will never compare with actually seeing the system do what you need it to do.

Here is a simple test that you can ask any of the marketing automation software vendors you are interested in evaluating to try during your demo with them.

Here is a link to a sample contact data file of 50,000 randomly generated contacts. Each one has basic information you would expect; name, address, company, email address. There are some duplicates in here, leaving just under 45,000 unique names. It’s not a huge file (only 5MB), so it should be easily accessible by anyone with Internet access.

When you are reviewing a demo of a marketing automation provider, give them the file (or the link) and ask them to do the following:

1. Upload the file to the marketing automation database (should take less than a few minutes, and you should be able to continue with the demo while it is being uploaded)
2. Ensure that the duplicates in the data are automatically handled (look for Claudia.Patterson@RainGolfEquipment.co.com as an example) to see if her data is clean (there are two records for her, which should be merged seamlessly)
3. Run a report that shows the breakdown in Job Titles within that set of data (to give you a sense of the platform’s ability to run rules against the data without impacting performance)
4. Export the full list again as a .csv file to test the system performance when doing a bulk export.
5. Delete the data that had been uploaded to ensure good performance when deleting in bulk


These 5 steps will tell you a lot about the performance of the marketing automation platform you are about to select. Things to look for are the following:
- How long does it take to upload the data?
- Is the platform usable while the upload is taking place?
- Are duplicates handled smoothly and accurately?
- Can the data be explored and reported on while in the marketing automation platform?
- Can data and reports be exported quickly and without causing platform performance issues?
- Can unneeded data be deleted in bulk, allowing you to keep your marketing database clean?

System performance is one of the most critical aspects of usability, and is very much worth evaluating. If you are considering an investment, this quick test is an easy one to perform and will give you very rich insights into whether the platform you are about to select will meet your needs.