Last summer, The Humane Society of the United States contacted our MiLo Intelligence team with a fundraising challenge: how could they maximize conversions in their sustainer conversion campaigns while optimizing mailing and production costs? The answer was to develop a model to accurately predict constituents most likely to become sustainers so that a smaller mailing could be targeted only to those most likely to respond.
As with all MiLo models, we started with client data and a specific methodology. Since HSUS has great historical data available in Revolution CRM about past sustainer conversion campaigns, we were able to analyze data as of the campaign date to get an accurate view of what the constituent looked like at the time of each campaign we were modeling. We collaborated on the appropriate model features and created a custom model for them that would predict conversions and allow them to focus on the highest ranked (most likely to respond) constituents.
Here are some of the components of the model we built:
- Total number of modeling data points: 82
- Total sustainer conversion contacts modeled: 1,070,000
- Number of machine learning models run to find best match: 56
- Selected machine learning model type: extreme Gradient Boosted Trees Classifier
How do we validate the model to make sure it will work in the real world? For every model we build, we pull a large dataset to evaluate and use 80% of the data to build the model. Once we have identified the best model, we run the remaining 20% through the model to test that the predictions made on the initial dataset hold up when looking at new data. Once we are satisfied that the model is trained and making accurate predictions, we move on to the backtest.
Backtesting the model involves running previously completed campaigns that were not part of the modeling universe through the model to see how well the predictions line up with the actual results of the campaign. We like to run more than one campaign as a backtest if possible so we can ensure that the model works for different time frames and does not have an overweighting toward one season or another. It’s important to build a model that will work throughout the year.
Once the backtests are complete, it’s time to run the model as a test in the real world so the client can evaluate performance and decide if the model will work in future campaigns. We always advise testing any new model to increase confidence and ensure that the results are in line with the organizations goals.
This is a brand new model so we don’t yet have extensive real world results, but we asked Emily Courville, Senior Director, Analytics at The Humane Society of the United States, about their modeling need and hopes for performance. “Asking donors at the right time to become part of a recurring giving program is vital to our success, but response rates are tiny in the mail. We need the insights a machine learning model can give us to optimize the resources we have to invest in these efforts. The plan is to take the typical huge invite audience and look to mail the top 50% which should get us 90% of the responders, per the model back testing.”
Given the importance of sustainers to nonprofits across the spectrum of causes, trying to optimize the audience for targeted conversion asks is a no brainer.
Please reach out to the MiLo team or your Account Director if you are interested in seeing if this model could work for you or if you have other modeling ideas.
