What Is a Recommendation System?
A recommendation system uses a subset of AI known as machine learning and data filtering techniques to make suggestions. When they work well, AI recommendation systems make accurate predictions about what types of content a user wants based on past choices, feedback, and engagement.
Netflix, Amazon, eBay, and most social media platforms have recommendation systems to help visitors sort through massive amounts of content to discover the specific items that are likely to be of interest. Artificial intelligence solutions like recommenders are common in day-to-day modern life, almost to the point that they are expected. Yet, to be optimally effective, they require substantial amounts of data, AI model training, and computing power.
Audiences sometimes depend on recommendation systems without being aware that they do. Choosing one movie to watch out of hundreds of thousands could be an insurmountable challenge, but choosing one movie out of 100 reduces the difficulty of the decision. When the recommendations are accurate and the system is responsive, the process of deciding what to consume becomes seamless.
Benefits of Recommendation Systems
For companies that want users to stay engaged, like social media platforms, or to continue to subscribe or make purchases, deploying an AI recommendation system can help them achieve those goals. In some instances, the recommendation system acts like the impulse-buy shelf in a grocery store. In others, it’s more like offering add-ons, such as tire protection at a car dealership.
Users benefit, too, because decision-making is difficult. Even when a user has a good idea of what they want, sifting through all the items they don’t want can be challenging. Searching a catalog of movies for “comedies” brings up a wide range of options with vastly different takes. Recommendation systems help users narrow their choices to find what appeals to them most.
Additionally, recommenders can help users discover new content they might not have found otherwise. Music streaming services such as Pandora and Spotify are examples of platforms that subscribers often use to find new content based on their preferences.
How Do Recommendation Systems Work?
AI Recommendation systems are complex and use several AI models, machine learning processes, and data analytics workflows. Most workflows include three general stages:
- Classification: In this stage, computer vision and natural language processing (NLP) are used to identify and classify different elements of pieces of content.
- Recall and similarity search: Next, items or objects are categorized by similar features.
- Ranking: Finally, wide and deep learning models sort the items or objects by relevance.
These three stages are necessary to deliver accurate and relevant results and reduce the risk of user frustration, not only with the recommendations but also with the company whose system provided them.
Types of Recommendation Systems
Recommendation systems are endlessly customizable and should be tailored closely to the specific use case. Generally, recommendation systems fall into three broad categories:
- Collaborative filtering: A collaborative data filtering recommendation system requires preference information from many users. The system recognizes patterns: People who like this movie also often like this other movie. It then recommends the other movie to people who like the first movie.
Similarly, if two users have two or more items in common, the recommendation system may recommend an item one person has rated highly or purchased to the other one with similar interests. - Content-based filtering: Content-based data filtering is helpful in situations where less information is available, such as when users are researching bigger-ticket items that are infrequently purchased, like new furniture or appliances. The recommendation system may suggest items of similar size, with similar features, or in a comparable price range.
Using the previous movie example, if a user has watched two action movies, the recommendation system may suggest other titles in that category. - Hybrid filtering: A recommender that uses both collaborative and content-based data filtering provides the elements of both types and sometimes also includes contextual information, such as location, time of day, and other data, to make more accurate and useful recommendations.
What Makes a Good Recommendation System?
A business goal for recommendation systems is repeated use. The more often a user perceives the recommendations they receive as “good,” meaning a close match to their preferences, the more likely they are to return to and reuse the system. Attributes of good recommendation systems include:
- Accurate: The suggestions must be personalized for the user. This can be achieved by incorporating a feedback system such as a simple thumbs up or down. Feedback reinforces and extends the AI’s learning.
- Responsive: Users are notoriously impatient and will not wait for recommendations. Load times must be fast and frictionless, or users will simply leave.
- Cost-effective: From a business standpoint, the investment in a recommendation system and related ongoing operational expenses must be balanced by increased user engagement, sales, or other business outcomes.
Recommendation System Use Cases
Recommendation systems are pervasive in the digital world. Nearly all e-commerce retailers have some kind of recommendation system, and most people use them seamlessly without even realizing what they are. Amazon was an early adopter, and without recommendations, discovering new products could be quite difficult.
Netflix also has an extensive and ever-evolving recommendation system. In fact, from 2007 to 2009, Netflix offered a prize to the team that could build a recommendation system that was just 10 percent more accurate. The winners created a pipeline of 107 different models that work together to offer a prediction.1
The two main uses of AI recommendation systems are personalized merchandising, like on eBay or Amazon, and personalized content, such as on social media platforms like Facebook or LinkedIn.
Personalized Merchandising
Most major retail brands have personalized merchandising recommendation systems. These can work in a couple of different ways.
For example, when a user browses listings on eBay, an “explore related items” recommendation system offers similar listings, with the option to give feedback on the suggestions. Clothing retailer Old Navy offers both a “customers also liked” recommender and “wear it with” recommendations.
As AI algorithm-powered recommendation systems become more common, they are also being used in new ways. In banking, recommendation systems may be used to securely suggest account types, services, or offers based on customer saving and spending behaviors, or in education, recommenders could help students decide what colleges to apply to.
Personalized Content
Users interact with recommendation systems seamlessly throughout a typical day, especially as they consume media and entertainment. Google uses a recommendation system to present ads to users; Meta uses recommendation systems across its products, including the Instagram Explore page, Facebook Reels, and the main feed users see on those platforms.
Both Amazon and Netflix use machine learning and vast amounts of data to recommend personalized content. User viewing information, search history data, ratings, date, time of day, and the type of device used are all folded into multilayered, hybrid recommendation engines.
Personalized recommendations, rather than only what’s popular at the moment, can lead to much deeper engagement and exploration—and provide additional data for even more relevant results.
Challenges of Recommendation Systems
Recommenders are useful for both organizations and users, but potential challenges also exist. Being aware of the challenges from the beginning of building a recommendation system offers an opportunity to avoid problems later. Here are a few of the known challenges associated with AI recommendation systems:
- Data sparsity: Early in the process, it’s possible that many items or products haven’t been rated or that the user is new, so the recommendation system doesn’t have much information to go on. Netflix, for example, asks new users to rate movies they’ve seen. Basic feedback mechanisms, such as a thumbs up or down or a star rating, can help provide a starting point.
- Cold start problem: New users and new items present a similar challenge. With insufficient data about the user’s preferences or about an item, the recommendation system can’t make accurate and useful recommendations. For example, imagine a user is shopping for a new refrigerator and visits a retail site they’ve never looked at before. The site’s recommendation system doesn’t have any information about what the user is looking for other than the current search.
- Scalability: Scaling from a relatively small amount of data to millions of users and items requires carefully planned technology infrastructure to balance use and accelerate time to results.
- Overfitting and diversity: These two challenges are related, and both result in the most popular items being recommended too often. When a recommendation system is overfitted, the training data fits the model too well, and new information isn’t incorporated easily. When the same products are recommended again and again, a lack of suggestion diversity emerges, and users may become disengaged. Using metrics such as entropy and novelty to measure the diversity of recommendations may be helpful.
- Algorithmic overreliance: As recommenders become more common and part of daily life, people can use them too often to make decisions and end up with filter bubbles or echo chambers. This is particularly problematic for more vulnerable people, such as minors, using social media platforms.
- Privacy: Recommendation systems must have access to users’ data, such as their browser and purchase histories, which can raise privacy issues. Companies implementing AI solutions should be aware of data security and privacy risks and implement security solutions to protect user and business data. Additionally, companies should be aware of responsible AI practices to ensure AI is being used in a safe, trustworthy, and ethical way.
The Future of Recommendation Systems
Generative AI, more-precise data filtering, increasing amounts of data to filter, and improved machine learning and large language models (LLMs) all point to a steady improvement in how well recommendation systems work for both users and companies. For companies that haven’t yet implemented recommendation systems, the tools to customize and train an effective recommender are becoming easier to access.