Recommending Tourist Attractions with Machine Learning

For my Metis capstone project, I decided to make a tourist attraction recommender, but unlike normal travel and vacation planning websites, this recommender uses an image as input.

Data Acquisition

The first step in creating this model is not only gathering the data set necessary for training our neural network but also defining the scope of the project. I decided to only look at attractions located inside the continental United States, for road trip purposes and also so recommendations were more affordable and feasible, for those looking to travel on a budget.

Web scraping Tripadvisor, using Selenium and Beautifulsoup

Attraction Labeling

At most 60 images were scraped from each attractions image gallery, some attractions only had 30 images. This is not enough data to train a neural network on to classify an image as a certain attraction, even excluding the fact that some of these images are already mislabeled or have no correlation to the attraction. So the decision was made to label attractions based on an assigned attraction class and then using a neural network, classify an image into an attraction class. After classifying, attractions within the specified attraction class will be recommended to the user.

WordCloud created after text pre-processing
The eight attraction class labels with example images from the dataset

Image Pre-Processing

After attraction labeling, a new data frame was created with each row representing a single image, with an image url, attraction name, location, and class label. Images are now ready to be loaded into an array and preprocessed.

Neural Network with Transfer Learning using VGG-16

Three initial neural networks were modeled: a Convolutional Neural Network (CNN), a CNN with regularization, and a neural network with transfer learning using the pre-trained VGG-16 computer vision model. Training and scoring all models on a small subset of the dataset showed that the neural network with transfer learning outperformed the CNN models. Looking at all models confusion matrices and accuracy and loss graphs, there appeared to be severe overfitting as well as class imbalance.

Initial NN with transfer learning results for small subset of dataset
Fixing class imbalance
Three variations on an image
Final confusion matrix and accuracy and loss graphs of the neural network

Finding Attraction Recommendations

Attraction recommendations are found by finding the closest attractions, within an attraction class to the input image. This is done by creating feature vectors for images and finding the distance between vectors using cosine distance. Two different feature vectors are used to calculate distance: the VGG-16 feature vector previously used to train the neural network and then a color distribution feature vector.

Example of RGB color distributions (12 bins) for two corner sections of an image
Calculating the TOTAL distances between an input image and images in an attraction class

Final Model

Testing the Final Model

Running the final model to show 3 recommendations for an input image

Data Scientist / Analyst

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store