Data Science Hobby Project: Web Scraping Book Ratings With BeautifulSoup

Tamas Ujhelyi was a participant in my 6-week data science course (the Junior Data Scientist’s First Month). After finishing the course, he started a cool hobby project and he sent it over to us for a review. After going through it, I found it just the perfect example of a hobby project that junior data scientists need to boost their portfolio. It’s about web scraping (with BeautifulSoup and Python) and analyzing customer reviews.
Anyways: I asked Tamas to write an article and guide us through his project — so every Data36 reader can benefit from it. Enjoy!

Just like many people, I love reading books – but I never trust what I’ll read next to luck, because I don’t want to waste my time. So before deciding which book to pick up next, I always visit Goodreads and Moly to consult the wisdom of other fellow readers.

I’m pretty sure you’re familiar with Goodreads – it’s a website where readers can rate the books they’ve read and write reviews about them to share their reading experience.

Moly is the Hungarian equivalent of Goodreads. It means moth, and it’s a nice little pun, because in Hungary we say bookmoth instead of bookworm about someone who reads a lot. 🙂

But why is this important, you may ask…?

As someone who reads lots of book reviews, I’ve always wondered when people are more likely to write a review.
To put it more specifically: are readers more motivated to share their opinion when they liked a book or when they disliked or even hated their reading?

That’s exactly what I wanted to find out.

1. Libraries used in the project

You’ll find no surprises here, just the usual stuff for web scraping and data analysis. Feel free to skip this part if you’re more interested in the actual project.

Anyway, here’s the list:

1.1 Requests: to get the HTML content of web pages.

1.2 BeautifulSoup: to extract data from the HTML content of the requested web pages.

1.3 Re: to create regular expressions.

1.4 Counter (from collections): to count the number of reviews and ratings by rating value.

1.5 Pandas: to create dataframes from the extracted data.

1.6 Matplotlib: to visualize data.

1.7 NumPy: to make charts more readable.

1.8 Random: to test the findings of the project by creating random samples.

2. Getting the data with web scraping (Requests + BeautifulSoup)

Before getting into the technical details, I’d like to share with you some statistics about the project. The results you’ll soon read about are based on:

• 875 books,
• 18,872 scraped URLs,
• 56,782 reviews,
• 183,764 ratings.

A book’s rating by a reader is expressed in stars. Its possible values are:

• 5.0 (highest), 
• 4.5, 
• 4.0, 
• 3.5, 
• 3.0, 
• 2.5, 
• 2.0, 
• 1.5, 
• 1.0, 
• 0.5, 
• 0 (lowest).

A review is a reader’s written opinion about a book. Reviews are always associated with ratings, meaning that if someone writes a review she is also required to give a rating. Whenever I refer to a rating with review in this article, I mean the rating that’s associated with a written review.

For the analysis, I collected the ratings and reviews of 875 books. On moly.hu, each book has its own separate page, but their ratings and reviews can be accessed on special “review” URLs  – these are what I call review pages from now on. I scraped 18,872 review pages in total.

The books I included in the analysis can be found on the 1001 könyv kitüntetés (in English: 1001 books award) list. I chose this list because it contains diverse books from different authors and of different genres, so the findings of my analysis could be more easily generalized.

Now that we got all this out of the way, let’s get that data, shall we? 🙂

2.1 Collecting the books’ URLs

First, I had to get every book’s URL from the list. The books are listed on 44 pages:

With a simple while loop it was easy to get the URLs, and save them to book_urls for later use:

book_urls = []
page = 1
while page != 45:
    result = requests.get(f"https://moly.hu/listak/1001-konyv-kituntetes?page={str(page)}")
    time.sleep(1)
    src = result.content
    soup = BeautifulSoup(src, "lxml")
    all_a = soup.find_all("a", class_="fn book_selector")
    book_urls.extend(["https://moly.hu" + a["href"] for a in  all_a])
    page += 1

I used Requests and BeautifulSoup to scrape the pages, and collect the URLs of the books.

In case you are wondering, time.sleep(1) was needed for safety reasons: it stopped the code from running for 1 sec, so the scraping process didn’t overload the web server with requests.

After this piece of code had finished running, I had all 875 book URLs in the book_urls list:

2.2 Extracting data for every book

Having gathered all books’ URLs, I could move on with the analysis by looping through book_urls with a for loop.

For every book I created a book dictionary in which I stored the following data:

• the book’s title (book[“title”], string variable),
• the book’s URL (book[“url”], string variable),
• the book’s ratings (book[“ratings”], list variable),
• the book’s ratings with reviews (book[“reviews”], list variable).

Just to make it easier for you to imagine, this is what the book dictionary looked like by the end of a loop iteration:

{
'title': 'A bambuszgyűjtő öregember meséi',
'url': 'https://moly.hu/konyvek/a-bambuszgyujto-oregember-mesei',
'ratings': [5.0, 5.0, 4.5, 5.0, 5.0, 5.0, 4.0, 4.5, 4.5, 4.0, 3.5, 4.0, 4.5, 5.0, 3.0, 4.5, 5.0, 4.0, 4.0, 5.0, 5.0, 4.0, 4.5, 4.5, 5.0, 5.0, 5.0, 5.0, 4.5, 4.0, 4.0, 0, 4.0, 5.0, 5.0, 5.0, 5.0, 3.0, 4.5, 4.0, 3.5, 4.0, 5.0, 4.5, 4.0, 4.0, 4.5, 5.0, 4.0, 4.5, 5.0, 4.0, 4.5, 4.5, 5.0, 4.0, 4.5, 3.0, 5.0, 5.0, 4.5, 4.0, 5.0, 4.0, 3.5],
'reviews': [5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.5, 5.0, 4.5, 5.0, 4.0, 4.0, 5.0, 5.0, 4.0, 4.5, 5.0, 5.0, 5.0, 4.5, 0, 5.0, 5.0, 5.0, 3.0, 4.0, 5.0, 4.0, 4.0, 5.0, 4.5, 5.0, 4.5, 4.0, 3.0, 5.0]
}

After a book dictionary was created, I added (books.append(book)) it to the books list which I’d created before and outside of the for loop, so that later I could access all books.

All you have to remember is that I created a book dictionary for all 875 books. If you’re interested in a more detailed explanation, here’s how I did it step by step. If not, it’s completely fine, too – just skip to the next section: 2.3 Creating dataframes. 

So, as the first step I created the books and the review_pages list, then the star_values dictionary:

books = []
review_pages = []
star_values = {
    "80" : 5.0,
    "72" : 4.5,
    "64" : 4.0,
    "56" : 3.5,
    "48" : 3.0,
    "40" : 2.5,
    "32" : 2.0,
    "24" : 1.5,
    "16" : 1.0,
    "8" : 0.5,
    "0" : 0
}

I needed books to store every book that I got after each loop iteration. In all honesty, I didn’t end up using review_pages, because I only needed it as a backup in case I had to redo the scraping process – I didn’t want to lose the URLs I had already collected, so I saved every review page URL here (remember, review pages are the pages where a book’s ratings and reviews can be found).

star_values was needed for a different purpose, just look at this screenshot from a review page’s source code:

Every rating is defined by its width in pixels: a rating of 5.0 is 80px wide, a rating of 4.5 is 72px wide, a rating of 4.0 is 64px wide, you get the idea…

With BeautifulSoup, I could only get the width of a rating, so I used star_values to translate the width of the rating HTML element to its real rating value (for instance, 80px was translated to a rating of 5.0):

star_value = star_values[review.find("span", class_="rater-on")["style"].split(" ", 1)[1].split("px;", 1)[0]]

Let’s keep moving. 🙂

In the for loop that I’ve already mentioned, I first created a book dictionary, a book_ratings and a book_reviews list:

book = {}
book_ratings = []
book_reviews = []

I’ve already discussed these, so on to the next section: this is where I set up the scraping part of the code. I requested a book’s URL with requests.get(book_url), parsed the book’s web page with BeautifulSoup, then saved its URL into book[“url”] and its title into book[“title”]:

result = requests.get(book_url)
src = result.content
soup = BeautifulSoup(src, "lxml")
book["title"] = soup.find("h1").find("span").get_text().rstrip().replace("\u200b", "")
book["url"] = book_url

After this, I saved the book’s first review page URL into the soup variable:

page = 1
result = requests.get(f"{book_url}/ertekelesek?page={str(page)}")
time.sleep(1)
src = result.content
soup = BeautifulSoup(src, "lxml")

As the next step, I checked with a try-except-else block if there was more than one review page for the book (this was necessary, because depending on the answer, a different code block had to run):

try:
    last_page = int(soup.find("a", class_="next_page").previous_sibling.previous_sibling.get_text())

If according to try there was no or at most one review page for a book, the except block ran:

except AttributeError:
    url = f"{book_url}/ertekelesek?page={str(page)}"
    reviews = soup.find_all("div", id=re.compile("^review_"))
    if len(reviews) != 0:
        review_pages.append(url)
    for review in reviews:
        star_value = star_values[review.find("span", class_="rater-on")["style"].split(" ", 1)[1].split("px;", 1)[0]]
        book_ratings.append(star_value)
        if review.find("div", class_="atom"):
            book_reviews.append(star_value)

If according to try there was more than one review page for a book, the else block ran:

else:
    while page != last_page + 1:
    url = f"{book_url}/ertekelesek?page={str(page)}"
    review_pages.append(url)
    result = requests.get(url)
    time.sleep(1)
    src = result.content
    soup = BeautifulSoup(src, "lxml")
    reviews = soup.find_all("div", id=re.compile("^review_"))
    for review in reviews:
        star_value = star_values[review.find("span", class_="rater-on")["style"].split(" ", 1)[1].split("px;", 1)[0]]
        book_ratings.append(star_value)
        if review.find("div", class_="atom"):
            book_reviews.append(star_value)
    page += 1

In either case, the result was the same: the review pages were added to the review_pages list (for safety reasons to not lose data), the ratings and the ratings with reviews were added to book_ratings and book_reviews respectively.

Then, as the final touch, book was finalized with book_ratings and book_reviews data, and appended to books, a list that contained all books after the for loop was done running:

book["ratings"] = book_ratings
book["reviews"] = book_reviews
books.append(book)

I can imagine this was a lot to take in, but bear with me, it gets better. 🙂

2.3 Creating dataframes with Pandas

By this time I had 875 book dictionaries in a list (books) with every book’s title, URL, ratings and ratings with reviews.

The next step was an easy one – I stored every rating and rating with review of each book in two separate lists:

all_reviews = []
all_ratings = []
for book in books:
    all_ratings.extend(book["ratings"])
    all_reviews.extend(book["reviews"])

After printing, all_reviews and all_ratings look like this:

all_reviews holds 56,782 ratings with reviews, while all_ratings contains 183,765 ratings.

I used Counter() to count the number of occurrences of the ratings in all_ratings and all_reviews, then converted the results to dictionaries with the dict() function:

c_all_ratings = Counter(all_ratings)
c_all_reviews = Counter(all_reviews)
all_ratings_dict = dict(c_all_ratings)
all_reviews_dict = dict(c_all_reviews)

The results look like this (the below code shows the result for all_ratings_dict):

{5.0: 76783, 4.5: 31517, 4.0: 35901, 3.5: 15286, 3.0: 11813, 0: 2237, 1.5: 945, 2.5: 3800, 2.0: 3261, 0.5: 983, 1.0: 1238}

This means that altogether there were, at the time of the project, 76,783 ratings with a value of 5.0, 31,517 ratings with a value of 4.5, and so on and so forth.

What’s more important is that based on all_ratings_dict and all_reviews_dict I could finally create the dataframes I needed for the later visualizations.

To create the initial dataframes I used the following code:

d = {"Rating" : [5.0, 4.5, 4.0, 3.5, 3.0, 2.5, 2.0, 1.5, 1.0, 0.5, 0.0], "# of ratings" : [76783, 31517, 35901, 15286, 11813, 3800, 3261, 945, 1238, 983, 2237], "# of ratings with reviews" : [23509, 9200, 9818, 4258, 3852, 1393, 1256, 389, 494, 440, 2173]}
df = pd.DataFrame(data=d)
df["# of ratings with reviews / # of ratings"] = df["# of ratings with reviews"] / df["# of ratings"]
df

And this is the dataframe I got:

The Rating column shows the possible values of the ratings in decreasing order (from 5.0 to 0.0).

The # of ratings column shows how many ratings there are for each rating (for example there are 2,237 ratings with the value of 0.0).

The # of ratings with reviews column shows how many ratings there are with reviews (for example there are 2,173 ratings where there’s also a review with the value of 0.0).

The # of ratings with reviews / # of ratings column shows for each rating what percentage of all ratings contains a review, too (for example around 30% of all ratings of 5.0 contained a review, which means that 70% of 5.0 ratings don’t contain a review).

You may already see the pattern here: as the value of Rating decreases (the ratings become more negative), people tend to write more reviews (as the increase of # of ratings with reviews / # of ratings indicates).

But let’s not get ahead of ourselves just yet. 😉

After a little tweaking I managed to improve on the previous dataframe by first creating two new variables (number_of_all_ratings which is 183,764 as we already know, number_of_all_ratings_with_reviews which is 56,782):

number_of_all_ratings = df["# of ratings"].sum()
number_of_all_ratings_with_reviews = df["# of ratings with reviews"].sum()

Then I did some new calculations with these variables:

df.insert(2, "% of all ratings", df["# of ratings"] / number_of_all_ratings * 100)
df.insert(4, "% of all ratings with reviews", df["# of ratings with reviews"] / number_of_all_ratings_with_reviews * 100)

Here’s the end result with two new columns (% of all ratings and % of all ratings with reviews):

The % of all ratings column shows for each Rating what percentage the given rating (# of ratings) takes up of all ratings (number_of_all_ratings). For example out of all ratings 3.0 accounts for 6.43%.

The % of all ratings with reviews shows for each Rating what percentage the given rating (# of ratings with reviews) takes up of all ratings with reviews (number_of_all_ratings_with_reviews). For example out of all ratings with reviews 4.0 accounts for 17.29%.

After I had created the above dataframes, I could finally start visualizing the data.

3. Visualizing the data with Matplotlib

First, I wanted to visualize # of ratings with reviews / # of ratings from my dataframe (remember, this column shows for each rating what percentage of all ratings contains a review).

This is the code I used for plotting the data:

plt.figure(figsize=(10, 6))
plt.title("Reviews / ratings by star rating")
plt.xticks(np.arange(0, 5.5, step=0.5))
plt.scatter(df["Rating"], df["# of ratings with reviews / # of ratings"])
plt.plot(df["Rating"], df["# of ratings with reviews / # of ratings"])
plt.xlabel("Rating")
plt.ylabel("Percentage")

And this is what the plot looks like:

The chart nicely shows which rating is associated with the most reviews (expressed in percentage). We can see that people are more inclined to write a comment if their experience with a book is negative (their rating is 2.5 or lower).

I’d like to mention that it needs further investigation why readers who gave a rating of 0 write reviews such a high percentage of the time (97%). During my project I’ve encountered both positive and negative reviews with a rating of 0, but I haven’t done extensive research into this, so this question remains to be answered.

Anyway, I made another chart that I find interesting:

This chart shows that the distribution of the reviews and the distribution of the ratings are not the same. If we compare % of all ratings with % of all ratings with reviews, we can suspect that worse ratings (typically from 3.0 to 0) tend to receive more reviews (compared to the number of all ratings).

Perhaps the dataframe version of this data is more illuminating:

This is the code I used to plot the above chart (in creating this chart, Python Tutorials’ article helped me a lot):

y = df["% of all ratings"]
y2 = df["% of all ratings with reviews"]
x = df["Rating"]
fig = plt.figure(figsize=(10, 6))
ax = plt.subplot(111)
ax.plot(x, y, label="% of all ratings")
ax.plot(x, y2, label="% of all ratings with reviews")
plt.title("% of all ratings vs % of all ratings with reviews by star rating")
plt.xticks(np.arange(0, 5.5, step=0.5))
plt.xlabel("Rating")
plt.ylabel("Percentage")
ax.legend()
plt.show()

4. Validating the results with random sampling

Naturally, the results I got from my analysis needed to be validated statistically. Tomi Mester suggested that I randomly select some books from all 875 books, carry out the same analysis I did for all books, and check if the results are more or less similar to my original results.

If I repeat this process 10 times, and the results I get are consistent, my original sample can be considered big enough to draw conclusions based on it.

That’s exactly what I did  – I created a randomize() function, where I randomly picked 400 books with random.sample(), and performed the same analysis I did for all 875 books.

Every randomize() function returned a dataframe, and added the sample’s value of # of ratings with reviews / # of ratings for each Rating to predefined lists. Each list was associated with a rating value (for example “3.0”), and held the related values received from the randomize() functions. After the 10 random samplings, I printed the results of each list to check if my original sample of 875 books was big enough.

Here’s what a dataframe returned by randomize() looked like:

And this is the code itself:

five = []
four_and_a_half = []
four = []
three_and_a_half = []
three = []
two_and_a_half = []
two = []
one_and_a_half = []
one = []
half = []
zero = []
def randomize():
    sample_books = random.sample(books, k=400)
    sample_ratings = []
    sample_reviews = []
    for book in sample_books:
        sample_ratings.extend(book["ratings"])
        sample_reviews.extend(book["reviews"])
    ratings_counted = dict(Counter(sample_ratings))
    reviews_counted = dict(Counter(sample_reviews))
    d = {"Rating" : [5.0, 4.5, 4.0, 3.5, 3.0, 2.5, 2.0, 1.5, 1.0, 0.5, 0.0], "# of ratings" : [ratings_counted[5.0], ratings_counted[4.5], ratings_counted[4.0], ratings_counted[3.5], ratings_counted[3.0], ratings_counted[2.5], ratings_counted[2.0], ratings_counted[1.5], ratings_counted[1.0], ratings_counted[0.5], ratings_counted[0]], "# of ratings with reviews" : [reviews_counted[5.0], reviews_counted[4.5], reviews_counted[4.0], reviews_counted[3.5], reviews_counted[3.0], reviews_counted[2.5], reviews_counted[2.0], reviews_counted[1.5], reviews_counted[1.0], reviews_counted[0.5], reviews_counted[0]]}
    df = pd.DataFrame(data=d)
    df["# of ratings with reviews / # of ratings"] = df["# of ratings with reviews"] / df["# of ratings"]
    v = df["# of ratings with reviews / # of ratings"]
    five.append(round(v[0], 4))
    four_and_a_half.append(round(v[1], 4))
    four.append(round(v[2], 4))
    three_and_a_half.append(round(v[3], 4))
    three.append(round(v[4], 4))
    two_and_a_half.append(round(v[5], 4))
    two.append(round(v[6], 4))
    one_and_a_half.append(round(v[7], 4))
    one.append(round(v[8], 4))
    half.append(round(v[9], 4))
    zero.append(round(v[10], 4))
    return df

After I ran randomize() 10 times, I printed the results:

Samples are shown for every rating value (for instance “Sample 5.0” means 10 samples for the rating of “5.0”).

The list values (for instance “0.3029”) are the ratio of # of ratings with reviews / # of ratings, created by randomize() 10 times.

The standard deviation shows that the distance between the values are very small, thus my sample of 875 books for the project can be considered big enough, and the conclusion that people are more likely to write a review if their rating of a book is rather negative can be accepted.

Conclusion

There’s nothing special left to say, so I’d like to repeat the main takeaway of my analysis:

• People are more inclined to write a review if their experience with a book is negative (their rating is 3.0 or lower).

This sounds nice, but we can never be 100% sure about any conclusion, so further analyses are always welcome. 🙂

---

I’d like to thank Tomi Mester for his help. He was kind enough to answer my questions regarding the project, and suggest to me what directions my project could possibly take next.

If you’re interested to learn more about data science, Tomi’s highly practical and beginner-friendly course should be the place where you start your data science journey. Good luck and have fun!

Cheers,
Tamas Ujhelyi