In this post, I will walk through the steps I took to grab data from an apartment leasing website, monitor changes, and send me email alerts.
Grabbing the Data
In this example we will take a dive into a Leasing Website in Glendale
A click on "Apartments + Pricing" will reveal a page that looks like this:
The web page is retrieving the apartment data from a remote server. Let's see if we can find it.
Using the Google Chrome web browser, select "View -> Developer -> Developer Tools", and refresh the web page.
In our Developer Console, we see dozens of javascript executions and connections to remote urls. One of the urls is called "ApartmentSearch". It contains all the beautiful apartment data we want.
Copy/paste the url into a new browser tab, and you'll find a handy, if messy, wall of JSON text.
Copy all the text. Let's visit jsonlint.com and pretty it up:
Fantastic! Scrolling through this data, we can find every apartment and its current lease price.
Monitoring Data with a Raspberry Pi
The website gives us a JSON object, which is very easy to read and manipulate in JavaScript. I wrote a server-side JavaScript script, "apartment.js", that parses the JSON data, and writes to a CSV file. Later, we use the CSV file to make pretty graphs in Microsoft Excel.
Here is a screenshot of the code parsing the JSON object:
When the data is saved to a CSV file, it looks like this:
I used a Raspberry Pi computer to run my "apartment.js" script every 6 hours. Each time the script ran, it would parse the JSON object, and add the current apartment prices to the CSV file. Server-side Javascript is possible thanks to the NodeJS project. Raspberry Pi is an inexpensive computer that is popular for open-source projects. What I like most about a Raspberry Pi is that I can leave it on all the time without worrying about an expensive energy bill.
Running a script every 6 hours with crontab
Here I log into my Raspberry Pi and execute "crontab -e". This can be done on any Unix-like machine such as Mac OS or Linux
Inside cron, the bottom line runs the "apartment.js" script every 6 hours.
Pretty Graphs
What did we learn?
Between September 13th and October 7th, apartment 106 bounced around in price. Eventually, it slid to a sweet $2150 before being sold. Whoever claimed it got a great deal.
Between September 13th and October 16th, apartment 109 slid to an even cheaper price at $2070. However, after 106 sold, it quickly shot up in price and stubbornly stayed there. 109 was sold on October 16th for $2200. Whoever bought it missed the optimal deal.
Apartment 117 appeared on November 4th, and quickly shot up in price. It's been more than 20 days and no one has bought it...
Later, I will share more graphs if I discover any interesting trends.
Email Alerts
As the Raspberry Pi collected data, I had it send me emails when something interesting happened:
- When any apartment reached its cheapest price ever
- When any apartment became vacant or sold
This way, I didn't need to refresh the leasing website like a crazy person.
Side note:
I implemented this algorithm for myself, so I took the lazy route and stored data in a text file for a single user. For a more scalable solution, I would recommend a MongoDB database. I chose to skip installing MongoDB at the time because the official MongoDB binaries did not support my Raspberry Pi.
Cheapest Price Ever
Prices dropped in September and October as vacancies opened up:
Eventually, a one-bedroom apartment bottomed out at $2070
Vacancies were sold and prices have climbed ever since.
Apartment Vacancy Changes
Starting on October 16th, I programmed the Pi to email me about vacancy changes
My email archives tell me a lot about how often vacancies were added or sold:
Notice that vacancies changed every 5-10 days until the middle of November. Since then, vacancies haven't changed for more than 20 days.
New apartments usually would popup with with much-higher-than-average prices. 😅
Source Code
I uploaded the source code onto GitHub!
Conclusion
My Apartment Hunting algorithm confirmed the obvious. When lots of apartments are vacant for a long time, their prices slowly drop until they are sold. After one sells, the remaining apartments spike in price for a while.The email alerts were certainly handy. I didn't need to worry about refreshing the web page every day for the best deal. With my Raspberry Pi collecting data directly, the apartment company could not track any web browsing data about me. That in itself was its own reward.
