craig-o-mation at CalHacks 2.0

Oct 11, 2015

This past weekend Josh and I participated in CalHacks 2.0 at UC Berkeley. It was announced during the hackathon that it was the largest hackthon to date with over 2000 people signing in. craig-o-mation is the project that we created. It won the prize for Best Use of Capital One API. The whole premise was based on the convenience of one hour delivery we have discovered since coming to intern in San Francisco. We wanted something similar to caviar, UberEATS, and etc, except for craigslist.

I’m gonna briefly skim over the technologies and then talk about why it’s the hackiest hack I’ve yet made at a hackathon. The backend is written in golang and hosted on Google App Engine. We also use Google for Cloud Datastore and email sending. The frontend is responsive HTML5/CSS3 with some Bootstrap 3 and jQuery. We used the Capital One API for payments, Postmates API for delivery, and Google Maps API for rendering maps and location autocomplete.

Now the hacky part; scraping craigslist. It doesn’t work on App Engine. When you make a HTTP request through App Engine, it adds the following line to the header.

User-Agent: AppEngine-Google

This is unfortunate because craigslist blocks that specific user agent. There’s no way around it. We tried manually setting an user agent, but App Engine overwrote the header. So to get around this, we wrote a simple python server that ran on Josh’s computer back in Canada. Below is a slightly modified version of the code.

import SimpleHTTPServer
import SocketServer
import urllib2

class RequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    def do_GET(self):
        craigslist_url = self.path[1:]
        contents = urllib2.urlopen(craigslist_url).read()

server = SocketServer.TCPServer(('', 80), RequestHandler)

This was the backbone of our whole application, and it’s interesting that it ran off a 10 year old computer with a 2 GHz AMD Athlon, and 1 GB of ram.

Since the hackathon we have shutoff the app because we do not wish to violate craigslist’s policies. If you would like to discuss the project in detail feel free to reach out.