Relaunching Twilert as a Subscription Service close
Twilert is a Twitter monitoring tool that helps people keep track of mentions and keywords. Users receive email alerts containing Twitter search results to help them participate more actively in conversations about their brand as well as to build a strong online reputation.
Twilert began as a client project at the tail end of 2008 which we took over as our own in 2010 and have built up since. Our initial intention was to make it a free service that was funded via advertising (Twilerts get a very high open rate) but as we scaled to having 70 000 users sending millions of emails a month our income from advertising fell short of our costs.
Coupled with the fact that Twitter deprecated their v1 API, effectively cutting off our access, Twilert had to face changes. In order to evaluate whether we should launch Twilert for a third time we surveyed our users last year and received a fantastic positive response. People really relied on this product to help them manage the tidal wave of info that Twitter churns out.
New features and business model
To justify a complex rebuild, ongoing customer support and high monthly running costs Twilert had to move to a paid subscription model. However in order to provide true value to our customers we addressed some of most frequently asked for features.
Luke, our technical director, devised a way to index all of the tweets we harvested into an archive of each search that our users could peruse in future. Twitter itself only retains a couple of weeks of data in it's search so an archive potentially spanning years seemed valuable. Users had previously been doing this by filing away emails in Gmail and using the search function to filter.
Additionally we increased the number of frequency options that each Twilert had to near realtime and made adding advanced search terms much easier. We also gave the user the option of how they wanted the results delivered – either by email, RSS, Atom or JSON.
There is still a free plan, but it's more limited than before. There are two core plans will cover the needs of most users and we are working on an agency model for marketing/pr/social media companies or departments that need many users under one account.
The techie bit
We wanted to build Twilert to be both cost effective and scalable. The big challenge was actually creating indexes of all the Twilerts for historical browsing and searching of the tweets. We ended up with a pretty distributed tech stack using the best services we could find for each task.
- The core web app is built in Python + Flask and is running on the Heroku platform with a Postgres database. To scale that we just increase dynos.
- Twitter searches are scheduled and run using PiCloud. This scales automatically to adapt to load :)
- Redis is used to keep state between searches so that we don't have to keep hitting the database.
- Emails are rendered on PiCloud and dispatched via the Amazon SES (simple email service) API.
- Tweets are then indexed into Elastic Search so they can be queried later. This has been a bit of a challenge. We started off with one instance but soon realised we needed a cluster and have spent the past few days tuning it. We are using SSDs rather than traditional hard disks to speed up indexing. As we scale we will add more nodes to the cluster.
The new platform soft launched this week and we are giving large discounts to existing users that take up subscriptions. We will drip feed access to all of our user base to monitor how we scale up and give us a chance to reply properly to users that get in touch. So far quite a few have upgraded to paid so things have started well! We will write another update in a few months to describe how the full process has gone.