@dansup you can prevent mass crawling without affecting regular users by setting reasonable rate limits. Twitter's limits are not reasonable and that's the entire problem.

For example, 1 post per second with a burst limit of whatever you need to load the front page is more than enough for humans but painfully slow for crawlers.