Taking a break from #pixelfed dev tonight to work on https://fedidb.org

The new design is much better, and includes more graphs and stats.

Also adding some logic to respect robots.txt rules, and an improved crawler info page to clarify the data we crawl!

It's important we respect the privacy of instances, ethical design is fundamental to all my projects. #fediDB