The backend has new API endpoints for retrieving the top projects on the Utopian. This can be used for statistical or analytical purposes or for a front end to display the top projects. Other bug fixes were made as well :)
Here's what can be be done out of the box
Endpoint to retrieve the top 3 projects based on number of contributions in the last week
/api/posts/top?limit=3&start_date=2017-12-1&end_date=2017-12-8
Endpoint to retrieve the top new 4 projects of the last week based on the number of contributions/rewards
/api/posts/top?limit=4&start_date=2017-12-1&end_date=2017-12-8&include_rewards=true&only_new=true
Endpoint to retrieve the top 10 contributions of the last week, based on number of rewards
/api/posts/top?limit=10&start_date=2017-12-1&end_date=2017-12-8&include_rewards=true&sort_by=rewards
Query parameters
limitHow many top projects to receive from the APIstart_dateAnalyze top projects starting from this date.end_dateAnalyze top projects ending to this day, used in combination with start_date to form a query for retrieving top projects in a given week.include_rewardsWhen set totruethe Steem API will be queried to aggregate the rewards across the top projects.sort_byCan be set tocontributionsorrewards. When set to rewards theninclude_rewardsmust be set to true for this to work. The Steem API will be queried for many more posts to get an accurate projection of top projects based only on rewards.
Technical details
Querying the local database is easy (as seen in the first endpoint sample). The server can aggregate the data quickly in less than a second. When rewards are involved, things are much trickier. We must now query the DB on many posts. The DB backend is MongoDB and data is queried through the aggregation pipeline.
First I tried response streaming but this wasn't playing well at all, it also had the downside of needing to query the blockchain again. This was then replaced with an async model. The API server will return a status and any data it completes.
There is a caching system in place for storing these expensive results. As a result data may become stale after a certain amount of time. So the caching system is capable of expiring the stale data after a certain amount of time, allowing for fresh data to be retrieved. The caching system is smart in regards to the parameters passed in and knows whether or not it is cached based on the query parameters parameter. A weak hash (md5) creates a unique identifier of the cache data in the backend. Caching happens entirely transparently to the user. Checks are added if the data has already been paid out or not, to avoid querying the Steem API if necessary.
A rate limiting system is in place as well to prevent abuse of this endpoint, but only applies if data needs to be queried from the blockchain. The limit is tied to a Steem account to prevent circumventing the system. The date range is limited to 8 days maximum and a maximum limit of 100 projects.
Refactoring
A universal post updater is now in use, allowing data queried from the Steem API to update the stale data in the DB, greatly simplifying the process and reducing coupling of the previous review function.
Bug fixes
- No longer does the front end lock you out when you revoke GitHub access to Utopian
- The auth system now uses refresh tokens, allowing the backend to nifty new things
- Querying the DB for flagged posts no longer returns an empty set
Commits
All commits can be found at https://github.com/utopian-io/api.utopian.io/commits/master
- https://github.com/utopian-io/api.utopian.io/commit/ecb9226d268394f7599dcc9f2be816ef1d74d74d
- https://github.com/utopian-io/api.utopian.io/commit/957c676c44110771b954f3a2638d9d19864b89bc
- https://github.com/utopian-io/api.utopian.io/commit/3226516efdbd1cd1922c0abce564d3efbdba84e4
- https://github.com/utopian-io/api.utopian.io/commit/4e87df768648aabdab855e7b0d78bc65bd7a97d7
- https://github.com/utopian-io/api.utopian.io/commit/4e87df768648aabdab855e7b0d78bc65bd7a97d7
- https://github.com/utopian-io/api.utopian.io/commit/0d5579ac637ef47490e093c7e0d181e6fa7f56dd
- https://github.com/utopian-io/api.utopian.io/commit/acbcb7faa9fca5053d3572f4cf14129cef31cbed
- https://github.com/utopian-io/api.utopian.io/commit/8c32f94802f7145c544842ca64f4cab8eb37e553
- https://github.com/utopian-io/api.utopian.io/commit/c600df2930fe25d399722ca7109d62a6467c5ab1
- https://github.com/utopian-io/api.utopian.io/commit/c215b6518f5132540053e0ea3284b1ffcd316c9e
- https://github.com/utopian-io/api.utopian.io/commit/92678f38f05d85525dcccc3ee95163f724372a31
- https://github.com/utopian-io/api.utopian.io/commit/8abf57b4ed86ea7c4144ad90ec5f56c64193f7b0
- https://github.com/utopian-io/api.utopian.io/commit/b879ec9e4dc637bb1592611cc697fb99f88b9115
Posted on Utopian.io - Rewarding Open Source Contributors