graphics by Hive account@etherdesign

Why steemtools

I have been pretty busy in the past month, working on several Steem related projects:

• SteemQ, a video platform on top of STEEM
• steem.li, real-time voting and curation tools
• IM market-maker, running 24/7
• a set of curation bots
• weekly reports on various topics

I had spent a fair amount of time writing code and looking for solutions that would enable me to in part do these things.

The need for the library emerged, as I realized I had scattered code over all of my projects, in many cases, implementing the same functionalities.

I have also been contacted by half a dozen people on https://steemit.chat, asking for advice on certain problems - problems I had already solved myself.

I think that different people solving the same problems makes no sense - when all we really want is to achieve a higher level goal.

The purpose of this library is to enable others to do what I do, and avoid the colossal waste of time.

Acknowledgments

steemtools is built on top of piston and python-steemlib, the awesome Python libraries made by Hive account@xeroc.

Hive account@jesta has been very helpful in #piston channel on https://steemit.chat, and steem.ws, a fast and reliable node cluster has saved me a lot of time, and allowed me to keep working while my local node was re-compiling/replaying.

And lastly, Hive account@burnin, a guy that reverse-engineered the voting mechanisms, and saved me so much time with the Converter module.

Thank you guys, I could not have done this without you.

Installation

I highly recommend starting off with Anaconda distribution of Python 3.5.

After that, we only need to run one command:

pip install steemtools

Modules

steemtools is currently comprised of 4 modules:

• blockchain: all utilities for blockchain traversal/parsing
• base: contains our Post, Account and Converter classes
• helpers: static helper functions
• node: a convenient way to connect to local RPC, with automatic failover to steem.ws

Blockchain

---

Replaying History:

from steemtools.blockchain import Blockchain

for event in Blockchain().replay():
    print("Event: %s" % event['op_type'])
    print("Time: %s" % event['timestamp'])
    print("Body: %s\n" % event['op'])

This function allows us to go back in time, and replay the entire STEEM blockchain from start to finish. Once it reaches the present moment, it will keep going, printing events with every new block (every 3 seconds or so).

The output will look a little bit like this:

Operation Types:
Perhaps we aren't interested in all the events, but just specific ones.
We can ask replay to only give us votes:

for event in Blockchain().replay(filter_by="vote")

Or a set of events, such as votes and comments:

for event in Blockchain().replay(filter_by=["vote", "comment"])

For the reference, the full list of currently available operations is:

blockchain_operations = [
    'vote', 'comment', 'delete_comment', 'account_create', 'account_update',
    'limit_order_create', 'limit_order_cancel',
    'transfer',  'transfer_to_vesting', 'withdraw_vesting', 'convert', 'set_withdraw_vesting_route',
    'pow', 'pow2', 'feed_publish',  'witness_update',
    'account_witness_vote', 'account_witness_proxy',
    'recover_account', 'request_account_recovery', 'change_recovery_account',
    'custom', 'custom_json'
]

Time Constraints:
Parsing the ENTIRE blockchain is often unnecessary. We can specify a desired range:

start_block = 3598980
end_block = 4260042
for event in Blockchain().replay(start_block, end_block=end_block, filter_by=["vote", "comment"]):
    pprint(event)

Working with block numbers is painful, and this is why blockchain module comes with 2 helpers:

get_current_block()
get_block_from_time("2016-09-01T00:00:00")

Putting it all together...

b = Blockchain()
history = b.replay(
    start_block=b.get_block_from_time("2016-09-01T00:00:00"),
    end_block=b.get_current_block(),
    filter_by=['transfer']
)
for event in history:
    payment = event['op']
    print("@%s sent %s to @%s" % (payment['from'], payment['amount'], payment['to']))

The above code will fetch all the transfers from September 9th going forward, up until present.

...
@victoriart sent 1.000 SBD to @null
@dude sent 5.095 STEEM to @bittrex
@devil sent 5.107 STEEM to @poloniex
@pinoytravel sent 0.010 SBD to @null
@aladdin sent 5.013 STEEM to @poloniex
@mrwang sent 31.211 STEEM to @blocktrades
@kodi sent 0.030 SBD to @steembingo
...

Account History

---

Account module allows us to lookup virtual operations, as well as some of the common operations for an individual account.
Usually it is more efficient to query the account history over parsing the blockchain block-by-block.

Looking up the account history:
Accounts module gives us 2 generators for the task, history, which gives us account history from inception forward and history2 which gives us account history newest to oldest. The interface should look familiar, as it is similar to the replay from the blockchain module.

from steemtools.base import Account
from steemtools.helpers import parse_payout

for event in Account("furion").history(filter_by=["transfer"]):
    transfer = event['op']
    if transfer['to'] == "null":
        print("$%.1f :: %s" % (parse_payout(transfer['amount']), transfer['memo']))

The code above will pull the transfer history for my account, find promoted posts by looking for transfers to Hive account@null, and finally print the $ amount spent as well as the permlink to the post.

$11.1 :: @furion/steem-analysis-ownership-distribution-and-the-whale-selling-pressure
$11.0 :: @furion/using-machine-learning-to-fight-plagiarism
$41.0 :: @furion/a-quick-look-at-null-and-the-profitability-of-promoted-posts

Virtual Operations:
As mentioned above, there are several operations that are virtual, and we cannot obtain these by parsing the blockchain itself. We can however lookup the history of virtual operations on a specific account with the history method.

Account("furion").history(filter_by=["curate_reward", "fill_order"])

Currently, the following types are available for the lookup (both virtual and not):

account_operations = {
    'account_create',
    'account_update',
    'account_witness_vote',
    'comment',
    'comment_reward',
    'convert',
    'curate_reward',
    'fill_order',
    'fill_vesting_withdraw',
    'fill_convert_request',
    'interest',
    'limit_order_cancel',
    'limit_order_create',
    'transfer',
    'transfer_to_vesting',
    'vote',
    'witness_update',
    'account_witness_proxy',
    'feed_publish',
    'pow', 'pow2',
    'withdraw_vesting',
}

Account Methods

---

Account has several helper methods. Here are a few:

from steemtools.base import Account

account = Account("furion")

account.get_sp()
#> 6211.590278675119

account.reputation()
#> 62.76

account.voting_power()
#> 80.75

account.avg_payout_per_post()
#> 142.7166

We can also easily obtain the latest blog posts. Lets get the titles of most recent 3:

blog = account.get_blog()
for post in blog[:3]:
    print(post['title'])

# outputs:
# A quick look at @null, and the profitability of Promoted Posts
# A quick look at the top curators and their rewards
# Homepage Payout Distribution, Power Law and Project Curie

How about a list of followers:

followers = account.get_followers()
#> ['anns', 'benjy33', 'negoshi', ...]

Lets obtain the curation stats:

account.curation_stats()
# outputs
# {'24hr': 9.627790750805277, '7d': 57.82547153222017, 'avg': 8.260781647460025}

Or get a basket of features:

account.get_features(max_posts, payout_requirement)

Outputs:

{'author': {'followers': 281,
  'post_count': 10,
  'ppp': 142,
  'rep': 62.76,
  'sp': 6211,
  'ttw': 281.0,
  'winners': 2},
 'name': 'furion',
 'settings': {'max_posts': 10, 'payout_requirement': 300}}

Check out the base.py module for all the methods.

Posts

---

Post is a superset of piston.steem.Post. This means that, it behaves the same way, and has all the niceties and helpers that piston's Post object has.

We can initialize it in any of these ways:
a) using the identifier string

Post("@furion/homepage-payout-distribution-power-law-and-project-curie")

b) using a piston.steem.Post object

last_post = Account("furion").get_blog()[0]
Post(last_post)

c) using a author+permlink containing dictionary, such as vote

for vote in s.rpc.stream("vote"):
        print(vote)
        print(Post(vote))
        
# {'voter': 'ats-david', 'author': 'whatsup', 'weight': 10000, 'permlink': 're-steve-walschot-investigating-the-wale-scam-assumptions-above-knowledge-20160912t020220232z'}
# <Steem.Post-@whatsup/re-steve-walschot-investigating-the-wale-scam-assumptions-above-knowledge-20160912t020220232z>

---

Now that our Post is initialized, we have access to extra methods. Here are a few:

p = Post("@furion/homepage-payout-distribution-power-law-and-project-curie")

p.payout()
#> 456.003

# if the post was 10 minutes old, this would output 33.33
p.calc_reward_pct()
#> 100

p.is_comment()
#> False

# not a spam tagged post
p.contains_tags(filter_by=["spam"])
#> False

p.get_votes()
#> [list of active votes]

Check out the base.py module for all the methods.

Converter

---

Converter is a class that tries to convert/calculate different units. I won't go into details in this post, aside from listing the available methods:

from steemtools.base import Converter

c = Converter()

c.sbd_median_price()
c.steem_per_mvests()

c.vests_to_sp(vests)
c.sp_to_vests(sp)
c.sp_to_rshares(sp)
c.steem_to_sbd(steem)
c.sbd_to_steem(sbd)
c.sbd_to_shares(sbd)
c.rshares_to_weight(rshares)

Helpers

---

Helpers is a set of static methods that are generally useful when dealing with Steem objects.

from steemtools import helpers

# get just the digit part of any asset (STEEM, VESTS, SBD)
helpers.parse_payout("12.3456 SBD") 
#> 12.3456

# break down our asset
helpers.read_asset("12.3456 SBD")
#> {'symbol': 'SBD', 'value': 12.3456}

# get time difference in seconds between lets say 2 posts
helpers.time_diff(time1, time2)
#> 1337

# determine if our object is a post or a comment
helpers.is_comment(object)
#> False

# time elapsed in seconds 
helpers.time_elapsed(post)
#> 9001

Going forward

I wanted to be as minimal as possible with the first version, and only include the most essential components.

There are many new features, and areas in which steemtools can be expanded upon. For instance, an exchange module, that would allow for quick bootstrapping of a market-maker or arbitrage bot. These features may be added later on.

For now, I am looking for feedback and new ideas on how to improve whats already there, and make the library stable, so people can depend on it.

Github: Netherdrake/steemtools

---

Don't miss out on the next post. Follow me.

---