This is the first post in a short series of posts about a python library that I'm working on. Right now the Hive Archeology Bot uses the lighthive python library. The lighthive library is a pretty good python library for HIVE, especialy for simple scripts, and its significantly better and more pythonic than the beem python library, but as I've noticed for the Hive Archeology Bot, its less of a good fit for stand alone bots or DApp back-ends.
Its been quit a while since the last version of asyncsteem, an asynchonic python library for HIVE (then STEEM). So long ago that asyncsteem was a python 2 library running on top of the Twisted async framework, because at that time Twisted didn't fully run on Pyhton 3 yet, and async support in Python3 was still inferior to what was available in Twisted. I really didn't want to build a new async library for HIVE again, but with the Hive Archeology Bot, and my desire to integrate coinzdense signing into some python HIVE library for proof of concept purposes, I'm now ariving at the point where I realize there is no way around it. I need to start work on a new async HIVE library for Python.
Things are still early, but I want to take you all with me on my journey as I continue building.
API nodes and APIs
According to the documentation there are 13 public API nodes for HIVE, but when running some code, some of these 13 don't seem to be operational. and others seem to be a bit flaky.
Next to this, not all of the APIs are run by all of the public API nodes. This can be a problem if your bot or DApp backend needs a specific API.
We are going to make out Python library take care of all of these details. It would be interesting to do all of this completely dynamically, and we may do so in futrure versions, but right now, we are going to start off with a little script that will generate a static configuration file for our library. It's so we have a start. Ideally, if the failing nodes would get fixed or if nodes added or removed certain API's from their supported subset, it should dynamicly get updated, but we are starting small and statically.
So lets look at a little class:
class PubNodeClient:
def __init__(self, clientset, nodenurl):
self._clientset = clientset
self._abandon = False
self._lastblock = None
self.nodenurl = nodenurl
headers = {"user-agent": "aiohivebot-build-api-json/0.1"}
self._client = httpx.AsyncClient(base_url="https://" + nodenurl, headers = headers)
self.api_info = {}
async def _initialize_api(self):
...
async def run(self):
await self._initialize_api()
A core little async class for handling a single API node. Let's fill out the _initialize api method:
async def _initialize_api(self):
global COUNT
try:
r2 = None
method = None
cont = True
trycount = 0
while cont:
r = await self._client.post("/", content='{
"jsonrpc":"2.0",
"method":"jsonrpc.get_methods",
"params":{},
"id":1}')
if r.status_code not in [408,420,429,500,502,503,504,529,598,599]:
rjson = r.json()
if "result" in rjson or "jsonrpc" in rjson and \
"error" in rjson and \
rjson["error"]["code"] not in [-32603]:
cont = False
elif "jsonrpc" in rjson and \
"error" in rjson and \
rjson["error"]["code"] not in [-32603]:
trycount += 1
if trycount > 5:
print(self.nodenurl,
"JSONRPC ERROR, code",
rjson["error"]["code"],
"too many tries")
return False
print(self.nodenurl,
"JSONRPC ERROR, code",
rjson["error"]["code"],
"retrying in 1 minute")
await asyncio.sleep(60)
else:
print(self.nodenurl,
"JSONRPC ERROR,
invalid jsonrpc response, abandoning.")
return False
else:
trycount += 1
if trycount > 5:
print(self.nodenurl,
"ERROR, status",
r.status_code,
"too many tries")
return False
print(self.nodenurl,
"ERROR, status",
r.status_code,
" retrying in 1 minute")
await asyncio.sleep(60)
else:
print(self.nodenurl,
"JSONRPC ERROR, invalid jsonrpc response, abandoning.")
return False
else:
trycount += 1
if trycount > 5:
print(self.nodenurl,
"ERROR, status",
r.status_code,
"too many tries")
return False
print(self.nodenurl,
"ERROR, status",
r.status_code,
" retrying in 1 minute")
await asyncio.sleep(60)
if "result" in r.json():
for method in r.json()['result']:
cont = True
while cont:
r2 = await self._client.post("/",
content='{
"jsonrpc":"2.0",
"method":"jsonrpc.get_signature",
"params":{
"method": "' + method + '"
},
"id":1}')
if r2.status_code not in [408,420,429,500,502,503,504,529,598,599]:
r2json = r2.json()
if "result" in r2json:
cont = False
self.api_info[method] = r2.json()['result']['args']
await asyncio.sleep(0.2)
elif "jsonrpc" in r2json and \
"error" in r2json and \
r2json["error"]["code"] not in [-32603]:
print(self.nodenurl,
"JSONRPC ERROR for",
method,
"code",
r2json["error"]["code"],
"retrying in 1 second")
await asyncio.sleep(1)
else:
self.api_info = {}
print(self.nodenurl,
"JSONRPC ERROR for",
method,
"invalid JSONRPC response, abandoning")
return False
else:
print(self.nodenurl,
"ERROR for",
method,
"status",
r2.status_code,
"retrying in 1 second" )
await asyncio.sleep(1)
COUNT += 1
print(self.nodenurl,"initialized", COUNT)
return True
except json.decoder.JSONDecodeError as exp:
print("### JSON ERROR")
print(self.nodenurl,"abandon")
print(r)
print(method)
print(r2)
if r2 is not None:
print(r2.status_code)
return False
except KeyError as exp:
print("### KEY ERROR")
print(self.nodenurl,"abandon")
print(r)
print(r.json())
print(method)
print(r2)
if r2 is not None:
print(r2.status_code)
print(r2.json())
return False
It's not the cleanest code, bit as it won't be part of our actual library, its not wortht the effort to clen it up.
What this code tries to do is call two JSON RPC methods on a giben node:
Basicly it first calls jsonrpc.getmethod and with the results it calls jsonrpc.get_signature for each of the methods returned by the first. Most of the code above is error handling because the flaky and broken public API nodes that are active.
That is for one node, let's add some code to get this info from all of the known nodes:
class PubNodeClientSet:
def __init__(self):
self._clients = []
self._abandon = False
for public_api_node in [
"api.hive.blog",
"api.deathwing.me",
"hive-api.arcange.eu",
"hived.emre.sh",
"api.openhive.network",
"rpc.ausbit.dev",
"rpc.mahdiyari.info",
"hive-api.3speak.tv",
"anyx.io",
"techcoderx.com",
"api.hive.blue",
"hived.privex.io",
"hive.roelandp.nl"]:
self._clients.append(PubNodeClient(self, public_api_node))
async def run(self, loop):
tasks = []
for nodeclient in self._clients:
tasks.append(loop.create_task(nodeclient.run()))
await asyncio.gather(*tasks)
result = {}
for nodeclient in self._clients:
node_url = nodeclient.nodenurl
apitree = nodeclient.api_info
for key, values in apitree.items():
namespace, method = key.split(".")
if namespace not in result:
result[namespace] = {}
result[namespace]["methods"] = {}
result[namespace]["nodes"] = []
if node_url not in result[namespace]["nodes"]:
result[namespace]["nodes"].append(node_url)
if method not in result[namespace]["methods"]:
result[namespace]["methods"][method] = values
with open("aiohivebot/api.json", "w") as outfil:
json.dump(result, outfil, indent=2)
pncset = PubNodeClientSet()
loop = asyncio.get_event_loop()
loop.run_until_complete(pncset.run(loop))
print("DONE")
Basicly we run the per node code with 13 instances in paralel, and when all 13 instances are done, we combine the result in one json file containing info on that node supports what API namespace and on what parameters each method of the HIVE API takes.
When we buikt our library we shall use the first to make sure we use an API on a node known to support that API.
Let's run the script as a whole and see what happens:
hived.privex.io ERROR, status 503 retrying in 1 minute
api.hive.blue JSONRPC ERROR, invalid jsonrpc response, abandoning.
hive-api.arcange.eu JSONRPC ERROR for account_by_key_api.get_key_references code 1000 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.broadcast_transaction_synchronous status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.find_proposals status 500 retrying in 1 second
hive-api.arcange.eu JSONRPC ERROR for condenser_api.broadcast_transaction code 1000 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_account_reputations status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_active_votes status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_block status 500 retrying in 1 second
hive-api.arcange.eu JSONRPC ERROR for condenser_api.get_account_votes code 1000 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_blog_authors status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_blog_authors status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_current_median_history_price status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_active status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_created status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_hot status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_hot status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_expiring_vesting_delegations status 500 retrying in 1 second
api.openhive.network JSONRPC ERROR for wallet_bridge_api.get_collateralized_conversion_requests invalid JSONRPC response, abandoning
rpc.ausbit.dev initialized 1
hive.roelandp.nl ERROR for condenser_api.get_feed status 500 retrying in 1 second
api.deathwing.me initialized 2
hive.roelandp.nl ERROR for condenser_api.get_feed_history status 500 retrying in 1 second
rpc.mahdiyari.info initialized 3
hived.emre.sh initialized 4
hive.roelandp.nl ERROR for condenser_api.get_followers status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_followers status 500 retrying in 1 second
hive-api.3speak.tv initialized 5
hive.roelandp.nl ERROR for condenser_api.get_market_history status 500 retrying in 1 second
hived.privex.io ERROR, status 503 retrying in 1 minute
anyx.io initialized 6
hive.roelandp.nl ERROR for condenser_api.get_reblogged_by status 500 retrying in 1 second
hive-api.arcange.eu initialized 7
hive.roelandp.nl ERROR for condenser_api.get_reblogged_by status 500 retrying in 1 second
api.hive.blog initialized 8
hive.roelandp.nl ERROR for condenser_api.get_state status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_transaction_hex status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witness_schedule status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witness_schedule status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses_by_vote status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses_by_vote status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses_by_vote status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.lookup_witness_accounts status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_change_recovery_account_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_hbd_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_savings_withdrawals status 500 retrying in 1 second
techcoderx.com initialized 9
hive.roelandp.nl ERROR for database_api.find_vesting_delegation_expirations status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_vesting_delegation_expirations status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_votes status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_witnesses status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.get_comment_pending_payouts status 500 retrying in 1 second
hived.privex.io ERROR, status 503 retrying in 1 minute
hive.roelandp.nl ERROR for database_api.get_required_signatures status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.get_reward_funds status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_accounts status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_change_recovery_account_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_change_recovery_account_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_proposal_votes status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_proposals status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_proposals status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.verify_signatures status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.get_resource_pool status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.get_resource_pool status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.get_resource_pool status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.list_rc_direct_delegations status 500 retrying in 1 second
hive.roelandp.nl ERROR for reputation_api.get_account_reputations status 500 retrying in 1 second
hive.roelandp.nl ERROR for reputation_api.get_account_reputations status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.find_rc_accounts code -32003 retrying in 1 second
hived.privex.io ERROR, status 503 retrying in 1 minute
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.find_rc_accounts code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.find_rc_accounts status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_account code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_account status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_account code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_account code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_chain_properties code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_chain_properties status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_collateralized_conversion_requests code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_collateralized_conversion_requests code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_conversion_requests code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_open_orders status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_open_orders status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_open_orders status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_version status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_withdraw_routes status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_witness code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_witness_schedule code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_my_accounts status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_my_accounts status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_proposals code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_proposals code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_proposals code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_proposals status 500 retrying in 1 second
hived.privex.io ERROR, status 503 retrying in 1 minute
hive.roelandp.nl ERROR for wallet_bridge_api.list_rc_direct_delegations status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_witnesses status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_witnesses code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_witnesses status 500 retrying in 1 second
hive.roelandp.nl initialized 10
hived.privex.io ERROR, status 503 too many tries
DONE
So what do we see?
One nodes just failed. The node api.hive.blue run by 
One other node, hived.privex.io run by 
Three other nodes showed more or less flaky bahaviour during the run. The node hive-api.arcange.eu run by 
A second node, hive.roelandp.nl seemed even more flaky as it gave a HTTP 500 error a total of 98 times, but in the end it completed the run. Thirdly api.openhive.network showed some flaky responses but also got through at the end.
The other eithr nodes, rpc.ausbit.dev run by 
But let's look a little bit closer looking at part of the produced JSON:
{
"account_by_key_api": {
"methods": {
"get_key_references": {
"keys": []
}
},
"nodes": [
"api.hive.blog",
"api.deathwing.me",
"hive-api.arcange.eu",
"hived.emre.sh",
"api.openhive.network",
"rpc.ausbit.dev",
"rpc.mahdiyari.info",
"hive-api.3speak.tv",
"anyx.io",
"techcoderx.com",
"hive.roelandp.nl"
]
},
"account_history_api": {
"methods": {
"enum_virtual_ops": {
"block_range_begin": 1,
"block_range_end": 2
},
"get_account_history": {
"account": "",
"start": "18446744073709551615",
"limit": 1000
},
"get_ops_in_block": {
"block_num": 0,
"only_virtual": false
},
"get_transaction": {
"id": ""
}
},
"nodes": [
"api.hive.blog",
"hived.emre.sh",
"api.openhive.network",
"anyx.io",
"hive.roelandp.nl"
]
},
..
}
When we look at the nodes sections, we see that not all nodes support all the API namespaces.
| api | number of nodes |
|---|---|
| account_by_key_api | 11/11 |
| account_history_api | 5/11 |
| block_api | 11/11 |
| condenser_api | 11/11 |
| database_api | 11/11 |
| market_history_api | 11/11 |
| network_broadcast_api | 11/11 |
| rc_api | 11/11 |
| reputation_api | 10/11 |
| transaction_status_api | 9/11 |
| wallet_bridge_api | 8/11 |
Lets look a bit closer at the API's not supported by one or more nodes.
| account_history | reputation | transaction_status | wallet_bridge | |
|---|---|---|---|---|
| api.hive.blog | YES | YES | YES | YES |
| api.deathwing.me | NO | YES | YES | YES |
| hive-api.arcange.eu | NO | YES | YES | NO |
| hived.emre.sh | YES | YES | YES | YES |
| api.openhive.network | YES | YES | YES | YES |
| rpc.ausbit.dev | NO | YES | YES | YES |
| rpc.mahdiyari.info | NO | NO | YES | YES |
| hive-api.3speak.tv | NO | YES | YES | YES |
| anyx.io | YES | YES | NO | NO |
| techcoderx.com | NO | YES | YES | YES |
| hive.roelandp.nl | YES | YES | NO | NO |
Looking at the methods sections, we see some info that will come in handy when we want to implement different API calls. In the mext post in this series we are going to look at using getattr in two Python classes to implement most of the API without any API specific code, just using something quite close to the JSON we produced today. We are likely going to remove the nodes part of the JSON as that part is quite doable in a dynamic way.
Available for projects
If you think my skills and knowledge could be usefull for your project, I am currently available for contract work for up to 20 hours a week. My hourly rate depends on the type of activity (Python dev, C++ dev or data analysis), wether the project at hand will be open source or not, and if you want to sponsor my pet project coinZdense that aims to create a multi-language programming library for post-quantum signing and least authority subkey management.
| Activity | Hourly rate | Open source discount | Minimal hours | Maximum hours |
|---|---|---|---|---|
| C++ development | 150 $HBD/€ | 30 $HBD/€ | 4 | - |
| Python development | 140 $HBD/€ | 30 $HBD/€ | 4 | - |
| Data analysis (python/pandas) | 120 $HBD/€ | - | 2 | - |
| Sponsored coinZdense work | 50 $HBD/€ | - | 0 | - |
| Paired up coinZdense work | 25 $HBD/€ | - | 1 | 2x contract h |
Development work on open-source project get a 30 $HBD discount on my hourly rates.
Next to contract work, you can also become a sponsor of my coinZdense project.
Note that if you pair up to two coinZdense sponsor hours with a contract hour, you can sponsor twice the amount of hours to the coinZdense project.
If you wish to pay for my services or sponsor my project with other coins than $HBD, all rates are slightly higher (same rates, but in Euro or euro equivalent value at transaction time). I welcome payments in Euro (through paypall), $HIVE, $QRL $ZEC, $LTC, $DOGE, $BCH, $ETH or $BTC/lightning.
Contact: coin<at>z-den.se