Hello everyone! I’ve been really busy with some larger, long-term Steem projects but I’ve still made some time to make updates to the Post Promoter software here and there and thought it would be a good time to make a post about it - both to let the community know about the changes and to support Utopian while they recover from the recent attack.
The GitHub repo for the project can be found here: https://github.com/MattyIce/postpromoter
New Blacklist & Whitelist Options
In the never-ending war against spam/plagiarism/junk content on the Steem blockchain, it’s of vital importance that operators of content promotion services have the tools they need to prevent that type of content from being promoted without limiting the options of legitimate publishers.
With that in mind, I have beefed up the blacklisting options offered by the Post Promoter software and also added a whitelist option at the request of some members of the community.
All of the existing and new settings have been grouped together and moved into the “blacklist_settings” object in the config.json file as shown below. This change is backwards compatible so you don’t have to worry about breaking anything if you update your software before your config.
"blacklist_settings": {
"flag_signal_accounts": ["spaminator", "cheetah", "steemcleaners", "mack-bot"],
"blacklist_location": "blacklist.txt",
"shared_blacklist_location": "http://sharedlist.com/list.txt",
"whitelist_location": "whitelist.txt",
"whitelist_only": false,
"refund_blacklist": false,
"blacklist_donation_account": "steemcleaners",
"blacklisted_tags": ["nsfw"]
}
You may notice there are now a few new settings there. In addition to the existing “blacklist_location” setting, there is now a “shared_blacklist_location” setting. This allows you to both use a shared blacklist which can either be a local file or a URL and a bot-specific blacklist if you want to blacklist any accounts not included in the shared list.
There is also now a “whitelist_location” setting which allows you to specify a whitelist for your bot. The whitelist can serve a few different purposes. If you set the “whitelist_only” setting to “true”, then only accounts on the whitelist will be able to use the service.
If “whitelist_only” is set to “false” then the whitelist can be used to override accounts that are on the blacklist, if, for example, you’re using a shared blacklist maintained by a third party and want to still allow certain accounts on the list to use your service.
Separate max bid setting for whitelisted accounts
Now that there is a whitelist option, a new max bid setting has been added for whitelisted accounts called “max_bid_whitelist”. This allows you to set a smaller max_bid for the majority of users but allow certain pre-qualified users to purchase larger amounts of promotion.
"max_bid": 50,
"max_bid_whitelist": 200,
In the above example regular users may only bid up to as much as 50 SBD or STEEM (if accepted) but users on the whitelist may bid up to 200. Any bids that exceed the maximum allowed will be automatically refunded.
Fix for Missed Transactions
This is a pretty major change to how the software determines which transactions it has processed vs which are new. It was made to fix a bug that has been puzzling me for quite some time where certain transactions were just completely missed by the software. There were no errors, or log entries or anything, it’s as if the software never saw them at all.
To get the transactions, the software polls the account history for the bot account every 10 seconds using the getAccountHistory() function call. Each transaction returned by this call includes an account-specific index which starts at 1 for the first transaction ever on the account and increases sequentially with each new transaction.
So to identify which transactions the software has already processed vs which are new, it used to just save the index of the last transaction it saw and then on subsequent calls check for any transactions with an index greater than that.
I knew that since transactions were getting missed, something was wrong with that logic, and I was finally able to catch the issue as it happened. It turns out that, for whatever reason, sometimes transaction indexes can change between calls to getAccountHistory() as shown below:
[ 299611,
{ trx_id: 'db53d503c3a2804802a14c7aa7ea49e575e603e4',
block: 22223537,
trx_in_block: 20,
op_in_trx: 0,
virtual_op: 0,
timestamp: '2018-05-07T14:20:15',
op: [ 'transfer', [Object] ] } ]
[ 299616,
{ trx_id: 'db53d503c3a2804802a14c7aa7ea49e575e603e4',
block: 22223537,
trx_in_block: 20,
op_in_trx: 0,
virtual_op: 0,
timestamp: '2018-05-07T14:20:15',
op: [ 'transfer', [Object] ] } ]
What you are seeing in the snippet above is the same transaction record from two separate calls to the getAccountHistory() method. In the first call that transaction had index 299,611 and in the second call the same transaction had index 299,616.
In this case it would actually cause that transaction to be processed twice, which was also happening along with some transactions being missed.
So it was finally clear what was happening, and that I could no longer use the transaction index on the account as a bookmark for which transactions the software already processed and which were new. Instead I have updated the software to keep track of the “trx_id” for the last 50 or so transactions it has processed, and that way whenever it sees another transaction it can check if it has already processed it or if it’s new - regardless of the order in which they get returned by the getAccountHistory() call.
You can see the transaction IDs getting saved also in the state.json file so they can be restored if the software is ever stopped and restarted:
"transactions": [
"cba86bcfbe1fa38bc9601026cc15b432187d5a03",
"7f3e4d51b8396b049ccfb660c273d19b8f9a30b1",
"ce94dfdf7ac1160aa6ffdb58b05460bdaf57c133",
"e3dab712c97f3212a47e21a364299ded2706b092",
"7be4b4800e0ab154423cb062e07cf13ce164dceb",
"eaa915a273e1078bde886d83d07ceb99e37d6e4c",
"5cd059662a18ffe148992df7cc6a04821ac239bd",
…
While it was somewhat rare, this issue seemed to happen in spurts and caused a lot of missed bids. I’m sorry it took so long but I’m happy to say that I believe it should be fixed for good now and hopefully we are closer to the goal of not missing a single transaction.
Thank you for your support!
As always I want to thank everyone who has helped and supported the development of this software, and to everyone who has submitted ideas, suggestions, bug reports, and contributions to the project!
Links to relevant commits: