Building an iota tangle from scratch in python and flask

Introduction
Before i start i want to talk about 2 very useful resources that was very influential on this project. first of all a post on hackernoon.com by an author Daniel van Flymen. He implements a blockchain similar to the one implemented by blockchain. This post can be found here https://hackernoon.com/learn-blockchains-by-building-one-117428612f46. I will not go into the definitions of what a blockchain is because this post gives all of the necessary details about what a blockchain is to how it works. He gives very clean code that is very easy to follow. A great post you should check it out.

Second is a video by siraj reval . He gives a very clear and easy to follow explanation of how the iota tangle works. His explanation is the way in which i have chosen to implement my code. If the implementation, or my understanding of how the tangle works is incorrect please leave a comment and i will update my code and repost it. For this reason i will not be going into depth about what the tangle is too much.

What is a tangle
Well now that the formalities are out of the way what is a tangle. Based on the video above a tangle is basically a directed acyclic graph, (DAG), which uses a proof of work algorithm. However instead of having miners to do the proof of work algorithm the act of making a transaction requires the client to validate 2 other random transactions which were made by some other random client. This DAG is then shared among every client in the network using a consensus algorithm

What will be implemented
This implementation is a basic implementation. It will focus on building the tangle and being able to share it with other clients. It will not be a way to share money reliably nodes will not have unique identifiers like wallet id’s etc. these may come later in follow up posts if people like it.

Who is this guide aimed at? You should be comfy reading and writing some basic Python, as well as have some understanding of how HTTP requests work, since we’ll be talking to our Blockchain over HTTP.

What do I need? Make sure that Python 3.6+ (along with pip) is installed. You’ll also need to install Flask and the Requests library. Also you will need an http client like postman or curl. I will be using postman as I simply like the gui. But you can use anything. Postman can be found here https://www.getpostman.com.

The final code can be found here https://github.com/ljlabs/tangle-pow

Step 1 building the tangle
Open a text editor and create a new files called tangle.py, api.py, and properties.py we will be using 3 files to help make the code easier to understand each part.

The file properties.py will be our basic configuration for the network and thus will have no logic only constants

RequiredProofs = 2 # the number of times a transaction needs to be processed
# befor it is seen as valid
numberOfValidationNodesNeeded = 2 # the number of nodes needed to connect to when
# addeing a new node to the network for the purpose
# of validating them when making a transaction

The skeleton for tangle.py will be

import properties as prop
import json
from time import time
import hashlib
import requests
from urllib.parse import urlparse

class Tangle(object):
def init(self):
pass # initialize the network

@staticmethod
def valid_proof(self):
   pass # used to make sure that the pow was correct

def proof_of_work(self):
   pass # calculate the proof of work

@staticmethod
def hash():
   pass # compute the hash
  
def createNode(self):
    pass # create a new node


def send_transaction(self, data):
   pass # send a transaction


###########################################################################
# this is the consensus algorithm
###########################################################################

def valid_tangle(self, tangle):
    pass # we will use this to ensure that the tangles we get from peers are correct

def register_peer(self, address):
    pass # we will use this a add new peers to our p2p network

And finally the skeleton for the api.py

import hashlib
import json
from textwrap import dedent
from time import time
from uuid import uuid4
from tangle import Tangle
from urllib.parse import urlparse

import requests
from flask import Flask, jsonify, request

Instantiate our Node

app = Flask(name)

Instantiate the Blockchain

tangle = Tangle()

Hive account@app.route('/transactions/new', methods=['POST'])
def new_transaction():
pass # make a new transaction on the network

Hive account@app.route('/tangle', methods=['GET'])
def full_chain():
pass # returns the current tangle

Consensus

Hive account@app.route('/peers/register', methods=['POST'])
def register_nodes():
pass # add new peers to the network

Hive account@app.route('/peers/resolve', methods=['GET'])
def consensus():
pass # check for other maybe newer tangles on the network

Hive account@app.route('/peers', methods=['GET'])
def list_peers():
pass # return a list of peers

if name == 'main':
app.run(host='0.0.0.0', port=5001)

What do the nodes look like
Each of the nodes in our tangle will look like this

Node = {
'index': "int value",
'timestamp': time(),
'data': "this is the transaction data that is being stored",
'proof': "proof of work",
'previous_hashs': "the hash of the previous 2 nodes that it is connected to",
'previous_nodes': 'index values',
'next_nodes': 'indexes of the nodes pointing to it',
'validity': the number of times the node has been proven
}

At this point the way the tangle works should be easy to see. Each node has its data and identifying characteristics as well as some information about how it fits into the tangle and how many times it has been validated by the network

Creating transactions
This is by far the most work that will have to be implemented as making a transaction also validates other transactions making it the most important step in the creation of our tangle

class Tangle(object):
def init(self):
self.nodes = []
self.peers = set()
for i in range(prop.numberOfValidationNodesNeeded):
# Create the genesis block
self.nodes.append(self.createNode(None, [], len(self.nodes), validity = prop.RequiredProofs))

  @staticmethod
  def valid_proof(last_proof, last_hash, proof):
      # ensures that the node has the correct number of zeros

      guess = (str(last_proof) + str(last_hash) + str(proof)).encode()
      guess_hash = hashlib.sha256(guess).hexdigest()
      return guess_hash[:4] == "0000"

  def proof_of_work(self, last_proof, last_hash):
      # computes the proof of work
      proof = 0
      while self.valid_proof(last_proof, last_hash, proof) is False:
          proof += 1
      return proof


  @staticmethod
  def hash(node):
      # make a hash of the block
      # We must make sure that the Dictionary is Ordered, or we'll have inconsistent hashes
      node_string = json.dumps(node, sort_keys=True).encode()
      return hashlib.sha256(node_string).hexdigest()

  def validate_node(self, node):
      if self.nodes[node['index']]['validity'] < prop.RequiredProofs:
          last_proof = self.nodes[node['index']]['proof'] # this nodes proof
          last_hash = ""
          for prevHash in self.nodes[node['index']]['previous_hashs']:
              last_hash += prevHash # the hashes of the nodes this node connects
          self.nodes[node['index']]['proof'] = self.proof_of_work(last_proof, last_hash)
          self.nodes[node['index']]['validity'] += 1

  def createNode(self, data, prevNodes, newIndex, validity=0):# these prevNodes are the indexes in the dag that points to the previous nodes
      prevHashes = []
      '''
      may need to update every node that points to this when sending transaction
      '''
      for i in prevNodes:
          prevHashes.append(self.hash(self.nodes[i]))
          # now we tell the nodes that we are pointing to that we are poiinting to them
          self.nodes[i]['next_nodes'].append(newIndex)

      Node = {
          'index': newIndex,
          'timestamp': time(),
          'data': data,
          'proof': 0,
          'previous_hashs': prevHashes,          # hashes of the nodes we are connecting to
          'previous_nodes': prevNodes,                # indexes of the nodes we are connecting to
          'next_nodes': [],                           # indexes of future nodes that will connect to us
          'validity': validity,
      }
      return Node


  def send_transaction(self, data):
      # find 2 nodes in the network that are un proven
      nodesToattach = []
      nodesIndexes = []
      newIndex = len(self.nodes)
      worstCaseScinario = []
      worstCaseScinarioindexes = []
      '''
      this function should be changed to search randomly
      '''
      for i in range(len(self.nodes)-1, -1, -1):
          node=self.nodes[i]
          if node['validity'] < prop.RequiredProofs:
              nodesToattach.append(node)
              nodesIndexes.append(node['index'])
          else:
              if worstCaseScinario == [] or len(worstCaseScinario) < prop.numberOfValidationNodesNeeded:
                  worstCaseScinario.append(node)
                  worstCaseScinarioindexes.append(node['index'])
          if len(nodesToattach) == prop.numberOfValidationNodesNeeded:
              break
      # if there are not enough un varified transactions then use verified transactions
      while len(nodesToattach) < prop.numberOfValidationNodesNeeded:
          nodesToattach.append(worstCaseScinario.pop())
          nodesIndexes.append(worstCaseScinarioindexes.pop())
      # process nodes to attatch
      for node in nodesToattach:
          self.validate_node(node)
      # now that those nodes are proven
      # we can now attatch our node to the dag
      self.nodes.append(self.createNode(data, nodesIndexes, newIndex))
      return newIndex

To break down the steps
First we have to find 2 nodes in the network which are unverified. If not enough unverified transactions are found then verified transactions will have to be used. A transaction is considered valid if its validity level is >= the property RequiredProofs set in the properties.py file earlier This should be done at random. If a transaction that has been previously verified is taken then we will not validate it again.

Then we need to verify those 2 chosen nodes:
We compute its proof of work using an algorithm very similar to hashcash described here https://en.wikipedia.org/wiki/Hashcash. We increment its validity of each of the nodes

We then create the new transaction node:
We set its previous hashes to a list of the hashes of the nodes that we just verified
We set its previous nodes to the index of the nodes that we had previously created.
We timestamp the new node.
We set its index to the index at the end of the list which holds all of the lists
Its next nodes are set to [] as there are no nodes pointing to this node yet
And the validity is set to zero as no node has validated it yet

And finally we tell the 2 nodes that we have just verified that the new node that we have created is pointing to it.

Building the api
To have other peers in our p2p network know about us, and to be able to make transaction we will need to have a network facing interface this is where flask comes in.

@app.route('/transactions/new', methods=['POST'])
def new_transaction():
# update tangle
tangle.resolve_conflicts()
# begin transaction
values = request.get_json()

# Check that the required fields are in the POST'ed data
required = ['sender', 'recipient', 'amount']
if not all(k in values for k in required):
    return 'Missing values', 400

# Create a new Transaction
index = tangle.send_transaction(values)

response = {'message': 'Transaction will be added to Block ' + str(index)}
# tell peers to update tangle
for peer in tangle.peers:
    requests.get("http://"+str(peer) + "/peers/resolve")
return jsonify(response), 201

Hive account@app.route('/tangle', methods=['GET'])
def full_chain():
response = {
'tangle': tangle.nodes,
'length': len(tangle.nodes),
}
return jsonify(response), 200

We create 2 end point
/transactions/new
Allowing us to create new transactions
/tangle
Allowing us to view our current tangle graph

Consensus
The final portion of the project
This is the more interesting portion. Till this point we have a tangle that accepts new transactions as well as allows us to view our transaction history now comes the part to make that network decentralised.

We begin by updating our api.py file to is final form

import hashlib
import json
from textwrap import dedent
from time import time
from uuid import uuid4
from tangle import Tangle
from urllib.parse import urlparse

import requests
from flask import Flask, jsonify, request

Instantiate our Node

app = Flask(name)

Generate a globally unique address for this node

node_identifier = str(uuid4()).replace('-', '')

Instantiate the Blockchain

tangle = Tangle()

Hive account@app.route('/transactions/new', methods=['POST'])
def new_transaction():
# update tangle
tangle.resolve_conflicts()
# begin transaction
values = request.get_json()

# Check that the required fields are in the POST'ed data
required = ['sender', 'recipient', 'amount']
if not all(k in values for k in required):
    return 'Missing values', 400

# Create a new Transaction
index = tangle.send_transaction(values)

response = {'message': 'Transaction will be added to Block ' + str(index)}
# tell peers to update tangle
for peer in tangle.peers:
    requests.get("http://"+str(peer) + "/peers/resolve")
return jsonify(response), 201

Hive account@app.route('/tangle', methods=['GET'])
def full_chain():
response = {
'tangle': tangle.nodes,
'length': len(tangle.nodes),
}
return jsonify(response), 200

Consensus

Hive account@app.route('/peers/register', methods=['POST'])
def register_nodes():
values = request.get_json()
print(values)
print("asdugdag")
peers = None
if type("") == type(values):
print(json.loads(values))
peers = json.loads(values)['peers']
else:
peers = values.get('peers')
if peers is None:
return "Error: Please supply a valid list of nodes", 400

for peer in peers:
    tangle.register_peer(peer)


response = {
    'message': 'New peers have been added',
    'total_nodes': list(tangle.peers),
}

return jsonify(response), 201

Hive account@app.route('/peers/resolve', methods=['GET'])
def consensus():
replaced = tangle.resolve_conflicts()

if replaced:
    response = {
        'message': 'Our chain was replaced',
        'new_chain': tangle.nodes
    }
else:
    response = {
        'message': 'Our chain is authoritative',
        'chain': tangle.nodes
    }

return jsonify(response), 200

Hive account@app.route('/peers', methods=['GET'])
def list_peers():
response = {
'known_peers': list(tangle.peers),
}

return jsonify(response), 201

if name == 'main':
app.run(host='0.0.0.0', port=5001)

As well as our tangle.py


import properties as prop
import json
from time import time
import hashlib
import requests
from urllib.parse import urlparse

"""
nodes in the dag are represented in json as folows
they can be transmitted across the network in this form and then can be
initialized in this form
Node = {
'index': "int value",
'timestamp': time(),
'data': "this is the transaction data that is being stored",
'proof': "proof of work",
'previous_hashs': "the hash of the previous 2 nodes that it is connected to",
'previous_nodes': 'index values',
'next_nodes': 'indexes of the nodes pointing to it',
'validity': the number of times the node has been proven
}

"""

class Tangle(object):
def init(self):
self.nodes = []
self.peers = set()
for i in range(prop.numberOfValidationNodesNeeded):
# Create the genesis block
self.nodes.append(self.createNode(None, [], len(self.nodes), validity = prop.RequiredProofs))

@staticmethod
def valid_proof(last_proof, last_hash, proof):
    # ensures that the node has the correct number of zeros

    guess = (str(last_proof) + str(last_hash) + str(proof)).encode()
    guess_hash = hashlib.sha256(guess).hexdigest()
    return guess_hash[:4] == "0000"

def proof_of_work(self, last_proof, last_hash):
    # computes the proof of work
    proof = 0
    while self.valid_proof(last_proof, last_hash, proof) is False:
        proof += 1
    return proof


@staticmethod
def hash(node):
    # make a hash of the block
    # We must make sure that the Dictionary is Ordered, or we'll have inconsistent hashes
    node_string = json.dumps(node, sort_keys=True).encode()
    return hashlib.sha256(node_string).hexdigest()

def validate_node(self, node):
    if self.nodes[node['index']]['validity'] < prop.RequiredProofs:
        last_proof = self.nodes[node['index']]['proof'] # this nodes proof
        last_hash = ""
        for prevHash in self.nodes[node['index']]['previous_hashs']:
            last_hash += prevHash # the hashes of the nodes this node connects
        self.nodes[node['index']]['proof'] = self.proof_of_work(last_proof, last_hash)
        self.nodes[node['index']]['validity'] += 1

def createNode(self, data, prevNodes, newIndex, validity=0):# these prevNodes are the indexes in the dag that points to the previous nodes
    prevHashes = []
    '''
    may need to update every node that points to this when sending transaction
    '''
    for i in prevNodes:
        prevHashes.append(self.hash(self.nodes[i]))
        # now we tell the nodes that we are pointing to that we are poiinting to them
        self.nodes[i]['next_nodes'].append(newIndex)

    Node = {
        'index': newIndex,
        'timestamp': time(),
        'data': data,
        'proof': 0,
        'previous_hashs': prevHashes,          # hashes of the nodes we are connecting to
        'previous_nodes': prevNodes,                # indexes of the nodes we are connecting to
        'next_nodes': [],                           # indexes of future nodes that will connect to us
        'validity': validity,
    }
    return Node


def send_transaction(self, data):
    # find 2 nodes in the network that are un proven
    nodesToattach = []
    nodesIndexes = []
    newIndex = len(self.nodes)
    worstCaseScinario = []
    worstCaseScinarioindexes = []
    '''
    this function should be changed to search randomly
    '''
    for i in range(len(self.nodes)-1, -1, -1):
        node=self.nodes[i]
        if node['validity'] < prop.RequiredProofs:
            nodesToattach.append(node)
            nodesIndexes.append(node['index'])
        else:
            if worstCaseScinario == [] or len(worstCaseScinario) < prop.numberOfValidationNodesNeeded:
                worstCaseScinario.append(node)
                worstCaseScinarioindexes.append(node['index'])
        if len(nodesToattach) == prop.numberOfValidationNodesNeeded:
            break
    # if there are not enough un varified transactions then use verified transactions
    while len(nodesToattach) < prop.numberOfValidationNodesNeeded:
        nodesToattach.append(worstCaseScinario.pop())
        nodesIndexes.append(worstCaseScinarioindexes.pop())
    # process nodes to attatch
    for node in nodesToattach:
        self.validate_node(node)
    # now that those nodes are proven
    # we can now attatch our node to the dag
    self.nodes.append(self.createNode(data, nodesIndexes, newIndex))
    return newIndex


###########################################################################
# this is the consensus algorithm
###########################################################################

def valid_tangle(self, tangle):
    for node in tangle:
        if node['index'] >= prop.numberOfValidationNodesNeeded: # dont test genesis nodes
            validitiyOfNode = node['validity']
            # make sure that the same number of nodes saying that they have
            # validated him as his validity level
            nextNodes = node['next_nodes']
            print(nextNodes)
            if validitiyOfNode < len(nextNodes):
                return False
            # make sure these nodes are pointing to him
            for Nnode in nextNodes:
                print(tangle[Nnode])
                if node['index'] not in tangle[Nnode]['previous_nodes']:
                    return False
    return True

def register_peer(self, address):
    parsed_url = urlparse(address)
    self.peers.add(parsed_url.netloc)

def resolve_conflicts(self):
    neighbours = self.peers
    new_tangle = None

    # We're only looking for chains longer than ours
    max_length = len(self.nodes)

    # Grab and verify the chains from all the peers in our network
    for peer in neighbours:
        response = requests.get("http://"+str(peer) + "/tangle")

        if response.status_code == 200:
            length = response.json()['length']
            tangle = response.json()['tangle']

            # Check if the length is longer and the chain is valid
            if length > max_length and self.valid_tangle(tangle):
                max_length = length
                new_tangle = tangle

    # Replace our chain if we discovered a new, valid chain longer than ours
    if new_tangle:
        self.nodes = new_tangle
        return True

    return False

This now allows us to make calls to our network register new peers to that network and check if our peers have newer versions of our tangle it does this by checking if their tangle is longer than ours

Finishing up

At this point we can spin up the api by calling python api.py on 2 computers or on 2 different port numbers which is what i had done one on port 5000 and the other on port 5001

Firstly we should make a call to our api at http://127.0.0.1:5000/peers/register
{
"peers": ["http://X.X.X.X:5000"]
}
Where X.X.X.X is the url of the second instance of the application on your lan that you had spin up

Using an http client make a post request to http://127.0.0.1:5000/transactions/new
With the json object

{
"sender": "this is such fucking bull shit",
"recipient": "anne",
"amount": 5
}

And we should see a response

{
"message": "Transaction will be added to Block 2"
}


Now we can either make a call to http://127.0.0.1:5000/tangle or http://X.X.X.X:5000/tangle and both will give us the same tangle

And this concludes the tangle. If this was helpful please tell me in the comments. This is my first post so please go easy on me. It is probably not up to the scratch of other posts i am a software developer and writing is new to me. Please feel free to take my code on github and edit it in any project. Please tell me if this was helpful and if i should keep making new posts about building blockchain technology