I have been posting a number of (loose) version 0.3 language specs posts in recent weeks, while I've started working on a first version of what I refer to as a Semantic Lexer for Merg-E in Python. The language is very much evolving as I write it, so some of the examples from my first post in the series won't make it through the lexer right now, but I plan to fix this after completing the first version of my lexer, hopefully in one or two weeks.

In this post I want to take a step back from the details of the Merg-E language and give a bit of a high level overview of what I am trying to build.

InnuenDo Stack context.

Merg-E doesn't aim to be a general purpose programming language like Python or C++ or Rust, nor does it aim (for now) to be a generally embeddable programming language. It (for now) aims to be a domain specific language or DSL especially tailored for the needs of non-layer-1 functionality within an InnuenDo stack ecosystem.

So what is the InnuenDo Stack? It is an envisioned multi-programming-language Web 3.0 stack on top of the ongoing CoinZdense project.Let's do a quick and superficial walkthrough of the envisioned stack just to get a feel for the context.

libcoinzdense

At the bottom of the stack is libcoinzdense. After I wrote most of a Python proof of concept and ran into the GIL when trying to do parallel computation with the standard hashing lib I was using, and after having given Rust a good long try, I recently started porting my Proof Of Concept functionality to C++, the ongoing libcoinzdense project. So what will libcoinzdense do? Well, in the first place it implements hash based signatures specifically geared towards the needs of Web 3.0 blockchain platforms. The algorithms used by libcoinzdense for that are loosely based on XMSS, the signature algorithm used by QRL, the Quantum Resistant Ledger. This is a high level overview post, so I won't go into much of the details why I chose not to use XMSS and instead used my own derived algorithm plus an extra layer on top, but I'll sum up a few details as how it is different for those interested in cryptography.

• Rather than using a single WOTS chain and basically a Cyclic Redundancy Check (as re-use attack protection) I used a double WOTS chain for the same purpose. I'm an engineer, not a cryptographer, so some Phd signing off on CRC usage being safe for this purpose just feels like using a tie wrap on a freight train bridge, I'm not doing it.
• Rather than a single huge merkle tree signing key that would be needed for the high-reuse that Web 3.0 needs, I am using a lazy tree of small merkle tree level keys that conceptually combine into a single massive-reuse key that would takes months to calculate and many gigabytes to store if I were to use a single merkle tree.
• Except for just a single signing key signing transactions, the authority a user has on the chain is modeled as a tree, and any user can share or publish or make and keep a receiver key where another user, or the user themselves can then delegate part of their authority tree to.

If you don't understand the technical details above, don't worry for now. The important thing is that this setup:

• Takes up pretty massive amounts of compute in bursts when the lazy layered key design needs a Just In Time new layer key (or two or three) to replace the old one.
• It is already very much a least-authority system at its core, even if the normal languages on top of it most definitely are not.
• Aims to provide a replacement path for Web 3.0 chains that currently rely on quantum vulnerable signature schemes like ECDSA in a way (with extensive key reuse) that currently makes Web 3.0 more vulnerable to future Quantum Computing attacks than most other blockchain technology (with the possible exception of BitCoin and BitCoin hard-forks). If this last bit is too technical, maybe you will enjoy reading my 2017 novel Ragnarok Conspiracy where such vulnerability leads to a quantum blockchain heist that triggers a war and the collapse of the global monetary system.

ports & Web 3.0 chains.

Not planned any time soon, but while HIVE, the first Web 3.0 platform I hope to find a connection with, has its core L1 nodes running on C++ code, other interesting chains use Go, Rust or Clojure as the L1 node language. Basically at the CoinZdense layer, if I "ever" have the time (or funding that could buy me time) to look beyond HIVE, I'll need to make ports of libcoinzdense to Go and Rust.

WASM & CoinZdense language bindings.

Mostly for the client side of things, both bots/L2/backend and frontend, C++ isn't the go to language, and I'll need to create language bindings. Python is top of my list there for backend and bot usage, but TypeScript will later become an important target too for supporting hash based signing up into the web client. Then, if in the future I'll have the time (or again, funding), I might work on language bindings for Clojure and ClojureScript to support amazing Clojure based blockchain technology like FlureeDB, a blockchain based graph database with some very special features that think many blockchain ecosystems may want to incorporate into their stack.

Except for Python, where the language bindings will be native C++ to Python, for all of the other languages I plan to use Web Assembly (WASM) as intermediate target for libcoinzdense so I can build the language bindings on top of WASM.

aiow3/tsw3/clsw3

A while ago I started working on a python library for writing HIVE bots called aiohivebot. The aiohivebot python library is a parallel HIVE bot library that connects to all of the public API nodes of the HIVE blockchain at the same time in order to have a truly robust Web 3.0 bot framework. It periodically tests the API nodes for speed and reliability and as such balances its work between the fastest handful non broken API nodes. It is an event driven asynchronous framework that next to the HIVE nodes connects to the hive-engine nodes if requested, and other event sources were being looked at before porting efforts to aiow3 started recently.

Right now I'm working on the API for aiow3 that aims to be a port and partial rewrite of aiohivebot that isn't HIVE centric by design (though the first version will still only support HIVE) but has an API that allows the use of a wide range of Web 3.0 chains, if needed in parallel. This aiow3 framework will be changed to include support for ECDSA and friends as well as libcoinzdense language bindings usage, and thus be the reference implementation of a InnuenDo stack framework for bots and backends in other languages.

After Python, two target languages for aiow3 are currently on my maybe, if I have time list, and these are:

• tsw3: For TypeScript
• clsw3: For ClojureScript

The chances of tsw3 ever seeing the light of day are a lot bigger than for clsw3 because as stated before, ClojureScript language bindings aren't currently high on my priorities list.

hive archeology bot and w3minorfs

These aren't actually part of the InnuenDo Stack, but they are still important as a proof of concept of the InnuenDo stack. The Hive Archaeology Bot is an existing HIVE bot, currently running on top of the lighthive library. I plan to port that bot to aiow3 as a test of the python column in the InnuenDo stack.

Now we get to the real subject of this blog post, Merg-E. The w3minorfs node will aim to be a simple file-storage layer-2 on top of HIVE that basically reimagines my old least authority project MinorFS as a Web 3.0 least authority blockchain based least authority file-system.

But with least authority on top (MinorFS logic) and least authority at the bottom (CoinZdense subkey management and delegation), having the middle made up by either Python or TypeScript just doesn't sit right. And I have too little experience with ClojureScript to put that option on the list.

Why no Monte or Elixir ?

As some of you may know, at first I was planning to add language bindings or ports for libcoinZdense. After my long struggles of porting my proof of concept to Rust and running into the troubles of language bindings on top of Rust compared to C++, and after looking into the WASM hack that opened up a lot of potential language bindings, both Monte and Elixir turned out to be quite challenging in terms of figuring out if clean language bindings would even be possible, or if I needed to do a full fork of Monte and a full port for Elixir. Add to that the desire to have a port of aiow3 to at least one of them, the thought came to me that replacing these four projects with just one: Merg-E

Merg-E: A language, a runtime, a framework, integration over language bindings

Consider that Merg-E aims to be both its own language bindings plus a port of aiow3 into its own runtime, and next to that Merg-E is it's own language, a Domain Specific Language for writing the same kind of things a user might write using the aiow3 framework or one of it's future ports. But also more. By aiming to bridge the gap between least authority below and least authority on top, Merg-E allows to make layer-2 nodes least authority top to bottom. So not just bots and backends, but L2 nodes too, going beyond what aiow3 aims to deliver.

Merg-E is a language, or at least it will be, but it's also the runtime, plus the language and ambient authority try to embed what the language bindings provide for other languages. You could say that in the InnuenDo stack, Merg-e is a merge of multiple stack layers into one.

The name Merg-E

Merg-E is more than a merge within the context of the InnuenDo Stack. It is also a merge of low level control and the least authority of object capability systems, of the actor model and the parallelism of ephemeral parallel running functions, of a programming language an aspects of data flow languages, and on top of that, merge is also the name of a key expression in the Merg-E language, an expression that allows for surgical localized imports.

So why is the word merge stylized as Merg-E? Well for two reasons:

• While many features like the borrows and the freezing rules, as well as the focus on a DAG based authority flow are very foreign to normal object capability languages, Merg-E does trace a big part of its least authority spirit back to the E language. As a minimal language we make no claim of Merg-E being anywhere close to E as a language, look at the naming as a tribute to E for the spirit of the language.
• The name merge or rather an overloaded term in tech, for one there is merge as used in version control systems like git, and then the combination of the words merge and language will have your google search or LLM prompt throw you deeply into linguistics. Google Merg-E and right now you will only find some non-technical financial organization with no risk of anyone getting confused.

So after checking with the designer and lead architect of the E language recently, who had no problem with me calling the language Merg-E, Merg-E is no longer a working title but the actual name for both the DSL and the runtimes.

So how to pronounce Merg-E? It's up to the user. Call it "merge" if you like, or "merge-E". I myself have a horrible accent when speaking English so I pronounce it like "mur-chee", you can join me in that pronunciation but "merge" and "merge-E" are just fine.

The Merg-E philosophy

Let's see if we can share what the Merg-E language is about from the design philosophy.

Opinionated

First of all, Merg-E is a highly opinionated language. This is partly out of necessity, and partially because after about 40 years of programming in some form or the other, exposure to dozens of languages and frameworks both mainstream and niche, I've grown quite a bit opinionated on some subjects myself. But the practical side is that I want to keep my implementation of Merg-E simple and the language small, and to do that I need to make choices.

The safest options are the least verbose

I really like the way libsodium (a cryptographic library that I'm using to implement libcoinzdense) and the philosophy it uses to make it harder to do the right or safe thing than to do the wrong, or dangerous to do the wrong or unsafe thing. For Merg-E I'm embracing that principle with a focus on robustness, least authority and system integrity. Sometimes there are minor diversions from that, like the use of the sensitive modifier on constants that might be a slightly controversial choice (I need to check if that is documented, basically a constant gets implicitly closure captured unless it is marked as sensitive), but in general Merg-E tries to make anything that is less secure, less robust, etc, require just that little bit more of verbosity to make people stand still and think: "Do I really need this variable to be shared" or "am I actually sure this function is ready to run parallel in dozens of instances?" In Merg-E you don't say things are const, you say they are mutable. And when Merg-E allows you to shoot yourself in the foot, you are forced to mark the hazardous construct you are using as hazardous to remind you that you better know what you are doing.

DAGs all the way down and all the way up.

With subkey management in CoinZdense being largely Directional Acyclic Graph (DAG) based, and with the planned proof of concept w3minorfs being all about DAGs, making Merg-E in the middle all about DAGs seems like the proper thing to do. Merg-E embraces DAGs to the fullest. The language itself is an arborescent DAG. All ambient authority the process as a whole is able to have is represented as an arborescent DAG. DAGs below DAGs up above and DAGs in the middle. Want a stuct? Nah, just use a DAG. Need a dict or a map? Sorry, not in the language, just use a DAG. If a DAG can do the job, then I'm not extending the language to do the job more conventionally. Keeping the language small is top priority and DAGs are helping with that.

If a DAG can't, then maybe a dataframe can.

Parallelism and asynchronous operations are very important for the domain where Merg-E is meant to work in. Having done a lot of data engineering work in the past, I've grown quite a fan of the numpy stack and of different dataframe oriented frameworks, both in Python and in more enterprise geared data streaming solutions. So next to DAGs I became convinced pretty quickly that Merg-E will need to support dataframes. In simple terms, a dataframe is like a page of a spreadsheet with rows and named columns where you can do operations on. So other than things like jsonrpc that we obviously need for communicating with for example HIVE public API nodes, making things like parquet support a part of the runtime has peak priority. Dataframes should be considered just as much of a first class citizen as DAGs for Merg-E.

And again this has consequences. We need dataframes, but do we need list and tuples if we have dataframes? Do we need sets if a dataframe has columns that can act like sets? Right now for Merg-E the answer is no. As said, we want the language to remain small and simple. The InnuenDo stack is big and very ambitious for a one man spare-time project. Or rather 20 one man spare time projects if I ever manage to complete the whole envisioned stack as a whole. So if I want to be able to continue working on libcoinzdense, on aiow3, and on all the currently not yet active other projects, I need to make sure Merg-E doesn't end up eating all of my available time for too long. For now that means being extremely strict with the idea of keeping Merg-E a small language.

Zero assumption parallelism.

Because, even while going for Merg-E closed the door on on Elixir or the InnuenDo stack (at least the part that I can even hope to complete myself in my lifetime), I don't quite want to close the door on BEAM yet, nor do I want to close the door on integrating SYCL in the future for pure compute parallel tasks, AND because I'm starting off with a scripting implementation without threads, with just a small taskpool, with in the middle the compiled CPU-only compiled target with a thread pool and a two stage scheduler, there are going to be at least two and possibly three or four different runtimes with greatly different parallelism properties.

Further the compiled CPU-only runtime will need to work together cooperatively with CoinZdense's compute need bursts for JIT layer-key replacement. As a result, parallelism in Merg-E should be seen as a hinting system. The user hints the runtime about intent with respect to parallelism, but the runtime might or might not be able to act on those hints. Either not now, because it has shrunken it's active thread pool to make way for a computational burst by libcoinzdense, or not at all because the runtime used is the initial scripting runtime made for language development, and there is no thread pool, just a task pool.

When you call a function, assume it will run in parallel. There is no return value, and unless you explicitly add the awaitable to a blocker, it will be fire-and-forget. The less blockers you introduce, the more efficient the scheduling, so aim to forget unless you really need to synchronise.

Note that the freedom to implement multiple runtimes is also the reason why Merg-E chooses for actor model adjacency, no return values except through using special constructs that simulate them and only in special cases (inline), and why locks aren't traditional locks but rather scheduling conditions. Things like "maybe one day I'll get to making a BEAM port" and "maybe I'll try to integrate SYCL for suitable parts of the compilation" are thoughts that keep Merg-E oriented toward being able to support multiple foreseeable and not yet foreseeable runtimes.

Don't hide the guts (thin abstractions)

Merg-E aims to be a small language, and if something is needed for implementation of the language runtime, we might as well expose it to the language too. But beyond the minimum that we need for language completeness, thin abstractions keep the codebase small. Things like operator extension, and integer-size generics, and explicit actorcitos are examples of such thin abstraction extras that make the language more interesting while remaining simple to make, but possibly a bit less simple to fully learn and master. But once you do master it, the extra exposed guts make it a more powerful language than its small size would suggest.

Closures

Many languages today implement the idea of closures. If you define a function within a closure, the inner function captures constants and variables from the outer closure. In Merg-E we use a least-authority variant of that. Constants are assumed not to hold authority and are thus implicitly frozen (unless they were marked as *sensitive). Variables are only captured explicitly as defined in the function declaration.

For the third time, closures are also a language feature that allows us to ask questions about other language features we need. If you have closures, do we need objects or classes? Again for Merg-E the answer is NO, if the language becomes bigger if we implement it and we don't really need it, then we won't implement it. We have closures, so no objects and classes. Just closures.

Move as default.

C++ developers reading this section may recoil in horror, but the ownership model of Merg-E is modeled after what the C++ committee considered a sort of a but in the now deprecated auto_ptr smart pointer. In Merg-E reference stealing is the default. Explicitly capture a standard mutable from the outer scope and you will steal ownership from your parent scope. We are going for the safe and least authority default, and in our opinionated language, in this case that means a non-shared non-borrowed mutable will have its ownership transferred unless, and this post is too high level to discuss the “unless”.

If you don't want this behaviour, use modifiers that force Merg-E to share or borrow out the mutable, or ask the runtime for a copy, but the default will always be transferred.

Always remember that Merg-E is just a DSL

Before you think: "what a limited or cumbersome language, I can't even do X", please remember that, at least for now, Merg-E isn't aiming to be a do-it-all general purpose programming language. It is being made specifically for the goal of having a least authority domain specific language for building Web 3.0 L2 nodes, bots and backends that run on top of and in subordinate cooperation with CoinZdense.

Merg-E is a "merge"

Merg-E may feel like it an implementation of some academically pure paradigm of least authority parallel and actor based paradigm, but this feel stems purely from the fact that Merg-E takes minimalism very seriously because of pure time resources constraints. Merg-E isn't pure in any sense. It is neither pure object-capabilities, pure actor model, pure data flow, nor pure functional. And it's not some novel new pure paradigm either. Merg-E is a practical merge of a little bit of all four of these. Practical in the sense that it's implementation needs to fit within my limited time resources, and practical in the sense that the language design should fit on all of the four envisioned runtimes, even if my time resources may only allow me to actually implement two or likely at most three of them. Further the language isn't purist in that it comes with practical escape hatches, partially to allow fix-later prototyping, and partially because sometimes the compiler simply isn't smart enough to know what you are doing is perfectly fine. When there is real tension between the four things that Merg-E tries to be just enough of, usualy the Principle of Least Authority and the multi-runtime vission prevail, but Merg-E is filled with small little compromises. I would have loved to be closer to E, especialy the idea of blockchain-backed persistent VATs was something I had to give up on reluctantly because of paralelism, and there are more cases like that. As a result, Merg-E is like E mostly in spirit. A little more than it is like Erlang in spirit or like Quix Streams is spirit. As such I'm very grateful that Mark allowed me to tribute E in naming Merg-E.

High level implementation

Before we deep dive into the language specs at a technical level in the spec posts listed later, let's outline a few important high-level implementation aspects of Merg-E that the user may care about.

Níðhöggr and Yggdrasil

While Merg-E aims for specs that allow more preemptive task execution at an implementation level—for example, mapping inline parallel lambdas to SYCL kernels or writing a runtime on top of the BEAM virtual machine with multitudes of long-lived processes (allowing actors without explicit scheduling, neither of which are currently on the roadmap)—conceptually, Merg-E is built around schedulable, task-like entities that are queued for a scheduler and distributed among worker event loops.

This scheduler, Níðhöggr, and its queue/database, Yggdrasil, form the core of any Merg-E runtime. At the core of this design is a queue that isn’t actually a queue: it is an in-memory DAG-shaped database with four zones (or “codominant stems”) on which the scheduler operates. Branches or sub-DAGs represent tasks with code, scope, continuation points, and metadata. These are temporarily delegated to event loops, processed, and then returned to one of the DAG’s zones—or permanently pruned when completed. Níðhöggr promotes tasks through different zones to prioritize work while respecting dependencies.

The hazardous / trustmebro design.

The modifier hazardous is sprinkled throughout Merg-E language, and (like other modifiers) the user can add more usages to extend the language. Basicly hazardous signals to the compiler, and to anyone reviewing the code: "I know this isn't the cleanest or safest acording to the compiler, but I know what I am doing, or this is temporary, and I'll fix this later.".

But this is a wide class of things where this can apply to:

• <scalartype> : When puting sensitive data in a scalar literal
• typecast : When casting an integer into an integer type the compiler thinks may be to small to hold the value
• prune : When soft-pruning a DAG and relying on non-deterministic restructuring of the remaining graph into a still rule-complient DAG.
• blocker : When using a blocker inside of a lock
• <custom operator> : Whatever meaning a user operator-extending the language assigns to hazardous for their operators.
• frozen : When the compiler can't figure out if a mutable is frozen or not, and the user is sure that it is.
• merge : When communicating to the lexer in a merge (surgical import) that an imported function or same-module code it invokes is allowed to use hazardous and for what.

In production the pervasive use of hazardous is considered non-idiomatic, and by default the lexer will error out on any hazardous, but the --trustmebro compile flag allows full control about what use of hazardous is deemed to be OK in the current stage of development.

Structural iterators and friends

In Merg-E, there is no for loop and if and while are just syntactic sugar around a Níðhöggr grounded abstraction that we name the structural iterator. While in many cases this will feel reasonable intuitive, once you as a user start diving into paralel code, the actual guts of the structural iterator will start showing, and you'll have to manualy juggle actorcitos (strange little callables with iterator bound lifetimes) and sorting functions. Actorcitos will often require explicit handling when running in parallel to maintain output order, something that may be less intuitive for people comming from languages where these things aren't a concern.

Two stage dataframes

After DAGs, dataframes are an important data container. But they aren't just containers, they are like a house you cant start living in untill it's ready. A dataframe starts its life empty, then is mutible in the sense that immutable rows can be added to it, but then, before it can actualy be used, at least before the data can be in any way accessed, it needs to be frozen first. After that, dataframes are one of the friendliest citicens when it comes to paralel processing of data, but they first need to be frozen.

The technical details

So far for the high level philosophy and the why behind a number of things. Plus a few noteworthy top level language properties. If you want to see what Merg-E code will look like, have a look at my more technical posts on the v0.3 language specification.

• part 1 : coding style, files, merging, scoping, name resolution and synchronisation
• part 2 : reverse markdown for documentation
• part 3 : Actors and pools.
• part 4 : Semantic locks, blockers, continuation points and hazardous blockers
• part 5 : Semantic lexing, DAGs, prune / ent and alias.
• part 6 : DAGs and DataFrames as only data structures, and inline lambdas for pure compute.
• part 7 : Freezing
• part 8 : Attenuation, decomposition, and membranes
• part 9 : Sensitive data in immutables and future vault support.
• part 10 : Scalars and High Fidelity JSON
• part 11 : Operators, expressions and precedence.
• part 12 : Robust integers and integer bitwidth generic programming
• part 13 : The Merg-E ownership model, capture rules, and the --trustmebro compiler flag.
• part 14 : Actorcitos and structural iterators
• part 15: Explicit actorcitos, non-inline structural iterators, runtimes, and abstract schedular pipeline.
• part 16 : Async functions and resources and the full use of InnuenDo VaultFS
• part 17 : RAM-Points, RAM-points normalization bag, and the quota-membrane.
• part 18: Literal operators & Rational and Complex numbers.