TUTORIAL: Using the Splinterlands API and KNIME to determine what cards you require to complete your untamed collection

About me

I've been playing Splinterlands since 2018.

I love data. I love crunching it. I love looking at it, storing it, and making decisions based on it. In my day job, I use a tool called KNIME which is an analytics tool that can transform and crunch data in any way shape or form. KNIME is freely downloadable, and available on all major operating systems.

About this tutorial

This tutorial is lengthy and advanced in nature. If you're a developer, you likely won't gain anything from this. If, however, you are an Excel junkie and have not yet graduated into the realm of data science and analytics, KNIME is a tool that can certainly take you to the next level. You'll learn about KNIME basics and apply it to something fun - Splinterlands cards.

Using Knime

Let's begin by downloading knime.

Once it is downloaded, install the software.

Run the software, set a directory to use as Knime's "scratch", and you're ready to go.

Knime has a variety of Windows. The top left is your explorer, where different KNIME flows live in your working directory. Here is also where you will find example analytical models from the community to learn how to do various things in KNIME.

The next window down is a workflow coach which helps you out with what sort of nodes can be linked together.

What are nodes? Nodes are blocks that perform operations on your data. They may modify it, search for something, filter, or do any of the other common operations you may want to do when working with large, small, or unfamiliar data sets. You can also perform a number of standard SQL functions like joins, which are incredibly useful.

The Node Library has hundreds of plug ins, and you can even drop R, Python, or JavaScript snippets into KNIME to perform operations for which there may not be a node. You've then got the convas in the middle of the screen, which is where you will do most of the work in KNIME.

KNIME sees you link a series of nodes to perform operations or analysis on data, and the processed result is then accessible at the end of the flow. For the rest of this tutorial, I will explain what I have created which can be built upon further in the future. The creation I've produced is indeed a minimum viable product, and just one of the many applications you could perform on Splinterlands card data.

Building the analytical flow

In order to determine the answer to the query: "How many regular foil cards do I require to max out my Untamed deck?" we need to know a number things.

1. What cards do I as a player have?
2. What cards exist in Untamed?
3. What is the number of cards required to max out my deck?
4. What is the difference between what I have and the max level cards?

It is important to know everything you need before starting any sort of analysis work - or you may have to start from scratch all over again. In my day job I make sure that parameters and scope is set before I start my research in order to avoid complexities in future iterations of reporting or analysis which means prior data sets are invalidated by iterative improvement (or changing) business requirements.

Here, we've got a fairly flat set of requirements - so let's get started.

Thanks to Hive account@kiokizz's excellent Splinterlands API documentation, a lot of the information can be extracted direct from the game's API.

What cards do I have as a player?

This is the first part of the puzzle. The Splinterlands API responds using a data format called JSON. Thankfully, Knime can import JSON data structures as well as CSV, Excel files, or even connect directly to a SQL database!

I create a JSON Reader Node by searching for JSON reader in the node repository window, and dragging it onto the canvas. I then right click to configure the node.

Right clicking the node gets me:

The next step is decoding and tabulating the JSON data to individual rows that can then be analysed. Otherwise, the response is a lengthy single table cell which lists all the individual cards I have:

Now, I don't need all the information about the card in order to answer my query - so I create a list of JSON paths (ie the variable structures) that I want to focus on. Think of this as a filter to only grab the variables and information that are relevant to the question I'm asking - which is "How many cards do I need to max out my Untamed regular foil collection?"

I need to know the following:

• uid

This is the unique ID for a card.

• card_detail_ids

This is the "serial number" for the card - it will tell me what number card it is - more on this later.

• golds

a boolean value that identifies if the card is a regular, or gold foil

• xps

The XP (how many cards) have been combined under the uid (or, as we know it, bcx)

• players

Who owns the card

• levels

What level the card is at currently

• editions

What edition the card is - Alpha, Beta, Untamed, Reward, Dice, Orbs, etc

The next operation in Knime is to "Ungroup" the JSON data of my card collection, so I get a list of each and every card that I have.

This gives me a table structure with each and every card I have. The UID is the card's unique "serial" - or the underlying NFT number. The card_detail_ids tells us what card it is - then if its gold, how much xp, who owns it, what level it is, and from which edition the card came...

But - it isn't very human readable. For that, we're going to need to use a really cool Knime feature called the Rule engine. Because I can go build a reference table to find out what edition a card is, I can make that "Editions" Column something that is very human readable.

This results in the previous table having a new column added:

However, we still don't know what card is what just by looking. This is where the card_id field will come in handy - but I certainly don't want to go and build a massive table listing each and every card's name, splinter, and other variables, when that already exists in the Splinterlands game API - I just need to pull it using a different part of the API documentation.

This is everything we've achieved so far:

Now - we'll look at the second part - getting the details of the cards from the API, which is the second line of the screenshot above.

What are the card details?

To get the card details, we need to make a call to the Splinterlands API.

Using the JSON Reader node as we did to get a user's card details, we simply make a different call to the game API:

The subsequent nodes transform the data into a table that looks a bit more useful...

For some reason, some of the number based variables came through as Strings - so I need to convert them to a number in order to further categorise the output, and make the table a bit more human readable.

Determining Rarity:

Determining the Splinter:

Next is where the major magic happens - a JOIN.

Joining is a function of SQL and data science / analytics where you take two disparate sets of data with a field that allows an association. In a list of people - this might be their full name. So, you might have a list of full names, and a list of drivers licences and full names, along with a date of birth. You might have another table that has the driver's licence number and the person's traffic infringement history.

By joining the driver's licence, and full name from the people table to the "fines" table, using the driver's licence number - you can enrich that data to learn about Bob's speeding tickets from the past.

Except here, we're wanting to take all the lovely details we have about Goblin Shamans, Hydras and Tyrus into our own card collection.

The top table is my card collection, as pulled from the API. The bottom table is the card details, as pulled from the API. By joining, or matching on the card details, I get an enriched data set of the cards I have.

You can see here that there's now card names, edition details, and a whole bunch more. However, this only works for the cards that I DO HAVE. If I'm missing anything from Untamed, it won't appear in this list. In order to figure this out, I need to calculate the difference - or, the cards that DIDNT match - where there's a card I don't have in my collection at all.

This doesn't really have a name for it, but I like how KNIME calls it an "Anti-Join" - this is the data from the secondary table that wasn't matched. I sort of build a cache to store this infomration in temporarily.

I then split out any gold / non gold cards,after only looking for untamed cards in the collection. That is the "Rule based Row Filter node"

The next step is to COUNT the individual cards I have. This is to determine how many of EACH Untamed card I have, and, if they're combined, how much XP (BCX) they have - to determine further calculations which will enable us to work out how many cards I need to buy to max out my particular card.

This is achieved by the GROUP BY node, which works much like a Pivot Table in Microsoft Excel, except it has dozens of powerful functions and features, and has the added advantage of processing in a fraction of the time.

I want to group by the individual card.

Then, I want to injet some meaning into the card...

I want a SUM of the xps - and the rarity and splinter details, so that I can sort it all later, that's the next tab in the Group By configuration.

The resulting table is as such:

A list of cards, and the XP, which splinter they belong to, and their rarity. Though these may be a combination of individual (1 BCX) and some combined cards - that doesn't matter, as the only thing I'm interested in is how many more BCX do I need to complete a max instance of an individual card.

The next step is to bring in some figures about what it takes to get a card to max level in the Untamed Edition.

I do this by using a subsequent Rule Engine, which explains the max levels:

Now of course, as I indiated earlier, this will only work for cards that are in my collection. For those that are not (those in the so called Anti-Join) - I can make the assumption that my owned quantity is zero, and that I therefore need the maximum amount to get them to the max level.

To find what cards I'm missing, I then concatenate the joined cards and the "Anti-Joined" cards (ie, the missing ones, into one tabulated list)

The output is a bit messy, so I use a few nodes to clean up the data set prior to completing my final calculations...

You can also see at the bottom of this output exactly which UNTAMED cards I am missing in my collection!

The final clean up is as follows - and I've documented it in my KNIME flow itself, as these are rather simple, explanatory nodes with basic configuration - up until the Math Formula Node. \

The Math Formula node is where this all comes together, to generate the final tables and calculations of what will be required to complete the collection at Max Level.

Here's the result of that formula:

I then tidy up that data to make it even more human readable....

From here, I have a list of what I need to complete a Max Level Untamed Collection.

Card Name	BCX Owned	Rarity	Splinter	Cards for Max	Cards to Buy / Sell
Albatross	485	Common	Water	400	-85
Azmare Harpoonist	3	Epic	Water	46	43
Beatrix Ironhand	47	Epic	Earth	46	-1
Biceratops	430	Common	Earth	400	-30
Bone Golem	115	Rare	Death	115	0
Bortus	135	Rare	Water	115	-20
Byzantine Kitty	1	Legendary	Dragon	11	10
Camila Sungazer	12	Legendary	Dragon	11	-1
Cave Slug	731	Common	Life	400	-331
Chain Golem	1	Legendary	Neutral	11	10
Chanseus the Great	11	Legendary	Life	11	0
Child of the Forest	123	Rare	Earth	115	-8
Contessa L'ament	115	Rare	Death	115	0
Coral Wraith	48	Epic	Water	46	-2
Cornealus	2	Legendary	Neutral	11	9
Crystal Jaguar	401	Common	Life	400	-1
Cursed Slimeball	49	Common	Death	400	351
Dark Astronomer	130	Rare	Death	115	-15
Dark Ha'on	3	Legendary	Death	11	8
Darkest Mage	46	Epic	Death	46	0
Death Elemental	115	Rare	Death	115	0
Diamond Dragon	3	Legendary	Dragon	11	8
Dragon Jumper	11	Legendary	Dragon	11	0
Drake of Arnak	115	Rare	Dragon	115	0
Elven Defender	444	Common	Neutral	400	-44
Elven Mystic	18	Rare	Neutral	115	97
Failed Summoner	49	Common	Earth	400	351
Feasting Seaweed	680	Common	Water	400	-280
Ferexia General	7	Epic	Fire	46	39
Fire Elemental	23	Rare	Fire	115	92
Giant Scorpion	527	Common	Death	400	-127
Giant Squid	115	Rare	Water	115	0
Goblin Chariot	23	Rare	Neutral	115	92
Goblin Fireballer	34	Common	Fire	400	366
Goblin Firemage	18	Epic	Fire	46	28
Goblin Thief	632	Common	Earth	400	-232
Herbalist	402	Common	Life	400	-2
High Priest Darius	11	Legendary	Life	11	0
Horny Toad	275	Common	Neutral	400	125
Ice Pixie	7	Rare	Water	115	108
Khmer Princess	160	Rare	Earth	115	-45
Kobold Bruiser	47	Common	Fire	400	353
Kron the Undying	2	Legendary	Earth	11	9
Light Elemental	4	Epic	Life	46	42
Lir Deepswimmer	2	Legendary	Water	11	9
Living Lava	20	Rare	Fire	115	95
Lobstradamus	46	Epic	Water	46	0
Lone Boatman	400	Common	Life	400	0
Luminous Eagle	115	Rare	Life	115	0
Maggots	605	Common	Death	400	-205
Magma Troll	63	Common	Fire	400	337
Mantoid	47	Common	Neutral	400	353
Mimosa Nightshade	2	Legendary	Death	11	9
Mitica Headhunter	1	Epic	Earth	46	45
Mother Khala	115	Rare	Life	115	0
Orc Sergeant	626	Common	Earth	400	-226
Parasitic Growth	1042	Common	Neutral	400	-642
Phantom of the Abyss	2	Legendary	Water	11	9
Pyre	115	Rare	Fire	115	0
Pyromaniac	13	Epic	Fire	46	33
Scale Doctor	1	Legendary	Dragon	11	10
Scarred Llama Mage	2	Legendary	Earth	11	9
Serpent of Eld	16	Rare	Water	115	99
Serpentine Spy	42	Common	Fire	400	358
Shieldbearer	1	Rare	Life	115	114
Sniping Narwhal	30	Common	Water	400	370
Soulstorm	5	Epic	Death	46	41
Spark Pixies	133	Rare	Fire	115	-18
Sporcerer	3	Epic	Earth	46	43
The Vigilator	7	Legendary	Dragon	11	4
Thunderbird	48	Epic	Life	46	-2
Tortisian Chief	23	Epic	Neutral	46	23
Tortisian Fighter	403	Common	Water	400	-3
Tower Griffin	18	Rare	Neutral	115	97
Truthspeaker	127	Rare	Life	115	-12
Undead Badger	477	Common	Death	400	-77
Unicorn Mustang	122	Rare	Earth	115	-7
War Chaang	22	Epic	Neutral	46	24
Wizard of Eastwood	115	Rare	Earth	115	0
Yodin Zaku	3	Legendary	Fire	11	8
Magnor	0	Legendary	Fire	11	11
Gloridax Magus	0	Legendary	Dragon	11	11
Gloridax Guardian	0	Legendary	Dragon	11	11

And I'm all done!

If you want me to calculate how many UNTAMED cards you need to MAX OUT your collection, please upvote and leave your username below. When I get some time, I'll generate a report for you!

Future improvements to the above flow can be made by incorporating current market prices, or looking at historical card versions in much the same way - I just need to build the logic. I've got all the cards I (probably) want from prior versions and editions, so I am completely unmotivated to do the work on that currently...

I hope to see you all on the battlefield!

TLDR: Want me to generate a list of cards you need to max out your Untamed REGULAR FOIL collection?

Drop your in-game username below, and please feel free to upvote or leave me a tip!