Hey everyone, today I want to talk about a few technical problems we had with sending structure files between clients and servers.
This post is going to be a bit more technical so, strap your seatbelts on.
First of all some explanations:
What are Structures?
In Minecolonies workers live in buildings made of of Minecraft blocks.
How those buildings look like is defined in so-called "Structure files" which use the Minecraft structure mechanism.
There are different styles of buildings and players can scan in all kinds of things which can be saved in those structure files.
What do we do with them?
The main worker of the Colony, the builder, is responsible for parsing those structure files and placing them for the player.
The player can select which structure he wants to be built at whatever place and the builder will build it for him.
This works with structure files which are only on the physical server and not on the client and also with structure files which are only on the client but not on the physical server.
What is the problem?
The problem is that there is a message size limit of 32kb from the client to the server (to prevent the client from spamming the server with huge messages for example).
Besides that, we had a similar limit from the server to the client.
What did I do?
I basically had two problems to solve.
Client - Server
Server - Client
Let's start with the easier one:
Server - Client:
Previous to this PR, players were able to scan arbitrarily huge structures which in some cases even caused crashes on the server.
For this reason, I added a max size in blocks which I set to 100k blocks for the time being.
Then, as I had written previously, the client-server communication was off.
The issue here was that we didn't compress the files before sending them after scanning to the client.
We didn't do that because when sending a message in bytes and we want to apply additional data to it, we run into some bad issues parsing it out later again.
The Idea I had was wrapping all the data in the compound and then storing and compressing altogether.
This way the data integrity is guaranteed without having to fiddle around with the byte arrays later.
Which makes writing simply by wrapping the data together and then compressing it.
And reading the same thing basically.
Client - Server:
The more difficult to do, was the Client-Server communication, due to the size limit.
The only reason therefore was, to cut the byte array into pieces and send it piece for piece and fiddle them back together on the other side.
This will leave us with two main pieces of code.
- Breaking the byte array into parts
Which is all of this.
Let's break it into small pieces.
First, we get the byte-stream on the client side from the file.
Then we generate a unique ID for the structure sending process and then we break the stream into a byteArray.
Then we check if we even have to cut it into pieces, meaning if it is too big to be sent.
Because then we can just send it right away.
If it is too big, we calculate the number of pieces we need.
Then we iterate through the number of pieces and calculate the start of the byte array each time. (Each iteration we take a max size chunk size of it).
If this is the last iteration of the loop, the size won't be the max size anymore because now we're dealing with the remaining part of it.
And then, finally, we copy over the part of the byte array which is of our interest and send it in a message.
- Fiddling it back together
On the other side, I created a hashmap which will take care of all those files we receive on the server side.
It has the unique ID of the structure we defined earlier and for each unique ID, we have a tuple which contains a timestamp (nano time, as a long) and a map of pieces by their piece id (1-20).
First of all, each time we receive a piece we check the remaining pieces timestamps and remove them if necessary to avoid any memory leaks.
Then once again, if it is only 1 piece, proceed as previously.
Then we check if the map has the key already.
If so, we retrieve the tuple and check if it is too old as well.
If not too old we check if we already received that piece.
Then we put the piece into the map of the structure and check if we now have all pieces.
If so we create a byte-stream and write each of these pieces in order (sorted by the key of the map) into the output stream.
Then we easily handle the schematic with the bytearray we get from the output stream.
If we don't have the piece yet, or if we don't have all yet. We store the pieces back into the hashmap and wait for the remaining ones.
I hope you enjoyed the more backend/technical pr this time.
I'll see you the next time!
Posted on Utopian.io - Rewarding Open Source Contributors