Exode Testers Report RAIDS

Hello again !! Welcome to the seventh Exode Testers Report RAIDS.

This report will present a basic description of the game play, then some writing about some of the testing issues and finally an attempt to provide a brief technical description of what testing is.

RAIDS

Raids is the first implementation of PvP in Exode. Being the first PvP interaction with Exode cards, there was a lot of work and still is, some work to be done on balance. Testing involved exploring each card and it's specific traits, evaluating if those traits help with combat, implementing an initial combat scenario and checking results. Results being a long text log of each action, for each facet of combat and mission resolution, all done by manual reading of the output data logging.

This game play is PvP. Players are grouped into teams, 2 vs 2 or 3 vs 3, depending on participation. The objective is to be in control of the sectors at the end of the season with enough points to have more than the other team. Each player is presented a screen with two areas to put team members in (cards, like challenges) and a nine sector map. Each sector on the map has a point value, meaning if your team controls it at the end of the season, those points are awarded. A sector can also have a mission, which resolves as is done in challenges.

The player sends in cards which will enable the mission to be successful. At the same time, the team may need a battle configuration to take on incoming or "sitting" teams. The new addition to missions is control points or pips. These are awarded when the mission is successful and the number of pip are the number of times your rolls beat the requirement. In order for the other team to gain control they must beat the pip count of the current controller. Putting this all together, we have combat and possible missions for each of the 9 sectors. In order to gain control you may have to coordinate your team so that it will be able to fight and have skills to complete the mission. This new game play has provided some extra fun for players as they get to compete and discover new ways that their collection can be used. This is also the first game play where players get to interact and plan the actions as a team.

Here, proto26 has written article with screenshots: @proto26/exode-finishes-its-first-pvp-raid-season

Testing began much like challenges, some API calls to invoke various base functions of Raids. A call to generate the Raid, a call to set the teams and the members, a call to send a player's card selections to sectors, a call to resolve the round and a call to save the results of a round. So to set up a test it takes (2 + (2 # players) + 1 + 1 ) 3 calls to finish a Raid season. All tests created were done manually, as I could not figure out a way to automate generation in a reasonable time. Plus the development was taking place as testing was being done. This means that it is hard to produce a test environment for something that is in such flux / change. The current document Raid Rules was the product of 3+ weeks of development and testing. An iterative process with bugs and feedback affecting both development and testing. An important thing to note is that the API calls go around all game UI and there is no filtered card selection for me to choose from. This means I have to know what is valid and only pick valid cards, which is an added complication when writing a test case.

At the start tests were easy to make as the valid card set is limited and basic functions could be tested with 2 vs 2 teams during a first round. During the first round, you do not have to worry about card choices other than adding a card twice. Using 2 vs 2 teams was limiting in some cases where I wanted more teams going into a battle than a 2 man team has. A 3 vs 3 test case for the first round was not too bad to create and gave more coverage than could be done with less teams. Moving a 3 vs 3 test case to the second round showed how complicated it gets. With the cards from the first round not usable in the next rounds, I had to keep track of used cards for each player's two teams. Getting all teams set for the second round took ~5 hours. Getting all the teams set for the third round took ~8 hr. This was quite taxing on my brain and really tired me out. In the end the 3 vs 3 test enabled many checks to be made. Once initially created, it was quite easy to change a couple cards here and there for further testing.

The following is an example of a test. It uses the commands defined in the python test bench, so should be kind of readable by anyone:

SET SEC1 1
SET SEC2 2
SET SEC3 3
SET SEC4 4
SET SEC5 5
SET SEC6 6
SET SEC7 7
SET SEC8 8
SET SEC9 9
SET SCOUT 0
SET STRIKE 1
SET ID combat6
SET_LOG_LEVEL 3
SET_RAID_ID $ID
SET_CHALLENGE_SEED 567987
SET_RAID_EXSEED 1234

GEN_RAID

SET_RAID green sckoarn,blockmonster,raudell
SET_RAID purple doubleozero,veryanprime,tyrnis
-- Raid_id, side, Player, team, Sector, cards
SEND_RAID_PLAYER $ID green blockmonster 0 $SEC2 exode_card_077_SyndicateAyumi,exode_card_065_SyndicateGeisha_ThirdSister,exode_card_074_SyndicateLeader,exode_card_070_SyndicateEquipmentAutoBlaster
SEND_RAID_PLAYER $ID green blockmonster 1 $SEC3 exode_card_081_RebelGeneral,exode_card_079_SyndicateYakuzaSniper,exode_card_080_TheKumicho,exode_card_069_SyndicateShipBlackLotus
SEND_RAID_PLAYER $ID green sckoarn 0 $SEC2 exode_card_077_SyndicateAyumi,exode_card_065_SyndicateGeisha_ThirdSister,exode_card_074_SyndicateLeader,exode_card_070_SyndicateEquipmentAutoBlaster
SEND_RAID_PLAYER $ID green sckoarn 1 $SEC3 exode_card_081_RebelGeneral,exode_card_079_SyndicateYakuzaSniper,exode_card_080_TheKumicho,exode_card_069_SyndicateShipBlackLotus
SEND_RAID_PLAYER $ID green raudell 0 $SEC2 exode_card_077_SyndicateAyumi,exode_card_065_SyndicateGeisha_ThirdSister,exode_card_074_SyndicateLeader,exode_card_070_SyndicateEquipmentAutoBlaster
SEND_RAID_PLAYER $ID green raudell 1 $SEC3 exode_card_081_RebelGeneral,exode_card_079_SyndicateYakuzaSniper,exode_card_080_TheKumicho,exode_card_069_SyndicateShipBlackLotus

SEND_RAID_PLAYER $ID purple doubleozero 0 $SEC2 exode_card_077_SyndicateAyumi,exode_card_065_SyndicateGeisha_ThirdSister,exode_card_074_SyndicateLeader,exode_card_070_SyndicateEquipmentAutoBlaster
SEND_RAID_PLAYER $ID purple doubleozero 1 $SEC3 exode_card_081_RebelGeneral,exode_card_079_SyndicateYakuzaSniper,exode_card_080_TheKumicho,exode_card_069_SyndicateShipBlackLotus
SEND_RAID_PLAYER $ID purple veryanprime 0 $SEC2 exode_card_077_SyndicateAyumi,exode_card_065_SyndicateGeisha_ThirdSister,exode_card_074_SyndicateLeader,exode_card_070_SyndicateEquipmentAutoBlaster
SEND_RAID_PLAYER $ID purple veryanprime 1 $SEC3 exode_card_081_RebelGeneral,exode_card_079_SyndicateYakuzaSniper,exode_card_080_TheKumicho,exode_card_069_SyndicateShipBlackLotus
SEND_RAID_PLAYER $ID purple tyrnis 0 8 exode_card_077_SyndicateAyumi,exode_card_065_SyndicateGeisha_ThirdSister,exode_card_074_SyndicateLeader,exode_card_070_SyndicateEquipmentAutoBlaster
SEND_RAID_PLAYER $ID purple tyrnis 1 9 exode_card_081_RebelGeneral,exode_card_079_SyndicateYakuzaSniper,exode_card_080_TheKumicho,exode_card_069_SyndicateShipBlackLotus

RAID_RESOLVE 1

EXIT

Some of the more interesting issues were "how powerful should an Elite character be?", like Kumicho. When should the transactor explode and how much damage does it do depending on the conditions triggering the explosion. Fireworks is an explosion weapon as well and some proof of explosions was specifically tested. As it turns out, even with characters that are very good, the player collection size has an affect on how really powerful character cards are. The introduction of research and buildings also introduces some variance between players. Due to all of the bonus from collection and research it is very difficult to say that a balance point has been reached. I think it will be very interesting to see more Raid seasons and how the all resolve.

Testing
This may not be for everyone, but some said they could not fully understand what was stated in a previous post.
Testing is the act of proving something is what it is expected to be. This does not mean the proven thing is correct in the end. What??? If the expectations are not correct and it is proven that the object you are testing meets those expectations, then the testing was correct. That may sound crazy or not but, testing is a way to increase the probability that what ever is produced meets some specification or requirement. For instance the universal specification of USB computer interface is something you could use to test against. A known standard with real requirements across a number of technical areas. (electrical, mechanical, data signalling)

In many cases a fully defined specification is not available as the item being developed, is in the development stage, but still needs testing to assist with quality and correctness. This level of testing is more like Development assistant than testing against specifications. In the case of Exode, the development effort is one thing and takes time to create and do very basic testing. After developing something within the Exode environment, we know is random, it is very time consuming to develop and run large numbers of tests on new features. Large numbers of tests are required as the system is random, and because of this, it is very hard to cover everything. Though I have done some testing the current players are also testing through game play. This testing is very valuable as the player is attempting to complete the mission using their card collection. The cards used and choices made by players are good for uncovering any real game play issues.

As an example, the last testing done was to ensure that the cards on a specific list of "special" cards never show as challenge rewards. A simple check was created to test for the condition after each mission generation. Then some 1500 tests were run to see if the issue ever shows up. A small fraction of what might be generated, showed no failures.

I hope you enjoyed this little view into the background testing of the Exode Raids production.

Best.