As I sort my account out in the aftermath of the Rebellion release, I find myself revisiting the top bots of the land. My account has been on ArchMage for the past year+, but a generous promotion by the XBOT team for a month free if you are an ArchMage user got me to try them out.
I have recently been investigating which is better (based on fees, results, etc.) and came to the annoying conclusion that it is really hard to tell. All of the metrics that are easily at hand (e.g., % of leaderboard using XBOT vs. AM) have issues.
So, the point of this post is essentially pre-registering an experiment I'm going to do. Writing all this out now will prevent any claims after the fact of choosing the numbers that make one of the services look better than the other.
The experiment
For the next two seasons, I will make 0 changes to my account (no buying or selling or renting of cards that are part of my Wild deck).
The only thing that will change is that the first season I will use XBOT, and the second season I will use AM.
This will give the cleanest comparison of the two services as it will be using the same set of cards and starting at the same place.
One note is that I will need to note the exact length of each season and adjust accordingly (14 days vs. 14.5 days may be a significant difference).
The outcome metrics
One of the important questions is how to measure success. I will show the main one I will use (total rewards), as well as a few secondary ones I am interested in.
Total rewards
I will count 1) the number of chests of each type (the first few days will be Diamond chests) and 2) the total amount of battle SPS won. I won't count the actual contents of the chests, as that is highly RNG-dependent. Instead, each chest will be valued at the median value for chests of that league.
I will then sum up the value of the chests and the SPS to arrive at a total value over the course of the season for the bot. I will subtract the fees the bot charges (15% of battle SPS and focus chest SPS for AM, 5% of the same for XBOT) to arrive at the net rewards the bot provided over a season.
I especially love this metric because it carries a lot of information; because RP go up as you increase rating, this number will be bigger if one bot vs. another gets you to the same point but faster.
Final rating
I will see which bot ends up at a higher rating. This will be somewhat RNG dependent as it is highly effected by the last few matches of the season, and a lucky winning streak could put one on top.
Performance of Rebellion cards
I will calculate 1) the total number of matches at least 1 rebellion card is used and 2) the rate of victories given at least 1 rebellion card is used. I want to see which bot is more likely to use the newer cards and is more effective at doing so.
Any other metrics?
See a metric you'd like to see added? Let me know! I'd love to add things to this list, as these were just the ones off the top of my head.
Shoutouts and referral
Special thanks to the devs of XBOT and AM: for making these bots that have kept me more engaged in the game than I would be without them, for providing a free trial that got me to try XBOT, and for engaging in discussions on effectiveness that inspired this post.
If you're interested in seeing more of this type of analysis or of getting in on the action, use my referral link below and I'll get you started with some cards!