The problem was that everybody seems to know how to implement Hoki's learning method except me. Of course I had read the papers, but I still didn't have a full grasp of the method in such a way that I could program it. The initial plan was to use the summer to try and implement Hoki's learning method, but at that time it became clear that I would start a new job from April 1st 2009 and work just seemed to snowball from there. During the summer, I generated a new opening book and set up a new evaluation function for Spear using hashed patterns, but there was no time to try and implement Bonanza's learning method.
After the summer, things only got worse and when the deadline for entering the CSA tournament arrived, I actually exchanged some E-mails with Jeff Rollason ending with the conclusion that it might be best if we both skipped the tournament. In the end, we both entered, but already seriously thinking about cancelling the entry later. There just seemed to be no point this year.
Then early February, just a few days before my 42nd birthday, I got an early present: the Bonanza source code was released! This was really a gift, because what I already suspected turned out to be true: the inner workings of Spear and Bonanza are very similar, in general based on the Crafty chess program. Therefore, for me the Bonanza source code was very easy to understand. Because of this, it took me less than two weeks (not full time, just a couple of hours in the evening) to get the learning code to work. As a result, I had more than two months to optimize the new evaluation function I had made. This got me excited, because I was now in full preparation of moving and the idea of Spear getting stronger while I was packing was almost too good to be true.
Unfortunately, things didn't go as well as hoped. The learning programming did its job, but the evaluation function I had made turned out to be unable to cover all the important aspects of shogi positions. The Bonanza method is based on the assumption that evaluation function features can be updated in such a way that the program will learn to select the moves that professionals played, but even after long learning sessions there were just too many position where the professional move was not selected by Spear. I had a version of Spear that just used the evaluation features of Bonanza and this version kept beating the other versions into the ground. This was the situation just a couple of days before moving from Yamagata to Tokyo (end of March). What to do?
In the end, I decided to further tune the original evaluation features of Bonanza, tweaking some of the learning parameters. After a few weeks of learning, this improved the program slightly over the original Bonanza evaluation features. I also added some of the hashed patterns for king defence to the evaluation and this was the version I entered in the tournament.
I had hoped for some more originality in my program and I also didn't like the fact that Spear played the Anaguma castle so often. As a shogi player, I don't like the Anaguma at all, because I think it is played by cowards who prefer not to lose than to win. My personal record (as a human) against the Anaguma is very good. The main reason is probably that when my opponent plays Anaguma, I more than anything want to win to show that this is an inferior castle. There is nothing like making one tokin after another to take away the Anaguma generals one by one (but I digress). When Spear was playing one game after another building an Anaguma, it felt like my son decided to enter a religious sect instead of going to medical school. This is not how I raised my shogi program!
Still, winning heals all wounds and I decided to enter the strongest new version, even though it kept playing Anaguma all the time. I also kept the evaluation function learning going until the morning the CSA tournament started.
Most participants were very happy about the new location being in Tokyo. For example, Jeff Rollason also had one of his cheapest visits to the tournament this year. Because of the weak pound, the plane ticket was cheap and because he could stay with me for the tournament, there were almost no extra costs. This was a big difference with the Kazusa Arc, which costs about 15,000 Yen a night. The disadvantage of the new location was that the playing halls were much smaller than at the Kazusa Arc, which meant that only a limited number of visitors were allowed. A former student of mine (and former participant with the program Avante) Yoshinori Higashiuchi had interest in visiting the tournament in person, but could only get permission for the first two days, not for the day of the final.
Full attention for one's program at all levels.
This year there were 42 participants, a number that has been quite stable over the past few years. However, there were some notable programs absent. First of all, last year's runner up Tanase Shogi was not among the participants. Tanase didn't feel his program was ready for this tournament and he decided to skip it. Also, Nara Shogi, which finished in seventh place last year didn't enter this year. And just a couple of days before the tournament, the program Kiri (seeded for the second day) cancelled its entry. As a result, instead of eight programs, nine programs would qualify for the second day in the first preliminary stage.
There were also a number of new or returning entries that caught the attention. First of all, KCC Shogi was back after a three year absence. I thought there were political reasons why the North Korean based KCC Shogi skipped the tournament for a while. As far as I know, these political reasons still exist, so I don't really know why it entered this time. In any case, its record in the CSA tournament is very impressive so it was immediately considered to be a favourite.
The second program that seemed very strong was the new program Monju, which had been playing on the floodgate server in the weeks before the tournament and had been about as strong a Bonanza, beating all the top programs there. Finally, the program Blunder also seemed a strong new program. On the floodgate server, its results were not as impressive as those of Monju, but it seemed to have a good chance of becoming a seeded program for the second preliminary round. When I made these assessments, I didn't know that Blunder was playing on the floodgate server on a very slow machine. I would find this out the hard way...
TACOS programmer Hashimoto trying some last-minute improvements.
Jeff and I took it easy on the first day, arriving at the tournament venue at about noon. KCC and Monju had already established themselves, winning their first three games. Also unbeaten was Yamada Shogi, which was a small surprise. It is a program that regularly makes it to the second day without making much impact there. Still, it seemed like it was improved considerably this year.
I ran some final tests on the learned evaluation features that I was planning to use in the tournament (tuning had finished that morning), sharing the room with Hashimoto (TACOS) and Otsuki (Otsuki Shogi), who were also doing some final tuning. Hashimoto seemed far from confident that TACOS would do well this year and Otsuki just left it all in the middle when asked how his program was doing. (I later heard that he had played his program under the names "test1" and "test2" on the floodgate server. The results were quite good, but not as good as for example the results of Monju.) I was also told that Kakinoki had already given up hope of reaching the finals this year. Kakinoki is famously pessimistic about the chances of his own program, but it was clear that the release of the Bonanza code had changed the perspective completely.
Going back to the first qualification round: KCC Shogi basically wiped the floor with everybody. Only the game against Monju was reasonably close. KCC Shogi's play was so impressive that people already starting talking about the possibility of winning every single game through three days, which had never been done before. In any case, KCC Shogi seemed to have returned with a vengeance.
Second place was for Monju, losing only against KCC Shogi. Seeing it in action, it became clear why Monju was so strong. It was running six Bonanza programs on three 8 processor machines (each Bonanza running on 4 processors). Each Bonanza program had a slight random distortion in its evaluation function. In the end, the move to play would be decided by a majority vote between these six Bonanza programs. A very simple idea, but surprisingly effective. Tests showed that Monju could win about 58% of the games against the original Bonanza. By the way, one of Monju's biggest supporters was Bonanza programmer Hoki, who generously gave a lot of advice about using his program in the weeks before the tournament. Also interesting was that one of the three machines running the Bonanza versions was actually a little more expensive (read: a little faster) than the other two. I wonder how that would have influenced the overall strength...
Third place was for Blunder, which only lost against KCC Shogi and Monju, but won all other games. Blunder was also using Bonanza's learning method, but at this stage of the tournament it was still unclear how strong it really was. The program Yumeki finished fourth, which was a big improvement from last year when it finished 13th. It also only lost against KCC Shogi and Monju, so it would be really tested on the second day.
The other programs who qualified from the first qualification round were veteran programs. WILDCAT, Yamada Shogi, Garyu, Auau Shogi have been around for a while, each qualifying for the second day multiple times. The final program to survive the first selection was ym-Shogi. Last year it finished 16th as a first year entry, so this was an important improvement. Still, it was lucky that Kiri cancelled at the last moment, or it would not have made the cut.
No. Program Name 1 2 3 4 5 6 7 Pt SOS SB MD 1* KCC Shogi 4+ 3+ 16+ 2+ 6+ 9+ 8+ 7.0 31.0 31.0 22.0 2* Monju 11+ 19+ 8+ 1- 5+ 4+ 3+ 6.0 32.5 25.5 18.0 3* Blunder 10+ 1- 5+ 22+ 8+ 6+ 2- 5.0 31.5 18.5 12.0 4* Yumeki 1- 10+ 23+ 13+ 11+ 2- 7+ 5.0 29.5 16.5 11.0 5* WILDCAT 12+ 15+ 3- 7+ 2- 13+ 6+ 5.0 29.0 18.0 11.0 6* Yamada Shogi 13+ 7+ 18+ 16+ 1- 3- 5- 4.0 30.0 13.0 6.0 7* Garyu 17+ 6- 14+ 5- 12+ 10+ 4- 4.0 28.0 14.0 7.0 8* Auau Shogi 24+ 21+ 2- 9+ 3- 11+ 1- 4.0 28.0 10.0 6.0 9* ym-Shogi 19+ 11- 21+ 8- 18+ 1- 15+ 4.0 25.5 10.5 5.5 10 Narikin Shogi 3- 4- 12+ 14+ 22+ 7- 19+ 4.0 25.0 11.0 5.5 11 Mattari-Yuuchan 2- 9+ 24+ 18+ 4- 8- 14+ 4.0 25.0 10.0 6.0 12 Tohske 5- 14+ 10- 17+ 7- 21+ 13+ 4.0 24.0 11.0 6.0 13 ponanza 6- 17+ 20+ 4- 16+ 5- 12- 3.0 26.5 8.5 3.0 14 Nanoha 15+ 12- 7- 10- 17+ 18+ 11- 3.0 25.0 9.0 3.0 15 Hakusa Shogi 14- 5- 17- 20+ 19+ 16+ 9- 3.0 23.0 8.0 2.5 16 Oki 23+ 20+ 1- 6- 13- 15- 21+ 3.0 23.0 6.0 2.0 17 GAsyou!!! 7- 13- 15+ 12- 14- 22+ 23+ 3.0 20.0 6.0 1.5 18 Suzuki Shogi 20+ 23+ 6- 11- 9- 14- 24+ 3.0 19.0 4.0 1.5 19 Daemon Shogi 9- 2- 22= 21+ 15- 24+ 10- 2.5 20.5 2.0 0.0 20 HIT Shogi 18- 16- 13- 15- 24+ 23= 22+ 2.5 15.0 1.5 0.0 21 Hayabusa 22+ 8- 9- 19- 23+ 12- 16- 2.0 20.5 3.0 0.0 22 Tohu 21- 24+ 19= 3- 10- 17- 20- 1.5 19.0 0.0 0.0 23 Inoue Shogi 16- 18- 4- 24+ 21- 20= 17- 1.5 18.5 0.0 0.0 24 Mangu Shogi 8- 22- 11- 23- 20- 19- 18- 0.0 19.0 0.0 0.0 * KCC Shogi, Monju, Blunder, Yumeki, WILDCAT, Yamada Shogi, Garyu, Auau Shogi and ym-Shogi qualify for the second qualification round.
Ym-shogi programmer Yamada happy with being the final program to go through to the second day.
It would have been a good idea to try and fix this opening book problem, but it was not trivial. I feared that I might break something in the process, so I left the book as it was. After all, I figured that there was not much chance of getting the same opening again, as Spear would pick moves randomly from a reasonably large number of alternatives in this position. I would later regret this line of thinking...
Spear again didn't make it to the final. Slight disappointment showing?
So, this tournament ended for Spear the same way as the previous two years: 5 wins and 4 losses in the second round. This time this meant 9th place, a slight improvement over last year's finish. Considering the high level of play in this group, this was a reasonable result, but I had hoped for a little more. Well, there is always next year...
Kinoa programmer Yamada Genki showing programmer's fatigue.
What happened to the other programs on this second day? Well, KCC Shogi just bulldozered its way to the finals, winning all nine games. Talk of winning this tournament by winning 23 straight games started to get stronger. It seemed that the closest KCC had come to trouble was the game against Spear, a clear indication of how dominating it was on the day.
Surprising runner-up was Monju, losing against Shueso and KCC Shogi (again), but winning every other game, including Otsuki Shogi, Kakinoki Shogi, K-Shogi, Dragon's Egg and GPS Shogi. Especially the win against GPS Shogi would become something to be proud of.
Third place was for GPS Shogi, which put in a solid performance, losing only to KCC Shogi and Monju, but winning all its other games. GPS Shogi is one of the under-achievers in the CSA tournament. It has a team of very good young programmers, but this time was only the second time it qualified for the finals. Not a surprise, really, because the new version of GPS was doing very well on the floodgate server (which the GPS team created, by the way).
KCC Shogi, Monju and GPS Shogi were the dominating programs on the second day, but there were two more spots in the final left. For a long time it looked like Shueso would easily claim one of these spots. The program started with five consecutive wins, including a win against Monju, but then lost three games in a row, including a game against Kakinoki Shogi it probably should have taken. Because of its strong opponents, it was paired in the final round against A-Class League, which must have been a relief. It won the game, finishing fourth and claiming a place in the final.
The luckiest program of the day was Otsuki Shogi. It started strong with three wins, but then lost against KCC and Monju and also dropping the eight round game against GPS Shogi. In the end, it didn't have its faith in its own hands. It had to beat Dragon's Egg in the final round and hope that K-Shogi would beat Blunder. If Blunder would have beaten K-Shogi, it would have finished higher than Otsuki Shogi. However, both Otsuki Shogi and K-Shogi won, and Otsuki Shogi was the last program to make it to the final. Otsuki Shogi finished with the same number of points as K-Shogi, but because K-Shogi had "walked the low road" (losing early and beating relatively weak opponents after that) Otsuki qualified based on Sum of Opponent Score.
Special accolades go to first year entry Blunder, beating former finalists Bingo Shogi and Dragon's Egg, just missing a place in the finals. Dragon's Egg did more or less as expected, playing for a place in the final until the last game. Disappointing results for Kakinoki Shogi (although Kakinoki himself was happy that his program had a theoretical chance of making it to the final until it lost the last game), Bingo Shogi and especially TACOS, which was expected to do much better. Yamada Shogi had its best result ever and clinched a seeded position, something that Shotest wasn't able to do. Still, Jeff was quite happy that even this version of Shotest running on a slow machine was able to win four games at this level. With some massive improvements in both soft- and hardware (very likely ready for next year's tournament), Shotest could be a program to be reckoned with again next year. Still, it will have to deal on the first day with the likes of Tanase Shogi, Nara Shogi and Kiri.
The Otsuki team had reason to smile after just barely making the final round.
No. Program Name 1 2 3 4 5 6 7 8 9 Pt SOS SB MD 1 KCC Shogi 9+ 12+ 17+ 5+ 3+ 4+ 2+ 7+ 10+ 9.0 49.0 49.0 38.0 2 Monju 6+ 21+ 10+ 4- 5+ 8+ 1- 14+ 3+ 7.0 51.0 36.0 26.0 3 GPS Shogi 20+ 14+ 11+ 15+ 1- 16+ 4+ 5+ 2- 7.0 48.0 32.0 23.0 4 Shueso 12+ 9+ 8+ 2+ 7+ 1- 3- 10- 13+ 6.0 51.0 30.0 19.0 5 Otsuki Shogi 23+ 16+ 7+ 1- 2- 11+ 14+ 3- 8+ 6.0 47.0 24.0 18.0 6 K-Shogi 2- 8- 24+ 13- 20+ 12+ 9+ 15+ 7+ 6.0 37.0 21.0 16.0 7 Blunder 14+ 20+ 5- 11+ 4- 9+ 8+ 1- 6- 5.0 49.0 22.0 14.0 8 Dragon's Egg 21+ 6+ 4- 10+ 15+ 2- 7- 13+ 5- 5.0 46.0 22.0 13.0 9 SPEAR 1- 4- 18+ 19+ 13+ 7- 6- 20+ 12+ 5.0 45.0 19.0 12.0 10 Kakinoki Shogi 24+ 17+ 2- 8- 14- 21+ 11+ 4+ 1- 5.0 43.0 18.0 12.0 11 Bingo Shogi 22+ 13+ 3- 7- 18+ 5- 10- 17+ 15+ 5.0 42.0 19.0 12.0 12 Yamada Shogi 4- 1- 21+ 14- 19+ 6- 22+ 16+ 9- 4.0 44.0 14.0 7.0 13 A-Class League 15- 11- 19+ 6+ 9- 17+ 16+ 8- 4- 4.0 43.0 18.0 8.0 14 TACOS 7- 3- 23+ 12+ 10+ 15+ 5- 2- 18- 4.0 43.0 14.0 8.0 15 Kinoa Shogi 13+ 22+ 16+ 3- 8- 14- 19+ 6- 11- 4.0 42.0 15.0 8.0 16 My Move 19+ 5- 15- 18+ 17+ 3- 13- 12- 23+ 4.0 38.0 13.0 8.0 17 Misaki 18+ 10- 1- 22+ 16- 13- 24+ 11- 20+ 4.0 37.0 10.0 6.0 18 Usapyon2 17- 24+ 9- 16- 11- 20- 21+ 23+ 14+ 4.0 29.0 8.0 4.0 19 Shotest 16- 23+ 13- 9- 12- 24+ 15- 21+ 22+ 4.0 28.0 7.0 4.0 20 Yumeki 3- 7- 22- 24+ 6- 18+ 23+ 9- 17- 3.0 35.0 5.0 1.0 21 WILDCAT 8- 2- 12- 23+ 22+ 10- 18- 19- 24+ 3.0 33.0 4.0 1.0 22 ym-Shogi 11- 15- 20+ 17- 21- 23+ 12- 24+ 19- 3.0 28.0 4.0 1.0 23 Auau Shogi 5- 19- 14- 21- 24+ 22- 20- 18- 16- 1.0 31.0 0.0 0.0 24 Garyu 10- 18- 6- 20- 23- 19- 17- 22- 21- 0.0 33.0 0.0 0.0 * KCC Shogi, Monju, GPS Shogi, Shueso and Otsuki Shogi qualify for the finals.
The second round pretty much went the same way as the first round. GPS Shogi beat Gekisashi, Monju beat YSS and KCC Shogi lost again, this time against Otsuki Shogi. Only Bonanza showed why it was a seeded program as it beat Shueso. These results meant that after only two rounds last year's champion Gekisashi would already need a miracle to defend its title. KCC Shogi's dream of winning the CSA crown also went up in smoke after only two games.
Humans get used to the amazing very quickly, because the third round results seemed suddenly quite natural. GPS Shogi beat Bonanza, Otsuki Shogi beat YSS and Monju beat KCC Shogi. Again, only one seeded program won as Gekisashi beat Shueso. This meant the third straight loss for KCC Shogi, against an opponent it had already beaten twice in this tournament. Time to ask the operator if this was just a coincidence. It turned out that it probably wasn't. It seemed that on the evening before the final round, a new version was sent from North Korea, supposedly fixing some problem with the use of time in the opening. The operator was asked to play the new version in the final, but that probably was a big mistake. Even when the operator suggested to change back to the previous version after losing in the first round, he was told to keep using the new version. Only after three straight losses, the North Koreans seemed to be coming to their senses and allowed the operator to change back to the program that had been so successful on the previous days.
Monju running 6 Bonanza programs on 24 processors.
In the fourth round more of the same. GPS Shogi beat YSS, Otsuki Shogi beat Gekisashi, KCC Shogi beat Bonanza and Monju beat Shueso. So, after this round GPS Shogi, Otsuki Shogi and Monju all had four wins, Bonanza, KCC Shogi, Gekisashi and Shueso had one win and YSS was at the bottom of the table with four losses. The CSA tournament had become predictable again, but this time in completely reversed order! Whatever would happen next, we would have a brand new champion because either GPS Shogi, Otsuki Shogi or Monju would win this tournament.
Round 5 was huge for GPS Shogi. It beat Otsuki Shogi and because Monju lost against Bonanza, it suddenly found itself leading the tournament on its own. There seemed to be some moral justice in a single Bonanza beating six Bonanzas, but when thinking a little about it, it was not really David beating Goliath. After all, the Monju Bonanzas were only running on 4 processors each, while the "real" Bonanza was running on eight. A faster machine without randomness beating slower versions with randomness doesn't seem that unnatural. At the bottom of the table YSS lost again, this time against Gekisashi. With games against Bonanza and KCC Shogi left to play, former champion YSS seemed in serious danger of losing all its games.
Even this special cooling device didn't help YSS.
The sixth round had leader GPS Shogi playing Shueso, which wasn't making much impact in this final round. Most spectators (including me) closely followed the game between Otsuki Shogi and Monju, because if Monju would lose and GPS Shogi would win, GPS would be the new champions with one round left to play. Otsuki Shogi indeed beat Monju and when Shueso resigned a few minutes later we had a new champion: GPS Shogi. A great team effort, again by a group from prestigious Tokyo University (like former champions IS Shogi and Gekisashi). At the bottom of the table also some drama. YSS won its first game, but needed some help to do it. For some inexplicable reason, Bonanza crashed during the game with YSS and lost the game by default.
The winning GPS team with Yonenaga Kunio.
With the champion decided and also the seeded programs decided (Otsuki Shogi and Monju) this exciting tournament ended a bit on a low. The only question that remained was if GPS Shogi would be able to break through the CSA jinx that a champion crowned before the final round will lose its final game. This has happened four times in the past (in the 12th, 13th, 14th and 18th CSA championships). Only Gekishashi managed to win all seven games in the 15th CSA tournament. Monju kept the jinx alive by convincingly beating GPS Shogi in the final round.
No. Program Name 1 2 3 4 5 6 7 Pt SOS SB MD 1 GPS Shogi 4+ 6+ 5+ 7+ 2+ 8+ 3- 6.0 22.0 17.0 10.0 2 Otsuki Shogi 5+ 4+ 7+ 6+ 1- 3+ 8+ 6.0 22.0 16.0 10.0 3 Monju 6+ 7+ 4+ 8+ 5- 2- 1+ 5.0 23.0 14.0 7.0 4 KCC Shogi 1- 2- 3- 5+ 8+ 6+ 7+ 4.0 24.0 7.0 3.0 5 Bonanza 2- 8+ 1- 4- 3+ 7- 6+ 3.0 25.0 8.0 2.0 6 Gekisashi 3- 1- 8+ 2- 7+ 4- 5- 2.0 26.0 2.0 0.0 7 YSS 8- 3- 2- 1- 6- 5+ 4- 1.0 27.0 3.0 0.0 8 Shueso 7+ 5- 6- 3- 4- 1- 2- 1.0 27.0 1.0 0.0
Results before the final round: all the top programs losing!.
In general the level of the programs has improved considerably since last year. GPS Shogi is estimated to have a rating that is among the 50 best players at the biggest Internet shogi site: Shogi Club24. There are quite a number of professional players anonymously active on this site, so it seems likely that GPS Shogi is already stronger than a number of professional players.
We may get a better indication of the strength of the top program next year. This year there was no exhibition match between computers and human players, but as JSR chairman Yonenaga announced at the party afterwards: it seems very likely that a professional will play a computer at a special event in 2010. Sponsorship is going to be a problem in the current recession, but it seems likely that this event will really take place.
GPS programmer Kaneko explaining to professional shogi player Sato Amahiko.
Another interesting aside was that only a few programs had actually used the code that Bonanza made public in February. Only Monju was using the full code, and most programmers even seemed to take some pride in not even reading the Bonanza code. I read the code and I can assure everyone that there are quite some gems in there that I didn't have time to add to Spear but that I will use for next year's version. As said, Spear's code is very similar to Bonanza, so this is not a difficult task.
I am sure that next year a lot of other programs will be using at least part of the Bonanza code. Bonanza will become the standard and everybody can build its own ideas on top of it. This opens the door for different programs and it will be interesting to see how this new wave of "Bonanza Children" will have an impact on next year's tournament. I truly hope that Spear will be one of the program benefiting from Hoki's generosity.
Yonenaga Kunio doesn't seem too worried about the threat to professional shogi players yet.
It is unclear how much impact this would have had in the actual tournament. On the floodgate server most programs play on hardware that is significantly slower than the hardware used at the CSA tournament. Furthermore, because the new Bonanza version was released so close to the CSA tournament, there would have been no time for trying any improvements, both in the learning or otherwise. Spear would have basically been a Bonanza clone, and playing it would be quite strange considering the fact that I vowed last year not to do like Bonanza at all costs.
So, maybe it was better not to know about the new version. I can now use this as a baseline and take some time until the next tournament to try to improve Spear. After the tournament Hoki said that computer shogi programming should be left to students, but after entering the tournament for 13 times, I still have hopes of making it to the final one day. Even at 42, this dream is very much alive!