[diana_eva]

We ran a simulation this month and we had 5 different types of robots replay December's Stars & Platinum Robot Individual to see how they do versus this strong field of human players. The monthly Stars & Platinum Robot Individual is a 3-day tournament that takes place at the beginning of each month. It's 16 boards every day, matchpoints, just-declare. Only BBO stars and players with at least 5 platinum acbl masterpoints can join.

The robots played all 320 hands available in the tournament. In a more realistic simulation, the robots would have played 48 boards each, like the players who participated.

The robots are:

• GIB Advanced
• GIB Basic
• Argine advanced
• Argine basic
• "ACBL-Ben" - a version of Lorand Dali's Ben robot trained on GIB for the bidding and on hands taken from ACBL human games for the play (more about that later).

You can see how the robots scored here.

For those who don't know, we're happy to announce that Lorand Dali has joined BBO's team this year.

Disclaimer: as this tournament is "just-declare", it only measures the play of hand, which put the "acbl ben" model at a disadvantage, as the play of hand is its weaker part. It plays like the average acbl field and, as per Lorand, it doesnt "think" too much. So take this experiment with a grain of salt, but we still thought it was fun to do.

Disclaimer 2: Lorand is experimenting with other variations of Ben, including one he calls 'the thinking Ben', which are likely stronger than this particular model. These various 'Bens' are not yet close to being implemented in BBO, and there is no intention to replace GIB with Ben. However, we believe that these experiments contribute to improving BBO's robots, and go a long way towards making bridge robots stronger, faster, and more likeable.

A summary here, if you don't want to click around through the tournament results:

| Rank      | MP  | Score (%) | Robot Type   | Robot Details                      |
|-----------|-----|-----------|--------------|------------------------------------|
| 60/618    | N/A | 56.30 	| GIB Advanced | Advanced GIB Robot         		|
| 93/618    | N/A | 55.23 	| Argine   	| Argine Robot               		|
| 123/618   | N/A | 54.29 	| GIB Basic    | Basic GIB Robot                    |
| 232/618   | N/A | 51.42 	| Argine Basic | Simplified Argine Robot            |
| 486/618   | N/A | 39.47 	| ACBL Ben 	| Ben model trained on ACBL human deals |

[pilowsky]

Or to put it another way:
Percentile.
10 GIB Advanced
15. Argine
20. GIB Basic
38. Argine Basic
79 ACBL Ben

Meaning GIB Adv is in the 10th percentile of card play compared to human experts (notwithstanding the opinion that some in the Forum have of robot tournaments).
Not sure what to conclude from that.

Could the same experiment be conducted using hands dealt to the top half of a Zenith field which might represent a decent representation of strong players bidding, declaring, and defending?

[smerriman]

pilowsky, on 2023-December-10, 21:24, said:

Not sure what to conclude from that.

It appears to show the advanced Argine is a considerably better player than GIB, when you consider that in Just Declare, every hand was bid by GIB using GIB's system, while Argine didn't have the remotest idea what half the bids would have meant and still nearly tied the result.

Though you couldn't know this for sure without repeating the other half of the experiment where all hands were bid by Argine.

[pilowsky]

smerriman, on 2023-December-10, 21:43, said:

It appears to show the advanced Argine is a considerably better player than GIB, when you consider that in Just Declare, every hand was bid by GIB using GIB's system, while Argine didn't have the remotest idea what half the bids would have meant and still nearly tied the result.

Though you couldn't know this for sure without repeating the other half of the experiment where all hands were bid by Argine.

Given that GIB's bid only bear a loose relationship to reality, it's surprising that any of the Declarer bots had any idea what was going on.
In any event, the percentile scale is non-linear - normally distributed.
This means that 10th percentile is substantially better than 20th and that 30th is closer to 20th than 20th is to 10th.

ATS, it's a great experiment and I hope we see more of them along these lines.

[pescetom]

diana_eva, on 2023-December-10, 18:05, said:

Lorand is experimenting with other variations of Ben, including one he calls 'the thinking Ben', which are likely stronger than this particular model. These various 'Bens' are not yet close to being implemented in BBO, and there is no intention to replace GIB with Ben. However, we believe that these experiments contribute to improving BBO's robots, and go a long way towards making bridge robots stronger, faster, and more likeable.

Thanks for the interesting information and the transparent approach.
I'm very glad to read that Lorand Dali is on board, and I hope that you *will* commit to replace GIB with Ben or something better, or in alternative revert to fixing the many well documented problems of GIB.

[smerriman]

pescetom, on 2023-December-12, 17:05, said:

Thanks for the interesting information and the transparent approach.
I'm very glad to read that Lorand Dali is on board, and I hope that you *will* commit to replace GIB with Ben or something better, or in alternative revert to fixing the many obvious and well documented problems of GIB as I retain is due.

With Ben learning how to bid from GIB, I don't think that would ever be a viable replacement. I agree it would be nice if GIB was replaced with something else, but for me that's solely because then BBO might be more open to releasing GIB's code