[diana_eva]

We periodically run simulations to see how different types of robots perform in games originally played by humans. By replaying these games with robots, we can evaluate their performance relative to the human field and to each other.

The robots competing in these simulations are Basic GIB, Advanced GIB, BBO's Ben AI and the latest candidate for a Ben upgrade, which we fondly call "Big Ben".

We plan to provide regular updates on these simulations, either monthly or once every few months, as well as after major robot upgrades. We'll kick off this thread with the latest results from our November simulations.


About the Robots

GIB is the classic BBO bridge robot, first integrated into BBO in July 2005. GIB, which stands for Ginsberg's Intelligent Bridgeplayer, was created by Matt Ginsberg. Over the years, BBO has maintained and refined GIB, with a focus mainly on updating the 2/1 system it plays.

• Basic GIB is the most used bridge robot on BBO -- you can try it in all our free robot games and as a substitute when a player leaves mid-hand; you can recognize from the name BasicGIB 2/1.
• Advanced GIB uses simulations during close decisions, in addition to the programmed rules, and analyzes multiple possible outcomes to choose the best course of action. If you see AdvGIB 2/1 on BBO, it's the advanced GIB robot -- you will have it as your partner and opponent in almost all premium games, as well as when you register with a robot partner in pair games.
• While both versions of GIB use the same bidding system, the difference lies in how much they "think" before acting.

Ben (short for Bridge ENgine) is a machine learning-based robot developed by Lorand Dali, who is now part of the BBO team. BBO's version of the Ben AI was trained on hundreds of millions of deals played by humans on BBO. You can play with Ben at any time by visiting Robot World, and then "Try our AI Bridge Engine".

Big Ben is an enhanced version of Ben, with more training on the bidding and numerous improvements to the play engine. 'Big Ben' will soon be introduced as an upgrade to the Ben AI. You can read more about Ben and its variations here.

[diana_eva]

November Highlights: Robot Performance at a Glance


Below are the BBO robot results in various games played in November 2024.

• Advanced GIB consistently outperformed the field, achieving the highest average matchpoint scores in nearly every game format. It stands out most in the Just Declare formats, where the focus is on declarer play.
• BBO Big Ben appears to be almost as good as Advanced GIB, and even surpassed it in specific games. Its results in the Zenith Daylong Reward — one of BBO's most popular and competitive formats — highlights the significant improvements made to its bidding and play, compared to the 'simpler' BBO Ben AI.
• Basic GIB continues to hold its own as a reliable and solid bridge partner, with above-average performance across most simulations.
• BBO Ben, our first AI child, excelled in massive free games like the Free BBO Super Sunday Daylong, where it remained competitive with Advanced GIB and Big Ben.

[johnu]

The like button is turned off for posts from BBO staff, so I'm posting this just to like your post.

Interesting results for the robots. Not much different results for Advanced and Basic GIB from previous comparisons, but Big Ben seems to be a noticeable improvement over Ben. Ben seemed very weak based on some of the earlier bug reports in this forum, so it's encouraging that Big Ben is scoring almost as well as advanced GIB.

[lorserker]

Updated the robot in Ben & Friends to be the new improved Ben

[pilowsky]

Is there an update log?
Do the improvements include bidding the same way in response to the same bids?

[diana_eva]

pilowsky, on 2024-November-19, 15:47, said:

Is there an update log?
Do the improvements include bidding the same way in response to the same bids?

If you mean for Ben, no release notes yet, as it's still in Beta, but I added the beta release to the Changelog section here: https://news.bridgeb...out-ben-on-bbo/

[pilowsky]

diana_eva, on 2024-November-19, 16:53, said:

From the link above.

Efficiency

• ...
• Simulation algorithm allows to do a lot of simulation during bidding (e.g basic GIB doesn't use any simulation at all).

Is this the reason for the variations in bidding during auctions that are otherwise identical?

[benellis58]

It's amazing that the GIB robot has these results, since it plays a very poor system, can't "think", doesn't signal, has abominable "judgment", seems to treat all spot cards interchangeably, is an incredibly poor leader, is not only a wildly untalented bidder but also a ridiculously erratic and inconsistent one, frequently misdefends atrociously, and can't even declare very well. I may have lived a sheltered existence, but I cannot ever remember having seen even a single human bridge competitor who played as consistently and constantly poorly in all facets of the game as the GIB robots.

[smerriman]

benellis58, on 2024-November-20, 21:48, said:

It's amazing that the GIB robot has these results, since it plays a very poor system, can't "think", doesn't signal, has abominable "judgment", seems to treat all spot cards interchangeably, is an incredibly poor leader, is not only a wildly untalented bidder but also a ridiculously erratic and inconsistent one, frequently misdefends atrociously, and can't even declare very well. I may have lived a sheltered existence, but I cannot ever remember having seen even a single human bridge competitor who played as consistently and constantly poorly in all facets of the game as the GIB robots.

I assume you've never played in the Main Bridge Club on BBO. GIB has its flaws, but I disagree that it is inconsistent; for those who have learnt its system - most humans in the MBC are far worse.

[benellis58]

Stephen, you're correct in assuming that I have never played in the Main Bridge Club on BBO, but I have played "real" bridge F2F for decades, and in that milieu I have never...ever...seen a human as completely and consistently incompetent as the GIB robots. If, as you say, "most humans in the MBC are far worse", then those poor souls have my deepest sympathy, because it is virtually impossible for me to imagine anyone playing worse than a GIB robot on a regular basis.

You're also VERY correct when you say that "GIB has its flaws". Lord KNOWS that you are correct with that statement!

I'm well aware that you are very knowledgeable about GIB and I always look forward to and appreciate your comments, but based on the many thousands of hands I have played in the robot world, I must disagree with you on the issue of robotic inconsistency. I will concede, however, that they are so abysmally weak in so many other areas that the inconsistency is one of their less egregious failings (bad as it nonetheless is). When I speak of their inconsistency, by the way, I am referring more to their inconsistency within a single auction than to inconsistency regarding their (very poor) system.

[johnu]

benellis58, on 2024-November-20, 21:48, said:

I may have lived a sheltered existence, but I cannot ever remember having seen even a single human bridge competitor who played as consistently and constantly poorly in all facets of the game as the GIB robots.

Yes, you have led a sheltered existence. Or maybe just have a bad memory. Have you ever played in a novice game, e.g. ACBL game with limit of 0-20 masterpoints?

Yes, there are actual world class players who play in some of the BBO games. They are a very small minority. The vast majority of players are very bad players. Some are new players who have only played online a short time, or maybe in home games. Others may have played a long time but they basically have never improved after years of playing.

Have you played in a robot tournament? For the most part, just bidding a game and taking all your tricks is like a 60-70% matchpoint score. A few people will end up in a ridiculous contract. Others will fail to take the obvious tricks.

[lorserker]

benellis58, on 2024-November-20, 21:48, said:

It's amazing that the GIB robot has these results, since it plays a very poor system, can't "think", doesn't signal, has abominable "judgment", seems to treat all spot cards interchangeably, is an incredibly poor leader, is not only a wildly untalented bidder but also a ridiculously erratic and inconsistent one, frequently misdefends atrociously, and can't even declare very well. I may have lived a sheltered existence, but I cannot ever remember having seen even a single human bridge competitor who played as consistently and constantly poorly in all facets of the game as the GIB robots.

I suggest you do an experiment.
Challenge GIB a few times (under Challenges->Challenge a robot). The basic GIB is free to challenge.
See how you fare. You may find it hard to beat.
If you can consistently beat it, try challenging in "Just Declare" mode. That is even harder to beat.
I regularly do this to practice.
Then try to challenge your friends who are much better than the robot. See how you fare against them.
(Irrelevant to the experiment, you may also challenge me, I would be happy to play)

My impression is that GIB is a pretty strong player. Yes, it plays very differently from humans.
With Ben we tried to build a robot that is supposed to be more "human". I invite you to try it and see what you think.

[lorserker]

pilowsky, on 2024-November-19, 15:47, said:

Is there an update log?
Do the improvements include bidding the same way in response to the same bids?

Yes, I have fixed the randomness.