[diana_eva]

At public's request, Lorand ran another simulation, this time with 6 types of robots, in December 10th Zenith Daylong.

These are the robot scores over 1,920 boards (all of the deal pools in Zenith)

Click for the leaderboard.

Zenith Dec 10th
Gib Advanced = 53.91%
Gib Basic = 52.58%
Argine Advanced = 50.96%
Thinking Ben = 50.32%
Argine Simplified = 48.38%
Instant Ben = 45.64%

Argine is set to play 2/1 (it played 2/1 in the previous simulations too).
The Ben model used is trained on GIB advanced for the bidding, and on ACBL human hands for the play of hand.
Instant Ben just plays based on what the neural network says
Thinking Ben runs some simulations using info from the neural network before making a move.

[pilowsky]

Remarkable how little difference there is between the two GIB's in this more "real" situation.

[mycroft]

Also interesting how it matches my (personally observed, so *so not* statistically relevant) memory that "no matter how bad 'the robots' bid and play, according to real club player humans, it's amazing how, when two robots fill in a half table, they end up 'third or fourth, at 54%'."

[pescetom]

 mycroft, on 2023-December-12, 10:08, said:

Also interesting how it matches my (personally observed, so *so not* statistically relevant) memory that "no matter how bad 'the robots' bid and play, according to real club player humans, it's amazing how, when two robots fill in a half table, they end up 'third our fourth, at 54%'."

May well be so if most or all club players are playing systems similar to Gib 2/1.
My experience is that the robot pair does much worse (low 40s) when they don't understand the opponent's bidding.

[pilowsky]

Just to be clear, in those simulations, was each engine playing South with GIB advanced at N,E and W to simulate the same conditions faced by human Souths in that tourney?

[diana_eva]

 pilowsky, on 2023-December-12, 21:07, said:

Just to be clear, in those simulations, was each engine playing South with GIB advanced at N,E and W to simulate the same conditions faced by human Souths in that tourney?

Yes.

[lorserker]

 pilowsky, on 2023-December-12, 21:07, said:

Just to be clear, in those simulations, was each engine playing South with GIB advanced at N,E and W to simulate the same conditions faced by human Souths in that tourney?

We are doing another run with the same bot playing for both North and South. That setting is better to compare bots to each other. Like how does a pair of gibs compare to a pair of argines to a pair of bens, etc. Making a bot play with another bot as partner puts the bot at a disadvantage (especially true for argine)

[pilowsky]

 diana_eva, on 2023-December-12, 23:10, said:

Yes.

Thanks.

[lorserker]

Hi, an update with a slightly different setting.
Both North and South are played by the same robot, so each robot plays with itself as partner.

Argine Advanced = 53.72%
Gib Advanced = 53.17%
Thinking Ben = 50.77%
Argine Basic = 50.62%
Gib Basic = 49.83%
Instant Ben = 45.47%

This new setup has helped Argine and has hurt Gib Basic.
My interpretation is that Argine has fewer misunderstandings now as she plays with herself as partner (Argine does play 2/1, but a slightly different flavor of 2/1 than gib).
Gib basic's performance dropped because now it has a weaker partner (itself).

[Ditherer]

 diana_eva, on 2023-December-12, 04:24, said:

At public's request, Lorand ran another simulation, this time with 6 types of robots, in December 10th Zenith Daylong.

These are the robot scores over 1,920 boards (all of the deal pools in Zenith)

Click for the leaderboard.

Here is another interesting thing you might do with the data from this experiment: for each human h, calculate Advanced GIB's percentage P_h on the set of 16 boards played by h. What percentage of humans were beaten by the robot when playing the same set of boards? How did the robot's performance vary over sets, e.g., what were the minimum and maximum values of P_h?

[Ditherer]

 diana_eva, on 2023-December-12, 04:24, said:

At public's request, Lorand ran another simulation, this time with 6 types of robots, in December 10th Zenith Daylong.

These are the robot scores over 1,920 boards (all of the deal pools in Zenith)

Click for the leaderboard.

Here is another interesting thing you might do with the data from this experiment: for each human h, calculate Advanced GIB's percentage P_h on the set of 16 boards played by h. What percentage of humans were beaten by the robot when playing the same set of boards? How did the robot's performance vary over sets, e.g., what were the minimum and maximum values of P_h?

[thepossum]

Is it possible to separate the variance in bidding and play?
Not sure how you would do it of course, and maybe very time consuming
Sorry for thinking idly - maybe even how often the contracts were the defining factor

Also pondering annoyingly. I wonder how the likes of Qplus and other 2/1 engines would stack up
- and I may be a contrarian but I like the idea of random pairings of different 2/1 bots - isn't that the point of bidding systems

Maybe a simple interpretative AI interface on each - what does GiB mean lol