Creating Strategy with AI
Gen AI
In a previous post, we talked about the Q-Learning model used in a game where players are to reach a goal cell without hitting a wall.
The backend logic of its workings is essentially strategy generation where an agent needs to make decision in a dynamic environment to maximize its long-term rewards. In this post, we will take a deeper dive into Q-learning in an interactive Tic-Tac-Toe situation.
In the code chunk above, __init__(self, alpha=0.5, gamma=0.95, epsilon=0.1) initializes the Q-learning algorithm with the learning rate alpha, discount factor gamma, and exploration rate epsilon.
Alpha is the learning rate that controls the impact of new information on the Q-values during the learning process
Gamma is the discount factor that determines the importance of future rewards relative to immediate rewards. A gamma value close to 1 indicates future rewards are considered to be important, and setting it closer to 0 places more emphasis on immediate rewards
Epsilon is the exploration rate that controls the balance between exploration and exploitation in the Q-learning algorithm. A higher epsilon value encourages more exploration, while a lower epsilon value favors exploitation of the learned Q-values, as the epsilon value represents the probability of the agent selecting a random section instead of one with the maximum Q-value (exploitation)
The code section that starts with update(self, old_state, action, reward, new_state, new_action) updates the Q-values based on the observed reward and the transition from the old state-action pair to the new state-action pair. A state can be represented by the current board configuration. Each position on the board can have one of three possible values: ‘X’, ‘O’, or blank. Therefore, there are 3^9=19,683 possible states in Tic-Tac-Toe.
The code section that starts with choose_action(self, state, actions) starts by checking if a random number between 0 and 1 is less than the exploration rate (epsilon). If it is, the agent takes a random action from the available actions using np.random.choice(actions). This random action allows the agent to explore different possibilities and learn from the outcomes.
On the other hand, if the random number is greater than or equal to epsilon, the agent selects the action that has the highest Q-value for the given state. It does this by first retrieving the Q-values for each action from the q dictionary using self.q.get((state, action), 0.0), where 0.0 is the default value if the state-action pair is not present in the dictionary. It then returns the action that has the highest Q-value determined by np.argmax(q_values).
This Q-learning algorithm combined with the algorithm that constructs the TicTacToe game environment and the user interactive feature makes the TicTacToe game in Python operational.
Please feel free to have a try yourself!
AIQ by Nick Polson and James Scott
Deep learning represents nothing less than a revolution in the visual competence of machines - and the core ideas and technology are diffusing everywhere. The limits here were once the availability of data, the speed of our computers, and the richness of our models. Today, there seems to be only one limit, and that’s the imagination of people.
CodeChat
CodeChat will continue this month and to be held on June 30 at 5pm EST. The topic is going to be on how to train a generative AI model, joiners’ perspectives related to the Generative AI, as well as any questions you may have related to coding. You may sign up here for a meeting reminder or the meeting link is here if you would like to join directly.
Feedback
The Substackers’ message board is a place where you can share your coding journey with me, so that we can exchange ideas and become better together.
Please open the message board and share with me your thoughts!