Kochi University of Technology Academic Resource Repository

高知工科大学博士（学術）doctoral thesi

谷本, 佑

Japanese

デジタル技術を用いたカンキツ類の出荷量予測に関する生理生態情報の測定技術

朝倉, 菜月

PNN-PZTとエポキシ樹脂からなる 圧電複合材料の特性評価

濵島, 岳

電力未整備地域での観測を目的とした省電力インフラサウンドデータロガーの開発

南部, 七音

熊本市坪井川における舟運計画と川まちづくり

秋山, 将太郎，伊藤, 祐太朗，恩藤, 慎太郎，川添, 崚平，篠原, 由樹，細川, 昌樹，高木, 方隆，村井, 亮介

2番池から南西部既存池における木灰コンクリートを用いた水路および里道の施工

中村, 直人，立田, 理紗

コミュニティ・スクールと学校運営協議会の活性化と教育経営構造の革新に関する一考察～高知県内のコミュニティ・スクールと学校運営協議会の事例研究を中心に～

Weikai Wang

－1－ 
（様式 ６） 
Form 6 
 
2025 年度 博士論文要旨 
Abstract of doctoral dissertation 
submitted in the fiscal year of    2025   
 
論文題目 ゲーム AIにおけるデータ利用の強化：2048ゲームのための深層 
強化学習戦略 
 
英文題目 Enhancing Data Utilization in Game AI : Deep Reinforcement 
Learning Strategies for the Game 2048 
 
学籍番号 1258010 
氏  名 WANG WEIKAI 
(ローマ字表記) ワン ウェイカイ 
 
 
（要旨）Abstract of dissertation 
 
 
Deep Reinforcement Learning (Deep RL) has emerged as a powerful paradigm in the field of artificial 
intelligence (AI), leveraging the representation capabilities of deep neural networks alongside the 
decision-making processes inherent in reinforcement learning. This combination enables agents to 
learn optimal policies directly from high-dimensional sensory inputs, facilitating remarkable 
advancements across various domains, including robotics, natural language processing, and notably, 
game AI. Game AI, in particular, serves as an ideal testbed for the development and evaluation of AI 
algorithms due to its well-defined rules, measurable outcomes, and the inherent complexity that arises 
from strategic depth and stochastic dynamics. 
The successes of DeepMind's AlphaGo and AlphaZero have been pivotal in demonstrating the potential 
of Deep RL agents to achieve superhuman performance in complex strategic games such as Go, Chess, 
and Shogi. These achievements are largely attributed to the agents' ability to effectively utilize vast 
amounts of data to learn sophisticated strategies, outperforming human experts and traditional AI 
approaches. However, despite these advancements, efficient data utilization remains a critical 
challenge in Deep RL, especially in environments characterized by high-dimensional state spaces and 
significant stochasticity. The methods by which agents utilize data significantly impact their ability to 
learn optimal policies and perform effectively, influencing both the learning speed and the stability of 
the training process. 
This thesis focuses on enhancing data utilization methods in Deep RL agents to improve their 
performance in game AI, specifically within the context of the game 2048. The game 2048 is a 
single-player sliding block puzzle that involves combining tiles with the same number to create 
－2－ 
higher-numbered tiles on a 4×4 grid. Despite its simple rules, 2048 is easy to play but hard to master, 
presenting substantial strategic depth due to the randomness of tile placements and the exponential 
growth of tile values. This high level of randomness closely mirrors the unpredictability found in 
real-world scenarios, making 2048 a valuable subject for study. Developing an AI agent capable of 
achieving high scores in 2048 requires sophisticated decision-making and efficient data utilization 
methods to handle these complexities effectively. 
Currently, methods based on N-tuple networks have demonstrated superior performance in the game 
2048 compared to existing Deep RL approaches. This performance gap suggests that there are areas 
where Deep RL methods can be enhanced to better exploit the available data and improve learning 
efficiency. Key areas for improvement in Deep RL approaches include: 
1. Enhanced Utilization of Global Information: Traditional neural network architectures, such as 
convolutional neural networks (CNNs), often focus on local feature extraction due to limited 
receptive fields. This limitation can hinder the network's ability to capture and utilize global 
contextual information essential for strategic decision-making in 2048. Enhancing the 
network's capacity to process global information may lead to better performance. 
2. Reduction of Variance and Inaccuracy in Training Data: The stochastic nature of the game 
introduces significant variance in the training data, which can lead to unstable learning and 
suboptimal policy estimation. Developing methods to generate and utilize more accurate data 
can improve the agent's learning process and performance. 
3. Improved Data Utilization Strategies: By refining the strategies for data utilization, agents can 
more effectively exploit the information available within the game. This includes better 
handling of the inherent randomness and incorporating deeper insights into future game states. 
Focusing on these aspects within Deep RL offers the potential to enhance its performance and possibly 
surpass existing methods in the game 2048. This leads to the central research question: How can 
improved data utilization methods enhance the scoring performance of Deep Reinforcement Learning 
agents in game AI? To address this overarching question, the research focuses on two sub-questions: 
1. Global Embeddings: How does the implementation of global embeddings affect the performance 
of Deep Reinforcement Learning agents in game AI? 
2. Accurate Data Generation and Utilization: How can generating and utilizing more accurate 
data affect the performance of Deep Reinforcement Learning agents in game AI? 
The primary objective of this research is to enhance the data utilization methods in Deep RL agents to 
improve their performance in the game 2048. Specifically, the research aims to: 
1. Enhance Global Information Utilization through Global Embeddings: Develop neural network 
architectures that effectively integrate global embeddings to improve the model's ability to 
utilize global information present in the game state. This addresses the first sub-question by 
exploring the impact of global embeddings on agent performance. 
2. Generate and Utilize More Accurate Training Data: Design advanced training methods focused 
on generating more accurate data and utilizing it effectively to train the agent. This addresses 
the second sub-question by examining how accurate data generation and utilization affect agent 
performance. 
－3－ 
3. Evaluate the Impact of Improved Data Utilization: Conduct comprehensive experiments to 
assess the effectiveness of the proposed data utilization methods, comparing them with baseline 
models to determine their impact on the agent's scoring performance. 
The significance of this research lies in its potential contributions to both theoretical and practical 
aspects: 
• Advancement of Deep RL Methodologies: By focusing on data utilization, the research 
contributes to improving learning efficiency and effectiveness in Deep RL agents, providing 
insights into how agents can better leverage available data. 
• Enhancement of Game AI Performance: Improved data utilization methods can lead to 
higher-scoring agents in 2048 and potentially other games, demonstrating the practical benefits 
of the research in developing more competent AI agents. 
• Broader Applications: The findings can inform data utilization strategies in other domains 
where efficient and accurate data usage is critical, such as robotics, finance, and autonomous 
systems. 
To address the research questions, the study proposes two main methods centered around enhancing 
data utilization: 
1. Global Embedding Techniques 
We introduce global embedding techniques to improve the model's utilization of global information in 
the game state. By embedding global contextual information into the neural network architecture, we 
aim to provide the agent with a more comprehensive understanding of the game environment, leading 
to better decision-making. The implementation involves integrating global embeddings into the 
baseline CNN22 model, resulting in enhanced architectures such as: 
• Bypass CNN (BP-CNN): Incorporates positional information directly into the input layer by 
concatenating positional embeddings, enabling the network to utilize spatial context effectively. 
• Fully Connected Embedding CNN (FC-CNN): Utilizes fully connected layers to create a global 
embedding vector that encapsulates the overall state of the game board, which is then 
integrated with the convolutional layers. 
• Convolutional Embedding CNN (CNN-CNN): Adds convolutional layers dedicated to processing 
and embedding global information, leveraging convolutional operations to capture complex 
patterns across the entire board. 
• Transformer Embedding CNN (T-CNN): Employs a Transformer Encoder to capture global 
dependencies through self-attention mechanisms, effectively modeling the relationships 
between all tiles on the board. 
These models are designed to investigate how the implementation of global embeddings affects the 
agent's performance by enhancing the utilization of global data, directly addressing the first 
sub-question. 
2. Generating and Utilizing More Accurate Data 
We develop advanced training methodologies focused on generating more accurate data and utilizing it 
effectively to train the agent. By reducing variance and incorporating deeper search strategies, we aim 
to improve the stability and quality of the training data, enhancing the agent's learning process. The 
－4－ 
methods include: 
• Temporal Difference Afterstate Full (TD-afterstate-full): Reduces variance in the training data 
by considering all possible future states from a given afterstate, leading to more accurate value 
estimates and better utilization of available data. 
• 2-Ply Temporal Difference Afterstate (2-ply TD-afterstate): Extends the search depth to two 
plies, allowing the agent to account for a broader range of future scenarios and utilize deeper 
information in the data. 
These methods aim to explore how generating and utilizing more accurate data affects the agent's 
performance by enhancing data quality and utilization, directly addressing the second sub-question. 
Experimental Setup 
We conduct comprehensive experiments to assess the effectiveness of the proposed data utilization 
methods. The experimental framework includes: 
• Datasets and Training Procedures: Agents are trained on numerous game episodes of 2048, 
utilizing the proposed architectures and training methods. We implement baseline models for 
comparison, including standard CNN-based Deep RL agents and N-tuple network approaches. 
• Evaluation Metrics: Performance is measured using average scores achieved, highest tile values 
obtained, and convergence rates during training. Statistical significance tests are conducted to 
validate the results. 
• Implementation Details: The models are implemented using standard deep learning 
frameworks, and hyperparameters are tuned for optimal performance. 
The experimental results demonstrate that the proposed methods significantly improve the 
performance of Deep RL agents in 2048: 
• Agents utilizing global embeddings outperform baseline models, achieving higher average 
scores and reaching higher tile values more consistently. 
• The Transformer Embedding CNN (T-CNN) shows the most substantial improvement, 
indicating the effectiveness of self-attention mechanisms in capturing global dependencies. 
• Advanced training methods that generate and utilize more accurate data lead to faster 
convergence and more stable learning processes. 
• Combining global embedding techniques with accurate data generation methods yields the best 
overall performance, highlighting the synergistic effect of enhancing data utilization on multiple 
fronts. 
We analyze the impact of each method on the agent's ability to make strategic decisions, handle the 
stochastic nature of the game, and generalize learned policies. 
This research demonstrates that improving data utilization methods in Deep RL agents leads to 
significant enhancements in game AI performance, particularly in complex and stochastic 
environments like 2048. By effectively integrating global embeddings and generating more accurate 
training data, agents can better leverage the information available, leading to superior decision-making 
and higher scores. 
The findings contribute to the advancement of Deep RL methodologies by providing insights into how 
－5－ 
agents can be designed to process global information and utilize data more efficiently. The implications 
extend beyond game AI, offering potential applications in other domains where efficient and precise 
data usage is critical. 
Future research directions include: 
• Extending the Proposed Methods: Applying the enhanced data utilization methods to other 
complex games and environments to assess their generalizability and effectiveness in different 
contexts. 
• Exploring Additional Neural Network Architectures: Investigating the integration of other 
advanced neural network architectures and training techniques to further enhance data 
utilization and agent performance. 
• Real-World Applications: Exploring the application of the research findings in real-world 
domains such as robotics and autonomous systems, where efficient data utilization and 
strategic decision-making are paramount. 
• Improving Computational Efficiency: Developing more computationally efficient algorithms to 
balance performance improvements with resource requirements, making Deep RL methods 
more accessible and scalable. 
• Enhancing Generalization and Transfer Learning: Implementing approaches that promote 
better generalization and transfer learning, enabling agents to adapt to new environments and 
tasks with minimal retraining. 
In conclusion, this research addresses critical challenges in data utilization within Deep RL, offering 
substantial improvements in agent performance in the game 2048 and laying the groundwork for 
future advancements in AI methodologies across various domains. 
 
 
 
 
 
 
 
 
 
 
 
 
 
（注）論文要旨：内容は4,000字程度とし、電子ファイル及び紙媒体で提出 
The abstract of doctoral dissertation should be approximately 2,000 words and must be submitted in both 
electronic and printed formats. 


English

Enhancing Data Utilization in Game AI: Deep Reinforcement Learning Strategies for the Game 2048

Boyue Song

Interpretability of Deep Neural Networks for Neurological Disorders Diagnosis using rs-fMRI

松本, 正弘

ステレオ画像処理を用いたユズ果実収量予測方法の検討

大原, 高秋

家計調査（加工食品）・品目別支出金額1位の2つのケースに関する考察

Kochi University of Technology

Not a member yet