کتاب یادگیری تقویتی

چرا یادگیری تقویتی؟

Why Now? • Machine Learning • Reinforcement Learning • When Should You Use RL? • RL Applications • Taxonomy of RL Approaches • Model-Free or Model-Based • How Agents Use and Update Their Strategy • Discrete or Continuous Actions • Optimization Methods • Policy Evaluation and Improvement • Fundamental Concepts in Reinforcement Learning • The First RL Algorithm • Is RL the Same as ML? • Reward and Feedback • Reinforcement Learning as a Discipline • Summary • Further Reading

فرآیندهای تصمیم‌گیری مارکوف، برنامه‌نویسی پویا و روش‌های مونت کارلو

Multi-Arm Bandit Testing • Reward Engineering • Policy Evaluation: The Value Function • Policy Improvement: Choosing the Best Action • Simulating the Environment • Running the Experiment • Improving the ϵ-greedy Algorithm • Markov Decision Processes • Inventory Control • Inventory Control Simulation • Policies and Value Functions • Discounted Rewards • Predicting Rewards with the State-Value Function • Predicting Rewards with the Action-Value Function • Optimal Policies • Monte Carlo Policy Generation • Value Iteration with Dynamic Programming • Implementing Value Iteration • Results of Value Iteration • Summary • Further Reading

یادگیری اختلاف زمانی، کیو-لرنینگ و الگوریتم‌های n-مرحله‌ای

Formulation of Temporal-Difference Learning • Q-Learning • SARSA • Q-Learning Versus SARSA • Case Study: Automatically Scaling Application Containers to Reduce Cost • Industrial Example: Real-Time Bidding in Advertising • Defining the MDP • Results of the Real-Time Bidding Environments • Further Improvements • Extensions to Q-Learning • Double Q-Learning • Delayed Q-Learning • Comparing Standard, Double, and Delayed Q-learning • Opposition Learning • n-Step Algorithms • n-Step Algorithms on Grid Environments • Eligibility Traces • Extensions to Eligibility Traces • Watkins’s Q(λ) • Fuzzy Wipes in Watkins’s Q(λ) • Speedy Q-Learning • Accumulating Versus Replacing Eligibility Traces • Summary • Further Reading

شبکه‌های کیو عمیق

Deep Learning Architectures • Fundamentals • Common Neural Network Architectures • Deep Learning Frameworks • Deep Reinforcement Learning • Deep Q-Learning • Experience Replay • Q-Network Clones • Neural Network Architecture • Implementing DQN • Example: DQN on the CartPole Environment • Case Study: Reducing Energy Usage in Buildings • Rainbow DQN • Distributional RL • Prioritized Experience Replay • Noisy Nets • Dueling Networks • Example: Rainbow DQN on Atari Games • Results • Discussion • Other DQN Improvements • Improving Exploration • Improving Rewards • Learning from Offline Data • Summary • Further Reading

روش‌های گرادیان خط‌مشی

Benefits of Learning a Policy Directly • How to Calculate the Gradient of a Policy • Policy Gradient Theorem • Policy Functions • Linear Policies • Arbitrary Policies • Basic Implementations • Monte Carlo (REINFORCE) • REINFORCE with Baseline • Gradient Variance Reduction • n-Step Actor-Critic and Advantage Actor-Critic (A2C) • Eligibility Traces Actor-Critic • A Comparison of Basic Policy Gradient Algorithms • Industrial Example: Automatically Purchasing Products for Customers • The Environment: Gym-Shopping-Cart • Expectations • Results from the Shopping Cart Environment • Summary • Further Reading

فراتر از گرادیان‌های خط‌مشی

Off-Policy Algorithms • Importance Sampling • Behavior and Target Policies • Off-Policy Q-Learning • Gradient Temporal-Difference Learning • Greedy-GQ • Off-Policy Actor-Critics • Deterministic Policy Gradients • Deterministic Policy Gradients • Deep Deterministic Policy Gradients • Twin Delayed DDPG • Case Study: Recommendations Using Reviews • Improvements to DPG • Trust Region Methods • Kullback–Leibler Divergence • Natural Policy Gradients and Trust Region Policy Optimization • Proximal Policy Optimization • Example: Using Servos for a Real-Life Reacher • Experiment Setup • RL Algorithm Implementation • Increasing the Complexity of the Algorithm • Hyperparameter Tuning in a Simulation • Resulting Policies • Other Policy Gradient Algorithms • Retrace(λ) • Actor-Critic with Experience Replay (ACER) • Actor-Critic Using Kronecker-Factored Trust Regions (ACKTR) • Emphatic Methods • Extensions to Policy Gradient Algorithms • Quantile Regression in Policy Gradient Algorithms • Summary • Which Algorithm Should I Use? • A Note on Asynchronous Methods • Further Reading

یادگیری تمام خط‌مشی‌های ممکن با روش‌های انتروپی

What Is Entropy? • Maximum Entropy Reinforcement Learning • Soft Actor-Critic • SAC Implementation Details and Discrete Action Spaces • Automatically Adjusting Temperature • Case Study: Automated Traffic Management to Reduce Queuing • Extensions to Maximum Entropy Methods • Other Measures of Entropy (and Ensembles) • Optimistic Exploration Using the Upper Bound of Double Q-Learning • Tinkering with Experience Replay • Soft Policy Gradient • Soft Q-Learning (and Derivatives) • Path Consistency Learning • Performance Comparison: SAC Versus PPO • How Does Entropy Encourage Exploration? • How Does the Temperature Parameter Alter Exploration? • Industrial Example: Learning to Drive with a Remote Control Car • Description of the Problem • Minimizing Training Time • Dramatic Actions • Hyperparameter Search • Final Policy • Further Improvements • Summary • Equivalence Between Policy Gradients and Soft Q-Learning • What Does This Mean For the Future? • What Does This Mean Now?

بهبود نحوه یادگیری عامل

Rethinking the MDP • Partially Observable Markov Decision Process • Case Study: Using POMDPs in Autonomous Vehicles • Contextual Markov Decision Processes • MDPs with Changing Actions • Regularized MDPs • Hierarchical Reinforcement Learning • Naive HRL • High-Low Hierarchies with Intrinsic Rewards (HIRO) • Learning Skills and Unsupervised RL • Using Skills in HRL • HRL Conclusions • Multi-Agent Reinforcement Learning • MARL Frameworks • Centralized or Decentralized • Single-Agent Algorithms • Case Study: Using Single-Agent Decentralized Learning in UAVs • Centralized Learning, Decentralized Execution • Decentralized Learning • Other Combinations • Challenges of MARL • MARL Conclusions • Expert Guidance • Behavior Cloning • Imitation RL • Inverse RL • Curriculum Learning • Other Paradigms • Meta-Learning • Transfer Learning • Summary • Further Reading

یادگیری تقویتی کاربردی

The RL Project Life Cycle • Life Cycle Definition • Problem Definition: What Is an RL Project? • RL Problems Are Sequential • RL Problems Are Strategic • Low-Level RL Indicators • Types of Learning • RL Engineering and Refinement • Process • Environment Engineering • State Engineering or State Representation Learning • Policy Engineering • Mapping Policies to Action Spaces • Exploration • Reward Engineering • Summary • Further Reading

یادگیری تقویتی عملیاتی

Implementation • Frameworks • Scaling RL • Evaluation • Deployment • Goals • Architecture • Ancillary Tooling • Safety, Security, and Ethics • Summary • Further Reading

نتیجه‌گیری و آینده

Tips and Tricks • Framing the Problem • Your Data • Training • Evaluation • Deployment • Debugging • ${ALGORITHM_NAME} Can’t Solve ${ENVIRONMENT}! • Monitoring for Debugging • The Future of Reinforcement Learning • RL Market Opportunities • Future RL and Research Directions • Concluding Remarks • Next Steps • Now It’s Your Turn • Further Reading