• Aishwarya Pothula

Interpretable Reinforcement Learning

Updated: Jan 27

The blog is part of a series of five posts in which I summarize academic papers and answer a few questions pertinent to them. These papers and related questions were given to my by my PhD committee as part of my comprehensive exam.


The paper I will be writing about in this post can be access using the following link.

Interpretable Reinforcement Learning Inspired by Piaget’s Theory of Cognitive Development


Summary


Synopsis

The paper entertains the idea of designing learning systems based on alternatives to connectionism theory such as Language of Though and Schema theory.


Review

The authors of the paper note that majority of the RL learning approaches to model human-level cognition are based on connectionist theory. As an alternative, the authors propose a a basic computational block and and a learning method inspired by Language of Thought (LOTH) theory and Script theory, in an effort to make the learning process more human-like and address some of the issues concerning connectionist theory.

Let's briefly look at a few theories and concepts described in the paper.


Connectionism theory: Thorndike's theory proposes that learning is a result of forming associations between stimuli and response, on which most RL algorithms are based.


Language of Thought (LOTH): To address some of the unanswered issues of Connectionism theory such as productivity, systematicity and inferential coherence, Fodor suggested LOTH. According to this theory, a linguistic structure, mentalese, forms the basis of the workings of the mind

Productivity: It refers to the ability of mind to have multiple distinct representations.

Systematicity: It refers to the mind's dependence on its ability to infer some propositions based on its ability to infer some other propositions

Inferential coherence: It refers to the mind's ability to create new propositions.


Cognitive Theory: "To Piaget, cognitive development was a progressive re-organization of mental processes as a result of biological maturation and environmental experience"[reference]



Schema Theory: The schema theory was introduced by Piaget as part of the Cognitive Development Theory. According to Piaget, "schema is a cohesive repeatable action sequence possessing component actions that are tightly interconnected and governed by a core meaning". He proposed two operations that are conducted by the human mind for learning and inference.

Assimilation: It refers to using existing schemas to understand the what is being perceived

Accommodation: Considering new object or events that do not fit into existing schema representations.


Behaviorism: This theory of learning proposed by Watson highlights the role of external environment in influencing behavior to the point of ignoring innate and inherited factors. This theory talks about reward and punishment from the environment influencing behavior, the idea behind reinforcement learning.


Top Down Learning: In Gregory's theory of top down learning, what is we perceive is influenced not only by what we see, hear, sense but also by our internal knowledge and hypothesis about what we are sensing.


Bottom Up Learning: In contrast to the top down theory, Gibson proposes the bottom up theory which proposes that minimal learning is involved in perception and that it is solely based on sensing the environment.


Perceptual Cycle Model (PCM): Neisser's PCM theory includes both top down and bottom up approaches and proposed learning to be a cycle represented in the diagram below. Neisser also proposed a schema theory which views schema as an organized mental pattern to understand.organize world knowledge.



Proposed Algorithm

The proposed algorithm in the paper is represented through the following algorithms. The algorithm is based on iteratively sampling from the environment, reforming schemas and modifying their weights. All operations in the schemas are based on finding the similarity between the schema options and current state of the environment. Trace length is used to regulate the updation of schema weights.





Strength

Learning in the proposed framework is interpretable, which is a great advantage in comparison with most neural network based algorithms in which their learning process is a black box. The strength of the paper also lies in its effort to find harmonies between different theories of learning to propose a proof-of-concept framework that takes the initial steps towards modeling human-like cognition.


Weakness

When this frameworks are deployed in the real world, the memory requirements to store huge schema structures may be enormous.


Questions


Discuss its use for developmental concepts inside the proposed learning approach. You should focus here mostly on the conceptual, higher level aspects rather than going into the mathematics of the underlying Reinforcement Learning approaches.

I understand the question as how components of the proposed learning approach are inspired by the developmental concepts.


The developmental concept used inside the approach is Language of Thought through cognitive development and schema theory.

The basic computational block is inspired from the schema theory. The computational block is defined as a tree of options. An options is a system's representations of a situation or event. A tree is constructed such that it is lead by a specific goal. Each leaf of the tree refers to an action that can be performed in a single timestep.



Manipulation of the computational block is based on the accommodation operation specified in the Schema theory, as described in Algorithm 2. It involves finding the most similar option for the current state of the system. If the similarity between them is below a threshold, the current state of the system is added as a new option to the schema tree. The weight updation of the tree structure is based on the assimilation operation specified in the Schema theory. The weights of the tree are updated to maximize rewards possible from chosen action at the current state.

Implementing the proposed learning method with developmental theories, alternative to the connectionist viewpoint, has the following benefits. First, the method's learning process is interpretable, which is not the case with most algorithms designed using neural networks with a connectionist view. Second, it addresses some of the unanswered issues in connectionism theory such as productivity, systematicity and cohesive inference.


Discuss potential relations, dependencies, or conflicts between the concept of developmental theory inside the algorithm and potential developmental evaluation schemes that you are looking into. In particular, could there be a potential for evaluation bias depending on whether the same or different concepts or theories of development are incorporated into the the algorithms and the evaluation ? Is there a way to potentially address such bias

I believe such conflicts may not arise. I may reflect on this statement as being naive in retrospect but here is my reasoning as per my current understanding. In my evaluation philosophy, in spite of the theory of learning followed, an agent must be able to acquire certain skills and abilities progressively. Consequently, I include as many evaluation instances as I can based on literature describing milestones in cognitive growth of humans. Since my evaluation approaches are not based on any particular theory of learning, the theory of learning implemented in the learning system should not be very relevant. The evaluations are not designed to test the mechanism of learning, they are designed to test if there is learning and whether it is progressing along the right path.


References


  1. Legg, Shane and Marcus Hutter. “Universal Intelligence: A Definition of Machine Intelligence.” Minds and Machines 17 (2007): 391-444.

  2. Law, James et al. “A psychology based approach for longitudinal development in cognitive robotics.” Frontiers in Neurorobotics 8 (2014): n. pag.

  3. Y. Wu, W. Ji, X. Li, G. Wang, J. Yin and F. Wu, "Context-Aware Deep Spatiotemporal Network for Hand Pose Estimation From Depth Images," in IEEE Transactions on Cybernetics, vol. 50, no. 2, pp. 787-797, Feb. 2020, doi: 10.1109/TCYB.2018.2873733.

  4. C. Doersch, A. Gupta, and A. A. Efros. Unsupervised vi- sual representation learning by context prediction. In ICCV,2015.

  5. D. Pathak, R. B. Girshick, P. Dolla ́r, T. Darrell, and B. Har- iharan. Learning features by watching objects move. In CVPR, 2017.

  6. X. Dong, S.-I. Yu, X. Weng, S.-E. Wei, Y. Yang, and Y. Sheikh. Supervision-by-registration: An unsupervised approach to improve the precision of facial landmark detectors.In CVPR, 2018.

  7. S. Tulsiani, A. A. Efros, and J. Malik. Multi-view consis- tency as supervisory signal for learning shape and pose pre- diction. CVPR, 2018

  8. A.J.Joshi, F.Porikli, and N.Papanikolopoulos,“Multi-classactive learning for image classification,” Computer Vision and Pattern Recognition (CVPR), 2009.

  9. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” International Conference on Machine Learning (ICML), 2009.

  10. G. Hacohen, L. Choshen, and D. Weinshall, “Let’s Agree to Agree: Neural Networks Share Classification Order on Real Datasets,” International Conference on Learning Representations (ICLR), 2020.

  11. Lee, M. H. 2020. How to Grow a Robot: Developing Human-Friendly, Social AI, 1–10

  12. Baldi, Pierre and Laurent Itti. “Of bits and wows: A Bayesian theory of surprise with applications to attention.” Neural networks : the official journal of the International Neural Network Society 23 5 (2010): 649-66

  13. Gemici, Mevlana, et al. "Generative temporal models with memory." arXiv preprint arXiv:1702.04649 (2017)

  14. Piloto, Luis, et al. "Probing physics knowledge using tools from developmental psychology." arXiv preprint arXiv:1804.01128 (2018)

2 views0 comments

Recent Posts

See All

The blog is part of a series of five posts in which I summarize academic papers and answer a few questions pertinent to them. These papers and related questions were given to my by my PhD committee as