Continual Learning after Model Deployment
- Learning on the Job

Learning Continuously During Model Application
A form of Lifelong Learning

Second Edition: "Lifelong Machine Learning." by Z. Chen and B. Liu, Morgan & Claypool, August 2018 (1st edition, 2016)
Continual Learning Dialogue Systems - Learning on the Job after Model Deployment. Tutorial @ IJCAI-2021, August 21-26, 2021, Montreal, Canada.
Continual Learning Dialogue Systems - Learning after Model Deployment. Invited talk @ ICLR-21 Workshop on Neural Conversational AI, May 7, 2021.
Learning on the Job in the Open World. Invited talk @ ICML-2020 Workshop on Continual Learning, July 17, 2020.
Lifelong Machine Learning Tutorial. Title: lifelong machine learning and computer reading the Web, KDD-2016, August 13-17, 2016, San Francisco, USA.
Lifelong Machine Learning Tutorial, IJCAI-2015, July 25-31, 2015, Buenos Aires, Argentina.

Motivation: It is known that about 70% of our human knowledge comes from “on-the-job” learning. Only about 10% is learned through formal education and the rest 20% is learned through observation of others (imitation). An autonomous AI system must have this capability - its machine learning algorithm must be able to learn on the job or while working after model deployment. As the real world is too complex and constantly changing, it is impossible to learn everything through offline training using manually labeled data. An autonomous AI agent has to explore and learn by itself in the real world, which is open and constantly change - full of unknowns. AI agents must be able to detect the unknowns and learn them in a self-supervised manner through its interaction with humans, other agents, and the real-world environment. It should not make the closed-world assumption any more. Here are two motiviating examples:

On-the-job learning: Like human on-the-job learning, it studies learning after model deployment or during model application (or testing) - after a good model has been built and deployed in an application. In classic machine learning, once a model is built, it is deployed in an application. During application, the model remains fixed or unchanged. On-the-job learning investigates continuous learning after model deployment. Specifically, apart from performing its task, it should (1) continuously discover new tasks by the agent itself, (2) gather training data through the agent’s own active effort via interaction with humans, other agents and the environment, which we call interactive self-suprevision and (3) incrementally learn the new tasks during model application without interrupting the application to adapt the model to the new environment and to become more and more knowledgeable. This is also called open-world learning, but here we emphesize interaction with humans and the environment to find new tasks and to naturally label training data (see the two motivating examples above).

Interactive Self-Supervision: Step 2 is the key for on-the-job learning, i.e., how to find the hidden classes and obtain labeled training data. This must be done through actions initiated by the system itself without interrupting the application. That is, it must learn actively on its own based on its prior knowledge by observing and interacting with the environment and humans to obtain explicit or implicit feedback to serve as supervision. The interaction with humans should be through natural language dialogues. It is not possible for an autonomous intelligent agent to rely solely on massive manual labeled training data to learn passively offline forever.


         TextBook: Zhiyuan Chen and Bing Liu. Lifelong Machine Learning. Morgan & Claypool, 2018 (2nd edition), 2016 (1st edition).

Created on July 15, 2020 by Bing Liu.