Continual Learning after Model Deployment
- Learning on the Job
Learning Continuously During Model Application
A form of Lifelong Learning
"Lifelong Machine Learning."
by Z. Chen and B. Liu, Morgan & Claypool, August 2018 (1st edition, 2016)
Continual Learning Dialogue Systems - Learning on the Job after Model Deployment. Tutorial @ IJCAI-2021, August 21-26, 2021, Montreal, Canada.
- Three new chapters have been added and others have been updated and/or reorganized.
- One Chapter is dedicated to Open World Learning
- Any AI system (e.g., chatbot and self-driving car) that cannot learn in deployment (e.g., chatting and driving) in the real-world open envoronment is not truly intelligent.
Continual Learning Dialogue Systems - Learning after Model Deployment. Invited talk @ ICLR-21 Workshop on Neural Conversational AI, May 7, 2021.
Learning on the Job in the Open World. Invited talk @ ICML-2020 Workshop on Continual Learning, July 17, 2020.
Lifelong Machine Learning Tutorial. Title: lifelong machine learning and computer reading the Web, KDD-2016, August 13-17, 2016, San Francisco, USA.
Lifelong Machine Learning Tutorial, IJCAI-2015, July 25-31, 2015, Buenos Aires, Argentina.
Motivation: It is known that about 70% of our human knowledge comes
from “on-the-job” learning. Only about 10% is learned through formal
education and the rest 20% is learned through observation of others
(imitation). An autonomous AI system must have this capability - its
machine learning algorithm must be able to learn on the job or while
working after model deployment. As the real world is too complex and constantly changing, it
is impossible to learn everything through offline training using manually
labeled data. An autonomous AI agent has to explore and learn by
itself in the real world, which is open and constantly change - full of
unknowns. AI agents must be able to detect the unknowns and
learn them in a self-supervised manner through its interaction with humans, other agents, and the real-world environment. It should not make
the closed-world assumption any more. Here are two motiviating examples:
- Greeting bot in a hotel: At any point in time, the bot
has learned to recognize all existing hotel guests. When it sees an
existing guest, it can call him/her by his/her first name and chat (e.g., Hi John, how are you today?). It must also detect any new guests that it
has not seen before. On seeing a new guest, it can say hello, ask for
his/her name (e.g., "Welcome to our hotel. What is your name, sir?"), take
pictures, and learn to recognize the guest. Next time when it sees the
new guest again, it can call him/her by his/her first name and chat
like an old friend.
- Self-driving car: I worked on self-driving car for a year. Once when we were
doing a road test, the car suddenly stopped and refused to move.
The road was completely clear and we could not see anything wrong.
Debugging back in the lab found that there was a small stone on the
road detected by a sensor. This made me think: why cannot the car tell us
what the problem was in natural language? Why cannot we tell the car to
go ahead also in natural language? Our instruction serves a piece of
supervisory information to enable the car to learn so that in the future
when a simialr situation occurs, it can behave correctly.
On-the-job learning: Like human on-the-job learning, it studies
learning after model deployment or during model application (or testing) - after a good model
has been built and deployed in an application.
In classic machine learning, once a model is built, it is
deployed in an application. During application,
the model remains fixed or unchanged. On-the-job learning investigates
continuous learning after model deployment. Specifically, apart from
performing its task, it should
(1) continuously discover new tasks by the agent itself,
(2) gather training data through the agent’s own active effort via interaction with humans, other agents and the environment, which we call interactive self-suprevision and
(3) incrementally learn the new tasks during model application without interrupting the application to adapt the model to the new environment and to become more and more knowledgeable. This is also called open-world learning, but here we emphesize interaction with humans and the environment to find new tasks and to naturally label training data (see the two motivating examples above).
Interactive Self-Supervision: Step 2 is the key for on-the-job
learning, i.e., how to find the hidden classes and obtain labeled training
data. This must be done through actions initiated by the system itself
without interrupting the application. That is, it must learn actively on
its own based on its prior knowledge by observing and interacting with
the environment and humans to obtain explicit or implicit feedback to serve
as supervision. The interaction with humans should be through natural
language dialogues. It is not possible for an autonomous intelligent agent
to rely solely on massive manual labeled training data to learn passively
TextBook: Zhiyuan Chen and Bing Liu. Lifelong Machine Learning. Morgan & Claypool, 2018 (2nd edition), 2016 (1st edition).
- Bing Liu and Sahisnu Mazumder. Lifelong and Continual Learning Dialogue Systems: Learning during Conversation. to appear in Proceedings of AAAI-2021. 2021.
- Sahisnu Mazumder, Bing Liu, Shuai Wang, and Sepideh Esmaeilpour.
An Application-Independent Approach to Building Task-Oriented Chatbots with Interactive Continual Learning. to appear at NeurIPS-2020 Workshop on Human in the Loop Dialogue Systems (HLDS-2020). 2020.
- Sahisnu Mazumder, Bing Liu, Nianzu Ma, Shuai Wang. Continuous and Interactive Factual Knowledge Learning in Verification Dialogues. to appear at NeurIPS-2020 Workshop on Human And Machine in-the-Loop Evaluation and Learning Strategies (HAMLETS-2020). 2020.
- Bing Liu and Chuhe Mei. Lifelong Knowledge Learning in Rule-based Dialogue Systems. arXiv:2011.09811 [cs.AI], 2020.
- Bing Liu. Learning on the Job: Online Lifelong and Continual Learning. Proceedings of 34th AAAI Conference on Artifical Intelligence (AAAI-2020), Feb 7-12, 2020, New York City. (This work was done while I was on leave in Peking University).
- Sahisnu Mazumder, Bing Liu, ShuaiWang, Nianzu Ma. Lifelong and Interactive Learning of Factual Knowledge in Dialogues. to appear in Proceedings of Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL-2019), 11-13 September 2019, Stockholm, Sweden.
- Hu Xu, Bing Liu, Lei Shu and P. Yu. Open-world Learning and Application to Product Classification. to appear in Proceedings of the Web Conference (formerly known as the WWW conference), San Francisco, May 13-17, 2019.
- Lei Shu, Hu Xu, Bing Liu. Unseen Class Discovery in Open-world Classification. arXiv:1801.05609 [cs.LG], 2018.
- Lei Shu, Hu Xu, Bing Liu. DOC: Deep Open Classification of Text Documents. Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP-2017, oral presentation short paper), September 7–11, 2017, Copenhagen, Denmark.
- Geli Fei, Shuai Wang, and Bing Liu. 2016. Learning Cumulatively to Become More Knowledgeable. Proceedings of SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2016), August 13-17, San Francisco, USA.
- Geli Fei, and Bing Liu. 2016. Breaking the Closed World Assumption in Text Classification. Proceedings of NAACL-HLT 2016 , June 12-17, San Diego, USA.
Created on July 15, 2020 by Bing Liu.