At the third CCAI Conference, Masashi Sugiyama, a new leader in artificial intelligence and machine learning from Japan, gave us a wonderful speech on weakly supervised machine learning, which is a rare voice in machine learning from Japan.
As director of riKEN’s Advanced Intelligence Research Center, Sugiyama will be one of the best-known scholars in the field. He has published many important theories in the field of machine learning, and his book, Illustrated Machine Learning, has long been translated into Chinese.
In order to give readers more information, AI Science Camp interviewed Sugiyama exclusively to discuss the details of weakly supervised learning and its implementation, as well as his latest research directions and achievements. Sugiyama also talked about the current state of AI and robotics research in Japan. enjoy!
| AI base of science and technology (rgznai100)
Edit | JeyZhang
AI Tech Base: You gave us the “AI Tech Base”Advances in weakly supervised machine learningCan you briefly talk about what weakly supervised learning is?
Sugiyama: Machine learning based on big data is a hot topic today. It is indeed a very important research direction, and its technology has been well applied in some fields. But in some areas, we sometimes don’t get enough data. At RIKEN Data Research In Japan, we work in areas such as health care, infrastructure and natural disasters, where it is also difficult to access large-scale data, but we still want to use machine learning to improve these areas.
All of these are hot topics, but there’s another hot topic: weakly supervised machine learning. We know that it’s not always easy to do machine learning on small data because of the statistical error of machine learning
Is proportional (where N represents the amount of data), so the smaller the amount of data, the greater the error. If there is not enough prior data on the target task, it is difficult for us to do the task well. However, as researchers in the field of machine learning, we want to have a universal, universal approach to solving problems on tasks that we see as independent of each other.
The approach we propose is to use machine learning on low cost data acquisition, and the simplest example is semi-supervised learning. Suppose we have a small amount of labeled data and a large amount of unlabeled data. Unlabeled data comes at almost no cost, thanks to the development of the Internet and sensor technology. Semi-supervised learning has been an area of research for the last 15 years, and there have been some fairly successful studies, and there have been some that seem less successful, but I think the ones that seem less successful are just not promising yet.
We looked at a number of different Settings this time. One is that we have two kinds of semi-supervised data, and sometimes this is a puzzle, and sometimes we can solve this problem very well without prior data. The other is PU learning, that is, when we only have positive example data and unlabeled data, it is often difficult to obtain this type of data.
AI Tech Base: Did you say that learning on PU data sets is better than PN data sets?
Sugiyama will: It depends on the ratio of positive and negative samples. One problem we consider is that the acquisition cost of both positive example data and unlabeled data is relatively low, while the acquisition cost of relevant data (PN data, positive and negative example data) is relatively high. In this case, if we have a large number of positive and negative example data, PN learning efficiency is not high; But if you have a lot of positive examples and unlabeled data, PU learning will be better than PN learning. This point has been verified in some theoretical research experiments.
You mentioned semi-supervised learning. Are semi-supervised learning and weakly supervised learning the same thing?
Mr. Sugiyama: Weakly supervised learning is a much broader concept. There are supervised and unsupervised learning approaches, and we’re interested in the technology in between. Semi-supervised learning is usually used in specific scenarios where there is a lot of unlabeled data and a little positive and negative example data. Semi-supervised learning is a well defined term, while weakly supervised learning is more general, and there should be three different approaches to weakly supervised learning.
AI Tech Base: I know that the cost of acquiring annotated data is very high, and this is a problem for most researchers. As you mentioned, some of them are trying to solve this by migrating learning and crowdsourcing data. May I ask why you chose to use weakly supervised learning method to solve it? Can you tell us the story behind this?
Sugiyama: As a researcher, I know that the field of weakly supervised learning is very important and has potential, and this is my motivation to study it. Of course, the crowdsourcing method of data in practice is very good and I am also interested in it. However, in the medical field, for example, medical data is usually not allowed to be uploaded on the Internet, so we cannot use the method of data crowdsourcing to obtain. The data is not universal, but very private. I have some industrial friends in Japan, and the relevant data of industrial projects cannot be uploaded on the Internet, so it is also impossible to crowdsource data. Over the past few years, I’ve worked in a variety of areas, some of which can be crowdsourced with data, and some of which can’t, so we have to try other approaches to deal with problems that can’t be solved with other approaches.
AI Tech Base: Early in your academic career, why did you choose machine learning as your field of study?
Satoshi Sugiyama: When I was a college student majoring in computer science, I became more interested in programming implementation. This was in the mid 90’s, and I was into programming and IT applications, but IT was just a hobby, and IT was hard for us to become programming professionals.
Programming was fun, but it wasn’t professional in a way, so I thought about doing some math research, which seemed professional, but I was still a college student so I didn’t know the details of the research. Some hobbies, such as photography and music, will always be my hobbies. But I wanted to be more professional, so I changed my major to the application of mathematics in computer science. I still majored in computer science, but I paid more attention to the application of mathematics. I was less interested in artificial intelligence and more interested in making computers more useful. At first, in Japan, we had a lot of cartoons about robots, and these cartoons grew up with us. We were already familiar with these intelligent robots, so it was natural that I chose this direction. But sometimes, I’m more interested in the math.
AI technology base: I remember there is a figure of speech PPT, the x axis is linear model, the kernel function model and deep learning model, on the y axis for supervised learning and a semi-supervised learning, we feel very novel, because we believe that the deep learning model is a typical supervised learning, which is why the annotation data acquisition cost is so high and must be the reason. You said that deep learning and unsupervised learning can also be combined. This is a new concept. Could you talk more about it?
Mr. Sugiyama: In a sense, deep learning is a very vague concept. The researchers’ point: Use deep models. But as I said, the X-axis is the model axis, and the deep learning model is one of the models. Models should be combined with learning methods, which are completely orthogonal, so there is supervised deep learning, unsupervised deep learning, and deep reinforcement learning. As long as we have new learning methods that can be combined with deep learning models, this should be called deep learning. At present, deep learning often only talks about deep learning models, in fact, models and learning methods are part of the research field of deep learning.
AI Tech Base: What can we do if your proposed weakly supervised learning approach matures and works? What can we do if an algorithm or method “grows”?
Gen. Sugiyama: At the RIKEN Center, we focus on research in the fields of medicine, management, infrastructure, natural disasters, etc. For these domains, however, we need machine learning approaches that can learn from small data. So, at least in these areas, there are hundreds of applications that could use our proposed approach.
AI Tech Base: Where are you looking for technology applications? Which is the most promising? Is it possible to talk about something?
Sugiyama: In terms of research applications, I can’t really say much at this point. We have a partner, but we haven’t decided yet. Personally, I’m a machine learning researcher, and I can talk about my part of the application that has been done with partners. We are doing medical diagnosis, such as cancer or dementia prediction and evaluation. We have some industrial partners in the financial sector, but it’s more like we’re contributing to our own projects.
AI Tech Base: So do you do the groundwork for them?
Will Sugiyama: At the RIKEN Center, we are more interested in public issues, such as healthcare, natural disasters and infrastructure management, which we also work on with some industrial partners. Some of the partners are also public, such as hospitals and natural disaster research centers. At the same time, some partners in industry have their own projects, and we contribute our own technology to promote them. Therefore, we hope that the application of basic machine learning techniques will help them in the future.
AI Tech Base: CSDN is currently the largest Chinese language community for developers in China, with around 12 million developers. The point is that they want to use AI technology to build the next generation of applications for different industries. However, most of them do not understand the underlying mathematics and prefer to use some off-the-shelf tool libraries and software packages. Is your team creating libraries and tools that people can use?
Sugiyama: Personally, I’ve created some simple MathLab toolkits and put them online. After that, the corporate partners I work with can download it and test it, but there are only a few smaller prototypes. When they really want to use our technology in their business, then they need to develop specialized software, that’s what they need to do. This is actually a good question: should we develop packages that can be used in industry? Right now, I’m not sure, because it’s not easy to decide, because we’re not sure which framework we should use.
AI Tech Base: Do you really think these tools can be used as a black box to get information, like a “magic” box?
Sugiyama: I can’t say “magic”, but we already offer packages for deep learning environments, such as PU learning, which are already online.
AI Tech Base: For Chinese developers, we wonder what AI development is like in Japan. Can you describe it to us?
Sugiyama: It’s not easy to describe the whole situation, but AI is very hyped in our country too! There’s a lot of interest in AI, so it’s very similar to China. We don’t have many AI engineers and researchers in Japan. We have a thriving robotics industry, but the number of researchers is not growing as much as it should be, and the same is true for machine learning. In China, most of the top students go to the United States to study, and then many of them come back to China to start their own companies, which are quite active. In Japan, the number of such students is very low because there are fewer teenagers and they are more interested in other fields such as medicine.
AI Tech Base: If we have Chinese students who are interested in studying in Japan, what advice would you give them?
Sugiyama: This is a great opportunity and we are very willing to accept people who work or study with us. At least in RIKEN, we have received some applications from Chinese PhD students, and even some introductions from Chinese friends. We also accept Chinese students for internships. I am also the director of the RIKEN AIP Center and a professor at the University of Tokyo. Even at the University of Tokyo, we accept students from China. Undergraduates must know Japanese. I actually have 7 or 8 Chinese students at the University of Tokyo and they are all doing very well!
AI Tech Base: If you have a particular advantage in AIP or RIKEN or Japan, how does that compare to the US, Europe or China?
Jai Sugiyama: In Japan, we have three AI research centers, and we are one of them. We have the support of the Ministry of Education, which means we can focus on basic technology research, and the other two research centers are supported by the Ministry of Economy and the Ministry of Transport, which are more applied in a sense. Specific applications are of course important, but in Japan for the last 20 years, basic research was not considered important and our basic research was underfunded. Now, the Ministry of Education has decided to invest in basic AI research to create a research center, which we are committed to doing for 10 years. Deep learning research began about a decade ago, starting with Professor Hinton’s 2006 paper. Deep learning has been hyped for a decade now, and it’s time to write a paper like Professor Hinton’s.
So now, standard deep learning is very popular, and we should also do standard things, nobody knows what will happen in the next 10 years. In about five to 10 years, there are a lot of other problems that can’t be solved by deep learning, and we may need completely different technologies. It’s rare that we have the opportunity to study completely different areas in the next 10 years. Our focus on basic research also attracts people from academia.
AI Tech Base: My last question is about robots in Japan. You may not know how much Japanese manga has influenced my generation and the next. In fact, we’ve seen a lot of Japanese comics about robots and intelligent machines in the future, and robots made by Japanese companies. Personally, do you think the intelligent robots depicted in Japanese comics are achievable? If so, when do you think we’ll see robots like this in the real world?
Sugiyama: I believe it will be possible in the near future. Scientists often have a hard time making predictions because so much changes happen from day to day. Who knows, maybe tomorrow, maybe after 2020? Technology is evolving, both software and hardware technologies are evolving globally. Just like in the last ten years, we have made great progress and technological development.