about image

Yong-Lu Li

Ph.D.

Email: yonglu_li[at]sjtu[dot]edu[dot]cn

[Google Scholar] [Github] [LinkedIn]

[ResearchGate] [dblp] [Semantic Scholar]

Machine Vision and Intelligence Group

Shanghai Jiao Tong University

About

I recieved a Ph.D. in Computer Science from the Shanghai Jiao Tong University (SJTU), under the supervision of Prof. Cewu Lu, in Machine Vision and Intelligence Group (MVIG). Prior to that, I worked and studied in Institute of Automation, Chinese Academy of Sciences (CASIA) under the supervision of Prof. Yiping Yang and A/Prof. Yinghao Cai (3D Reconstruction). My primary research interests are Computer Vision and Robotics. Currently, I focus on developing a knowledge-driven vision system (HAKE) that learns to effectively perceive human activities, reason human behavior logics, and interact with objects and environments.



News



2021.05: Selected as the Chinese AI New Star Top-100 (Machine Learning).

2021.02: Upgraded HAKE-Activity2Vec is released! Images/Videos --> human box + ID + skeleton + part states + action + representation. [Description] Demo: [YouTube] [bilibili]

2021.01: TIN (Transferable Interactiveness Network) is accepted by TPAMI!

2021.01: Recieved Baidu Scholarship (10 recipients globally).

2020.12: DecAug is accepted in AAAI 2021.

2020.11: The code and paper of HOI Analysis (IDN) are released! IDN is maintained under our new project HAKE-Action-Torch.

2020.09: Our work HOI Analysis will appear at NeurIPS 2020. Code and arXiv paper are comming soon.

2020.07: Fortunate to recieve WAIC YunFan Award and be among the 2nd A-Class Project.

2020.06: Larger HAKE-Large (>120K images with activity and part state labels) is released!

2020.04: Code, data and paper of "Detailed 2D-3D Joint Representation for Human-Object Interaction (CVPR2020)" are released!

2020.04: Code, data, paper of image-based HAKE, "PaStaNet: Toward Human Activity Knowledge Engine (CVPR2020)", are released!

2020.04: The code and paper of our CVPR 2020 paper Symmetry and Group in Attribute-Object Compositions are available now!

2020.02: Three papers Image-based HAKE: PaSta-Net, 2D-3D Joint HOI Learning, Symmetry-based Attribute-Object Learning are accepted in CVPR 2020! Papers and corresponding resources (code, data) will be released soon.

2019.08: Code for our ICCV2019 paper InstaBoost is available now!

2019.07: Our paper InstaBoost is accepted in ICCV 2019.

2019.06: The Part I of our HAKE : HAKE-HICO which contains the image-level part-state annotations is released!

2019.06: Code for our CVPR2019 paper on Human-Object Interaction is available now!

2019.04: Our dataset Instance-60k & 3D Object Models in ECCV2018 paper SRDA is available!

2019.04: Our project HAKE: Human Activity Knowledge Engine begins trial operation!

2019.02: Our paper on Interactiveness is accepted in CVPR 2019.

2018.11: We released our new paper on Interactiveness.

2018.07: Our paper on GAN & Annotation Generation is accepted in ECCV 2018.

2018.05: Presentation (Kaibot Team) in TIDY UP MY ROOM CHALLENGE | ICRA 2018.

2018.02: Our paper on Object Part States is accepted in CVPR 2018.

2017.07: I joined the MVIG lab.




Publications





HAKE: Human Activity Knowledge Engine

Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Mingyang Chen, Ze Ma, Shiyi Wang, Hao-Shu Fang, Cewu Lu.

Preprint  [arXiv] [PDF] [Project] [Code]

Main Repo:

Sub-repos: Torch TF Halpe List

Canonical Voting: Towards Robust Oriented Bounding Box Detection in 3D Scenes

Yang You, Zelin Ye, Yujing Lou, Chengkun Li, Yong-Lu Li, Lizhuang Ma, Weiming Wang, Cewu Lu.

Preprint  [arXiv] [PDF] [Code]

UKPGAN: Unsupervised KeyPoint GANeration

Yang You, Wenhai Liu, Yong-Lu Li, Weiming Wang, Cewu Lu.

Preprint  [arXiv] [PDF] [Code]

Transferable Interactiveness Knowledge for Human-Object Interaction Detection

Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Xijie Huang, Liang Xu, Cewu Lu.

TPAMI 2021  [arXiv] [PDF] [Code]

An extension of our CVPR 2019 work (Transferable Interactiveness Network, TIN).

DecAug: Augmenting HOI Detection via Decomposition

Yichen Xie, Hao-Shu Fang, Dian Shao, Yong-Lu Li, Cewu Lu.

AAAI 2021  [arXiv] [PDF]

HOI Analysis: Integrating and Decomposing Human-Object Interaction

Yong-Lu Li*, Xinpeng Liu*, Xiaoqian Wu, Yizhuo Li, Cewu Lu (*=equal contribution).

NeurIPS 2020  [arXiv] [PDF] [Code] [Project: HAKE-Action-Torch]

PaStaNet: Toward Human Activity Knowledge Engine

Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang, Ze Ma, Mingyang Chen, Cewu Lu.

CVPR 2020  [arXiv] [PDF] [Video] [Slides] [Data] [Code]

Oral Talk, Compositionality in Computer Vision in CVPR 2020.

Detailed 2D-3D Joint Representation for Human-Object Interaction

Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, Cewu Lu.

CVPR 2020  [arXiv] [PDF] [Video] [Slides] [Benchmark: Ambiguous-HOI] [Code]

Symmetry and Group in Attribute-Object Compositions

Yong-Lu Li, Yue Xu, Xiaohan Mao, Cewu Lu.

CVPR 2020  [arXiv] [PDF] [Video] [Slides] [Code]

InstaBoost: Boosting Instance Segmentation Via Probability Map Guided Copy-Pasting

Hao-Shu Fang*, Jianhua Sun*, Runzhong Wang*, Minghao Gou, Yong-Lu Li, Cewu Lu.

ICCV 2019  [arXiv] [PDF] [Code]

Transferable Interactiveness Knowledge for Human-Object Interaction Detection

Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yan-Feng Wang, Cewu Lu.

CVPR 2019  [arXiv] [PDF] [Code]

SRDA: Generating Instance Segmentation Annotation via Scanning, Reasoning and Domain Adaptation

Wenqiang Xu*, Yong-Lu Li*, Cewu Lu. (*=equal contribution).

ECCV 2018  [arXiv] [PDF] [Dataset](Instance-60k & 3D Object Models) [Code]

Beyond Holistic Object Recognition: Enriching Image Understanding with Part States

Cewu Lu, Hao Su, Yong-Lu Li, Yongyi Lu, Li Yi, Chi-Keung Tang, Leonidas J. Guibas.

CVPR 2018  [PDF]

Optimization of Radial Distortion Self-Calibration for Structure from Motion from Uncalibrated UAV Images

Yong-Lu Li, Yinghao Cai, Dayong Wen, Yiping Yang.

ICPR 2016  [PDF]



Projects





HAKE: Human Activity Knowledge Engine

Human Activity Knowledge Engine (HAKE) aims at promoting activity understanding with a large-scale knowledge base and visual reasoning. To afford knowledge-based learning/reasoning, HAKE provides not only 26M+ human body part-level atomic action labels (Part States) but also holistic object knowledge annotations. It boosts the performances of several widely-used activity benchmarks (HICO, HICO-DET, V-COCO, AVA, etc), and can convert human boxes into multi-modal representations for diverse downstream tasks, e.g., image/video action recognition/detection, captioning, VQA, visual reasoning, etc. Now, the HAKE project contains the works of 8 papers (CVPR'18/19/20, NeurIPS'20, TPAMI'21). We will keep enriching HAKE to make it a general research platform of knowledge mining, visual reasoning, and causal inference. Come and join us!

[Website] [Result Description] [YouTube] [bilibili]

Contents:

1) HAKE-Image (CVPR'18/20): HAKE-HICO, HAKE-HICO-DET, HAKE-Large, Extra-40-verbs.

2) HAKE-Action-TF, HAKE-Action-Torch (CVPR'18/19/20, NeurIPS'20, TPAMI'21): SOTA action understanding methods and the corresponding HAKE-enhanced versions (TIN， IDN).

3) HAKE-3D (CVPR'20): 3D human-object representation for action understanding (DJ-RN).

4) HAKE-Object (CVPR'20): object knowledge learner to advance action understanding (SymNet).

5) HAKE-A2V (CVPR'20): Activity2Vec, a general activity feature extractor and PaSta (human body part states) detector based on HAKE data, converts a human (box) to a fixed-size vector, PaSta and action scores.

6) Halpe: a joint project under Alphapose and HAKE, full-body human keypoints (body, face, hand, 136 points) of 50,000 HOI images.

7) HOI Learning List: a list of recent HOI (Human-Object Interaction) papers, code, datasets and leaderboard on widely-used benchmarks.



Public Services





Reviewer

  • Conference: CVPR'20/21/22, ICCV'21, NeurIPS'20/21, ICLR'22, ICML'21, AAAI'21/22, ACCV'20, WACV'21/22.

  • Journal: Neurocomputing, JVCI.



    • Teaching





    Assistant

  • Problem Solving and Practice of Artificial Intelligence, Shanghai Jiao Tong University, 2020-2021 (Spring)

    • The second Artificial Intelligence Class

    Mentor

  • Problem Solving and Practice of Artificial Intelligence, Shanghai Jiao Tong University, 2019-2020 (Spring)

    • The first Artificial Intelligence Class

  • Guided Paper:

    • Rb-PaStaNet: A Few-Shot Human-Object Interaction Detection Based on Rules and Part States

    Shenyu Zhang, Zichen Zhu, Qingquan Bao (freshmen)

    IMVIP 2020, Press Coverage: SEIEE of SJTU



    Talks





  • 2021.3.05: Human Activity Knowledge Engine (updated)

    • SJTU Computer Science Global Lunch Series. [Video]

  • 2020.8.23: Knowledge Driven Human Activity Understanding

    • The 3rd International conference on Image, Video Processing and Artificial Intelligence, IVPAI 2020.

  • 2020.7.12: Human Activity Knowledge Engine

    • Student Forum on Frontiers of AI, SFFAI. [Slides]

  • 2020.6.16: PaStaNet: Toward Human Activity Knowledge Engine

    • Compositionality in Computer Vision in CVPR 2020 Virtual. [Video] [Slides]



    Honors





  • Shanghai Outstanding Doctoral Graduate, Aug. 2021.

  • PhD Fellowship, The 85th Computer Department Education Development Fund and Yang Yuanqing Education Fund, Jun. 2021.

  • Chinese AI New Star Top-100 (Machine Learning), May. 2021.

  • Baidu Scholarship (Top-10, worldwide), Jan. 2021. Press Coverage.

  • WAIC Outstanding Developer, Dec. 2020. Press Coverage: 机器之心, 上海临港

  • China National Scholarship, Sep. 2020.

  • WAIC YunFan Award, Rising Star, July. 2020 (World Artificial Intelligence Conference, Shanghai). Press Coverage: 机器之心

  • The 2nd A-Class Project , July. 2020.

  • Annual Scholarship, SJTU, Nov. 2019.



    • Personal Interests





  • Chinese History Reading

    • Twenty-Four Histories: 5/24 (･ิω･ิ)

    Early Four Historiographies: Records of the Grand Historian, Book of Han, Records of the Three Kingdoms, Book of Later Han;
    The Book of Jin;

  • Travel around China

    • 34 Provinces: 20/34 (･ิω･ิ)