Department of Computer ScienceAcronym | D-INFK | Homepage | http://www.inf.ethz.ch/ | Country | Switzerland | ZIP, City | | Address | | Phone | | Type | Academy | Parent organization | ETH Zurich | Current organization | Department of Computer Science | Child organizations | |
Open OpportunitiesEstimating human poses within global trajectories is critical for applications such as augmented reality and sports analytics, yet it often demands precisely calibrated cameras and significant computational efforts. With advancements in deep learning and pose estimation technologies, various models can be trained using 2D or 3D motion data. However, effectively integrating these models to predict and analyze human movement trajectories in a continuous and dynamic environment remains challenging. This project aims to create a robust system that estimates and predicts human poses accurately, facilitating advancements in dynamic pose analysis and real-world applications. - Information, Computing and Communication Sciences
- ETH Zurich (ETHZ), Lab Practice, Master Thesis, Semester Project
| Reinforcement learning (RL) can potentially solve complex problems in a purely data-driven manner. Still, the state-of-the-art in applying RL in robotics, relies heavily on high-fidelity simulators. While learning in simulation allows to circumvent sample complexity challenges that are common in model-free RL, even slight distribution shift ("sim-to-real gap") between simulation and the real system can cause these algorithms to easily fail. Recent advances in model-based reinforcement learning have led to superior sample efficiency, enabling online learning without a simulator. Nonetheless, learning online cannot cause any damage and should adhere to safety requirements (for obvious reasons). The proposed project aims to demonstrate how existing safe model-based RL methods can be used to solve the foregoing challenges. - Engineering and Technology
- Master Thesis
| While camera-based capture systems remain the gold standard for recording human motion, learning-based tracking systems based on sparse wearable sensors are gaining popularity. Most commonly, they use inertial sensors, whose propensity for drift and jitter have so far limited tracking accuracy. Ultra-wideband (UWB) is a radio technology that offers precise ranging capabilities and is integrated into modern smart devices such as iPhones and Apple AirTags. Our recent work, Ultra Inertial Poser, accepted by SIGGRAPH'24, has shown great potential to combine IMU with UWB sensors to constrain drift and jitter in inertial tracking via inter-sensor distances. As we are in the early stages of development, there is still significant room for improvement in the methodology. In this project, we aim to design a deep learning model to improve our dataset's human motion tracking results.
Note: This project will focus on developing a novel supervised learning-based method. We have a clean and synchronized dataset ready (UIP-DB) for training and testing alongside ground-truth tracking data for all joints. - Engineering and Technology, Information, Computing and Communication Sciences
- Master Thesis, Semester Project
| 3D hand pose forecasting is a new benchmark introduced by HoloAssist [1]. Existing action forecasting work mostly focuses on providing semantic labels of future actions and does not provide explicit 3D guidance on hand poses. Predicting 3D hand poses can be useful for various applications, and it can augment instructions and spatially guide users in different tasks. In this benchmark, we take 3 seconds inputs similar to other 3D body location forecasting literature and forecast the continuous 3D hand poses for the next 0.5, 1.0, and 1.5 seconds. The evaluation metric is the average of mean per joint position error over time in centimeters compared to ground truth. To have a proper evaluation metric that can help 3D action guidance, we remove the mistakes from the action sequences and only forecast 3D hand pose for the correct labels.
[1] Wang, X., Kwon, T., Rad, M., Pan, B., Chakraborty, I., Andrist, S., ... & Pollefeys, M. (2023). Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20270-20281). - Computer Vision, Virtual Reality and Related Simulation
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| Action recognition is an essential task in computer vision and has numerous applications in various fields, including robotics, surveillance, and healthcare. The recognition of actions involves the analysis of temporal and spatial information within a video sequence. Current state-of-the-art methods use 3D hand and object poses for action recognition, where the object's corners are commonly used for representation. However, this approach has limitations in accurately modeling the hand-object interaction. In [1], we show that leveraging hand-object contact-map representation helps improve action recognition. However, this representation can be learned implicitly for the task of action recognition.
[1] https://arxiv.org/pdf/2309.10001.pdf - Computer Vision, Virtual Reality and Related Simulation
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| The recent development of LLMs (Large Language Models), such as ChatGPT and Llama, opens up new possibilities for understanding procedural actions. In the past, action recognition was restricted to the classification of visual frames. However, with LLMs, the model can observe the whole action sequence in a more effective way and even predict the future actions [1]. In this project, students will explore how LLMs can improve action recognition in procedural tasks. Specifically, given a high-level procedural task (e.g., making coffee, copying a paper), students will use existing pretrained action recognition models to predict the top 5 actions for each clip and feed them into the LLMs to refine and correct the predicted actions. As a comparison, students will also establish a baseline using simple machine learning and statistical methods to correct actions.
[1] Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023, CVPR'23 workshop
- Computer Vision, Text Processing
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| Reading text manuals to set up and manipulate devices takes a lot of time and is not intuitive when it comes to 3D instruction. Despite the advent of Mixed Reality (MR) devices, 3D instruction is still limited and expensive to set up. In this project, we will develop an app, an adaptive 3D hand guidance system that projects instructional 3D hand poses in MR devices with pre-recorded instructional videos using MR devices. - Computer Vision, Virtual Reality and Related Simulation
- ETH Zurich (ETHZ), Master Thesis, Semester Project
| The goal of this project is to use language prompts to help find object parts in 3D. - Computer Vision
- Master Thesis, Semester Project
| The objective of this project is to determine the metric relative pose between two images using object-to-object matches. - Computer Vision
- Master Thesis, Semester Project
| We extend the lamar.ethz.ch benchmark to develop accurate SLAM methods that can co-register drones, legged robots, wheeled robots, smartphones, and mixed reality headsets based on visual SLAM. - Computer Vision, Intelligent Robotics
- Bachelor Thesis, Master Thesis, Semester Project
|
|