telekinesis-examples

Telekinesis Agentic Skill Library for Computer Vision, Robotics and Physical AI

The Telekinesis Agentic Skill Library is the first large-scale Python library for building agentic robotics, computer vision, and Physical AI systems. It provides:

• Skills: a broad set of AI algorithms for perception, motion planning, and control.
• Physical AI Agents: LLM/VLM agents for task planning across industrial, mobile, and humanoid robots.

The library is intended for robotics, computer vision, and research teams that want to:

• Speed up development by integrating production-grade robotics, computer vision, and AI algorithms
• Add intelligence to robots with LLM/VLM-driven task planning tied to real perception and control systems
• Iterate quickly on Physical AI systems using a single, consistent Python library

This repository provides Python examples demonstrating Telekinesis Skills—atomic operations you can chain into pipelines for real-world applications. Each example is standalone, readable, and composable.

Tip: A free API key is required. Create one at platform.telekinesis.ai. See the Quickstart Guide for details.

The Telekinesis Community

The Agentic Skill Library and Brainwave are just the beginning. We're building a community of contributors who grow the Physical AI Skill ecosystem—researchers, hobbyists, and engineers alike. If you have a Skill, we want to see it. Release it, let others use and improve it, and watch it deploy in real-world systems.

Join our Discord community to connect, share, and build together.

Table of Contents

• What is a Skill?
• What is a Physical AI Agent?
• How do Physical AI Agents use Skills?
• What Can You Build?
• Control Any Robot
• Production-Grade Computer Vision
• Getting Started
• Example Categories
• Directory Structure
• Who We Are

What is a Skill?

A Skill is a reusable operation for robotics, computer vision, and Physical AI. Skills span 2D/3D perception (6D pose estimation, 2D/3D detection, segmentation, image processing), motion planning (RRT*, motion generators, trajectory optimization), and motion control (model predictive control, reinforcement learning policies). Skills can be chained into pipelines to build real-world robotics applications.

Example 1: Segment image using SAM — docs

from telekinesis import cornea                                # Import Cornea - Image segmentation module

# Executing a 2D image segmentation Skill                     
result = cornea.segment_image_using_sam(                      # Executing Skill - `segment_image_using_sam`
    image=image,
    bboxes=[[400, 150, 1200, 450]]
)
# Access results
annotations = result.to_list()                

Example 2: Detect objects using RF-DETR — docs

from telekinesis import retina                                   # Import Retina - Object detection module

# Executing a 2D object detection Skill
annotations, categories = retina.detect_objects_using_rfdetr(    # Executing Skill - `detect_objects_using_rfdetr`
    image=image,
    score_threshold=0.5,
)
# Access results
annotations = annotations.to_list()
categories = categories.to_list()  

Skills are organized in Skill Groups. Each can be imported from the telekinesis library:

from telekinesis import cornea     # image segmentation skills
from telekinesis import retina     # object detection skills
from telekinesis import pupil      # image processing skills
from telekinesis import vitreous   # point cloud processing skills
from telekinesis import illusion   # synthetic data generation skills
from telekinesis import iris       # AI model training skills
from telekinesis import neuroplan  # robotics skills
from telekinesis import medulla    # hardware communication skills
from telekinesis import cortex     # physical AI agents

Cornea: Image segmentation

from telekinesis import cornea

• Color-based segmentation: RGB, HSV, LAB, YCrCb
• Region-based segmentation: Focus region, Watershed, Flood fill
• Deep learning segmentation: BiRefNet (foreground), SAM
• Graph-based segmentation: GrabCut
• Superpixel segmentation: Felzenszwalb, SLIC
• Filtering: Filter by area, color, mask
• Thresholding: Global threshold, Otsu, Local, Yen, Adaptive, Laplacian-based

Retina: Object detection

from telekinesis import retina

• Classical shape detection - Hough Transform, Contours
• 2D Object detection - YOLOX, RF-DETR
• Open-Vocabulary detection - Qwen-VL, Grounding DINO

Pupil: 2D image processing

from telekinesis import pupil

• Morphology: erode, dilate, open/close, gradient, top-hat
• Structure: Frangi, Hessian, Sato, Meijering
• Edges: Sobel, Scharr, Laplacian, Gabor
• Denoising: Gaussian, median, bilateral, box filters
• Enhancement: CLAHE, gamma correction, white balance
• Transforms: pyramids, mask thinning

Vitreous: 3D point cloud processing

from telekinesis import vitreous

• Point cloud: centroids, normals, bounding boxes, principal axes
• Filtering: masks, outliers, downsampling, plane & cylinder removal
• Segmentation: DBSCAN, density, color, plane-based clustering
• Transforms: rigid transforms, scaling, projection
• Registration: ICP (P2P, P2Plane), global registration, cuboid sampling
• Meshes: shapes, mesh to point cloud, convex hull, Poisson reconstruction

Illusion: Synthetic data generation

from telekinesis import illusion

• Synthetic image data generation for AI model training

Iris: AI model training and deployment

from telekinesis import iris

• AI model training pipelines • Fine-tuning and evaluation of foundation models

Neuroplan: Robotics

from telekinesis import neuroplan

• Kinematics • Motion planning • Control • Robot database

Cortex: Physical AI Agents

from telekinesis import cortex

• Action & Skill graphs • Physical AI Agents for skill graph generation and execution

Brainwave: Physical AI Platform

• Skill & Agent orchestration • Simulation, digital twins, sim-to-real workflows • Monitoring and logging

What is a Physical AI Agent?

Recent advances in LLMs and VLMs have shown the potential of learned models to perform semantic reasoning, task decomposition, and high-level planning from vision and language inputs.

In the Telekinesis library, a Physical AI Agent—typically a Vision Language Model (VLM) or Large Language Model (LLM)—autonomously interprets natural language instructions and generates high-level Skill plans. In autonomous Physical AI systems, Agents continuously produce and execute Skill plans, allowing the system to operate with minimal human intervention.

To learn more, explore Cortex.

How do Physical AI Agents use Skills?

Telekinesis Agentic Skill Library Architecture

Flow Overview

1. A user prompts the robot with a task: "Hey, can you do a CNC machine loading task?"
2. The Agent interprets the intent and reasons over the given task and sequencing.
3. The Agent constructs a high-level plan and orchestrates the Skills—selecting order and parameters.
4. Skills are executed on the robot using production-grade perception, motion, and control algorithms, with continuous feedback from the environment.

What Can You Build?

Telekinesis Agentic Skill Library helps you build real-world robotics and Physical AI applications for industries such as manufacturing, automotive, aerospace, and others. Below are use cases the Telekinesis team has deployed using the skill library.

Example Use Case	Description
Carton Palletizing	Vision-guided palletizing that adapts to changing layouts and product variations. Object detection, pose estimation, and motion planning for accurate placement.
Automated Assembly	Multi-step assembly combining task planning, coordinated manipulation, and precise motion execution.
Vision-Based Quality Control	Industrial computer vision for defect detection, dimensional verification, and surface analysis.
Automated Basil Harvesting	Vision-based manipulation for agricultural robotics in unstructured outdoor environments. Detect plants, estimate grasp poses, execute adaptive closed-loop motions.

Carton Palletizing	Automated Assembly
Vision-Based Quality Control	Automated Basil Harvesting

Control Any Industrial Robot, Mobile Robot & Humanoid with a Unified Python Interface

One of the biggest pains of robotics is that each robot provider has their own interface. Telekinesis offers Neuroplan—a skill group that provides a unified interface to control any industrial, mobile, or humanoid robot.

Supported: Universal Robots, KUKA, ABB, Franka Emika (real & simulation); Boston Dynamics, Anybotics, Unitree (simulation).

from telekinesis import neuroplan  # robotics skills

Prototype on any robot, perform any task on the same platform, and deploy the same Skill Groups anywhere—any robot, any task, on one Physical AI platform.

Production-Grade Computer Vision Models for Robotics and Physical AI Systems

The library offers production-grade computer vision Skill Groups for object detection, segmentation, pose estimation, synthetic data generation, and AI model training.

from telekinesis import cornea     # image segmentation skills
from telekinesis import retina     # object detection skills
from telekinesis import pupil      # image processing skills
from telekinesis import vitreous   # point cloud processing skills
from telekinesis import illusion   # synthetic data generation skills
from telekinesis import iris       # AI model training skills
from telekinesis import medulla    # sensor interface skills

	2D Image Processing, Object Detection & Segmentation Build reusable 2D vision pipelines using Pupil for low-level image processing, Retina for object detection, and Cornea for segmentation and mask generation. These Skill Groups can be composed into standalone perception pipelines for images, video, or sensor data.
3D Point Cloud Processing & Mesh Generation Develop geometric alignment pipelines using Vitreous to register point clouds or meshes against reference models or scenes. Vitreous provides reusable registration Skills—ICP-based alignment, global registration—enabling precise localization and model-to-scene matching.
	3D Object Detection & 6D Pose Estimation Create 3D object detection and 6D pose estimation pipelines by combining Retina for object detection with Vitreous for point cloud filtering, registration, and geometric pose estimation—for grasp planning, inspection, and vision-guided manipulation.
Synthetic Data Generation & AI Model Training Generate photo-realistic synthetic image datasets for training object detection, segmentation, and classification models using the Illusion skill group. Train state-of-the-art AI models in the cloud and deploy them to real-world systems using the Iris skill group.

Brainwave: the Telekinesis Physical AI Platform

Brainwave is the Telekinesis Physical AI cloud platform for managing skill orchestration, simulation, digital twins, and robot deployments from a single system. It enables agent-based robotics systems to be developed, deployed, and operated at scale across heterogeneous robots and tasks.

Develop and simulate digital twin workflows to validate, stress-test, and optimize Skill Groups. Deploy the same Skill Groups to real-world robots using a simulation-to-real transfer pipeline.

CNC Machine Tending	Pick and Place	Surface Polishing
Robotic Welding	Metal Palletizing	Palletizing

Getting Started

You can easily integrate the Telekinesis Agentic Skill Library into your own application. Set up the library in just 4 steps and start building!

Requirements

• Python 3.11 or 3.12
• A Telekinesis account and API key

Step 1: Create an API Key

Since all Skills are hosted in the cloud, you need a free API key to access them securely. Create a Telekinesis account and generate an API key: Create API Key

Store the key in a safe location, such as your shell configuration file (e.g. .zshrc, .bashrc) or another secure location on your computer.

Step 2: Configure the API Key

Export the API key as an environment variable. Open a terminal and run the command below for your operating system.

Note: Replace your_api_key with the key you generated in Step 1.

# macOS / Linux
export TELEKINESIS_API_KEY="your_api_key"

# Windows
setx TELEKINESIS_API_KEY "your_api_key"

Note: After running setx on Windows, restart your terminal for the changes to take effect.

The Telekinesis Agentic Skill Library uses this API key to authenticate requests and automatically reads it from your system environment.

Step 3: Install the Telekinesis Agentic Skill Library

1. Create an isolated environment so that there is no dependency conflicts. We recommend installing Miniconda environment by following instructions from here.

2. Create a new conda environment called telekinesis:

   conda create -n telekinesis python=3.11
   

3. Activate the environment:

   conda activate telekinesis
   

4. Install the library using pip:

   We support Python 3.11 and 3.12. Ensure your environment uses one of these versions.

   pip install telekinesis-ai
   

Step 4: Run Your First Example

1. Clone the telekinesis-examples repository (including the data submodule):

   git clone --depth 1 --recurse-submodules --shallow-submodules https://    github.com/telekinesis-ai/telekinesis-examples.git
   

   This also downloads the telekinesis-data repository, which contains sample point clouds, meshes, and images. You can replace this with your own data when using Telekinesis in your projects. Download time may vary depending on your connection.

2. Change into the repository directory:

   cd telekinesis-examples
   

3. Install example dependencies:

   pip install numpy scipy opencv-python rerun-sdk==0.27.3 loguru pycocotools
   

4. Run the segment_image_using_sam example:

   python examples/cornea_examples.py --example segment_image_using_sam
   

   If the example runs successfully, a Rerun visualization window will open showing the input and filtered point cloud. Rerun is a visualization tool for 3D data and processing results.

List Available Examples

python examples/cornea_examples.py --list     # Cornea (2D image segmentation)
python examples/retina_examples.py --list     # Retina (2D image object detection)
python examples/pupil_examples.py --list      # Pupil (2D image processing)
python examples/vitreous_examples.py --list   # Vitreous (3D point cloud & mesh)

Run a Specific Example

python examples/cornea_examples.py --example segment_image_using_rgb                    # Cornea
python examples/retina_examples.py --example detect_objects_using_grounding_dino        # Retina
python examples/pupil_examples.py --example filter_image_using_morphological_gradient   # Pupil
python examples/vitreous_examples.py --example estimate_principal_axes                  # Vitreous

Use Your Own Data

1. Create a directory with images/, point_clouds/, and meshes/ subdirectories. See the Skills documentation for more on data formats.
2. Update DATA_DIR in vitreous_examples.py or pupil_examples.py or cornea_examples.py or retina_examples.py
3. Update the file_name variable to your input file.
4. Run the example.

Example Categories

Telekinesis Agentic Skill Library Modules

Module	Description	Status
Cornea	Image segmentation	Released
Retina	Object detection (foundation models, classical)	Released
Pupil	2D image processing	Released
Vitreous	3D point cloud & mesh processing	Released
Illusion	Synthetic data generation	Planned — Feb 2026
Iris	Model training & fine-tuning	Planned — Feb 2026
Neuroplan	Motion planning, kinematics, control	Planned — Feb 2026
Cortex	Task planning & skill composition	Planned — Mar 2026
Brainwave	End-to-end robotic solutions	Planned — Apr 2026

Directory Structure

telekinesis-examples/
├── examples/
│   ├── cornea_examples.py      # Image segmentation Skills
│   ├── datatypes_examples.py   # Data types & transformations
│   ├── pupil_examples.py       # 2D image processing Skills
│   ├── retina_examples.py      # Object detection Skills
│   └── vitreous_examples.py    # 3D point cloud & mesh Skills
├── telekinesis-data/           # Git submodule (sample data)
│   ├── images/
│   ├── point_clouds/
│   └── meshes/
├── assets/
├── README.md
├── DEVELOPMENT.md
└── LICENSE.txt

Who We Are

We're a team of passionate robotics and computer vision experts who care about the details. Industry veterans who know the frustration of systems that just don't work. All of us asking the same question: why is robotics still so hard to use?

Telekinesis began as a spin-off from the Intelligent Autonomous Systems Lab at TU Darmstadt, led by Prof. Jan Peters, and is supported by people with years of experience at KUKA and Universal Robots.

Next Steps

Ready to go further? Explore the Agentic Skill Library, dive into specific modules, or connect with the community.

• Documentation — Full Agentic Skill Library reference and guides
• Quickstart — Step-by-step setup
• Cornea Skills — Image segmentation
• Pupil Skills — 2D image processing
• Vitreous Skills — 3D point cloud processing
• Discord — Ask questions, share feedback, connect with users

Support

• GitHub Issues — Report bugs or request features
• Create API Key — Get started with the Telekinesis platform
• Discord — Community support and discussions