PhD researcher · Automation & Robotics · FERI, University of Maribor

I build robots that
refuse to fall over.

I combine reinforcement learning with classical control for legged and wheeled-biped robots — hybrid architectures where learned policies and hand-tuned loops work together, not against each other. The current one balances on two wheels using a policy of 5 186 parameters, running on a single microcontroller with no computer on board.

↓ This is not an animation. It's a cart-pole simulation running a real state-feedback controller at 240 Hz. Click the robot to shove it. Then break the controller with the sliders.

wheeled_biped.sim inverted pendulum on wheels · u = k₁x + k₂ẋ + k₃θ + k₄θ̇
🖍️ Explain it like I'm five

Balancing this robot is like balancing a broom standing up on your palm. You can't hold the broom still — you have to keep moving your hand so it stays under the broom. The robot does the same thing with its wheels, hundreds of times per second.

The four sliders are the robot's four little worries:

  • k₃ — how hard it panics when it starts tipping. This is the big one.
  • k₄ — how much it notices the tipping getting faster. This stops it from wobbling forever.
  • k₁ — how much it wants to come back to its home spot.
  • k₂ — how much it dislikes going fast.

Set k₃ to zero and it stops caring about falling. It falls. Try it — it's fine, it gets back up.

∑ The actual math (for the curious)

Single inverted pendulum on a cart, integrated with semi-implicit Euler at 240 Hz:

θ̈ = [ g·sinθ − cosθ·(u + m·l·θ̇²·sinθ)/(M+m) ] / [ l·(4/3 − m·cos²θ/(M+m)) ]
ẍ = (u + m·l·(θ̇²·sinθ − θ̈·cosθ))/(M+m)

The controller is full-state feedback, u = k₁x + k₂ẋ + k₃θ + k₄θ̇, saturated at ±25 N — the structure an LQR design produces. Note the non-minimum-phase behaviour: to drive the cart back to the origin, the controller must first steer it away to lean the body toward home. In "RL-style" mode the same state vector is fed through a saturating nonlinearity, u = umax·tanh(·) — a hand-written stand-in that mimics the aggressive, saturation-riding character of trained policies (it's not a live neural network; the real one is on the robot, and in the talk below).

01

What I work on

The field tends to frame learned and classical control as rivals. My research treats them as layers of one architecture: hand-tuned impedance handles what it's provably good at, and reinforcement learning handles what can't be modelled cleanly — contact, impacts, recovery from the unexpected.

hybrid architectures

Learned policies and classical loops in the same control stack, each doing the part it's best at. Not "RL replaces control" — coordination between them, with the classical layer as a verifiable fallback.

minimalist rewards

Reward only what actually matters and let balance emerge as an instrumental strategy instead of hand-coding it. The flip side is that a policy will exploit every loophole you leave open — most of my debugging time is spent finding them.

sim-to-real, embedded

Policies trained with PPO across thousands of parallel environments in Isaac Lab, then deployed on an STM32 — the same class of microcontroller running the classical loops. No offboard compute, no GPU on the robot.

02

The robot: two loops, one machine

A two-wheeled self-balancing robot with articulated legs, built to be dropped, shoved and thrown off things — and to keep balancing anyway. What makes it interesting isn't either loop on its own. It's that the legs are hand-tuned and the wheels are learned, and neither one is told what maneuver to perform.

Driving off a table and landing. The jump was never trained in simulation — balancing is a learned policy, the legs are hand-tuned impedance control, and neither loop specifies this maneuver. The two together just make it possible.

policy loop · learned

Two hub motors, nothing else

  • 13 observations in, 2 wheel currents out
  • 5 186 parameters at 66.7 Hz (15 ms)
  • One STM32F446RE — no companion computer
  • Trained with PPO in Isaac Lab, then quantized and flown

leg impedance · hand-tuned

Four leg joints, fixed stance

  • MIT-mode PD closed inside each CyberGear drive
  • Its own loop rate, independent of the policy
  • Stiffness and damping set by hand, not learned
  • Absorbs impact so the wheel loop never sees the spike
the three pieces: platform, leg stiffness and damping, learned wheel currents
a push: legs absorb the impact, the policy keeps the platform upright
jumping off a table — momentum and compliance do the work a reference trajectory would have had to specify
standing itself back up after a fall

Why this platform

A wheeled biped is the cheapest possible legged robot that still has the hard problems: it is statically unstable, underactuated, and its contact state changes the moment it leaves the ground. A table drop is a plant-model discontinuity — airborne, the wheel loop has nothing to push against; on impact, states jump. That boundary between regimes is exactly where the learned layer earns its place.

Splitting the machine in two is what makes it tractable. The legs are a spring and a damper with numbers I chose; the wheels are a policy that learned what to do about whatever the legs let through. Neither half needs a model of the other, and only one half needed a GPU.

leg actuatorsCyberGear geared BLDC, MIT mode
wheelsDDSM115 direct-drive hub motors
IMUBNO086 (onboard sensor fusion)
computeSTM32F446RE — classical loops & policy
policy13 obs → 2 currents, 5 186 params, 66.7 Hz
trainingIsaac Lab, PPO, thousands of parallel envs
legshand-tuned impedance, fixed stance

03

Simulation to reality

Simulation is the idealized system. The robot is not. The gap between them is where most of the work actually happens — and the fix turned out not to be a better model of the robot, but refusing to commit to one.

measure first

Before randomizing anything, characterize the actual hardware. Twelve nominally identical motors, measured on a bench, have current deadzones spread across 36–78 mA. That spread is the thing to reproduce — not its average.

train against a population

Each parallel environment draws its own masses, frictions and actuator parameters. The policy never meets the same robot twice, so it can't overfit to one. Read the widths of those ranges, not the names of the parameters.

widen in stages

Domain randomization applied all at once stops the policy learning anything. Applied in stages, each widening the world a little further, it survives the ranges that would have prevented it from ever getting off the ground.

don't grade in the training simulator

Isaac Lab, MuJoCo and the robot disagree about the same controller — and the simulator you trained in is the one most likely to flatter you. If the score comes from the same physics that produced the policy, it isn't evidence.

a wheel changed for a different diameter while it is balancing — the policy never retunes
an off-centre mass dropped on top, shifting the centre of mass
the ground itself tilting underneath it
driving between obstacles under hand commands
mismatched diameter and friction: one wheel breaks traction, the other holds
the same push, slowed and zoomed — the slipping wheel spins up

Change one thing at a time, twice

Three policies, each differing from the previous one in a single respect, tested on the real robot rather than in the simulator that trained them.

variantnetworkactuatorsRMS pitch on hardware
AMLP, 5 186 paramspoint estimate did not start
BMLP, 5 186 paramsrange randomized 5.04°
CGRU, 23 618 paramsrange randomized 2.42°

A vs B isolates actuator randomization; B vs C isolates the recurrent policy. The policy trained against a single best-guess actuator model never started on hardware at all — its fall detector inhibited actuation before the first stage. Randomizing around the measured actuators is what transfers.

04

The other robot

A legged robot, trained the same way, kept deliberately separate — it's where I test whether a reward that worked on two wheels means anything on four legs. Mostly it's where I learn how creatively a policy will cheat.

upright at 108% of standing height, and actually walking — this one scores worst
moving forward in a low, seated posture — technically correct, and not what anyone wanted

Those two clips are the whole lesson, and the scoreboard has them the wrong way round. The upright gait on the left comes from an older reward function and scores 46.53. Two policies trained on the newer reward beat it — 51.86 and 54.10 — and they get there by crouching to 42% and 34% of standing height. The better network did not learn to stand up straighter; it got better at finding the crouch.

Nothing in that reward ever said standing up was the point, so nothing made it worth doing. You specify; it negotiates. Every reward term is a loophole until proven otherwise — and the policy that looks right to you may be the one your own metric ranks last.

05

Playground: PID for plants

Control theory isn't only for robots. One of my first projects, years ago, was an MQTT plant monitor — so here is a plant with a proper PID controller on its watering pump. The sun rises and sets, the soil dries out, and the controller tries to hold the moisture you ask for. Tune it badly and you can drown the poor thing.

plant_pid.sim soil-moisture control · u = Kpe + Ki∫e + Kdde/dt
🖍️ Explain it like I'm five

The pump has three feelings about the thirsty plant:

  • Kp — "the thirstier the plant, the harder I pump, right now."
  • Ki — "if the plant has been a little thirsty for a long time, I get more and more impatient until it's fixed." Without this, the plant stays forever slightly thirstier than you asked.
  • Kd — "whoa, the soil is getting wet fast — I should ease off before I overshoot." The calm one.

Crank Ki way up and watch the pump get so impatient it floods the pot, then sulks, then floods it again. That's called integral windup, and real engineers fight it every day.

06

Earlier work

Before the wheels, an arm. My mechatronics diploma thesis: a fully 3D-printed 4-DOF manipulator, designed, simulated, built and controlled end-to-end — with a machine-vision demo where the end effector follows your hand in real time.

Render of the 3D-printed 4-DOF robotic manipulator in grey and orange
4-DOF manipulator, Fusion 360 render
Wireframe CAD view of the manipulator
wireframe — every part designed from scratch

trajectory generation

Custom point-to-point and linear moves with quadratic, cubic and sinusoidal acceleration profiles — written from scratch, not taken from a library.

kinematics

Inverse and forward kinematics via the Robotics Toolbox for Python; kinematic validation in MSC Adams, structural FEM validation in Fusion 360.

embedded link

A custom UART protocol between the Python host and the STM32 that drives the joints — the same host/microcontroller split the wheeled biped uses today.

machine vision

YOLO models trained for object detection, and a MediaPipe hand-tracking demo where the arm's end effector mirrors the user's hand live.

07

Papers & talks

  1. Policy learning for complex maneuvers

    J. Tofant, A. Sarjaš · Styrian Workshop on Automatic Control, Retzhof 2026 · invited talk

    slides workshop

  2. Učenje in vgrajena implementacija nevronskega regulatorja za stabilizacijo dvokolesnega robota z zavračanjem motenj (Learning and embedded implementation of a neural controller for disturbance rejection in a two-wheeled self-balancing robot)

    J. Tofant, A. Sarjaš · PPO policy deployed on STM32

  3. Diferenčno vodenje agilnega dvokolesnega mobilnega robota (Differential control of an agile two-wheeled mobile robot)

    J. Tofant, D. Gleich · ERK 2025, Portorož · pp. 597–600

08

Say hi

Junior researcher (mladi raziskovalec) at the Faculty of Electrical Engineering and Computer Science, University of Maribor. Happy to talk about hybrid control, wheeled bipeds, reward design, or why your robot keeps falling over.

jon.tofant@um.si github.com/JonTofant