The Society of Robotics and Automation is a society for VJTI students. As the name suggests, we deal with Robotics, Machine Vision and Automation
McQueen: From Perception to Motion
Source code: McQueen on GitHub
Future Nostalgia
Think of the future. Maybe 5 years down the line. Maybe 10. Maybe 100. What’s the first thing that comes to your mind?
I bet it involves autonomous systems.
I dream of a world with autonomously driven cars ruling the streets. A world where human lives involve much more comfort. Safer roads. Efficient travelling. That’s the goal.
Do you want a glance at my dream of the future?
Lay your eyes on McQueen, an attempt to look at how far we can explore autonomous driving using a single RGB camera on an RC car. It’s simple and gets the job done. Our hardware stack is completely grounded - a basic RGB camera, a simple motor driving system, and offboard compute. If we can produce a working pipeline with such constraints fueled by just our ambitious ideas, the possibilities might just be endless.
Shooting for the Stars
Self driving is overwhelming. Multimillionaire companies like Tesla, Waymo, NVIDIA and many more have invested unthinkable amounts of money into solving this. But still, self driving remains an open problem.
While researching for McQueen, we came across the company named comma.ai. Their work greatly motivated us and I distinctly remember having countless discussions about how revolutionary their idea of one device fitting all cars is. Additionally, the X account simcity was a great inspiration for us.
Beauty lies in the constraint. And we were dealing with many of them. I didn’t want to be bottlenecked by ambition. And so ,we worked tirelessly and went about solving each and every problem. We only had one dream in mind while doing so - expanding the bounds of autonomous driving systems.
Pitstop
We started with an RC car that barely budged, let alone drove itself. Our first task was to get it moving, which meant replacing its old motors with our own and figuring out how to control them. We named it McQueen after its bright red colour, which, fortunately, does a pretty good job of distracting people from the mess of wires underneath.
On paper, the hardware seemed simple enough: an RGB camera, a Jetson Nano, and two motors. But the hardware gods had other plans. What we expected to be a straightforward setup turned into countless hours of debugging, and getting McQueen to move reliably became an adventure of its own.
Lap 1 : openpilot
What is openpilot?
openpilot is an open-source driver-assistance system developed by comma.ai. For us, the most interesting part of openpilot was its vision stack: a system that takes in camera frames and learns a representation of the driving scene.
It does not treat every frame independently. Instead, it reasons across time, which lets it understand not just what the road looks like right now, but also how the scene is changing. That temporal awareness matters a lot for driving, where the system has to react to curves, motion, and context rather than isolated images.
Of course, openpilot is much bigger than a single neural network. It is a full driver-assistance stack with perception, planning, control, safety logic, and vehicle interfaces. But for McQueen, what mattered most was the pretrained visual backbone at the center of that stack.
That gave us a huge head start. Instead of building visual perception from scratch, we could reuse a backbone that had already learned a strong representation of driving scenes and adapt it to the much smaller problem of driving an RC car.
Frozen Backbone
McQueen’s policy is built around openpilot’s actual pretrained driving backbone, kept frozen during both training and inference. We use comma.ai’s larger big_driving_supercombo model and treat it as the visual-and-temporal trunk of our system.
The backbone expects structured inputs similar to the original stack: two YUV-packed camera frames, a rolling buffer of previous hidden states for temporal context, and a few auxiliary inputs. Those auxiliary signals were not meaningful for McQueen, so we zeroed them out.
The network produces a large native output vector. openpilot normally uses that output for things like lane geometry, lead vehicles, and trajectory-related predictions. McQueen does not need that full planning stack, so we only extract the 512-dimensional recurrent hidden state.
That hidden state acts like a compact summary of what the backbone understands about the scene. We feed it into a small trainable action head, which maps that representation directly to the two actions McQueen needs: steering and throttle.
This makes the split very clean. The large pretrained backbone handles scene understanding and temporal context, while our lightweight head learns how the RC car should act based on that representation.
Lap 2 : The Link
Getting the connection
McQueen runs on a Jetson Nano connected through a mobile hotspot, while the RTX 4090 handling inference sits elsewhere on the internet. Getting these two machines to communicate directly was our first challenge, especially with the Jetson sitting behind carrier-grade NAT.
Our initial approach was WebRTC, a framework designed for real-time peer-to-peer communication. However, the versions of GStreamer and libnice available on our Jetson couldn’t reliably gather the server-reflexive ICE candidates needed to establish the connection. So we decided to implement the connection mechanism ourselves.
Both machines first contact a STUN server to discover their public IP addresses and ports. A lightweight WebSocket broker then exchanges these addresses, allowing both endpoints to begin sending UDP packets to each other and establish a direct connection through their NATs.
Once connected, the broker steps aside. McQueen and the 4090 communicate directly over a single UDP socket, without routing the driving data through an intermediary server.
Closing the Loop
With the connection established, we needed to stream camera frames to the 4090 and receive steering and throttle predictions in return, all over the same UDP socket.
The Jetson captures frames and encodes them into H.264. We initially attempted hardware encoding through NVENC, but it repeatedly stalled on our Jetson, forcing us to switch to CPU-based encoding. We also implemented RTP packetization ourselves to maintain control over frame timestamps. Before transmitting each frame, the Jetson sends a metadata packet containing its frame ID and capture timestamp.
On the 4090, incoming frames are passed through a jitter buffer and decoded before being forwarded to a separate inference process. We isolated inference because running CUDA and GStreamer in the same process caused interference between the two pipelines.
The model then predicts steering and throttle, which are transmitted back to the Jetson through the same UDP connection. The Jetson maps these predictions to servo and motor commands, moving McQueen and changing what the camera sees next.
Lap 3: Learning Curves
First Attempts
As stated before, we use comma.ai’s driving_supercombo as a frozen vision-and-temporal backbone and train only a small action head on top of it. The first baseline was deliberately simple: a small MLP trained on DonkeySim data to map the frozen representation to steering and throttle. The zero-shot model barely drove at all, but the trained head was already a large improvement. Its validation loss dropped substantially, and the difference was visible in simulation too. Without a throttle cap it would accelerate straight into a wall; with one, it could actually stay on the track and make reasonable steering decisions.
We then experimented with the larger big_driving_supercombo model, applying low-rank adapters to selected linear layers of its frozen network. Unlike our initial approach, which trained a separate action head on top of frozen features, this experiment adapted parts of the pretrained model itself using LoRA.
Around the corner
The version we eventually settled on went back to the simpler frozen trunk and small action head, but fixed the temporal part of the training setup. We revisited the temporal handling of the pretrained model. In the frozen-head trainer, we retained a 24-step feature buffer across minibatches and reset it at session boundaries, rather than discarding all temporal information after each batch. This still wasn’t equivalent to strictly sequential recurrence, because our batching strategy advanced each buffer position across spaced-apart frames. During live inference, however, we maintained a rolling feature buffer that was updated after each processed frame. At that point, the curves we cared about weren’t just the ones in the training logs. We looked at whether improvements in validation loss actually translated into better trajectories in simulation: holding a line, taking turns, and recovering from small deviations. That became the more useful measure of whether a checkpoint was worth putting on the car. The slice of the model output we use as the hidden state differs between the export we trained against and the one currently deployed. The model is learning. The remaining question is whether it has learned the right thing well enough to take a real McQueen around a real curve.
Murphy’s Law
Can’t check enough times
Hardware gave us a few failures that were much harder to diagnose than they should have been. We spent a full bench session convinced the encoder setup was faulty, even though the encoder was wired directly to the GPIO and the motor driver wasn’t involved. The actual problem was our documentation: the wire colours we’d written down were wrong for the specific encoder we were using. We were debugging a hardware fault that existed because we had documented the hardware incorrectly.
The Jetson caused a different kind of problem. It would periodically disappear from the network, and after checking the software side we eventually traced it to a dead power bank. There was another problem waiting behind that one: our autostart script lived in /tmp, which is cleared on reboot. So whenever the Jetson restarted, the script responsible for bringing the system back online was gone as well. Neither issue was particularly complicated once we knew where to look, but both cost us time because we were initially looking at the wrong layer.
We also managed to kill a motor driver early on by powering it directly from the battery. The board was not rated for the current the motor could draw, particularly during a stall. That pushed us toward separating the motor and servo power supplies and paying much closer attention to the actual current requirements of the components rather than just their nominal voltage ratings.
Read between the curves
The most important failure was much less visible. We trained the larger Supercombo-based model and watched the validation loss fall consistently across the epochs. On paper, the training run looked great. In simulation, the model barely drove. The problem was in how we converted the simulator’s steering labels into the model’s action space. An earlier version applied an assumed-speed scaling to curvature targets, making them incompatible with the outputs we expected from the model. The validation loss could still fall because the model was being trained against those same incorrectly scaled targets, but the resulting actions were unusable in simulation.
These were probably the most useful failures in the project because they were both examples of the same problem at different levels: the system can behave consistently and still be wrong. Sometimes the thing that needs debugging isn’t the model or the hardware, but the assumption sitting between them.
You've read enough! Enjoy a video of McQueen taking this turn completely by itself
If the video does not load, open it directly.
Finish Line
McQueen is now a working vision-based autonomous driving system running on a physical car. An RGB camera provides its only visual input, the Jetson handles the onboard processing and communication, and the RTX 4090 runs the neural-network inference. The model uses a frozen pretrained vision-and-temporal backbone with a McQueen-specific action head trained on driving data, and the complete pipeline has been run end-to-end on real hardware. The policy has already produced steering and throttle commands from live camera input and successfully taken turns on the track. What we haven’t achieved yet is a consistently clean lap from start to finish. The system can drive, but it still has room to become reliable enough to handle the entire track without intervention. This isn’t the end, it’s merely the start.
Acknowledgements
comma.ai — For open-sourcing the pretrained driving models that made this project possible.
simcity — For sharing inspiring work in autonomous driving that motivated us throughout the project.
RunPod — For providing the GPU compute that supported our experiments.