# π¦Ύ NeuralNexus Arm
### A 6-DOF robotic arm built from scratch β mechanics, firmware, kinematics, and a live digital twin.
[](#)
[-A8B9CC?style=for-the-badge&logo=c&logoColor=black)](#)
[](#)
[](#)
[](#)
[](#)
---
## π Table of Contents
| # | Section |
|---|---------|
| 1 | [Motor Selection](#1--motor-selection) |
| 2 | [Motors, Gearing, Speeds & Joints](#2--motors-gearing-speeds--joints) |
| 3 | [Simulation, IK & Path Planning](#3--simulation-ik--path-planning) |
| 4 | [Mechanical & Electrical Challenges](#4--mechanical--electrical-challenges) |
| 5 | [Hardware Stack](#5--hardware-stack) |
| β | [Software Architecture](#-software-architecture) Β· [Control Panel](#-web-control-panel) Β· [Quick Start](#-quick-start) Β· [Roadmap](#-roadmap) |
---
## π― What This Is
A fully custom 6-axis robotic arm β **not a kit**. Every layer was designed, debugged, and integrated in-house:
```
CAD (SolidWorks) β Simscape Multibody twin β Inverse Kinematics solver
β
USB CDC @ 115200 baud
β
STM32H743 @ 420 MHz β 6 stepper axes + gripper
β
AS5047P absolute encoders (SPI feedback)
```
**The headline feature:** a single command drives the **physical arm and the 3D digital twin simultaneously**, in sync, in real time.
---
### π₯ Demos
[](https://drive.google.com/file/d/1faK7I7ZX_agUlC0IrAXy0S09C4y3s5h2/view?usp=sharing)
[](https://drive.google.com/file/d/1S6juB1wD2G7o0wvhlD4bZA8bZekDnkWw/view?usp=drive_link)
[](https://drive.google.com/file/d/1cnsTrxQivk0x1Dtu1PmsmbPZ_AezZNNF/view?usp=sharing)
[](https://drive.google.com/file/d/1yWmGitRZMLtR4NfW3XfLVypYUs1kBSMH/view?usp=drive_link)
[](https://drive.google.com/file/d/1Re87R6V8fl_Va-D2lkvHRHDZWcIKDiEP/view?usp=drive_link)
---
## 1. π§ Motor Selection
Motor choice was driven by one principle: **torque scales with how much arm you're carrying.** The base carries everything; the wrist carries almost nothing. So the arm uses a *tiered* motor strategy rather than six identical motors.
### Selection Logic
```mermaid
graph TD
A[Joint torque requirement] --> B{Carries how much
downstream mass?}
B -->|Base + all links| C[NEMA 23/24
High torque]
B -->|Wrist only| D[NEMA 17
Low torque, compact]
C --> E{Positional risk
if steps lost?}
E -->|Severe| F[Closed-loop driver
CL57T]
E -->|Moderate| G[Open-loop driver
DM542]
D --> H[Onboard TMC2209
silent, microstepping]
```
### Why Each Choice
| Tier | Joints | Motor | Reasoning |
|:-----|:-------|:------|:----------|
| **Heavy** | J2, J3 | **NEMA 24** | Shoulder & elbow lift the entire outboard arm at maximum lever arm. Highest torque demand in the system. |
| **Medium** | J1 | **NEMA 23** | Base rotation carries full mass but works against *inertia*, not gravity β lower continuous torque than the lifting joints. |
| **Light** | J4, J5, J6 | **NEMA 17** | Wrist joints position only the end-effector. Compact size matters more than torque β mass here penalises every joint below it. |
> π‘ **Design insight:** every gram added at the wrist is a gram J2 and J3 must lift at full extension. Motor *mass* is a design constraint, not just torque.
### Driver Pairing
| Driver | Joints | Type | Why |
|:-------|:-------|:-----|:----|
| **CL57T** | J2, J3 | Closed-loop | Missed steps on the lifting joints = the arm silently drops. Closed-loop drivers detect and correct stalls internally. |
| **DM542** | J1 | Open-loop | Base rotation is more forgiving β a lost step is a heading error, not a collapse. |
| **TMC2209** | J4, J5, J6 | Onboard, silent | Integrated on the custom PCB. Quiet operation, fine microstepping, no external driver boxes for the wrist. |
---
## 2. βοΈ Motors, Gearing, Speeds & Joints
### Full Drivetrain Map
| Joint | Role | Motor Slot | Motor | Driver | Signal Type | Gear Ratio | Pulses / Joint Rev |
|:------|:-----|:-----------|:------|:-------|:------------|:-----------|:-------------------|
| **J1** | Base rotation | M6 | NEMA 23 | DM542 | Open-drain | **1 : 7** | 2800 |
| **J2** | Shoulder | M4 | NEMA 24 | CL57T | Open-drain | **1 : 30** | 5000 |
| **J3** | Elbow | M5 | NEMA 24 | CL57T | Open-drain | **1 : 30** | 5000 |
| **J4** | Wrist pitch | M2 | NEMA 17 | TMC2209 | Push-pull | **1 : 1** | 6400 |
| **J5** | Wrist roll | M1 | NEMA 17 | TMC2209 | Push-pull | **1 : 1** | 1600 |
| **J6** | End-effector | M3 | NEMA 17 | TMC2209 | Push-pull | **1 : 1** | 1600 |
> β οΈ **Joints are NOT wired in motor order.** The jointβmotor mapping is `jointToMotor[6] = {5, 3, 4, 1, 0, 2}`. This is baked into firmware so every layer above it can think in clean joint numbers (J1β¦J6, baseβtip).
### J2 / J3 Compound Drivetrain
The two lifting joints use a **two-stage reduction** β and where the encoder sits inside that chain turns out to matter enormously (see Β§4):
```
Motor ββ[ 1:10 planetary gearbox ]βββΊ ENCODER ββ[ 1:3 belt/pulley ]βββΊ Joint
β²
magnet mounted HERE
(3 encoder turns per joint turn)
```
### Joint Limits & Reference Pose
| Joint | Min (Β°) | Max (Β°) | Range | Stable Pose (Β°) | Notes |
|:------|--------:|--------:|------:|----------------:|:------|
| J1 | β180 | +180 | 360Β° | 0 | Full rotation |
| J2 | β70 | +70 | 140Β° | β5 | Gravity-limited |
| J3 | **+120** | +240 | 120Β° | **120** | β οΈ Starts *at* its lower limit β can only move **up** |
| J4 | β180 | +180 | 360Β° | 0 | Full rotation |
| J5 | +60 | +300 | 240Β° | 160 | 180Β°-centred band |
| J6 | β180 | +180 | 360Β° | 0 | Full rotation |
**Stable reference pose:** `[0, β5, 120, 0, 160, 0]` β the hand-matched starting configuration that keeps the physical arm and the simulation aligned for all relative motion.
### Motion Control Parameters
| Control loop | TIM6 ISR @ 2 kHz (PSC 74, ARR 499) |
| Tick period | 500 Β΅s |
| Max step rate | 2000 steps/s (1 step per tick β hard ceiling) |
| Working speed | 400 steps/s per joint |
| Acceleration | 150 steps/sΒ² |
| Profile | Trapezoidal ramp with decel-to-stop lookahead |
### The Motion Profile
Every axis runs a non-blocking trapezoidal velocity profile inside the ISR β accelerate, cruise, decelerate β computed independently per joint:
```
speed
β²
β βββββββββββββββββββ
β β± β²
β β± β²
β β± β²
βββ΄ββββββββββββββββββββββββββ΄βββΊ time
accel cruise decel
(stepsToStop lookahead)
```
> π **The scaling trap:** when the ISR was retuned from 50 kHz β 2 kHz, the hardcoded constants `dt = 0.00002` and `stepInterval = 50000/speed` were **25Γ wrong**, making every motor crawl. Correct values: `dt = 0.0005`, `stepInterval = 2000/speed`. Tick rate and motion math must always be changed together.
### Command Protocol
| Command | Format | Example | Action |
|:--------|:-------|:--------|:-------|
| **Joint move** | 6 comma-separated degrees | `0,-50,60,-80,-90,50` | Relative move on all six joints |
| **Gripper** | `G,<0\|1>` | `G,1` | Open / close servo |
| **Home** | `HOME` | `HOME` | Encoder-based homing (J4/J5/J6) |
| **Test** | `TEST` | `TEST` | 30Β° encoder calibration measurement |
---
## 3. π§ Simulation, IK & Path Planning
### Pipeline
```mermaid
flowchart LR
A[SolidWorks CAD] -->|export| B[Simscape Multibody]
B -->|importrobot| C[rigidBodyTree]
C --> D[inverseKinematics solver]
E[Target XYZ + orientation] --> D
D --> F[6 joint angles]
F --> G[Quintic blend
trajectory]
G --> H[Mechanics Explorer
3D twin]
G --> I[USB CDC β STM32
physical arm]
style H fill:#0076A8,color:#fff
style I fill:#03234B,color:#fff
```
### Kinematic Model
The CAD assembly exports directly into a **`rigidBodyTree`**, with real joint limits applied so the solver never proposes an unreachable pose:
```matlab
robot = importrobot('Assem1');
robot.DataFormat = 'row';
limitsDeg = [-180 180; -70 70; 120 240; -180 180; 60 300; -180 180];
for i = 1:6
robot.Bodies{i}.Joint.PositionLimits = deg2rad(limitsDeg(i,:));
end
ik = inverseKinematics('RigidBodyTree', robot);
```
### Trajectory Generation
Paths are interpolated with a **quintic (5th-order) blend** rather than linear interpolation β this gives zero velocity *and* zero acceleration at both endpoints, so motion starts and stops smoothly instead of jerking:
```matlab
b = (k-1)/(steps-1);
blend = 10*b^3 - 15*b^4 + 6*b^5; % smoothstep: S-curve, CΒ² continuous
pose = startPose + (targetPose - startPose) * blend;
```
### Planned Paths & Demos
| Demo | What It Proves | Method |
|:-----|:---------------|:-------|
| π― **Position Lock** | Tip holds a fixed point in space while the arm reconfigures around it | IK solved for constant XYZ, sweeping orientation |
| π§ **Orientation Lock** | Tool orientation stays fixed while the tip sweeps X, then Y, then Z | IK solved for constant rotation matrix, moving XYZ |
| β¬ **Vertical Square** | Coordinated multi-joint straight-line tracing | 4 corners in the YβZ plane, edges interpolated, IK per point |
| π **Bye Wave** | Balanced choreography that returns exactly to start | Hand-tuned deltas, every joint column sums to zero |
| π€ **Pick & Place** | Full task sequence with gripper actuation | Recorded relative poses + `G,0` / `G,1` |
> π¬ **A fundamental result we proved (and had to design around):**
>
> **A 6-DOF arm cannot hold tip position AND tool orientation fixed simultaneously while the links reconfigure.** That needs a 7th degree of freedom. Six DOF = exactly six constraints = a unique solution with no redundancy left to exploit. Every "lock" demo therefore locks *one* constraint and varies the other. This wasn't a bug β it's the kinematics telling us the truth.
### Synchronised Twin
The `moveBoth()` function is the centrepiece β one call, two arms:
```matlab
moveBoth(s, [0 10 0 0 0 0]) % J2 +10Β° on BOTH the real arm and the simulation
```
It tracks accumulated pose in a persistent variable, runs a soft-limit check before committing, streams the delta over serial, and animates the Simscape model β all from a single command.
---
## 4. π₯ Mechanical & Electrical Challenges
> *This is the section that actually took the time.* Every entry below cost hours of debugging and is documented so it never has to be rediscovered.
### π₯ The Big One: Phantom Encoder Wander
**Symptom:** encoder readings on a *stationary* joint appeared to swing by **60β70Β°** between samples. Completely unusable for absolute positioning.
**Failed hypotheses** (each cost a debugging cycle):
| Theory | Test | Result |
|:-------|:-----|:-------|
| SPI too fast β signal integrity | Slowed prescaler 32 β 64 | Slightly better |
| Still too fast | Slowed further β 128 | β **Got WORSE** |
| ISR corrupting SPI mid-frame | Moved reads to main loop | β No change |
**Actual root cause:** the read function returns **`β1` on a failed read**, and those `β1` sentinels were being **averaged in with valid readings**. A few error codes in a 30-sample mean dragged the average wildly. The sensor was perfect the whole time.
**Fix:** filter `β1` before averaging.
```matlab
if row(j) >= 0 % β the entire fix
sums(j) = sums(j) + row(j);
counts(j) = counts(j) + 1;
end
```
**Result:** readings repeatable to **Β±0.01Β°** across runs.
```
BEFORE (raw mean): J4 = 310.40 β 274.49 β 346.27 (Β±72Β° spread) β
AFTER (filtered): J4 = 181.88 β 181.88 β 181.88 (Β±0.01Β°) β
```
> π§ **The lesson:** *"slowing it down made it worse"* was the decisive clue. A genuine signal-integrity problem always improves with a slower clock. When it didn't, the problem was never analogue β it was a data-hygiene bug in software.
---
### β‘ Electrical Failures
| Failure | Cause | Resolution |
|---|
π 3 GPIO pins destroyed
PE0, PD7, PD3 |
5 V back-feed during miswired open-drain testing β external pull-ups fed 5 V into 3.3 V-tolerant-only pins |
Remapped in .ioc β PE7, PD11, PD12. Dead pins left unassigned (Hi-Z). One bodge wire via the M3_MS1 pad. |
π LDO destroyed
ST1L05CPU33R (3.3 V rail) |
Failed on connection to the main electronics compartment β suspected input overvoltage, inductive transient from the steppers, or insufficient local decoupling |
Open. Measure actual VIN at the LDO footprint before fitting a replacement. |
---
### π The Driver Topology Trap
The custom PCB mixes **two electrically incompatible driver families**, and getting the GPIO mode wrong produces a *silent* failure β no error, the motor just doesn't move.
| | J4 / J5 / J6 (TMC2209) | J1 / J2 / J3 (CL57T Β· DM542) |
|:--|:--|:--|
| **Mounting** | Onboard | External boxes |
| **GPIO mode** | `OUTPUT_PP` (push-pull 3.3 V) | `OUTPUT_OD` (open-drain) |
| **Wiring** | Direct logic | Common-anode via optocoupler |
| **Enable** | `SetEnableM1M2M3(true)` β LOW | `SetEnableM4M5M6(false)` β **HIGH** |
> β οΈ **Counter-intuitive but correct:** `Stepper_SetEnableM4M5M6(false)` is what **enables** the external drivers. With common-anode wiring, driving the open-drain pin HIGH releases it (Hi-Z) β opto LED dark β drivers enabled. This looks like a bug in code review. **It is not. Do not "fix" it.**
---
### π Firmware Bugs Worth Remembering
Double direction handling β homing only converged from one side
`Stepper_MoveAllJoints()` applies `jointDir[]` internally. The homing routine *also* computed direction from the measured `encStepSign`. Two sign flips = they cancelled on one side of the target and doubled on the other.
**Fix:** homing and encoder-driven motion must use `Stepper_MoveAll()` β the raw, motor-indexed mover with **no** `jointDir` applied.
The unreachable gripper branch β G,0 silently did nothing
```c
if (rxFlag) {
rxFlag = 0; // β cleared here
...
else if (rxFlag) { // β always FALSE. Dead branch.
if (rxTemp[0] == 'G') { Gripper_Open(); } // never reached
}
}
```
The gripper worked perfectly when called directly from the loop β which proved the hardware was fine and pointed straight at command dispatch.
**Fix:** flatten the handler; make the gripper check a top-level `else if`.
Off-by-one on joint index β J6 refused to home
`uint8_t j = 6;` β but joints are indexed **0β5**. J6 is index **5**. Every array access (`homeAngle[6]`, `motors[6]`, `js[6]`) read past the end of its array β undefined behaviour that happened to produce silence instead of a crash.
Two infinite loops β commands ignored while encoders streamed
A leftover `while(1)` encoder-print loop sat *above* the main loop, trapping the CPU forever. The command handler below it was never reached. Symptom looked like "the arm reads encoders but ignores serial" β actually the CPU never got there.
Serial handshake deadlock β MATLAB froze mid-demo
A periodic `enc |` status print flooded the serial line, so MATLAB's `readline() == "END"` handshake never matched β infinite wait, sim and arm both frozen.
**Fix:** telemetry streaming and command handshaking cannot share one wire. Pick one mode at a time, or make the reader tolerant with a timeout.
---
### β±οΈ The Timing Discovery
**Symptom:** commanding 60Β° produced only ~30Β° of motion. Increasing the amplitude didn't help.
**Measurement:** a 60Β° move physically takes **~4 seconds** at 400 steps/s. The demo scripts were sending the next command after **1 second** β every move was being interrupted at ~25% completion and replaced with a fresh target.
**Solution (user-devised):** never send a move larger than **10Β°**. Split a 60Β° move into six 10Β° commands, each of which completes comfortably within the 1 s window.
```matlab
nChunks = ceil(max(abs(delta)) / 10);
stepDelta = delta / nChunks;
for c = 1:nChunks
writeline(s, sprintf('%.2f,%.2f,%.2f,%.2f,%.2f,%.2f', stepDelta));
pause(1.0);
end
```
> β
Elegant because it needs **no firmware change and no handshake** β it just respects the physics of how long a move actually takes.
---
### π© Mechanical: The Simscape Over-Constraint
The imported model wouldn't move at all. Cause: **over-constrained SolidWorks mates**.
| Problem | Effect |
|:--------|:-------|
| Extra Parallel / Perpendicular mates | Collapsed revolute joints into rigid welds β zero DOF |
| Base mated to non-adjacent links | Created closed kinematic loops the solver couldn't resolve |
**Rule established:** exactly **one concentric + one coincident/distance** mate per joint. Nothing more. Clean serial chain: `Base β P1 β P2 β P3 β P4 β P5 β EndEffector`.
---
### 𧲠The Encoder Mounting Constraint
The most consequential *mechanical* decision in the project β and it was discovered after the fact.
An AS5047P is absolute over **one magnet revolution**. Whether a joint can determine its own position at power-on depends entirely on **what gearing remains between the magnet and the joint**:
```mermaid
graph LR
A[Encoder magnet] -->|remaining ratio 1:1| B[β
TRUE ABSOLUTE
one unique reading]
A -->|remaining ratio 1:N| C[β N-fold ambiguity
N poses share a reading]
```
| Joint | Ratio *after* encoder | Absolute on boot? | Consequence |
|:------|:---------------------|:------------------|:------------|
| J1 | 1 : 7 | β | 7 candidate positions per reading |
| J2 | 1 : 3 | β | 3 candidates |
| J3 | 1 : 3 | β | 3 candidates |
| **J4** | **1 : 1** | β
| Unique β true absolute |
| **J5** | **1 : 1** | β
| Unique β true absolute |
| **J6** | **1 : 1** | β
| Unique β true absolute |
**Workaround in use:** hand-position the arm *roughly* near home, then command `HOME`. Pre-positioning within one encoder revolution removes the ambiguity entirely, letting the encoder resolve the fine error. Clever, free, and it works today.
**Permanent fix (future revision):** relocate the J1/J2/J3 magnets to the *joint side* of the final reduction. All six joints then become absolute and the whole problem disappears.
---
### π» Host-Side Gotchas
| Issue | Symptom | Fix |
|:------|:--------|:----|
| **USB CDC won't enumerate** | Device Manager shows nothing / "Unknown Device" | Route **HSI48** to USB clock + call `HAL_PWREx_EnableUSBVoltageDetector()` **before** `MX_USB_DEVICE_Init()` |
| **Board freezes at boot** | No LED, no USB, totally dead | `MX_SDMMC1_SD_Init()` traps in `Error_Handler()` with no SD card. Keep SDMMC1 + FATFS **disabled in the `.ioc`** |
| **COM port keeps changing** | COM9 β COM12 β COM14β¦ | Windows caches stale USB devices. Device Manager β *Show hidden devices* β uninstall greyed entries β replug |
| **`importrobot` fails** | "Variable `jointData` has been deleted" | Simscape *From Workspace* blocks are evaluated during import. Run `jointData = zeros(2,7);` **first** |
| **Timer resource conflict** | Gripper twitched whenever the buzzer sounded | Both were on TIM2. `Buzzer_SetFreq()` reprograms PSC/ARR, destroying the servo's 50 Hz frame. **Moved gripper to TIM1_CH3** β conflict eliminated |
---
## 5. π οΈ Hardware Stack
### Compute
| Component | Spec |
|:----------|:-----|
| **MCU** | STM32H743VITx, Cortex-M7 |
| **Clock** | 420 MHz (25 MHz HSE β PLL) |
| **PCB** | Custom board (migrated from WeAct MiniSTM32H743 dev board) |
| **Toolchain** | STM32CubeIDE + CubeMX (HAL) |
### Actuation
| Component | Detail |
|:----------|:-------|
| **Steppers** | 2Γ NEMA 24 Β· 1Γ NEMA 23 Β· 3Γ NEMA 17 |
| **Closed-loop drivers** | 2Γ CL57T (J2, J3) |
| **Open-loop driver** | 1Γ DM542 (J1) |
| **Onboard drivers** | 3Γ TMC2209 (J4, J5, J6) |
| **Gripper** | Hobby servo Β· TIM1_CH3 Β· 50 Hz PWM Β· 930 Β΅s open / 1100 Β΅s close |
### Sensing
| Component | Detail |
|:----------|:-------|
| **Encoders** | 6Γ AS5047P magnetic absolute, 14-bit (16384 counts/rev) |
| **Bus** | SPI1 β 16-bit frames, CPOL HIGH / CPHA 2EDGE (mode 1) |
| **Chip selects** | `PB0` `PB1` `PB2` `PC7` `PC6` `PD15` (active-LOW) |
| **Noise floor** | Β±0.04Β° at rest (after `β1` filtering) |
### Peripherals & I/O
| Function | Resource |
|:---------|:---------|
| Stepper control loop | TIM6 @ 2 kHz interrupt |
| Gripper servo PWM | TIM1_CH3 |
| Buzzer | TIM2_CH2 / PA1 |
| Host link | USB CDC Virtual COM @ 115200 |
| Status | Onboard LED, user buttons |
### Peripheral Allocation
```
TIM1 βββ CH3 βββΊ Gripper servo PWM (50 Hz)
TIM2 βββ CH2 βββΊ Buzzer (variable frequency)
TIM6 ββββββββββΊ Stepper ISR @ 2 kHz βββ the heartbeat
SPI1 ββββββββββΊ AS5047P encoder chain Γ6
USB ββββββββββΊ CDC Virtual COM Port
```
> π§― **Peripheral conflicts are real.** The gripper and buzzer originally shared TIM2 and were mutually exclusive β playing a tone corrupted the servo signal. Separating them onto different timers was a genuine architectural fix, not cosmetic.
---
## ποΈ Software Architecture
```mermaid
graph TB
subgraph HOST["π₯οΈ Host Layer"]
M[MATLAB
IK Β· Simscape twin]
W[Web Control Panel
Web Serial API]
end
subgraph FW["βοΈ STM32 Firmware"]
P[Command Parser]
S[Stepper_MoveAllJoints
joint-indexed + jointDir]
R[Stepper_MoveAll
raw motor-indexed]
I[TIM6 ISR @ 2 kHz
trapezoidal profile]
E[Encoder read + tick accumulate]
H[Home_All Β· Encoder_Test30]
end
subgraph HW["π¦Ύ Hardware"]
D[6Γ Stepper drivers]
G[Gripper servo]
A[6Γ AS5047P]
end
M -->|USB CDC| P
W -->|USB CDC| P
P --> S
P --> G
P --> H
H --> R
S --> I
R --> I
I --> D
I --> E
A --> E
E --> H
style I fill:#c62828,color:#fff
style M fill:#0076A8,color:#fff
style W fill:#f7df1e,color:#000
```
### Two Movers, One Critical Distinction
| Function | Indexing | Applies `jointDir`? | Use for |
|:---------|:---------|:-------------------|:--------|
| `Stepper_MoveAllJoints()` | **Joint** (J1β¦J6) | β
Yes | Commands from MATLAB / UI |
| `Stepper_MoveAll()` | **Motor** (M1β¦M6) | β No | Homing & encoder-driven correction |
> βοΈ Mixing these up causes the **double-direction bug** β the single most time-consuming class of error in this project. Encoder feedback already knows the true physical direction via `encStepSign`; applying `jointDir` on top of it inverts the correction on one side of the target.
---
## ποΈ Web Control Panel
A standalone **Web Serial** control panel that talks to the arm directly β **no MATLAB, no Python, no install.** Just open the HTML file in Chrome or Edge.
```
ββββββββββββββββββββββββββββββββββββββββββββββββ
β π Connect arm β Connected β Stop β
ββββββββββββββββββββββββββββββββββββββββββββββββ€
β RECORDED SEQUENCES Delay [4]s β
β ββββββββββββ¬βββββββββββ¬ββββββββββββββββββββ β
β β Show off β Pick β Place from pole β β
β ββββββββββββΌβββββββββββ΄ββββββββββββββββββββ€ β
β βPick pole β Pick and place pole β β
β ββββββββββββ΄βββββββββββββββββββββββββββββββ β
ββββββββββββββββββββββββββββββββββββββββββββββββ€
β JOINT CONTROL β relative move β
β J1 βββββββββββββββββββββββββββ 0Β° β
β J2 βββββββββββββββββββββββββββ 0Β° β
β J3 βββββββββββββββββββββββββββ 0Β° β¦ β
β [ Send move ] [ Reset ] β
ββββββββββββββββββββββββββββββββββββββββββββββββ€
β [ Open ] [ Close ] [ Home ] β
β > 0,-50,60,-80,-90,50 [ Send ] β
ββββββββββββββββββββββββββββββββββββββββββββββββ
```
| Feature | Detail |
|:--------|:-------|
| **Transport** | Web Serial API @ 115200 β direct browser-to-MCU |
| **Sequences** | All five recorded motions as one-click buttons |
| **Jog control** | Six centred sliders (β90Β° β¦ +90Β°), sent as one relative move |
| **Safety** | Stop button cancels a running sequence mid-flight |
| **Live log** | Timestamped record of every command sent |
> π **Why this matters for a finished product:** the demo machine no longer needs a MATLAB licence. MATLAB remains the *authoring* environment (IK, trajectory design, the digital twin) β the browser is the *operator* interface.
---
## π Quick Start
### Firmware
```bash
# 1. Open the project in STM32CubeIDE
# 2. If regenerating from CubeMX, re-comment MX_SDMMC1_SD_Init() β it traps without an SD card
# 3. Build (π¨) and flash via ST-Link (βΆ)
# 4. Confirm the board enumerates as a USB Virtual COM Port
```
### MATLAB Session
```matlab
jointData = zeros(2,7); % MUST come first β From Workspace blocks need it
load_system('Assem1');
robot = importrobot('Assem1'); robot.DataFormat = 'row';
limitsDeg = [-180 180; -70 70; 120 240; -180 180; 60 300; -180 180];
for i = 1:6
robot.Bodies{i}.Joint.PositionLimits = deg2rad(limitsDeg(i,:));
end
ik = inverseKinematics('RigidBodyTree', robot);
weights = [1 1 1 1 1 1];
q0 = homeConfiguration(robot);
s = serialport("COM9", 115200); % update port as needed
```
### Everyday Commands
```matlab
armIK(robot, ik, weights, q0) % interactive IK slider tool
goToXYZ(robot, ik, weights, q0, [-0.15 0.30 0.10], s) % XYZ β arm + synced twin
moveBoth(s, [0 10 0 0 0 0]) % relative move on BOTH
armWave(s) % goodbye choreography
objectPicking(s) % pick sequence
writeline(s, "HOME") % encoder homing
```
### Browser Panel
```
1. Close the MATLAB port first: clear s
2. Open NeuralNexusArmControl.html in Chrome / Edge
3. Click "Connect arm" β select the COM port
4. Drive it.
```
---
## π Project Status
| Subsystem | Status | Notes |
|:----------|:------:|:------|
| 6-axis stepper control | β
| Trapezoidal profile, 2 kHz ISR |
| USB CDC command link | β
| 115200, parsed & verified |
| Joint mapping | β
| `jointToMotor` verified on hardware |
| Direction signs | β
| `jointDir` verified via A/B sim-vs-hardware |
| MATLAB IK pipeline | β
| `rigidBodyTree` + solver + limits |
| Synchronised digital twin | β
| `moveBoth()` β arm & sim in lockstep |
| Gripper servo | β
| TIM1_CH3, serial-commanded |
| Encoder read (AS5047P) | β
| Β±0.01Β° repeatable after `β1` filtering |
| Recorded motion library | β
| 5 sequences + wave + square |
| Web control panel | β
| Web Serial, MATLAB-free |
| Homing β J4 / J5 / J6 | π‘ | Works; scaling verification ongoing |
| Homing β J1 / J2 / J3 | π΄ | Blocked by encoder mounting ratio |
| `pulsesPerJointRev` calibration | π‘ | J6 measured at 3200 (not 1600) β others pending |
| Absolute positioning | π‘ | Viable for the 1:1 wrist joints |
| 3.3 V LDO replacement | π΄ | Measure VIN before fitting |
| Closed-loop step correction | βͺ | Planned |
| Vision pick-and-place | βͺ | Planned |
---
## πΊοΈ Roadmap
```mermaid
gantt
title Development Phases
dateFormat X
axisFormat %s
section Complete
Simscape model fix :done, 0, 1
MATLAB IK pipeline :done, 1, 2
STM32 6-axis firmware :done, 2, 3
Encoder integration :done, 3, 4
Synced twin and demos :done, 4, 5
Web control panel :done, 5, 6
section In Progress
Per-joint calibration :active, 6, 7
Absolute positioning :active, 7, 8
section Planned
Closed-loop correction :8, 9
Encoder remount J1-J3 :9, 10
Vision pick-and-place :10, 11
```
---
# ποΈ Computer Vision & Object Detection
> **The eyes of the NeuralNexusArm.** This subsystem turns raw camera pixels into real-world `(X, Y)` coordinates that feed straight into the inverse kinematics solver β closing the loop between *seeing* and *reaching*.
---
## π Pipeline Overview
```mermaid
flowchart LR
A([π· Camera Feed]) --> B[π§Ή Undistort
Camera + Distortion Matrix]
B --> C{π― Detection Mode}
C -->|Primary| D[π¨ OpenCV
Colour Segmentation]
C -->|Fallback| E[π§ Trained Models
YOLO / Custom]
D --> F[π Object Centroid
in pixels]
E --> F
F --> G[πΊοΈ Homography
Pixel β World mm]
G --> H([π¦Ύ IK Solver β Arm Motion])
style A fill:#1f77b4,stroke:#fff,color:#fff
style D fill:#2ca02c,stroke:#fff,color:#fff
style E fill:#9467bd,stroke:#fff,color:#fff
style G fill:#ff7f0e,stroke:#fff,color:#fff
style H fill:#d62728,stroke:#fff,color:#fff
```
---
## Object Detection
### Primary β OpenCV Colour Identification
The main detection path uses **classical colour segmentation** (HSV thresholding β contour extraction β centroid).
Why classical over learned, as the default?
| Latency | Determinism | Compute | Tunability |
|:---:|:---:|:---:|:---:|
| Millisecond-scale | Same input β same output, every time | Runs comfortably on CPU | Live HSV slider tuning |
### Secondary β Trained Detectors
For objects that colour alone cannot separate, the repository also ships learned detectors:
| Model | Target | Type |
|---|---|:---:|
| π¦ Custom trained | Cardboard box detection |  |
| π¬ Custom trained | Candy detection |  |
| `yolo26n.pt` | General-purpose objects |  |
| `yolo11n.pt` | General-purpose objects |  |
---
## Camera Calibration
Calibration runs in **two independent stages** β first fix the *camera*, then fix the *world*.
```mermaid
flowchart TB
subgraph S1["π¬ Stage 1 β Intrinsic"]
A1[βοΈ 20 Chessboard Images] --> A2[Camera Matrix]
A1 --> A3[Distortion Coefficients]
A2 --> A4[β
Reprojection Error
0.6863 px]
A3 --> A4
end
subgraph S2["π Stage 2 β Extrinsic"]
B1[π Fixed Mount @ 41 cm] --> B2[Known World Points]
B2 --> B3[3Γ3 Homography Matrix]
B3 --> B4[β
Pixel β mm Mapping]
end
S1 --> S2
style A4 fill:#2ca02c,stroke:#fff,color:#fff
style B4 fill:#2ca02c,stroke:#fff,color:#fff
```
### βοΈ Stage 1 β Chessboard Calibration *(Intrinsic)*
Standard OpenCV chessboard calibration recovers two matrices:
|
**π Camera Matrix**
Describes the camera's *own* optical properties:
- Focal lengths `fx`, `fy`
- Principal point `cx`, `cy`
|
** Distortion Coefficients**
Model and cancel lens distortion:
- Radial (barrel / pincushion)
- Tangential (sensor misalignment)
|
**Method** β 20 chessboard images captured at varying angles, orientations, and positions across the frame.
> [!TIP]
> **Result: mean reprojection error = `0.6863 px`** β comfortably sub-pixel, which is a healthy calibration for a 20-image set.
### Stage 2 β Homographic Calibration *(Pixel β World)*
With the camera **rigidly mounted at a fixed height**, a single `3Γ3` homography maps undistorted pixel coordinates to real-world planar coordinates in the arm's base frame.
Because the pose is fixed and every target lies on one plane (the table surface), **no per-frame depth estimation is required** β one matrix does the whole job.
```
βββββββββββββββββββββββββββββββββββββββββββ
β π· Camera β
β β β
β β β Fixed height = 41 cm β
β β β
β βββ§ββββββββββββββββββββββββββββββββββ β β Work surface (Z = 0)
β π― Targets live on this plane β
βββββββββββββββββββββββββββββββββββββββββββ
```
> [!WARNING]
> The homography is valid **only** for this exact mounting height and pose. Move or re-aim the camera and the homographic calibration **must be redone** β the intrinsic calibration, however, stays valid.
---
### π Repositories
[](https://github.com/Lasan-Perera/NeuralNexusArm_CodeBase)
[](https://github.com/Lasan-Perera)
---
## π₯ Team
| Index No. | Name | GitHub |
|:----------|:-----|:-------|
| **230487D** | Lasan Perera | [@Lasan-Perera](https://github.com/Lasan-Perera) |
| **230502X** | Deneth Priyadarshana | β |
| **230492M** | Dulana Pitiwaduge | [@DulanaPiti](https://github.com/DulanaPiti) |
| **230051L** | Isitha Dinujaya | β |
| **230108U** | Mokshan Colombage | β |
---