June 16, 2018

Integrated Motor Controllers V2: Searching for the noise

Remember how I said I was going to re-do the integrated motor controller using the new TI DRV8323 chip?  Well, I did that.  First round of boards came in and had a few errors, but I think I've found them all. 

Changes:

  • DRV8323RS for gate drive, buck converter, and current shunt amplifiers
  • Smaller footprint - 37x37mm
  • Reasonable shape!  Square mounting hole pattern, centered position sensor
  • All components (except for the position sensor) on the top side - the underside is flat for easy mounting and heat-sinking
  • Designed for through-board cooling, with FET pads thermal-via'd to the underside
  • True 4-layer design, 0402 package passives wherever possible, 6 mil trace/spacing, to make it small
  • Smaller buck passive, smaller CAN transceiver

Front:



Back side.  Notice how empty it is.


Attached to a motor.  Unfortunately I forgot to take a picture of the board before doing a lot of reworking on it, so it's kind of ugly now:


After correcting a couple easy-to-fix mistakes with bodge wires (forgot to connect the CAN transceiver standby pin to ground, routed one of the wrong current sense amplifier outputs to the A/D, since I'm only using 2 out of the 3), the controller worked (as in, it was able to spin a motor), but it was clear immediately that something was wrong with the current sensing.  There was tons of audible noise once the current loop was closed, so I took a look at the measured phase currents.  Here's a plot of the output from the 2 A/Ds, while setting a constant d-axis voltage and slowly rotating the motor by hand:


The first current measurement looks beautiful, but the second one is absolute garbage.  The big spikes are ~750 A/D counts, which would correspond to 15 amps.  Time to break out the scope:

Here's the output of the amplifier:  The big spikes are at around 8.7 kHz, but there's also a lot of higher-frequency noise:


Zooming in:  There are also smaller pulses at 675 kHz.  That's almost certainly the switching frequency of the buck converter on the DRV chip:


Next step was to see if this noise actually showed up on the input to the amplifier, or whether it was showing up after the amplifier or inside the chip somehow.

Test points attached:


I measured in a few places.  First, directly across the shunt.  As expected the noise didn't show up directly across the shunt - unless there were actually many amps of noise flowing through the 1 milli-ohm resistor, there shouldn't be any volts across it.  

I moved the test points ot the two vias directly before the input to the DRV:


And the noise appeared.  For these measurements, the Rigol didn't really have the resolution, so I broke out the Yokagawa DL708E, which has 8 isolated channels with 12-bits of resolution (4 bits more than the Rigol).


After two evenings of probing around and making very little progress, I finally found the two layout errors that combined to cause this problem.

Here's the board with ground highlighted.  The logic ground up top is on an inner layer, and the power ground below is on both the bottom plus an inner layer.  Take a look at the border between the two:


The three highlighted traces crossing the border are the traces for sensing across the current shunts (and from the sources of the un-shunted FET).  Yep, I accidentally forgot to connect the logic ground to the power ground.

Which brings up the question:  How did the board even turn on at all?  The traces from the shunts are inputs to amplifiers, and should be high impedance.  It shouldn't be possible to power all the logic through them.  And why didn't Eagle yell at me when I ran DRC, or show any air wires?

Taking a closer look at the noisy shunt, this time with the inner layers visible:


Oh.  Oops.  When I dropped in the vias for the shunt, the ones on the ground-side of the shunt got automatically connected to the internal ground plane.  Which is also why the logic ground plane didn't show up as being disconnected from the power ground plane in the DRC.  So all the power for all the logic was passing from the power ground, through the current sensing trace, through the vias, to the logic ground.  That'll do it.  

At least I've learned a good lesson from this:  Always add kelvin connections to the shunt footprint in Eagle, even if the actual shunt resistor doesn't have them.   This way, the current sensing traces won't be part of the ground net, and this kind of mistake would be impossible.

New boards are on the way, so hopefully I'll be able to do some stress-testing in the next few weeks.  On the firmware side, I plan on finishing all my autocalibration stuff soon.  Then I'll easily be able to slap this controller on all my small motors, and collect a huge pile of motor data.

May 27, 2018

Big Dyno Beginnings

Bayley and I have been thinking about building a bigger motor dyno for a while now, based off the same power system as the Big Kart.  Over the last year we've been slowly collecting parts for it - a torque sensor, a couple more Sonata HSGs, Prius inverters, etc, and I've finally started putting together the mechanical side of things.

Whereas the small dyno is good for 10 N-m of torque and 6000 RPM, this one will be good for 60 N-m and 15,000 RPM (although not simultaneously), and somewhere around 30 kW power.

First step was acquiring the appropriate torque sensor.  The DC-output one I have on the small dyno is extremely convenient, because you can just measure the output with a DAQ or A/D, but the AC coupled ones are much more common on Ebay.  Unfortunately, the readouts for the AC torque sensors are very expensive and hard to come by.  A pile of really big, really cheap DC-operated torque sensors showed up on ebay for $28 apiece, so we got one and the MIT FSAE team collected the rest of the lot.

10,000 inch-pounds! (~1100 N-m) and 8500 RPM.  With ~2" shafts on either end.  Not actually very useful.


We gutted the readout electronics from it:


And shoved them into this AC-operated 500 in-lbs, 15k RPM torque sensor.  This thing is pretty much perfectly matched to the Sonata HSGs.


Several months ago I went on a shaft-coupling machining spree, making the two lovejoy couplings for the torque sensor, and two more for a pair of the HSGs:





The shaft couplings all have internal keyways, which I manually shaped on the lathe.  I ground HSS inserts for a boring bar, locked the lathe spindle, and shaped the keyways with the carriage feed, retracting the cross slide ~.005" per stroke, with occasional spring passes.  The keyways can be easily widened by shifting the tool up a little on the toolpost.


I've been pleasantly surprised by how nicely the keyways turn out with this process.  Even for some big 6mm keyways for pulley hubs for the Big Kart, the results were very clean.




The shape of the Sonata HSGs leaves much do be desired.  While at least they are completely sealed and have real shafts, unlike the motors in most hybrid transmissions (i.e. the Prius or Fusion motors, which are tightly integrated into the powertrain, and have to be freshly rehoused to be used at all), they are very strangely shaped.  Their cast aluminum housings have almost no flat or square surfaces, making them very challenging to measure.  I was able to eventually find the positions of all three mounting feet and the shaft by using a surface plate, a height gauge, a set of adjustable parallels, some custom-turned shafts which press-fit into the bolt holes, and one sacrificial motor housing which was bandsawed in half.  The last measurements, which located the motor shaft with respect to the feet, could not be reasonably made without referencing some of the machined bores on the inside of the housing.


I machined a pair of HSG mounts.  The feet were turned on MITERS's  newly acquired Hardinge HLV lathe, and the base was CNC milled on the Super Mini Mill.  The turned feet press-fit into locating pockets on the base, and are fastened from underneath.  The two diagonal feet have 14mm bosses on top, which mate with corresponding 14mm locating bores in the bottom HSG casting.


The underside:  Since the parts needed to get flipped over anyways, I did some gratuitous pocketing on the bottom:


The painful measurement process paid off.  The motors dropped into place perfectly:


Like the small dyno, the base of the big dyno is a huge slab of aluminum.  I started out with a big optical breadboard I got for free last fall.  It's a roughly 1'x3'x1" cast aluminum sheet, with a 2" grid of 1/4-20 holes in it.  Somehow this plate was extremely warped, with roughly a 1.5mm height difference between the center and the edges, in the long direction.  I set up the plate on the bridgeport, and faced the hole thing:

The plate was substantially longer than the x-axis travel of the mill, so I had to swivel the column around half way through to face the whole thing.


To fixture it, I machined a little step into each end.  This way the step clamps hold it down hold right at the edges, and don't put any bending stress into the plate:



Here you can see the huge gap between the center of the plate and the table of the mill:


I was expecting the plate to continue warping as I machined it from residual stress in the material, but by some miracle it didn't.  Once I faced the first side, it just stayed perfectly flat.  I didn't even bother facing the underside, because nothing will be mounted to it.

Pile of chips post-facing:



I squared up all the edges as well, and gave the corners a healthy chamfer. 

Here's the absorber mounted:


And a preview of the whole assembly.  I still need to make the foot to space the torque sensor up appropriately.  The first motor we plan on dyno-ing on this thing is the HSG, so we can squeeze every last drop of performance out of the Big Kart, but after that , the motor on the right will be swappable for any other motor which fits.  The plate is mounted to the top of a cart by some rubber shock mounts to vibration isolate it.


In related news, the Big Kart is frighteningly fast now, thanks to the double-motors, Polychain-based drivetrain, and new lookup tables based on the stall data and better inductance measurements.  Hopefully we'll get some decent videos of it, now that the weather's shaping up.  Here it is after some thrashing by Michael

March 7, 2018

The Rubik's Contraption

A collaboration with Jared.

Before any build log, here's a video:


That was a Rubik's cube being solved in 0.38 seconds.  The time is from the moment the keypress is registered on the computer, to when the last face is flipped.  It includes image capture and computation time, as well as actually moving the cube.  The motion time is ~335 ms, and the remaining time image acquisition and computation.  For reference, the current world record is/was 0.637 seconds.

The machine can definitely go faster, but the tuning process is really time consuming since debugging needs to be done with the high speed camera, and mistakes often break the cube or blow up FETs.  Looking at the high-speed video, each 90 degree move takes ~10 ms, but the machine is actually only doing a move every ~15 ms.  For the time being, Jared and I have both lost interest in playing the tuning game, but we might come back to it eventually and shave off another 100 ms or so.  

Here's the contraption:




The contraption is built from:

Several cubes were broken and rebuilt in the process:





Build Log

The hardware started out with a 1-axis test setup cobbled together out of spare parts.  Below is a big ServoDisc U12 and cruft encoder found lying around MITERS, the motor controller I built for 6.131 a few years ago, with everything wired into a Nucleo for control.  I used this setup to implement the minimum time controller in hardware, and figure out a ballpark for how fast this thing could go.


The answer was pretty darn fast.  


The plot shows a sequence of 90 and 180 degree moves.  Each 90 degree move here took a little under 15 ms.  And that was only at 40 volts.

We tested with an actual cube, using Bayley's Photron for high-speed recording. The "speed cubes" we used have pop-off caps at the center of each face.  Beneath the cap is a screw you can use to adjust the amount of slop in the cube.  We machined some square-drives that press into the recess beneath the cap, so no cube modification was required.




One counterintuitive trick for getting things to work well was to make the cube really tight.  When the cube is loose (like it would be if a person were trying to solve it fast), the outer faces just cam outwards when you try to turn the center faces quickly - see the Cubsplosion video for what this looks like.  It took tightening the cube way past what intuitively felt appropriate, in order to stop the camming action from happening.

From this point, the hardware was relatively straight forward.  The big ServoDisc was swapped for six smaller ones;  2 from my old robot arm, and 4 more from ebay.  The smaller ones have the advantage of having higher torque-to-inertia, so should are capable of even greater angular accelerations, and draw much less power in the process.  I machined a bunch of motor-to-cube couplings, and threw together a box to hold everything out of some laser cut acrylic and scavenged 80/20.

To be specific, the motors are 4 ServoDisc N9M4T motors, and 2 UD9-E's.  These two motors have identical performance, but the N-ones use neodymium magnets, so they're much thinner.


Each motor has a US Digital 2000 line optical encoder strapped on the back.  While that much resolution definitely isn't necessary, we found the encoders for $14 apiece on ebay, brand new, which was an excellent deal.  The encoders only had 1/4" bores through the optical discs, and none of our motors had 1/4" shafts.  In fact, the N9's didn't have any shaft whatsoever protruding beyond the back of the motor.  We stuck each motor in the lathe, held by the front shaft, and reamed 1/4" holes in the backs.  1/4" dowel pins were pressed into the holes, to add enough shaft to mount the encoders to.  The N9's also had no features to mount the encoder sensor and housing to, so they were held on with some VHB.


I  built some custom DC motor controllers to do the servo-ing:


The motor controllers aren't anything too fancy - in fact, they don't even have current sensors on them.  Gate drive uses and external 12V power supply and bootstrapped gate drive optoisolators.  They've each got an STM32F303K8 doing all the control.  Commands are sent to each controller over a differential serial interface, and there are 2 extra GPIO pins broken out, which are used for synchronization between the 6 motors.  The controller is built out of 100V FETs, so I was expecting it to run up to 60V with no issues, but we've managed to blow up one of them twice at that voltage, when the cube locked up.  I'm not sure if it's a conventional FET-blew-up-from-too-much-current situation, or ringing on the bus causing the FETs to avalanche and die, or something like that.  It is suspicious that the same one has died twice though. I also have a set of lower-Rds(on), higher gate charge version of these FETs I could swap to, which should help on both accounts.

Here's the firmware running on them

Jared came up with an excellent system for synchronization/anti-collision between the 6 faces.  On the hardware side, this consists of the AND BOARD, which, as you might expect, basically is just an AND gate (it also does the logic power distribution).  Each motor controller has a digital output going to the AND BOARD, which ANDs the six signals, and a digital input driven by the output of the AND gate.


When a motor starts a move, it turns off the output of the and board, and turns it back on once it has completed the move.  When none of the motors are moving, the output of the AND gate goes high, signalling to the next motor in the move sequence that it's safe to start moving.

After the solve sequence is found, the sequence of moves is sent to all the motor controllers at once over differential serial.  Each motor controller has a different ID number, so they can pick out the moves corresponding to their ID number.  Using the output of the AND Board to keep the count of elapsed moves, the six motor controllers step through the sequence of moves until the cube is solved.

Two PlayStation Eye cameras are positioned pointing towards opposite corners of the cube, so that all 6 faces can be seen with only 2 cameras.  Jared was able to get these things running at ~150 fps under Linux, and at very low latency compared to your typical webcam


Here's part of the software stack ;)
Up top is a monitor showing the camera's views of the cubes.  Jared's software allows you to draw polygons on the screen, and assign them to different faces of the cube.  The pixels inside each polygon are averaged and compared to some thresholds to determine the color of each face.  We found that distinguishing the red faces from the orange ones was finicky, thanks to the low-quality of the cameras, so we sharpied the orange faces black.  Apparently that's allowed though.

Once the faces are identified, the solve sequence is computed using the min2phase algorithm.  This returns a solution which we've observed to be 19-23 moves for a well-scrambled cube.  The fastest solve we've done was actually a 21-move solve, so just getting a little lucky could easily shave off another ~30ms from the time.

And like all good projects, this thing requires a tall stack of power supplies to run - 5V logic power, 12V gate-drive power, 24V LED power, and motor power.


For consistent lighting, there's a bright LED next to each camera, lighting up the cube from the corners:


Here's a better view of how the motors interface with the cube.  Popping the cap off each center face reveals an adjustment screw, and a square cavity which the shafts interface with.  Each of the shafts can be loosened to slide out of the cube, so that the cube can be removed from and inserted into the box.



For added legitimacy, here's a scope shot of some signals during the fastest solve.  The blue trace is one of the pins on the AND Board.  When it toggles high at the end, the motor drivers think they are all done moving.  From the high-speed video at the beginning of the post, you can see that the AND board actually turns on a few ms after the cube could be counted as being solved.  The yellow trace is connected to the DTR pin on the USB FTDI adapter going from the computer to the motor drivers.  When the solve button is pressed, the first thing the software does is toggle this pin, before even capturing the images from the webcams.


Here's another entertaining scope shot.  The yellow trace is the AND board, blue is the DC bus current measured with the LEM-Stick.  The purple trace is the integral of the blue trace.   During this particular test, the contraption drew 6 Amp-seconds of current (i.e. 6 coulombs of charge) at 40 V, meaning that the solve took 240 Joules of energy, and peak power was ~1300 watts.


February 17, 2018

Small motor controller with integrated position sensor

A while ago I added the hall effect encoder IC I've been using directly to the motor controller PCB.  The controller sits directly on the back of the motor (with a magnet added to the motor shaft), and the phase wires solder straight in.  I also have a pair of board-mounted XT30 connectors on the DC bus for easy daisy-chaining.  Otherwise, the board is basically identical to the previous version of this controller.  I've now built over a dozen of these, and have had no problems.


I plan on doing another revision of this controller using the new replacement to the DRV8301/2, the DRV8323, which Austin has played around with already.  That should let me shrink the controller even more.  I also plan on designing some general-purpose mounting brackets that fit a wide variety of hobby motors, to make it really easy to strap these controllers on the back of basically any small motor.  In a similar vein, I've also mostly written an auto-tuner for automatically measuring motor parameters for tuning the current loops.  I've verified that the measurement works, but haven't integrated it with the rest of my motor control code yet. 

Once I get those things done, it should be super easy for me to throw all the small motors I have onto the motor dyno and fill out the motor database quite a bit.

For testing and easy dyno-ing I machined some mounts for attaching the controller on the back of my T-Motor U8's and U8 knock-offs.  This isn't the intended use of this controller, but I'll get into that in a later post.


Parallel XT-30's for power, and parallel SPOX for the CAN bus.



Same old serial and programming connectors up top:


The whole reason for the linearization step in my autocalibration procedure was because I misplaced the encoder IC in my first revision of these boards, so the encoder sat .5mm off-axis.  So rather than waiting on the board revision to get things up and running, I fixed the problem in software.  The version of the controller here is fixed though.

Here are the eagle files and gerbers for the boards if you're interested, although I'll hopefully get to the DRV8323 version soon, making these obsolete.
2-layer
4-layer (2 extra layers on the power planes are way cheaper than 4-oz boards.  Helpful for daisy chaining)