Using robot framework for embedded hardware acceptance testing

I've been developing the Buddy DAQ project over the past few years.  The Buddy DAQ is a low cost and open source data acquisition device that connects to a host computer using the USB HID protocol.  During firmware development I was routinely getting hit with regressions.  Minor modifications in the device firmware were throwing the performance out of specification.

Figure 1: Picture of the Buddy DAQ.  The host computer attaches to the micro USB port and the flexible IO pins are exposed on the right 10-pin male IDC connector. 

The hardware in the Buddy DAQ uses a C8051 micro-controller with a small 4k RAM and 48 MHz system clock.  Most of the computational effort in the DAQ firmware is packing and unpacking unsigned integers to and from a frame buffer.  These codec operations dominate the execution cycles so changes to this section of code are very sensitive.  I needed some kind of quick check during development to ensure that the project was still meeting the performance specification.

The best way to ensure the device is meeting performance spec is to push the device at the rated specification and analyze the performance.  The idea here was to push the sample frequency up on the Buddy in DAC or ADC mode of operation and drive or watch the flex inputs to quantify performance.  In DAC mode, we generate a sinusoidal wave of fixed frequency using the DAC and digitize the signal with another instrument.  In ADC mode, we analyze a sinusoidal waveform being generated by a reference test instrument.  We convert the digitized waveform into the frequency spectrum and look for the frequency delta between the true tone frequency and the digitized and recorded signal.  If the measured delta frequency exceeds some threshold value then the test fails and the ADC or DAC is not performing at the specification.

The communication between host PC and Buddy DAQ uses a common codec library where channel values are packed in a 64 byte frame buffer.  Memory copy operations incur a hefty run penalty so we build the frame in one memory location before performing a single copy operation to the destination USB endpoint buffer.  This leaves a lot that can be get whacked during development as simple thing likes accidentally changing the buffer memory model or adjusting alignment can cause big slowdowns in codec operations leading to degradation in the actual sample rate.

I use Robot Framework as the automated test framework and tool on the host PC.  Using Robot Framework, you write test suites and support routines in an expressive verb syntax that makes clear the test intent.  The framework is python-based and that's awesome because Buddy DAQ already has existing SWIG python bindings.

Test List

The following test cases are implemented.  Some are general and meant to check version and identification information while the others are mode-specific and meant to validate performance specification.  

• Exercise each channel by cycling through each flex pin and performing an all channel test.
• Exercise each resolution (8 bit, 16 bit, and 32 bit) modes where appropriate.

Test Name	Test ID	Description	Result
Device Version	BUDDY_FW_VER_CHECK	PC requests device version information from Buddy device.	Check if application and bootloader version are sane.
Device Information	BUDDY_FW_SERIAL_NUM_CHECK	PC requests device information from Buddy device.	Check build datetime, version, and DAC type.
USB Identification	BUDDY_USB_ID_CHECK	PC returns OS device information on USB device from Buddy device.	Check USB manufacturer and product string.
External Teensy Version	BUDDY_TEENSY_ID_CHECK	Teensy returns the version information when requested.	Check if Teensy MCU version are sane.
ADC static simple	BUDDY_DAC_STATIC_CHECK	Buddy supplies static DAC output voltages. A step cycle is repeated for each channel with DAC output voltages (0, 256, 512, 1024, 2048, and 4095).	Check if digitized static voltage are valid on the Teensy.
ADC dynamic	BUDDY_DAC_DYNAMIC_CHECK	Buddy supplies dynamic input voltage waveform.  A sinusoid of fixed generation frequency with a fixed sample rate is sent using stream mode.	Record ADC samples on Teensy.  Take FFT of samples collected and validate tone frequency.
DAC static	BUDDY_ADC_STATIC_CHECK	Set a series of static output voltages on the Teensy.	Check if digitized output voltage as detected by Buddy are valid.
DAC dynamic	BUDDY_ADC_DYNAMIC_CHECK	Set a dynamic voltage waveform.  Use a sine wave being driven from the Teensy.	Record ADC samples on the Buddy.  Take FFT of samples collected and validate tone frequency.
PWM frequency	BUDDY_PWM_FREQUENCY_CHECK	Set a series of static PWM frequencies over a given clock range on the Buddy.	Check if detected frequency matches expected frequency on the Teensy.
PWM duty cycle	BUDDY_PWM_DUTY_CHECK	Set a series of static PWM duty cycles with a fixed base frequency on the Buddy.  Utilize all base frequencies.	Check if detected duty cycle matches the expected duty cycle on the Teensy.
Tick counter	BUDDY_COUNTER_CHECK	Set a series of static frequencies (1 kHz, 10 kHz, and 100 kHz) on the Buddy.	Check if detected tick period matches the value expected on the Teensy.

Table 1: Description of the test cases implemented to test the Buddy DAQ.

Teensy test jig design

The test jig board needed a microcontroller with a fast clock speed, large memory, and rich on-chip peripherals (1x DAC, 8x ADC, 8x PWM).  The Teensy 3.2 was used in the first iteration of the hardware as it is easily the most flexible, documented, easy-to-source, and miniature solution out there.

Figure 2: Picture of the configuration used for test.  The Buddy DAQ board is connected to the Teensy test jig board using a 10 pin IDC ribbon cable. 

For the second iteration of the hardware, I moved to the Teensy 3.6 as it runs at more than 2x the clock frquency (180 MHz v. 72MHz) with 4x the RAM (256k v. 64k).  The additional RAM allows the Teensy software to buffer larger stream waveforms.  This is a major benefit as the ADC or DAC on the Teensy can sample at a higher rate allowing capture and generate of waveforms with better time resolution.

A block diagram showing the hardware design of the test jig is provided in Figure 3.  The Buddy DAQ device is connected to the test jig (FIO_0 - FIO_7).  The Teensy drives a DAC signal into an 8:1 multiplexer (MAX4617) with GPIO outputs from Teensy acting as selectors and an enable line.  Two 8 channel single-pole-single-throw SPST (MAX335) chips are used to route incoming ADC, PWM, and counter into the Teensy.  The Teensy acts as a SPI master and interfaces with the two SPST chips using two GPIO lines as chip enables.

Figure 3: Block diagram of the Teensy test jig board connected to the Buddy DAQ.

Figure 4: (A) Back and (B) Front picture of the Teensy test jig board.  The Buddy DAQ board connects to the test jig board by the IDC ribbon header on the left or the high density mezzanine connector.

FFT Calculation

A sine waveform is generated and output by a DAC.  This test is run with the Buddy operating in ADC and DAC modes with the Teensy configured in the opposing mode to monitor or provide stimulus.  

In the first test (BUDDY_ADC_DYNAMIC_CHECK), the Teensy test jig generates the sine waveform on its DAC output and directs the 8:1 Mux to the desired flex channel. The Buddy DAQ is then instructed to sample on the same flex channel using the Buddy ADC.

In the second test (BUDDY_DAC_DYNAMIC_CHECK), the Buddy generates a sine waveform using the DAC and the waveform is sampled by the test jig board by routing the flex channel output through the SPST circuit and into an ADC channel on the Teensy micro-controller.


The recorded waveform data has an FFT operation run to analyze waveform data in the frequency domain. The FFT magnitude data is then fed into a peak detector to find the frequency of the input tone (Figure 5).


The dynamic frequency test expects the detected tone frequency to fall within the following range, otherwise the test is deemed a failure.

• BUDDY_DAC_DYNAMIC_CHECK: 10 ± 15 Hz.  
• BUDDY_ADC_DYNAMIC_CHECK: 50 ± 4 Hz

Figure 5: A block diagram showing the FFT, peak magnitude detection and expected frequency evaluation operation.

(a) A single channel on the Buddy DAC generates a 10 Hz tone and the waveform is recorded by the Teensy.

(b) The Teensy uses it's onboard DAC to generate a 50 Hz sinusoidal tone and the signal is digitized by the Buddy DAQ operating in ADC mode.

Figure 6: FFT magnitude spectrum plots showing a sampled and recorded sinusoidal time waveform that when transformed shows the tone frequency.

File Logging

I quickly realized I desperately needed logs when running these tests.  When running the FFT analysis I wanted to be able to go back and understand why the FFT and threshold detection was failing.  We also save the matplotlib generated temporal and FFT magnitude-frequency plots.  The CSV files are saved in the `csv` directory in the standard `reports` directory.  The generated plot image files are saved into an images directory on disk.

Caveats

Initially I had the test cases defined in a single robot framework (.robot) file.  I was having issues with intermittent test failures when all the tests were combined in this fashion.  I attribute this mostly to the USB HID backend not getting properly cleaned up after each test suite ran.  I saw occasional timeout errors with the serial link to the Teensy but most of this was memory overruns related to the RingBuffer.  I added the ability to reset both the Buddy and Teensy on a software command from the robot framework tests using the watchdog timer in both firmware.  

Future

• Refactor teensy software.  Break the functionality into separate source files.  Add error handling and reporting.
• Refactor robot framework support libraries.  Move all threshold checks into the test suite but keep all the heavy lifting behind the scenes in support *.robot or wrapper python libraries.

References

1. Buddy DAQ Robot Framework tests

2. Buddy DAQ project on github

3. Robot Framework official website
4. Teensy 3.6 pinouts