posted on: 02.02.2017
Tomaž Nahtigal

Author:

Tomaž Nahtigal

Lead Engineer

Automated Tests do not Work in Embedded Development or do They?

A common preconception is that “automated tests do not work in embedded development”.

Due to the diversity of embedded systems there is no general approach to test automation and it requires much effort. Sometimes it is not feasible or possible to automate tests at all. But on the other hand, because it is quickly dismissed, people don’t see the real benefits of test automation.

On a previous project, we were responsible (with a partner company) for redesigning an Electronic Shelf Label (ESL). The ESL is composed of a custom ASIC chip that includes a microcontroller, an LCD display and a few passive components. It supports full graphic imaging and animated sequences for displaying product pricing.

The complexity of the software, time constraints and the requirement that the new ESL behave identically to its predecessor presented significant challenges for software verification. The following constraints had to be considered while selecting a viable testing approach:

  • No detailed specifications or reference data were available
  • A normal transmission of data to the ESL can take several minutes
  • The only way to compare the behavior to the old ESL is through visual inspection

 

Early enough we recognized that traditional “test after” approach won’t work.

Seeing that the traditional “test after” approach would not have worked as the time constraints for the project were too strict and would allow for just one or two such test cycles, we had to come up with a different solution.

Our solution was to build a computer vision based test automation system that performs differential testing. The automation system needed to be robust, reliable and low-cost.

The resulting test automation system is comprised of the following components:

  • Windows system with the test automation framework that handled the overall test execution
  • RF transmitter, hardware that enables us to send commands to the ESL via RF (using a proprietary protocol); communication with the transmitter is done using a so-called “script sender”
  • ESL – Electronic Shelf Label
  • LCDrip, a tool developed to capture, rotate, normalize and compare the image
  • Test Automation Artifacts, such as test scripts, expected results (images) and test logs

 

We developed a specific computer vision based test automation system that significantly reduced the testing time.

Picture: Block diagram

 

The computer vision based test automation system significantly reduced the testing time. The complete test suite takes only 21 hours to execute. For comparison, our customer needed 2 test engineers working for 4-6 weeks in order to execute a similar test cycle (on the older label).


Takeaways

1. Look for open source tools available on the market

Building the test automation system was challenging, especially considering we had no prior experience with image recognition. Fortunately, there are numerous open source tools available, such as OpenCV for image processing.

OpenCV allowed us to speed up the development of the image recognition part. What is important is that we tried to make it as simple as possible, investing more time in making the test automation system physically robust to ensure the image recognition algorithm remained simple.

2. Investing time and effort into a test automation system – can be worthwhile

Looking back at the project the key moment was the decision on the verification approach, which we did before writing even a single line of code. The key was that our PM understood the importance and value of a test automation system, because the team “culture” was open and allowed for open discussions.

Test automation takes effort and is not always possible, but it is not as difficult as you may think and can make a huge difference on the project.

There are several vendors offering tools that can help you with this and sometimes making an automated test tool is actually more fun than you ever expected.


Embedded corner is intend as a series of short blog posts, where each post will be presenting some examples of good or bad practices. There is no particular order in which the topics will be presented and I will also invite more contributors. So stay tuned!