feature article
Subscribe Now

Are You Covered?

Software Test Coverage Isn’t as Straightforward as You Might Hope

LDRA issued a press release sometime back on their work with the Mathworks; I addressed the overall topic at the time, but there was something else that caught my eye.

In the press release was the statement, “The LDRA tool suite provides full code coverage whether statement, branch or decision, linear code sequence and jump (LCSAJ), or modified condition/decision coverage (MC/DC) of code created from Simulink models and hand-written code.”

I thought, if there are these various criteria for considering software covered, then is there a matrix of who or what standard requires what level of coverage?

As you might expect, the answer isn’t simple. Actually, the specific answer to that specific question is simple: it’s “No.” There is no such matrix.

Before going into the nuance, I thought it useful to present the Wikipedia definitions of the various standards as a baseline (combining the contents of a couple of entries)*.

Condition: A condition is a leaf-level Boolean expression (it cannot be broken down into a simpler Boolean expression).

Decision: A Boolean expression composed of conditions and zero or more Boolean operators. A decision without a Boolean operator is a condition.

Condition Coverage: Every condition in a decision in the program has taken all possible outcomes at least once.

Decision Coverage:  Every point of entry and exit in the program has been invoked at least once, and every decision in the program has taken all possible outcomes at least once.

LCSAJ: An LCSAJ is a software code path fragment consisting of a sequence of code (a Linear Code Sequence, or basic block) followed by a control flow Jump, and consists of the following three items:

    the start of the linear sequence (basic block) of executable statements
    the end of the linear sequence
    the target line to which control flow is transferred at the end of the linear sequence.

Condition/Decision Coverage:  Every point of entry and exit in the program has been invoked at least once, every condition in a decision in the program has taken all possible outcomes at least once, and every decision in the program has taken all possible outcomes at least once. 

Modified Condition/Decision Coverage:  Every point of entry and exit in the program has been invoked at least once, every condition in a decision in the program has taken on all possible outcomes at least once, and each condition has been shown to affect that decision outcome independently. A condition is shown to affect a decision’s outcome independently by varying just that condition while holding fixed all other possible conditions. The condition/decision criterion does not guarantee the coverage of all conditions in the module because in many test cases, some conditions of a decision are masked by the other conditions. Using the modified condition/decision criterion, each condition must be shown to be able to act on the decision outcome by itself, everything else being held fixed. The MC/DC criterion is thus much stronger than the condition/decision coverage.

I had a conversation with LDRA’s Mark Pitchford to get some more clarity (or at least color) on how this stacks up in real life.

While some standards may have their own matrices for coverage according to safety integrity level (SIL), there is much that’s left to the dude that adjudicates whether or not you’ve met your duty under whichever standard you’re working towards; this is the so-called designated engineering representative (DER).

In particular, MC/DC has some surprising subtleties associated with it, depending on a couple of ways of interpreting the requirements – some of which are, of course, much more conservative than others.

One subtlety involves the “short-circuit” feature of different languages. In languages like C or C++, if a decision involves multiple conditions ANDed or ORed together, then evaluation goes only as far as necessary to determine the overall outcome. If A, B, and C are ANDed together and A is FALSE, then B and C don’t have to be evaluated; you already know the result is FALSE. Ada is more conservative: all conditions have to be evaluated regardless of outcome.

This issue was addressed by the DO-178B Certification Authority Software Team (CAST), and they defined DC and MC/DC as follows, given a series of increasingly tough requirements:

a)  Every statement in the program has been invoked at least once;
b)  Every point of entry and exit in the program has been invoked at least once;
c)  Every control statement (i.e., branchpoint) in the program has taken all possible outcomes (i.e.,branches) at least once;                
d)  Every non-constant Boolean expression in the program has evaluated to both a True and a False result;
e)  Every non-constant condition in a Boolean expression in the program has evaluated to both a True and a False result;
f)  Every non-constant condition in a Boolean expression in the program has been shown to independently affect that expression’s outcome.

Based upon these definitions:

  • Statement Coverage requires (a) only
  • DC requires (b, c, d)
  • MC/DC requires (b, c, d, e, f)

There’s yet another nuance when it comes to hardware registers. There’s disagreement as to whether assigning a value to a register should constitute a decision. Which, on its face, sounds bizarre: how could assignment be considered a decision?? But here’s the reasoning: this applies in particular when writing to memory-mapped registers that hand off some result to a peripheral.

A conservative approach would assume that the peripheral will be making some decision based on the written value, and that decision is outside the scope of the program being tested – and yet it depends on the value created by the program being tested. Therefore, some DERs will require that all values for that register be covered as a proxy for covering the decisions the peripheral will make.

Less conservative DERs will require condition coverage only when (and if) the hardware register value is actually involved in a decision somewhere later in the code being tested (which is the more intuitive approach).

One of the key take-aways here is that DER discretion can make a difference. The conservative interpretations take more work (although LDRA does support them with switches in their tools). If two competing companies have two different DERs, one of which is nice and one of which is harsh, then the unlucky company will have higher costs due to the greater rigor required of them, even though both companies are working to the same standard.

DO-178C is supposed to close many of the loopholes and make more concrete some of the vaguer aspects of DO-178B, but for now, at least, these “clear-cut” test coverage definitions are anything but.

 

*At least, this is what Wikipedia says as of 11/2/11…

One thought on “Are You Covered?”

Leave a Reply

featured blogs
Jul 6, 2020
If you were in the possession of one of these bodacious beauties, what sorts of games and effects would you create using the little scamp?...
Jul 3, 2020
[From the last episode: We looked at CNNs for vision as well as other neural networks for other applications.] We'€™re going to take a quick detour into math today. For those of you that have done advanced math, this may be a review, or it might even seem to be talking down...
Jul 2, 2020
In June, we continued to upgrade several key pieces of content across the website, including more interactive product explorers on several pages and a homepage refresh. We also made a significant update to our product pages which allows logged-in users to see customer-specifi...

Featured Video

Product Update: New DesignWare® IOs

Sponsored by Synopsys

Join Faisal Goriawalla for an update on Synopsys’ DesignWare GPIO and Specialty IO IP, including LVDS, I2C and I3C. The IO portfolio is silicon-proven across a range of foundries and process nodes, and is ready for your next SoC design.

Click here for more information about DesignWare Embedded Memories, Logic Libraries and Test Videos

Featured Paper

Cryptography: How It Helps in Our Digital World

Sponsored by Maxim Integrated

Gain a basic understanding of how cryptography works and how cryptography can help you protect your designs from security threats.

Click here to download the whitepaper

Featured Chalk Talk

Demystifying Wireless! Key Considerations for Adding Wireless to Your IoT Solution

Sponsored by Mouser Electronics and Digi

Adding wireless connectivity to your IoT design can be a major challenge in an already-crowded schedule. And, doing it as an afterthought can cause all kinds of trouble. In this episode of Chalk Talk, Amelia Dalton chats with Andrew Reiter from Digi International about a holistic approach to designing wireless connected designs.

Click here for more information about Digi International XBee® XBIB-C Development Boards