Code should be tested

Rule ND1600 Code Coverage

Lists the methods not covered or partially covered by tests, one issue per method — also in a type no test reaches yet: writing the first tests of a type fixes issues, it never turns one issue into several.

The technical debt (effort to write tests) is the effort to develop the code element multiplied by factors in ]0,1.3] based on:

- the method code size and complexity

- the actual percentage coverage

- the abstractness of types used (classes are harder to test than interfaces)

- the method visibility (private/protected harder to test)

- reads of mutable static fields (state shared across tests executions)

- JustMyCode or not (NotMyCode is often generated, with generatable tests)

This rule is a large debt source: untested code is by definition part of the technical debt. The annual interest (annual cost of leaving code uncovered) is based on the effort to develop times usage factors.

The Testability column (0 to 100, higher = easier to unit test) and the TestHurdle column help to pick the quick wins first. Testability decreases with the number of methods called (the strongest sign), with a hurdle reached (hard: a process, the network, a database or console key input, up to 3 calls deep; soft: the file system, HttpClient, the environment, a sleep or console line input, called directly), with the method complexity, when tests cannot call the method (private, or internal without InternalsVisibleTo) and when no test reaches its type yet; it increases for a static method without mutable state. TestHurdle names the hurdle reached and, deeper than one call, the first call of the chain.

How to fix

Write unit tests covering the matched methods.

CQLinq source

This rule is written in CQLinq, C# LINQ over the NDepend code model. It ships with every new NDepend project: you can read it, adapt it, or use it as a starting point for your own custom rules.

// <Name>Code should be tested</Name>
// <Id>ND1600:CodeShouldBeTested</Id>

warnif count > 0

// This lambda infers a factor in the range [0,1] from a sequence of distinct types used,
// based on how many of these types are actually concrete (i.e are not interfaces or enumeration).
// Primitive types (int, bool...) are eliminated from the sequence of types used 
// by filtering types declared in the namespace System.
// Namespaces are per assembly: several "System" namespaces, compared by reference in a HashSet
let systemNamespaces = Namespaces.WithName("System").ToHashSetEx()
let abstractionUsageFactorFormula = new Func<IEnumerable<IType>,double>(typesUsed => 
            typesUsed.Count(t => !systemNamespaces.Contains(t.ParentNamespace) && !t.IsInterface && !t.IsEnumeration) 
     / (1 + typesUsed.Count(t => !systemNamespaces.Contains(t.ParentNamespace))))

// methodsUntested is all methods not 100% covered by tests
let methodsUntested = (from method in JustMyCode.Methods
  where !method.IsExcludedFromCoverage && 
         method.NbLinesOfCode >= 0 && 
         method.NbILInstructions > 0 &&
         method.PercentageCoverage < 100 

  // Factor in case method is partially covered
  let uncoverageFactor = ((100 - method.PercentageCoverage) / 100).Value

  // Complexity factor
  let complexityFactor = ((method.CyclomaticComplexity ?? method.ILCyclomaticComplexity) + method.ILNestingDepth).Linear(0, 0.1,    10, 1).Value

  // Not my code is often generated code and is in general easier to get tested since test can be generated as well.
  let justMyCodeFactor = (JustMyCode.Contains(method) ? 1 : 0.4)

  // abstractionUsageFactor reflects the fact that code that relies on interfaces
  // is easier to test that code that relies on concrete classes.
  let abstractionUsageFactor = 0.7 + 0.3 *abstractionUsageFactorFormula(method.MembersUsed.ParentTypes())

  // The usageFactor depends on the method 'rank' that is a value
  // indicating if the method is often used or not
  let usageFactor = (method.Rank / (method.Rank + 4)).Value

  // It is more complicated to write tests for non publicly visible methods
  let visibilityFactor = method.Visibility.EqualsAny(Visibility.Public, Visibility.Internal) ? 1 : 
                         method.Visibility != Visibility.Private ? 1.1 : 1.2
  
  // Is is more complicated to write tests for methods that read mutable static fields 
  // whose changing state is shared across tests executions.
  let staticFieldUsageFactor = method.ReadsMutableTypeState ? 1.3 : 1.0


  // Both "effort to write tests" and "annual cost to not test" for a method
  // is determined by several factors in the range [0,1] that multiplies the effortToDevelop
  let effortToDevelopInMinutes = method.EffortToDevelop().Value.TotalMinutes

  let effortToWriteTests = Math.Max(2, // Minimum 2 minutes per method not tested
        effortToDevelopInMinutes * 
        uncoverageFactor * 
        complexityFactor * 
        justMyCodeFactor * 
        abstractionUsageFactor * 
        visibilityFactor *
        staticFieldUsageFactor).ToMinutes().ToDebt()

  let annualCostToNotFix = Math.Max(2, // Minimum 2 minutes per method not tested
        effortToDevelopInMinutes * 
        usageFactor *
        uncoverageFactor * 
        justMyCodeFactor).ToMinutes().ToAnnualInterest()

select new { 
    elem = method, 
    debt = effortToWriteTests, 
    ai = annualCostToNotFix 

  // Uncomment the line below to tinker with various factors
  //    uncoverageFactor, complexityFactor , justMyCodeFactor , abstractionUsageFactor, visibilityFactor, staticFieldUsageFactor 
}).ToArray() // enumerate once

// TESTABILITY: the columns Testability (0-100, higher = easier to unit test) and TestHurdle, to pick
// the quick wins first. Computed for the issues only, from lookups built once per query: it changes
// neither the debt, nor the annual interest, nor the order. Few 'let' on purpose: each one, even
// unused, costs at every record of the query.
//
// The hurdles: framework APIs that tie a unit test to infrastructure. HARD (a process, the network,
// a database, console key input): their callers up to 3 calls deep are hard to test. SOFT (the file
// system, HttpClient - fakeable through its message handler -, the environment, sleeps, console line
// input - fed through Console.SetIn -): direct callers only.
// A call through an interface stops the chain: an interface is a seam a test can fake.
// Members: the hurdle members of the type, all of them when empty. Value: 1 hard type, 2 hard member,
// 3 soft type, 4 soft member.
let hurdleApis = ThirdParty.Types.Join(new[] {
      new { Type = "System.Diagnostics.Process", Hard = true, Members = "" },
      new { Type = "System.Net.HttpListener", Hard = true, Members = "" },
      new { Type = "System.Net.WebClient", Hard = true, Members = "" },
      new { Type = "System.Net.WebRequest", Hard = true, Members = "" },
      new { Type = "System.Net.HttpWebRequest", Hard = true, Members = "" },
      new { Type = "System.Net.Sockets.Socket", Hard = true, Members = "" },
      new { Type = "System.Net.Sockets.TcpClient", Hard = true, Members = "" },
      new { Type = "System.Net.Sockets.TcpListener", Hard = true, Members = "" },
      new { Type = "System.Net.Sockets.UdpClient", Hard = true, Members = "" },
      new { Type = "System.Data.Common.DbConnection", Hard = true, Members = "" },
      new { Type = "System.Data.Common.DbCommand", Hard = true, Members = "" },
      new { Type = "System.Data.SqlClient.SqlConnection", Hard = true, Members = "" },
      new { Type = "System.Data.SqlClient.SqlCommand", Hard = true, Members = "" },
      new { Type = "Microsoft.Data.SqlClient.SqlConnection", Hard = true, Members = "" },
      new { Type = "Microsoft.Data.SqlClient.SqlCommand", Hard = true, Members = "" },
      new { Type = "System.Console", Hard = true, Members = " ReadKey " },
      new { Type = "System.Console", Hard = false, Members = " ReadLine Read " },
      new { Type = "System.IO.File", Hard = false, Members = "" },
      new { Type = "System.IO.Directory", Hard = false, Members = "" },
      new { Type = "System.IO.FileInfo", Hard = false, Members = "" },
      new { Type = "System.IO.DirectoryInfo", Hard = false, Members = "" },
      new { Type = "System.IO.FileStream", Hard = false, Members = "" },
      new { Type = "System.Net.Http.HttpClient", Hard = false, Members = "" },
      new { Type = "System.Threading.Thread", Hard = false, Members = " Sleep " },
      new { Type = "System.Threading.Tasks.Task", Hard = false, Members = " Delay " },
      new { Type = "System.Environment", Hard = false, Members = " GetEnvironmentVariable SetEnvironmentVariable Exit " } },
   t => t.FullName, h => h.Type,
   (t, h) => t.MethodsAndConstructors.Where(m => h.Members.Length == 0 || h.Members.Contains(" " + m.SimpleName + " "))
              .Select(m => new { m, Value = (h.Hard ? 1 : 3) + (h.Members.Length == 0 ? 0 : 1) }))
   .SelectMany(ms => ms)
   .ToDictionary(x => x.m, x => x.Value)

// The hurdle depth of the methods calling them: 1 to 3 for the hard hurdles (0 = the API), then 11
// for a direct caller of a soft one; the hard depth wins.
let hurdleDepth = Enumerable.Range(1, 4).Aggregate(
      hurdleApis.Where(api => api.Value <= 2).ToDictionary(api => api.Key, api => 0),
      (depthOf, step) => depthOf
         .Concat((step <= 3 ? depthOf.Where(e => e.Value == step - 1).SelectMany(e => e.Key.MethodsCallingMe)
                            : hurdleApis.Where(api => api.Value >= 3).SelectMany(api => api.Key.MethodsCallingMe))
                 .Where(m => !depthOf.ContainsKey(m)).Distinct()
                 .Select(m => new KeyValuePair<IMethod, int>(m, step <= 3 ? step : 11)))
         .ToDictionary(e => e.Key, e => e.Value))

// Testability of a method: simpler, no hurdle, calling few methods, callable by tests (public, or
// internal in an assembly with InternalsVisibleTo), in a type tests already reach, static without
// mutable state = easier. The weights were calibrated on 173 test-writing outcomes of an AI agent
// (methods it covered vs. methods it declared integration-only): fewer calls out is the strongest sign.
let testabilityOf = new Func<HashSet<IAssembly>, Func<IMethod, int>>(assembliesOpenToTests => m =>
   (int)Math.Round(100 * Math.Min(1.0,
      1 / (1.0 + 0.03 * ((m.CyclomaticComplexity ?? m.ILCyclomaticComplexity ?? 0) + (m.ILNestingDepth ?? 0))) *
      1 / (1.0 + 0.05 * (m.NbMethodsCalled ?? 0)) *
      (!hurdleDepth.ContainsKey(m) ? 1 : hurdleDepth[m] == 1 ? 0.15 : hurdleDepth[m] == 2 ? 0.5 : hurdleDepth[m] == 3 ? 0.7 : 0.8) *
      (m.IsPubliclyVisible ||
       (m.Visibility.EqualsAny(Visibility.Public, Visibility.Internal, Visibility.ProtectedOrInternal) &&
        m.ParentType.Visibility.EqualsAny(Visibility.Public, Visibility.Internal, Visibility.ProtectedOrInternal) &&
        assembliesOpenToTests.Contains(m.ParentAssembly) &&
        m.ParentType.OutterTypes.All(t => t.Visibility.EqualsAny(Visibility.Public, Visibility.Internal, Visibility.ProtectedOrInternal)))
         ? 1 : 0.8) *
      (m.ParentType.PercentageCoverage > 0 ? 1 : 0.75) *
      (m.IsStatic && !m.ReadsMutableTypeState ? 1.1 : 1))))
   (Application.Assemblies.Where(a => a.HasAttribute("System.Runtime.CompilerServices.InternalsVisibleToAttribute".AllowNoMatch())).ToHashSetEx())

// The hurdle as text: the API reached and, deeper than 1, the first call of the chain.
let hurdleText = new Func<Func<IMethod, int, IMethod>, Func<IMethod, string>, Func<IMethod, string>>((calleeAt, apiText) => m =>
   !hurdleDepth.ContainsKey(m) ? "" :
   hurdleDepth[m] == 1 ? apiText(calleeAt(m, 0)) :
   hurdleDepth[m] == 2 ? apiText(calleeAt(calleeAt(m, 1), 0)) + " via " + calleeAt(m, 1).ParentType.Name + "." + calleeAt(m, 1).SimpleName + "() (2 calls deep)" :
   hurdleDepth[m] == 3 ? apiText(calleeAt(calleeAt(calleeAt(m, 2), 1), 0)) + " via " + calleeAt(m, 2).ParentType.Name + "." + calleeAt(m, 2).SimpleName + "() (3 calls deep)" :
   apiText(m.MethodsCalled.FirstOrDefault(c => hurdleApis.ContainsKey(c) && hurdleApis[c] >= 3)))
   // the callee of m at that hurdle depth
   ((m, depth) => m == null ? null : m.MethodsCalled.FirstOrDefault(c => hurdleDepth.ContainsKey(c) && hurdleDepth[c] == depth),
   // a hurdle type in full (any of its members), a hurdle member with its name
    api => api == null ? "" : hurdleApis[api] == 1 || hurdleApis[api] == 3 ? api.ParentType.FullName : api.ParentType.FullName + "." + api.SimpleName + "()")

// One issue per method, also in a type no test reaches yet: the first tests of a type fix issues,
// they never turn one type issue into one issue per method still uncovered.
from x in methodsUntested
orderby x.ai.Value descending

select new { 
  x.elem, 
  x.elem.PercentageCoverage,
  x.elem.NbLinesOfCode,
  x.elem.NbLinesOfCodeNotCovered,
  x.elem.CyclomaticComplexity,
  Debt = x.debt, 
  AnnualInterest = x.ai,
  Testability = testabilityOf(x.elem),
  TestHurdle = hurdleText(x.elem)
}

//<Expl>The *{0}* is {1}% covered by tests. {3} statement{3?:s} {3?is:are} not executed at test-time.</Expl>
//<Desc>
// Lists the methods not covered or partially covered by tests, one issue
// per method — also in a type no test reaches yet: writing the first tests
// of a type fixes issues, it never turns one issue into several.
//
// The **technical debt** (effort to write tests) is the effort to develop
// the code element multiplied by factors in ]0,1.3] based on:
//
// - the method code size and complexity
//
// - the actual percentage coverage
//
// - the abstractness of types used (classes are harder to test than
// interfaces)
//
// - the method visibility (private/protected harder to test)
//
// - reads of mutable static fields (state shared across tests executions)
//
// - *JustMyCode* or not (*NotMyCode* is often generated, with generatable
// tests)
//
// This rule is a large debt source: untested code is by definition part of
// the technical debt. The **annual interest** (annual cost of leaving code
// uncovered) is based on the effort to develop times usage factors.
//
// The **Testability** column (0 to 100, higher = easier to unit test) and
// the **TestHurdle** column help to pick the quick wins first. Testability
// decreases with the number of methods called (the strongest sign), with a
// hurdle reached (*hard*: a process, the network, a database or console
// key input, up to 3 calls deep; *soft*: the file system, HttpClient, the
// environment, a sleep or console line input, called directly), with the
// method complexity, when tests cannot call the method (private, or
// internal without InternalsVisibleTo) and when no test reaches its type
// yet; it increases for a static method without mutable state. TestHurdle
// names the hurdle reached and, deeper than one call, the first call of
// the chain.
//</Desc>

//<HowToFix>
// Write unit tests covering the matched methods.
//</HowToFix>

Open in the interactive Rules Explorer · All default rules

Code Coverage: all 15 items