Test methods should assert something

Rule ND1611 Code Coverage

Matches test methods (methods tagged with a test attribute like [Test], [Fact] or [TestMethod]) that call no assertion API, directly or through helper methods. Such a test covers code without checking what this code does: coverage figures overstate how well the code is tested.

Assertion APIs recognized: the Assert classes of NUnit, xUnit, MSTest and TUnit, FluentAssertions, Shouldly, NFluent, Verify, ApprovalTests, mock verifications (Moq Verify(), NSubstitute Received(), FakeItEasy MustHaveHappened()), BDD runners (SpecFlow, Reqnroll, LightBDD), Roslyn analyzer and code fix tests and any third-party method named Assert... or declared in a type named ...Assert.... Assert.Pass(), Assert.Ignore(), Assert.Inconclusive(), Assume, Warn and Debug.Assert() check nothing.

Not matched: tests whose returned value is checked by the test framework (NUnit ExpectedResult, FsCheck properties), tests tagged with [ExpectedException] and tests invoking a member dynamically (a call with a dynamic argument is bound at run-time and might assert).

Unlike other default rules, this rule reports issues on test code.

How to fix

Assert the observable outcome of the code exercised: returned values, state changes or expected interactions. A test meant to only check that no exception is thrown can say so, like with NUnit Assert.DoesNotThrow().

If your tests check through your own helpers that throw an exception instead of calling an assertion API, add these helpers to the list at the top of the rule source.

Debt is 5 minutes per test: understand what the test exercises and assert it.

CQLinq source

This rule is written in CQLinq, C# LINQ over the NDepend code model. It ships with every new NDepend project: you can read it, adapt it, or use it as a starting point for your own custom rules.

// <Name>Test methods should assert something</Name>
// <Id>ND1611:TestMethodsShouldAssertSomething</Id>

warnif count > 0

// Assertion APIs: adapt this list to your own assertion library
let assertionApis = ThirdParty.Methods.Where(m =>
   // Test frameworks: Assert, ClassicAssert, CollectionAssert, AssertionHelper...
   // Pass(), Ignore(), Inconclusive(), Skip(), Assume and Warn check nothing.
   m.ParentNamespace.Name.StartsWithAny("NUnit.Framework", "Xunit", "Microsoft.VisualStudio.TestTools.UnitTesting") ?
      m.ParentType.SimpleName.Contains("Assert") &&
     !m.SimpleName.StartsWithAny("Pass", "Ignore", "Inconclusive", "Skip") :

   // Assertion and snapshot libraries, BDD runners (steps are bound at run-time)
   m.ParentNamespace.Name.StartsWithAny(
      "TUnit.Assertions", "FluentAssertions", "AwesomeAssertions", "Shouldly", "NFluent",
      "Verify", "ApprovalTests", "Snapshooter", "TechTalk.SpecFlow", "Reqnroll", "LightBDD") ||

   // Mock verifications, property-based checks, Roslyn analyzer and code fix tests
   (m.ParentNamespace.Name.StartsWith("Moq") && m.SimpleName.StartsWithAny("Verify", "InSequence")) ||
   (m.ParentNamespace.Name.StartsWith("NSubstitute") && m.SimpleName.StartsWithAny("Received", "DidNotReceive")) ||
   (m.ParentNamespace.Name.StartsWith("FakeItEasy") && m.SimpleName.StartsWith("Must")) ||
   (m.ParentNamespace.Name.StartsWith("FsCheck") && m.SimpleName.Contains("ThrowOnFailure")) ||
   (m.ParentNamespace.Name.StartsWith("Microsoft.CodeAnalysis") && m.ParentNamespace.Name.Contains("Testing") &&
    m.SimpleName.StartsWithAny("RunAsync", "Verify")) ||

   // Any other assertion API: AssertEx.Equal(), Mock.Assert(), AssertWasCalled()...
   // Debug.Assert() is removed from release builds.
   ((m.SimpleName.StartsWith("Assert") || m.ParentType.SimpleName.Contains("Assert")) &&
    !m.ParentNamespace.Name.StartsWith("System")))
   // Your own assertion helpers that throw instead of calling an assertion API:
   // .Concat(Methods.Where(m => m.ParentType.FullName == "MyTests.Check"))

// A dynamic call like Assert.Equal(expected, (dynamic)x) is bound at run-time:
// a method invoking a member dynamically might assert.
let assertingDynamically = ThirdParty.Methods.Where(m =>
   m.SimpleName == "InvokeMember" && m.ParentType.FullName == "Microsoft.CSharp.RuntimeBinder.Binder")
   .SelectMany(m => m.MethodsCallingMe)

// Test code asserting directly, through helpers (lambdas and async bodies
// included) or through overrides: a base fixture calling an abstract
// CheckResult() implemented by derived fixtures.
let assertingMetric = assertionApis.SelectMany(api => api.MethodsCallingMe).Where(m => m.IsTestCode)
   .Concat(assertingDynamically).Distinct()
   .FillIterative(methods => methods.SelectMany(m => m.MethodsCallingMe.Concat(m.OverriddensBase)).Where(m => m.IsTestCode))
let asserting = assertingMetric.DefinitionDomain.ToHashSetEx()

from m in TestCode.Methods
where m.IsTestMethod && !m.IsAbstract && !asserting.Contains(m) &&
      // The test framework checks the value returned: NUnit ExpectedResult, FsCheck properties...
      m.ReturnType != null &&
      m.ReturnType.FullNameEqualsAny("System.Void", "System.Threading.Tasks.Task", "System.Threading.Tasks.ValueTask") &&
      // The exception expected is the assertion
     !m.HasAttribute("Microsoft.VisualStudio.TestTools.UnitTesting.ExpectedExceptionAttribute".AllowNoMatch()) &&
     !m.HasAttribute("NUnit.Framework.ExpectedExceptionAttribute".AllowNoMatch())
orderby m.NbLinesOfCode descending
select new {
   m,
   exercised = m.MethodsCalled.Where(mc => !mc.IsThirdParty && !mc.IsTestCode),
   m.NbLinesOfCode,
   Debt = 5.ToMinutes().ToDebt(),
   Severity = Severity.Medium
}

//<Expl>
// The test method *{0}* calls no assertion API: it passes as long as no
// exception is thrown, whatever the code it exercises does.
//</Expl>

//<Desc>
// Matches **test methods** (methods tagged with a test attribute like
// *[Test]*, *[Fact]* or *[TestMethod]*) that call no assertion API, directly
// or through helper methods. Such a test covers code without checking what
// this code does: coverage figures overstate how well the code is tested.
//
// Assertion APIs recognized: the *Assert* classes of NUnit, xUnit, MSTest and
// TUnit, FluentAssertions, Shouldly, NFluent, Verify, ApprovalTests, mock
// verifications (Moq *Verify()*, NSubstitute *Received()*, FakeItEasy
// *MustHaveHappened()*), BDD runners (SpecFlow, Reqnroll, LightBDD), Roslyn
// analyzer and code fix tests and any third-party method named *Assert...*
// or declared in a type named *...Assert...*. *Assert.Pass()*, *Assert.Ignore()*, *Assert.Inconclusive()*,
// *Assume*, *Warn* and *Debug.Assert()* check nothing.
//
// Not matched: tests whose returned value is checked by the test framework
// (NUnit *ExpectedResult*, FsCheck properties), tests tagged with
// *[ExpectedException]* and tests invoking a member dynamically (a call
// with a *dynamic* argument is bound at run-time and might assert).
//
// Unlike other default rules, this rule reports issues on **test code**.
//</Desc>

//<HowToFix>
// Assert the observable outcome of the code exercised: returned values,
// state changes or expected interactions. A test meant to only check that
// no exception is thrown can say so, like with NUnit *Assert.DoesNotThrow()*.
//
// If your tests check through your own helpers that throw an exception
// instead of calling an assertion API, add these helpers to the list at the
// top of the rule source.
//
// **Debt** is 5 minutes per test: understand what the test exercises and
// assert it.
//</HowToFix>

Open in the interactive Rules Explorer · All default rules

Code Coverage: all 15 items