Automated code quality testing using Roslyn
Roslyn is an exciting project from Microsoft which after years in the making is currently available as a community technology preview. It is described as a “compiler as a service” and promises to give you as a developer access to a rich set of APIs exposing all the interesting stuff that goes on inside the C# and Visual Basic compilers.
How is Roslyn useful
There are numerous useful applications for Roslyn. It seems a lot of emphasis is being put on the integration with Visual Studio since Roslyn greatly reduces the pain of writing tools for Visual Studio, making it easier than ever to create interesting and helpful plugins. But there are plenty of other areas that can benefit from Roslyn, including code generation, scripting, domain specific languages and code analysis.
The Syntax API
So what are all these interesting things that goes on inside the compiler? Roslyn exposes a number of different APIs, ranging from code parsing and analysis to integration with Visual Studio. For this blog post however I will be using only the Syntax API which will allow me to read, parse and query source code
Before we dig into the real topic of the post, let’s take a brief look at what the Syntax API offers at the most basic level.
var tree = SyntaxTree.ParseText(@"
using System;
using System.Collections.Generic;
using System.Linq;
class Program
{
static void Main(string[] args)
{
Console.WriteLine(""Hello, Cruel World"");
}
}
");
In this example we are asking Roslyn to parse a piece of code and provide us with a SyntaxTree that accurately and completely describes the code it was given as input using SyntaxNodes, SyntaxTokens and SyntaxTrivia.
Part of what makes this so interesting is the ability to query SyntaxTrees using LINQ providing us with a great foundation for analyzing code.
Rather than giving you a thorough intro to all the Syntax API concepts I will refer you to the Roslyn home page, where you will find many interesting walkthroughs and articles.
Automated code quality testing
As we have just seen, code analysis is an area where Roslyn could prove useful. To illustrate this let me introduce you to the concept of “automated code quality testing”.
Automated testing should always be part of our software development routine, but while we test a lot of things that has to do with functionality, we don’t often test the actual craftsmanship of the code.
Let’s pretend a subcontractor has been working on our software project and has delivered a new module
namespace NAcqt.Demo
{
public class Subcontractor
{
public void SomeMethod()
{
}
public string someOtherMethod(string someString)
{
return someString;
}
public void AndFinallyThisMethod()
{
var shouldHaveBeenInjected = new InjectMePlease();
var t = shouldHaveBeenInjected.ReturnTrue();
}
}
}
The delivered code works as intended, but there are a couple of non-functional problems with it.
- It doesn’t adhere to our coding style guidelines (see that public method name beginning with a lower case letter)
- It doesn’t have a class level summary comment (required for our automated documentation generation)
- It creates an instance of the InjectMePlease class which should always be provided using dependency injection. This is bad for testability.
These are merely examples, whether or not this is something you actually want (or need) to test for is another discussion. The point is, if we can describe our code quality guidelines in code using Roslyn, we are able to write test cases that can be run to automatically test whether or not our project source code adheres to our guidelines.
Here is the source code for testing the three cases above
using System;
using System.Linq;
using NAcqt.Demo.Tests.Properties;
using NUnit.Framework;
using Roslyn.Compilers.CSharp;
namespace NAcqt.Demo.Tests
{
[TestFixture]
public class All
{
private SourceReader _source;
[SetUp]
public void SetUp()
{
_source = NAcqt.Analyze(Settings.DemoProjectPath);
}
[Test]
public void LoadedAndValid()
{
Assert.That(_source, Is.Not.Null);
Assert.That(_source.SyntaxTrees.Count, Is.GreaterThan(0));
}
[Test]
public void Classes_Always_ShouldHaveSummary()
{
var classes = _source.OfType<ClassDeclarationSyntax>();
var noSummary = classes.Where(
c => !c.GetLeadingTrivia().Any(
trivia => (
trivia.Kind == SyntaxKind.DocumentationCommentTrivia &&
trivia.ToString().Contains("<summary>")
)
)
).ToList();
Assert.That(
noSummary.Count(),
Is.EqualTo(0),
String.Format(
"These classes do not have summary comments: {0}",
String.Join(", ", noSummary.Select(c => c.ToString()))
)
);
}
[Test]
public void Methods_WhenPublic_ShouldNotStartWithLowerCase()
{
var methods = _source.OfType<MethodDeclarationSyntax>();
var publicMethods = methods.Where(
m => m.Modifiers.Any(
mod => mod.Kind == SyntaxKind.PublicKeyword
)
);
var withLowerCase = publicMethods.Where(
x => x.Identifier.Value.ToString().StartsWithLowerCase()
).ToList();
Assert.That(
withLowerCase.Count(),
Is.EqualTo(0),
String.Format(
"Methods with lower case: {0}",
String.Join(", ", withLowerCase.Select(m => m.ToString()))
)
);
}
[Test]
public void InterfaceImplementations_Always_ShouldBeInjected()
{
var interfaces = _source.OfType<InterfaceDeclarationSyntax>().Select(
x => x.Identifier.ToString()
);
var classes = _source.OfType<ClassDeclarationSyntax>();
var implementsInterface = classes.Where(
c => (
c.BaseList != null &&
c.BaseList.Types.Select(t => t.ToString()).Any(
t => interfaces.Any(i => i.Contains(t))
)
)
).Select(i => i.Identifier.ToString()).ToList();
var notInjected = _source.OfType<ObjectCreationExpressionSyntax>()
.Where(
o => implementsInterface.Contains(o.Type.ToString())
);
Assert.That(notInjected.Count(), Is.EqualTo(0));
}
}
}
Let’s take a closer look at one of the tests
Notice first the constructor which initializes a “SourceReader”. The SourceReader is not part of Roslyn itself but is basically a small tool I have written that, when given a path to a C# project file, will attempt to read all the source files in that project and load their respective syntax trees. The returned source reader gives us access to Roslyn syntax trees for all the files within the project.
After loading the syntax trees we can start querying them and test if they adhere to our standards and guidelines.
var methods = _source.OfType<MethodDeclarationSyntax>();
var publicMethods = methods.Where(
m => m.Modifiers.Any(
mod => mod.Kind == SyntaxKind.PublicKeyword
)
);
Here we ask for all syntax nodes of type “MethodDeclaration”. The source reader will go through all the syntax trees for our project and return a complete list of method declarations within the project (notice that .OfType() is a custom extension method that wraps Roslyn functionality to make it more readable in this context).
We are now able to query the list of method declarations, in this case we use LINQ to find only the method declarations that are public. Next we filter again to find only those that start with a lower case letter (again using a custom extension).
var withLowerCase = publicMethods.Where(x =>
x.Identifier.Value.ToString().StartsWithLowerCase()
).ToList();
Since we are testing to make sure no public methods start with a lower case letter we can now make the following assertion
Assert.That(
withLowerCase.Count(),
Is.EqualTo(0),
String.Format(
"Methods with lower case: {0}",
String.Join(", ", withLowerCase.Select(m => m.ToString()))
)
);
If our project contains any public method name starting with a lowercase letter this test will fail and we will be notified about it.
By using the code parsing and querying features provided by Roslyn we have now created a simple suite of tests that if run as part of a build will ensure all our code adheres to our coding guidelines and best practices.
Beyond the proof of concept
The example tests provided in this proof of concept take advantage of the functionality in the Syntax API only. If you take a closer look you will see that the tests are in fact quite brittle. A natural next step would be to start using the Workspace API and the Semantics API as well. These tools combined would give you the means to write powerful and meaningful tests that can actually be used to make sure the quality of your source code is not neglected or allowed to slide.
I hope you enjoyed the introduction to Roslyn and the Syntax API. If you are interested in trying it out for yourself you will find the entire test project available at https://github.com/braincell/NAcqt.