This catchy title hides behind it a rather interesting tale of debugging. A while back I encountered a case where two seemingly entirely unrelated changes in two different branches led to one of the changes no longer working. That change was the inclusion of Entity Framework Profiler (EFProf) to a console application.
(Note for the sake of completeness: for historical reasons, we are dealing with a .NET 4.x codebase that uses EF5 and EFProf version 3.x.)
TL;DR: This problem was solved by initializing EFProf before referencing any .NET Standard assemblies. Keep reading for the dirty details.
Symptomy of Destruction
The error was rather non-obvious: we were getting a compiler error at runtime, and the error details looked like this:
System.InvalidOperationException: Failed to compile the following symbols: 'EF4;NET40;NET45' The type name 'DbProviderFactory' could not be found. This type has been forwarded to assembly 'System.Data, Version=126.96.36.199, Culture=neutral, PublicKeyToken=b77a5c561934e089'. Consider adding a reference to that assembly. (... lots of repetitive errors elided ...) at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount) at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder)
From the error message, it’s clear that we’re dealing with a type forwarding error, but there was no clear explanation as to why we were suddenly seeing this.
The branch that triggered this issue contained a number of references to .NET Standard assemblies, and as usual, whenever the two worlds of desktop .NET and .NET Standard collide, sparks can be expected to fly. After banging my head against the problem for a while, I decided to debug the error in Rider.
One of the things I so love about Rider is how easily it lets me debug inside code I don’t own. Most of the time, when dealing with exceptions from third-party dependencies, the best way to figure what’s going on is to inspect the local variables around the code where the exception is thrown, just like with code I write myself. So I poked around a bit, and then decided to set a breakpoint at the happy path of the code and see how things looked in the version that worked.
After a moment of comparing things, I noticed that on the original branch, the list of referenced assemblies looked different than with the breaking change. Originally, the assembly list looked like this:
After the change, it looked like this:
The difference is pretty easy to spot: originally, we were referencing System.Data, but after the change, it was netstandard instead. So… why is that?
Again, Rider’s tools helped me locate the crucial clues. I used Find Usages to go up the call stack and noticed that the reference in question was found by scanning the assemblies currently in memory, looking for the one that contained a declaration for DbConnection and using that as a reference.
Two minutes of furious brow-wrinkling later, a lightbulb appeared over my head. I moved the code that triggered EFProf’s initialization to happen before any references to .NET Standard assemblies… and poof. The problem was gone. The changes had nothing obvious to do with each other, but the internal state of the runtime can and does affect your code too.
So uh… what’s up with the type forwards?
Type Forwards are a mechanism that allow the implementations of types to be moved to another assembly without breaking the runtime contract of “this assembly provides these types”. In compiled .NET code, assemblies are typically loaded in the order they are encountered in the code, so the first usage of a type from a particular assembly will also trigger that assembly to be loaded. If the type is forwarded to another assembly, the runtime will then also load that one, and everything is fine.
However, since type forwarding is a runtime mechansim, what it doesn’t do is provide the full metadata description of the type. In practice, this means that when you compile against an assembly that has forwarded types, in order to use those types in your code, you also need to reference the assembly that actually contains those types.
In this case, the EFProf code that does runtime compilation didn’t take that into account. When it found the declaration for DbConnection from within netstandard.dll, it included that as a reference. However, when the Roslyn compiler encountered code that actually uses the types from System.Data, it didn’t know what to do with them. Which is why reordering the method calls in a way that forced the actual System.Data to be loaded first fixed the issue.