Out of the box, NETLink in Mathematica 8.0 will use version 2.0 of the .NET Framework. To use a newer version, you must edit the .NET configuration files InstallableNET.exe.config and InstallableNET32.exe.config. These files can be found at the location returned by this Mathematica expression:
FileNameJoin[{$InstallationDirectory, "SystemFiles", "Links", "NETLink"}]
Add a new supportedRuntime element for the desired runtime version (4.0 in this example):
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<!-- The supportedRuntime lines control which version of the .NET Framework will
be used when .NET/Link is launched with InstallNET[].
.NET/Link requires at least .NET 2.0. If you have .NET 3.0 installed,
it will be used (note that the 3.0 version is really just the 2.0
version with some extra assemblies).
-->
<supportedRuntime version="v4.0" />
<supportedRuntime version="v2.0.50727" />
</startup>
</configuration>
Note that supportedRuntime elements should be listed in order of preference.