The short answer is, no, there's no way to do what you're asking.
The longer answer is this: There is a special assembly loading method, Assembly.ReflectionOnlyLoad(), which uses a "reflection-only" load context. This lets you load assemblies that cannot be executed, but can have their metadata read.
In your case (and, apparently, in every use case I could come up with myself) it's not really that helpful. You cannot get typed attributes from this kind of assembly, only CustomAttributeData. That class doesn't provide any good way to filter for a specific attribute (the best I could come up with was to cast it to a string and use StartsWith("[System.Diagnostics.Debuggable");
Even worse, a reflection-only load does not load any dependency assemblies, but it forces you to do it manually. That makes it objectively worse than what you're doing now; at least now you get the dependency loading automatically.
(Also, my previous answer made reference to MEF; I was wrong, it appears that MEF includes a whole ton of custom reflection code to make this work.)
Ultimately, you cannot unload an assembly once it has been loaded. You need to unload the entire app domain, as described in this MSDN article.
UPDATE:
As noted in the comments, I was able to get the attribute information you needed via the reflection-only load (and a normal load) but the lack of typed attribute metadata makes it a serious pain.
If loaded into a normal assembly context, you can get the information you need easily enough:
var d = a.GetCustomAttributes(typeof(DebuggableAttribute), false) as DebuggableAttribute;
var tracking = d.IsJITTrackingEnabled;
var optimized = !d.IsJITOptimizerDisabled;
If loaded into a reflection-only context, you get to do some work; you have to figure out the form that the attribute constructor took, know what the default values are, and combine that information to come up with the final values of each property. You get the information you need like this:
var d2 = a.GetCustomAttributesData()
.SingleOrDefault(x => x.ToString()
.StartsWith("[System.Diagnostics.DebuggableAttribute"));
From there, you need to check the ConstructorArguments to see which constructor was called: this one with one argument or this one with two arguments. You can then use the values of the appropriate parameters to figure out what values the two properties you are interested in would have taken:
if (d2.ConstructorArguments.Count == 1)
{
var mode = d2.ConstructorArguments[0].Value as DebuggableAttribute.DebuggingModes;
// Parse the modes enumeration and figure out the values.
}
else
{
var tracking = (bool)d2.ConstructorArguments[0].Value;
var optimized = !((bool)d2.ConstructorArguments[1].Value);
}
Finally, you need to check for NamedArguments that might override those set on the constructor, using for example:
var arg = NamedArguments.SingleOrDefault(x => x.MemberInfo.Name.Equals("IsJITOptimizerDisabled"));
var optimized = (arg == null || !((bool)arg.TypedValue.Value));
On one final note, if you are running this under .NET 2.0 or higher, and haven't already seen in, MSDN points this out in the DebuggingModes documentation:
In the .NET Framework version 2.0, JIT tracking information is always generated, and this flag has the same effect as Default with the exception of the IsJITTrackingEnabled property being false, which has no meaning in version 2.0.
