Sunday, January 31, 2010

A Look at the New DotNetMigrations

One aspect of .Net that a lot of people struggle with is how to manage database changes.  If anyone has looked beyond .Net into the realms of Ruby on Rails or Python's Django project, database management is one of the first things that becomes obviously different.  Thankfully, there's been a number of migration utilities that have been ported or recreated in the .Net world.  After trying out a couple of them, I began to notice that there was a wide variety of how each attempt to accomplish this pain point.  Some require IronRuby scripting, many use a custom C# DSL, and some use external build utilities like NAnt to fully manage the changes.  One that I was introduced to by a friend and began to really enjoy was DotNetMigrations - an OSS project over at CodePlex.

An Overview of DotNetMigrations

DotNetMigrations started as an OSS project over at CodePlex by Joshua Poehls in early 2008 when he liked what he saw in Ruby on Rails and wished there was a sql script-based version of such a utility in .Net.  Unlike Rails' migration strategy which used a DSL written in Ruby to do all of the database object changes, DotNetMigrations was focused around native Sql scripts that would be read and executed by the migration engine.  As the project evolved, new commands were added and updates to the Rails platform's migration strategy were also made to DotNetMigrations.  Soon, it was obvious that DotNetMigrations had a lot of potential when it comes to new commands and other things.  I ended up joining the project in mid 2009 to make some updates and add some commands that I required for a project I was using it on.

After my project was finishing up, I ended up beginning to think of ways to make the application easier to maintain.  After some conversations with Joshua about this, we decided to go ahead with rewrite with the goals of making it easier to maintain and extend while keeping the same spirit which it was started in.  This past weekend, the finishing touches were placed on the rewritten code and a new release was established with some great features.

Extending DotNetMigrations Thanks to MEF

Any reader of this blog knows that I've been addicted to MEF these past few months.  When I looked at the architecture of DotNetMigrations and the goals of such, the decision to use MEF was an obvious one.  MEF provides the ability to not only extend but also compose internal components of an application.  One of the things we wanted to do was allow other people to create new commands that fit their needs without requiring them to recompile the application's source.  MEF allowed us to do such for external commands and because of such, we also turned all internal commands to conform to the same contracts. 

In addition to the command aspect of DotNetMigration, we also changed the way messages could be logged.  In the older versions of the application, it acted like a standard console application with all messages being output to the console application window.  This works for many scenarios; however, if you needed a "silent" program for your task scheduler or wanted to log all messages to a text file, you'd have to do a lot of additional work with the source code or pipe the command to the text file at the command line.  Through using MEF and the custom configurable type catalog, creating new logs becomes easy as well as being able to utilize one or many different logging mechanisms.

What Else Is New?

In addition to the core code being rewritten to leverage MEF, we also launched the DotNetMigrations-Contrib project.  The focus of this project was to allow others to suggest and share new logs and commands they have written.  The components found in this project also will help allow companies who are locked into specific versions of software upgrade with new commands without having to upgrade the core app.  As time passes in, some elements from the Contrib project may find their way into the core application; however, things are just starting for now.

 

As new features and such are added to each project, I'm sure I'll be talking about them here.  Until then, feel free to head over to the CodePlex projects' sites and check it out for yourself. 


Shout it

Tuesday, January 26, 2010

Making Part Declarations Easier with InheritedExports

After playing around with MEF for a while, you begin to find more and more locations where it could be used to extend your application.  It works great for a large number of scenarios and provides a method for others to extend your apps in a way as simple or complex as you want to make it for them.  However, anyone who has sought community participation can tell you that the best way to get such is to make thing as easy as possible for a member of the community to participate.  If you have the greatest, extensible application in the world but require 50 steps to get a part added to the application, the user community may not be very prone to build extensions for such.  Alternatively, if the work required to extend the application was whittled down to only a few steps, the probability of participation rises.  In this blog post, we'll look at this pattern and show how using MEF's InheritedExport attribute can be used to make part creation extremely simple for their authors.

Reviewing Previous Examples:

In previous examples of this series, we created parts that would be placed into a directory and imported into the app based on how the classes were decorated.  By explicitly decorating our parts with the Export attribute works well since it gives us the ability to say which classes to import and which not to; however, in order to use the Export attribute we also had to include a reference to the System.ComponentModel.Composition assembly & namespace.  Depending on how we setup our part contacts, we would also need to add a reference to our application as well.  While that's not a lot of setup, it adds up since it has to be done every time a new part has to be created.  If the parts were created as part of a framework or engine, then it would be in a person's best interest to create a VS template or some other form of code generation or snippets to automate as much of that ceremony as possible.  But what if there was a simpler way to address this from within the application we're trying to extend?

Simplification through Inherited Exporting:

One of the lesser-documented items contained within MEF is the InheritedExport attribute.  This attribute flags all derived classes as parts that fit the contract defined by it.  So instead of requiring every part that implements a particular interface or base class to use the Export attribute and settings, you can just decorate the interface or base class with the InheritedExport attribute and then every class is considered to be an Exported part by default.  If you are familiar with WCF, this is similar to the practice of creating an Interface for declaring your data contracts instead of directly decorating concrete classes.  Below is an comparison between the way that has been show thus far and using the InheritedExport attribute.

Previous Example:

   1:  [Export("commands", typeof(ICommand))]
   2:  public class Command1 : ICommand
   3:  {
   4:      // Implementation of the Interface
   5:  }
   6:   
   7:  [Export("commands", typeof(ICommand))]
   8:  public class Command2 : ICommand
   9:  {
  10:      // Implementation of the Interface
  11:  }

Inherited Example:

   1:  [InheritedExport("commands",typeof(ICommand))]
   2:  public interface ICommand
   3:  {
   4:      // Interface Declaration
   5:  }
   6:   
   7:  public class Command1 : ICommand
   8:  {
   9:      // Implementation of the Interface
  10:  }
  11:   
  12:  public class Command2 : ICommand
  13:  {
  14:      // Implementation of Interface
  15:  }

Without the requirement of marking all of the parts with the Export attribute, the developers who are creating extensible parts for your application will no longer need to reference the System.ComponentModel.Composition namespace.  In addition, the risk of mistyping the contract information in the Export attribute is removed.  By using the InheritedExport attribute, the only requirement for authoring parts becomes just a reference to the assembly containing the decorated base class or assembly.

A Useful Pattern of Inherited Exporting:

While the InheritedExport attribute provides a simplified and safer method of handling part creation, it is an All-or-Nothing mechanism.  While there are ways to exclude parts using different catalogs, there is no way to explicitly exclude a part like you could when we were using the Export attribute on all parts.  A good way around this issue is to base your part contract/type on an interface and add the InheritedExport attribute to an abstract base class that implements said interface.  This provides the blanketed support and convenience of using InheritedExport while providing authors an advanced mechanism by directly implementing the interface and adding the Export attribute manually if needed.  Below is an example of this pattern.

   1:  public interface ICommand
   2:  {
   3:      // Interface Declaration
   4:  }
   5:   
   6:  [InheritedExport("commands", typeof(ICommand))]
   7:  public abstract class CommandBase : ICommand
   8:  {
   9:      // Abstract implementation of the Interface
  10:  }
  11:   
  12:  public class Command1 : CommandBase
  13:  {
  14:      // Implementation of the Abstract Class
  15:  }
  16:   
  17:  public class Command2 : CommandBase
  18:  {
  19:      // Implementation of the Abstract Class
  20:  }

Summary:

In this post, we looked at how to streamline and simplify our part definitions by using the InheritedExport method instead of just the Export attribute.  We looked at the benefits of this simplicity and how it can help make things easier for part authors as well as identified a possible fault with using the InheritedExport attribute at the wrong level of abstraction.  In the corresponding example project linked below, the concepts discussed above are applied to a previous example project in order to illustrate some of the simple benefits that can be used within your application.

Resources:

kick it on DotNetKicks.com Shout it

Tuesday, January 12, 2010

A Configurable Type Catalog for MEF

Over the past month or so, I've had  some time to focus on a few different projects.  In one of these new projects, I was using MEF in order to provide a number of extensibility points.  Everything was going well until I started to dive into a place where I wanted it to be extensible but also configurable like a traditional provider model scattered throughout the .Net framework. With MEF, there's a couple of different ways of handling this situation; however, after playing around with a few of them, I decided to extend the TypeCatalog in order to provide another layer of abstraction that allows me to harness the power of MEF with the ability of outside configuration. In this article, we'll diving into the code that provides provider-like configuration aspects to MEF as well as shine the light at 2 possible pitfalls of using something like this.  Before we do that though, let's examine what MEF has to offer out of the box in order to understand why I felt another layer was needed for my project.

Looking in the Box and at MEF-Contrib

So far in my MEF posts, I've really been focused on simple solutions that utilize the AssemblyCatalog and DirectoryCatalog objects.  These specific catalogs provide a very fast and easy way of composing your various parts together. Part of their simplicity comes from how they handle the parts that it finds in the assemblies you provide it; namely, they take an All-or-Nothing approach.  If they find parts that meet the defined contract criteria, they assume that they are going to be part of some composition.  But what if you only wanted to use only a few parts located in a loaded assembly? 
This question was half of the problem I was trying to solve for my project.  Looking around what's been done so far with MEF, I saw that I could address this part of my problem using a TypeCatalog.  A TypeCatalog differs from other ComposablePartCatalogs in that it allows you to explicitly specify which Types you want to use as Parts.  For example, in previous posts we showed "Help Text" of various parts that represented different commands/arguments of a command line application.  If you didn't want Command #2, you could simply load all other commands into a collection explicitly and then pass the collection into a TypeCatalog.
Great, one problem solved, right?  I can now explicitly load which parts I want to import; however, the second part of my problem was that I didn't want to recompile  the code every time I wanted to change which parts were imported.  At this point, I started to look around at what other people might have already done that would allow this type of configuration.  I ended up looking at the Provider Model for MEF that was a part of the MEF-Contrib project out on CodePlex. What the Provider Model from MEF-Contrib does is move the part declarations (i.e. the Import and Export Attributes) to a custom configuration section.  This held a lot of promise; however, ran into some trouble while working with it.  After about a day of working with the MEF-Contrib provider model, I decided to try my hand at a different approach.

Adding a Layer on top of MEF

While I'm still determined to look into the MEF-Contrib provider model deeper in the near future, I also began to feel that mixing the attributed model and a provider model in the same application felt a bit dirty or smelly. I wanted the leverage the power and simplicity MEF's attributed model but be able to push it to the level of configuration that comes with a provider model.  In order to reach this goal, I decided to add another layer of abstraction onto of the great foundation MEF provides for extensibility.
In order to do this, I needed 2 things:
  1. A Custom Configuration Section to Identify the specific Types to import
  2. A ComposablePartCatalog that will grab these parts.

Creating the Configuration Section

I'm not going to go into a large amount of information on how to create a custom configuration section.  There's a large amount of resources out on the Internet that can assist with such.  What I needed was a simple section that took a collection of types.  I wanted to truly keep things simple.  What came out mapped to the following:
1:  <configuration>
2:      <configSections>
3:          <section
4:            name="mef.configurableTypes"
5:            type="MEFExample8.Core.Provider.ConfigurableTypeSection, MEFExample8.Core" />
6:      </configSections>
7:   
8:      <mef.configurableTypes>
9:          <parts>
10:              <part type="MEFExample8.TestCommand1, MEFExample8" />
11:              <part type="MEFExample8.Commands.TestCommand3, MEFExample8.Commands" />
12:          </parts>
13:      </mef.configurableTypes>
14:  </configuration>


Creating the Catalog

Now that I have the section how I needed it, I had to create a catalog that will utilize it.  While I could have create a custom catalog using the ComposablePartCatalog base class, I decided to just extend the TypeCatalog that comes with MEF.  The TypeCatalog works great for such and the only thing I had to do was bring in the Types from the config file, load them into an IEnumerable<Type> collection, and then pass it to the TypeCatalog base.  Simple right?
As it turned out, I looked beyond my original needs and found two issues.  The first was addressing what to do if the configuration section was named differently.  The second issue was dealing with what to do if the catalog is instantiated more than once.  Luckily, a simple solution to both was to allow the developer to pass in the section name into the catalog's constructor.  This addressed the renaming concern while allowing multiple catalogs to point to different configuration sections.
Below is the catalog's code:
1:  public class ConfigurableTypeCatalog : TypeCatalog
2:  {
3:      public ConfigurableTypeCatalog()
4:          : base(GetTypes())
5:      {
6:      }
7:   
8:      public ConfigurableTypeCatalog(string sectionName)
9:          : base(GetTypes(sectionName))
10:      {            
11:      }
12:   
13:      private static IEnumerable<Type> GetTypes()
14:      {
15:          return GetTypes("mef.configurableTypes");
16:      }
17:   
18:      private static IEnumerable<Type> GetTypes(string sectionName)
19:      {
20:          var config = GetSection(sectionName);
21:   
22:          IList<Type> types = new List<Type>();
23:   
24:          foreach (ConfigurableTypeElement p in config.Parts)
25:          {
26:              types.Add(Type.GetType(p.Type));
27:          }
28:   
29:          return types;
30:      }
31:   
32:      private static ConfigurableTypeSection GetSection(string sectionName)
33:      {
34:          var config = ConfigurationManager.GetSection(sectionName) as ConfigurableTypeSection;
35:   
36:          if (config == null)
37:          {
38:              throw new ConfigurationErrorsException(string.Format("The configuration section {0} could not be found.", sectionName));
39:          }
40:   
41:          return config;
42:      }
43:  }

As I stated before, I decided to extend the TypeCatalog class. In order to extend it and meet the needs of my project, I overloaded the constructor so that I could set the default configuration section name (used in the configuration file snippet above) while allowing the second constructor to specify the section name.  Since the constructor of the TypeCatalog class takes either a single Type or a collection of Types, I turned the code that retrieves that information from the configuration section into a static value.  Since I'm not overriding any aspects of the TypeCatalog other than the constructors, I needed a way to pass the information to it as such and the static methods provided.

Notes about the Catalog:

At this point, I have a fully functional catalog that allows me to specify which types to include into a certain piece of my application. After I played around with it a bit I found that it worked well but had a few shortcomings that may cause people to wonder what's going on.
The first scenario I came across was working with the catalog to reference parts in a subdirectory.  This issue doesn't exist if the types you are referencing are in the same assembly or an assembly that's in the same directory as the executing program; however, if the assembly is in a subdirectory, you'll need to probe the directory as described in my previous MEF post - Using MEF and Custom Configuration Sections.
The second scenario came with the need for Type or Part exclusions.  The code above uses the same attributed model that all out of the box catalogs use.  This means that if you created an assembly with 3 attributed parts in it and used an AssemblyCatalog or similar one, all 3 parts would be identify and consumed.  The custom catalog outlined in this post focuses on Type inclusion and not exclusions. Because of this, if the above catalog is combined in an AggregateCatalog, all of the parts that meet the contract will still be imported.  Through this fact, I highly recommend using the above catalog separately from the others.

So What's Next?

As for this catalog, I need to do a bit more refining and possibly add a few more catalogs that follow a similar manner.  Ultimately, I'm going to be wrapping some unit tests around it and packaging it up separately.  Beyond that, there's a lot of MEF content coming in the future so stay tuned.

Resources:


kick it on DotNetKicks.com Shout it

Wednesday, January 6, 2010

Goals for the New Year (2010 ed.)

When I started to write down my professional goals for 2010, I ended up doing the same things as I did last year; just create a list of things to learn.  Looking back over my experiences while attempting to accomplish my goals for 2009, I began to believe that this was the wrong approach.  My initial goals were very focused on learning different technologies or hitting certain milestones in general.  By the end of 2009, I found that sure I may have hit most of my goals; however, something was lacking.

For example, one of the items that I completely wasn't anticipating becoming so passionate about was the possibilities of MEF.  This lead to a large number of rapid fire posts in March on some pretty rudimentary tutorials on the basics of MEF.  While I was working on creating these tutorials, I constantly wished I was able to work on the latter posts (the ones that I'm slowly working on currently).  It wasn't because I didn't think providing tutorial information wasn't important but being able to provide a better context was.  This was also true earlier in the year during my initial F# exploits.  Doing tutorials is good but sometimes showing the overall value before the tutorials can provide the basis as to the WHY someone can learn something instead of just the HOW to use it.  If you have ever looked into becoming a better speaker, this practice is sometimes described as "showing the sex first".  Get a person interested and intrigued first and then provide them the information and tools necessary to run with what you showed.

With this in mind, it's what my goals now resolve around.  While I'll be working hard on learning new technologies or tying down loose ends on technologies I haven't fully learned, my focus is going to be on context.

Things I'm focusing on in 2010:

Getting More Involved
I'm a firm believer that teaching is one of the greatest ways of evaluating how well you know a topic. One of the things I discovered a few years back is a passion for presenting and teaching. Combine that with a desire to get more involved in the development community as a whole and it leads to a lot of opportunities.  Right now I'm staring at about 4-5 presentations just in the first half of the year to various user groups and code camps all over the midwest.  I'm hoping to expand this more by the end of the year.  Sure it'll keep me busy and traveling but it's fun and I would am looking forward to meeting a lot of the community leaders at each location.  As my calendar solidifies, I'll make sure to update things here.

More Contextual Value on Posts
Looking over the past 2-3 years that I've been writing in this blog, the posts that have the deepest impact are also the ones that provide the richest examples or specific focus.  Some of the basic tutorial posts have appeared to be viewed well; however, the some of the posts that took me a significantly longer amount of time to write and develop the samples for have seen the best amount of feedback.  With this in mind, I'm going to focusing more on intermediate examples more than introductory posts.  The introductory elements will still remain; however, I'm hoping to change the quantity of the two types of posts in order to provide more balance and value to the readers.

Tying up Loose Ends
Last year I worked a lot on learning a number of new tools; however, some of them I only learned enough to understand where they may fit into my current situation (a.k.a. just learning what value is in it for me).  This is usually a pretty good way of learning things; however, it can limit the potential. After changing jobs at the end of 2009, being able to reevaluate what I learned last year and expand on such will allow me to adapt better to different scenarios.  This also will help me provide even better context when learning other things. An example of this is the fact that while I know the basics of writing in Silverlight, I need a little bit deeper understanding of it so that I can start applying MEF into my Silverlight applications when applicable.

Redefining a Dream
When I started getting serious about my development career, I wondered what I would consider to be my dream job.  This idea always shifted with every career progression on my way towards my dream job.  Recently, I landed a job that is the closest match thus far and am begin to evaluate the environments and companies I've worked for thus far as well as my current one.  While I don't plan on leaving my current employer since I just came onboard and they are a very good company thus far, I'm a person that loves striving for what's next. Perhaps what's next is not a different company but something that augments my day job like getting more involved. Perhaps is back to a more remote working environment.  Lots of ideas to consider and think about.  Ultimately though, it's the journey that drives me, not what I have currently obtains. Anything is possible as long as you keep dreaming in my opinion.