Showing posts with label Automation. Show all posts
Showing posts with label Automation. Show all posts

Friday, October 30, 2009

Reviewing UppercuT - A Build Framework Based On NAnt

Earlier this week I went to the Kansas City .Net User Group meeting where the topic of discussion was UppercuT, a build management framework created by Rob Reynolds. The discussion was pretty good and Rob did a good job at showing a number of demonstrations on how to use the various aspects of UppercuT. All in all, I feel like the presentation got the point across as to the typical why/when/how/etc. questions.


What Is UppercuT?


UppercuT is a build management framework based on NAnt. It was created in order to establish a consistent way to build .Net applications with a minimum amount of configuration while providing areas for extensibility. In addition to just compiling, UppercuT provides a number of options to manage application versioning, automated testing, and code packaging. While it can be extended to include such steps through NAnt, UppercuT doesn't deploy code...it only stages it since every company/project's deployment strategies may be different. One nice thing that UppercuT does do in terms of deployments though is provide you instructions on how to wire it up to CruiseControl.net.


When Should UppercuT Be Used?


UppercuT provides a way for people who have not started to take advantage of automated builds to do so quickly. Because of this, it's best suited for projects that don't have automated build systems integrated into them yet. With some configuration changes, it is possible to integrate with your current scripts; however, changes would need to be made on both side to the point where it would honestly be easier to convert your current scripts to custom steps for UppercuT to take advantage on. If you currently use some build framework like MSBuild or NAnt, I would probably recommend at least checking out UppercuT to see if it's a good fit; however, if you or your company already have some system that works for you in place, it may be better to just keep with what you have.


Where Can I Download and Learn More?


You can find more information on Uppercut by going to it's project website at http://ProjectUppercuT.org. In addition, you can check out Rob's blog over at http://ferventcoder.com/ as well as on Twitter by following @ferventcoder and @ProjectUppercuT.


kick it on DotNetKicks.comShout it

Wednesday, July 29, 2009

Writing a Custom NAnt Task (Part 3)

Over the past couple of months, I have been working on writing the SchemaSpy Task for NAnt located on CodePlex.  When I began to create a few custom tasks for NAnt in the past, I found very little documentation about how to create one.  Much of the documentation surrounding the process how to create a custom task is based solely on examples without any supporting information on how to expand such.  While this works very well for basic things, there were some experiences I had to work through while writing the task for SchemaSpy that I would like to talk about here.  In this, the third of four parts, we'll be diverting away from the example created by part 1 and 2 and dive straight into examples from the SchemaSpy Task for NAnt code in order to examine how to create a custom element collection for your task.  In this post, we'll be reviewing the Schemas property of the SchemaSpy Task for NAnt.

 

Why Do I Need a Custom Element Collection?

For a very large number of tasks, basic attribute-based task properties should be enough.  However, if you find yourself where you need to allow a list of inputs into your task, you are stuck with two different options.  The first option is to simply create a basic property just like those described in the first two parts in this series in order to set a delimited string from the build file.  While this is easy to code and implement, it is not the most user friendly after the list grows in size.  The second option is to create a child element that represents a collection of elements.  This is a little bit more complex to develop; however, it provides a much better experience for the people who have to maintain the build file.  Unlike a simple decorated class property, a custom element collection requires things to be implemented in the code.

 

Creating Collection Items

The first item that we need to create is a simple object that will represent the children nodes of your custom element collections.  Looking at the code, this is represented by the Schema class in the Schema.cs file.  This class has to inherit from the NAnt.Core.Element class and be decorated with the ElementName attribute, the attribute is used to identify the name of the XML node in the build file.  In the example (see the code below), the name of the element will be "schema".  After the class is established, we are able to add decorated properties that will represent the different attributes for the element. 

   1:  [ElementName("schema")]
   2:  public class Schema : Element 
   3:  { 
   4:      /// <summary>
   5:      /// The name of the schema to analyze and document. 
   6:      /// </summary> 
   7:      [TaskAttribute("schemaName")]  
   8:      [StringValidator(AllowEmpty = false)]  
   9:      public string SchemaName { get; set; } 
  10:  }
 

Creating the Strongly-Typed Collection

The second item we need to add to our task is a strongly-typed collection to store instances of our Schema object we just created. Looking at the code, this is represented by the SchemaCollection class in the SchemaCollection.cs file.  Like many strongly-type collections created in .Net, this class inherits from the System.Collections.CollectionBase class.  Simply by implementing the abstract base class and filling in the code, the custom, strongly-typed collection of Schema objects will be created and ready to use.

 

Implementing the Collection

The third and fourth tasks left to add our collection of custom elements to our NAnt task is to update the task code itself.  Now that we have the collection and child elements created, we have to first create a private variable to hold a new instance of our collection object.  If the collection is not created and instantiated, NAnt will not be able to add items to the collection and throw an error. 

The last thing to do is to create a new decorated property to tell NAnt to access and expect the collection.  Unlike the other properties that we declared that describe node elements, we need to decorate a property with the BuildElementCollection attribute.  The attribute requires three parameters.  The first describes the name of the collection node.  The second describes the name of the children nodes. The last parameter describes the task options.  Once all of these items are set, the new code should look like the below:

   1:  /// <summary>   
   2:  /// Instantiates a default collection to add elements to.   
   3:  /// </summary>   
   4:  private SchemaCollection _schemaCollection = new SchemaCollection();   
   5:       
   6:  /// <summary>   
   7:  /// Gets or sets the collection of schemas to use.  
   8:  /// </summary>   
   9:  [BuildElementCollection("schemas", "schema", Required = false)]  
  10:  public SchemaCollection Schemas   
  11:  {  
  12:       get { return _schemaCollection; }      
  13:       set { _schemaCollection = value; }  
  14:  }

 

A Look at the Build File

With everything completed, the code can be compiled and installed for NAnt to begin using it.  Once all of that is setup, you can access and update the build file to take advantage of the customer collection similar to the example below:

   1:  <example prop1="test">   
   2:      <schemas>   
   3:          <schema schemaName="dbo" />   
   4:          <schema schemaName="sys" />   
   5:      </schemas>   
   6:  </example>

 

Summary

In this section we reviewed the code implemented by the SchemaSpy Task for NAnt I wrote and placed out on CodePlex.  We looked at what it takes to implement a custom collection.  In the next and final installment of this series, we'll be looking at writing a custom task that is used to call an external application and look at what differences there are between it and the task we have been working with in the first two segments of this series.



kick it on DotNetKicks.comShout it

Thursday, July 23, 2009

Writing a Custom NAnt Task (Part 4)

In this final installment of a series of posts looking into how to create a custom NAnt task, we'll dive into how to create a task that executes an external application.  While NAnt has the built in <exec> task to handling command line programs, there may come a time where the possible arguments or the command itself is just too much for such a generic task.  This was the case that I ran into while trying to integrate SchemaSpy, the Java-based database analysis and documentation tool, into my build scripts.  With the number of arguments and complexities of the program, I decided to create my own custom task.  We'll be diving into this task while exploring the differences between a basic task and an executable task.  The source code for this project will refer to the SchemaSpy Task for NAnt located out on CodePlex.

 

A Look at the Default <exec> Task.

NAnt comes bundled with a task that can be used to call external programs as a way to extend the functionality of the build script.  For some applications that take only a few arguments, this is really easy to manage.  This can be seen using a command line program for the DotNetMigrations project also located out on CodePlex

   1:  <exec program="db">
   2:      <arg value="migrate dev 2" />
   3:  </exec>

However, if you are using an item like SchemaSpy, the cleanliness of the solution begins to deteriorate fairly quickly with only a partial set of its argument.

   1:  <exec program="java">
   2:      <arg value="-jar SchemaSpy.jar -t mssql05 -host serverName -db myDb -port 1433 -sso -all -o ..\output" />
   3:  </exec>

When there is a need to integrate a program like SchemaSpy into a tool like NAnt, creating a custom task to streamline the arguments into a more user-friendly manner can greatly ease script authoring and debugging.

 

Task vs. ExternalProgramBase

In previous examples in this series, we would create our custom tasks using the NAnt.Core.Task abstract base class.  This base class gave us everything we needed to do to handle properties, children elements, and similar items.  One item that the Task base class did not handle though was running external processes though.  If you were to attempt to start a new process, an error would occur or the script would hang since the Task base class isn't thread-safe by default.  If you wanted to create the multi-threading code in order to make the process run successfully and safely, then you can definitely do such while inheriting from the Task base class; however, NAnt has already done this for you by providing the ExternalProgramBase abstract base class.  The ExternalProgramBase class inherits from Task and enhances it by providing specific methods and properties for working with external programs.  In addition, it has all of the threading logic established for you.

 

Diving Deeper into ExternalProgramBase

Since ExternalProgramBase inherits from Task, every thing we can do when we inherited from Task previously can still be done when we inherit from ExternalProgramBase.  Where the differences come into play is when we overwrite the ExecuteTask() method.  This method is called by the NAnt engine to start the task's specific function.  In the ExternalProgramBase base class implementation, this method executes the actual external application we want to start.  It does so by looking at the values of 2 properties; ExeName and ProgramArguments.  The ExeName property provides the string name (with extension) of the executable program to run.  The ProgramArguments property provides a string representing the fully list of arguments.  We can still provide our own custom code inside of the ExecuteTask() method, and once we're ready for the external program to start, we can simply set the ExeName property and call base.ExecuteTask().  The base class's ExecuteTask() method will read both properties, create the necessary threads, and execute the program.  Below is a snippet from the SchemaSpy task that illustrates this process.

   1:  /// <summary>
   2:  /// The Program Arguments listing used by the ExternalProgramBase class.
   3:  /// </summary>
   4:  public override string ProgramArguments
   5:  {
   6:      get { return BuildArgumentList(); }
   7:  }
   8:   
   9:  /// <summary>
  10:  /// Executes the task's operation for NAnt.
  11:  /// </summary>
  12:  protected override void ExecuteTask()
  13:  {
  14:      bool isValid = ValidateAttributes();
  15:   
  16:      if (!isValid)
  17:      {
  18:          Log(Level.Error, "Task Attributes are not valid.");
  19:          return;
  20:      }
  21:   
  22:      this.ExeName = "java.exe";
  23:      base.ExecuteTask();
  24:  }

Notice that in the SchemaSpy task for NAnt, I did not directly set the ProgramArguments property prior to calling base.ExecuteTask().  Instead, I had the property execute a method that would do it when the base class is ready.  This method evaluates the various properties/attributes of the task in order to create all of the arguments required by SchemaSpy to run properly.

 

The Updated Build Script

With the custom task created, we can now update the build script to use it instead of relying on the <exec> task.  Below is an example of the SchemaSpy call previously shown.

   1:  <schemaSpy
   2:      jarPath="..\SchemaSpy.jar"
   3:      dbType="mssql-jtds"
   4:      host="MyDatabaseServer"
   5:      port="1433"
   6:      dbName="MyDatabase"
   7:      schemaName="dbo"
   8:      outputDirectory="..\MyDatabaseDocumentation"
   9:      singleSignOn="true" />

 

Summary

Throughout this series of blog posts, we've covered how to make very simple custom tasks for NAnt up through the ability to create complex tasks that contain child elements and/or simplifies running external programs.  Through the steps listed in this series, hopefully all of information you need to create a custom NAnt task will be available to you.  If a scenario comes up that you need assistance with, feel free to post a comment and I'll see what I can do to assist.


kick it on DotNetKicks.com Shout it

Wednesday, May 27, 2009

SchemaSpy for NAnt

Earlier this year, I talked about a couple of tools to help automate pieces of the development process in some fashion.  One such tool was SchemaSpy, a Java-based database documentation tool.  SchemaSpy is a tool that I love working with; however, always was annoyed of using it in a manual method.  I streamlined this a little bit using a batch script but it still didn't feel quite right when I hooked it up to my automated deployments.  So, I did what any process driven developer would do; I made it work more naturally into NAnt.

I started a custom NAnt task for SchemaSpy.  Given the complexity of using SchemaSpy to begin with, I decided to not seek to have it added to the NAnt contrib or similar libraries around the web.  However, even though I wanted it separate, I did decide to open source it.  So, I create a CodePlex project to host my NAnt task for SchemaSpy.  In it's current version, it only has the bare minimum of options that I have required; however, I am hoping to continue working on it until it's feature complete.  If you are looking for a great way to document your databases in associated with your automated builds with NAnt, take a look at it since it hopefully will provide you options for what you need.

I'm always open for feedback and recommendations on the task.  Hopefully I'll be posting the next release with in the week.


kick it on DotNetKicks.comShout it

Thursday, March 12, 2009

Tackling Anxiety Against Automation

A few years ago, the company that I worked for was preparing to upgrade from MS Sql Server 2000 to Sql Server 2005.  While there wasn't too much abnormal concern about upgrading the databases themselves, there was a lot of concern about the large number of DTS packages that the company created for the majority of their B2B processes.  Their solution until they could come up and complete a test strategy to ensure the DTS packages would run under Sql 2005's DTS runtime, just incase the packages couldn't be converted to SSIS in some fashion, was to stop writing new DTS packages and push the processes into .Net console applications that would be scheduled through a batch processing system.  Personally, I liked this solution since it allowed me to write .net code in VS and not VBS code inside of the DTS Package designer in Enterprise Manager.  Actually abstracting out processes into a more reusable and testable state was a great benefit over VBS and DTS as a whole.

One of the first console applications written under this initiative was to automate a series of manually ran Sql Scripts that people were running against production to accompany a file import process.  It was a pretty simple project and ran, still to my knowledge, bug free, even when tested by my QA team.

Now fast forward to today.  The need for the process is still in place and the code hasn't been updated since no changes have been required; however, I found out that the application was only ran once in the past few years and then the person who used to manually go through the steps and scripts didn't want to automate it and has been doing the manual steps ever since.  Even today, this person has continued to manually spend a few hours every other day going through the same ceremony.  I think my brain gave me a BSOD from what I find highly illogical thinking.

So I approached my coworker who has been doing this and start asking the number of "why" questions I had.  Here are some of the responses I was treated to:

  • "I don't trust what I cannot see or watch."
  • "I like being able to watch and correct where necessary."
  • "The process had a bug."
  • "I won't ever know when it breaks or succeeds."
  • "What else would I work on?"

From the discussion around these themes, there are a few design and development themes that can/should be implemented when faced with such an anxious coworker.

 

Establishing Trust through Quality

While I see James Bach's point in his "Quality Is Dead" blog post, I disagree with the blanket of the statement.  There are always constraints when you're talking about software development (usually Time/deadlines being the largest constraint), it doesn't mean that every piece of code that you churn out will be of poor quality.  Practices such as TDD or just plain Unit Testing and reporting can be implemented in order to demonstrate and achieve a higher level of quality to someone who doubts quality.  Now, I believe that any developer can write code with and without unit tests and still achieve the same level of quality; however, it's easier to validate that level of quality through unit tests and TDD practices.  This level of validation is essential to people who do not trust something to be automated.

 

Establishing Awareness through Instrumentation

The bulk of the other themes that I received surrounded the concept of being able to step through the process and watch/correct the data as necessary.  This is a nice thing for debugging; however, if an automated process has to be stepped through, then it's not automated.  One way to handle this issue is to ensure you have a high level of coverage of instrumentation in the code/process.  This means making sure the proper types of notification is provided, auditing is turned on to the degree needed, and any errors (be it bugs or bad data-related) are captured and reported appropriately.  I cannot confirm my coworker's claim that there was a bug in the code since I was unable to find one in the bug database.  If an email to capture the error existed, at the very least, I would begin having my own doubts about the quality of the code; however, even that was not found.  If instrumentation was implemented to a better degree than what I put into it for error logging and tracking, I would be able to verify the claim.  As far as bad-data and catastrophic scenarios (i.e. server loses network connection mid-process), those items can (and was) implemented through logging or some for of process auditing infrastructure.  The more data that can be provided, the more confidence the person will have when automated.

 

Establishing Automation Can Assist with Workload

The last theme that came from the conversation dealt with how the person's routine became.  This person was a very routine-oriented individual who knew exactly when to do when and how long it would take.  This person was a person of ceremony and possibly a bit fearful of being automated out of a job.  After being in the industry for a number of years, I have came to the conclusions that those nightmare stories of people being replaced by programs rarely happen to the developers or analysts that implement the program or automated process.  The automation allows for new, different tasks to be done.  Even if the automated process is kicked off manually, it still frees up time to do other things.  This means you can do more easily or in general transition onto the next big thing.  I do not really know any company out there that would say they have a truly shortage of work.  Even if it's wish-list, lowest priority, internal cost projects, there's always something to do.

 

Summary

Automation is a beautiful thing that I have grown more and more fond with.  I have grown into a developer where I look for things to automation.  I have seen the themes I talked about here at a couple of different companies I've been with and people that I've talked with.  Each time, I find more and more ways to automate certain elements of a work day and also make the work flow easier.  Hopefully a few things here will help others as well.  Now if only I could find a way to automate my chores at home. :-)



kick it on DotNetKicks.comShout it