the ramblings of a .net dude

Exception Usage Considerations

2011-08-10T07:26:00.004-07:00

It seems that I always have the following conversation with different developers.

Them: This method shouldn't throw an exception, it should return an error code.
Me: OK. Why?
Them: Because... umm... exceptions are a terrible means to communicate errors. I want to see a status flag and an error message.

I kinda get the point, but in frameworks such as .NET and Java, it really doesn't make sense to use anything but exceptions.

First, if you use error codes or anything of the sort, you now have the worst of both worlds: the runtime and class library code will throw exceptions and your code will return error codes. Now you have to deal with both.

Second, when returning error codes and messages you have to make assumptions about how application developers will use your code. It's impossible for you to determine under what conditions your code will be used and what assumption the application developer will make about it. It's best to just be consistent and throw an exception when you're not able to complete the work you "contracted" to do through your method signature.

In the interest of completeness: there's perhaps one case in which you should return a flag instead of throwing an exception. When you have a method that gets called frequently and it also has a high failure rate, you'll likely experience a performance hit (from throwing/handling so many exceptions) that won't be tolerable.

Microsoft, for example, had the above problem with the Int32.Parse method. To address the issue MS introduced TryParse. This new method returns a flag telling you whether worked and returns the value of the parse in and output parameter called result.

It's important to note, however, that the boolean flag returned only indicates one type of failure: the method's inability to parse the string into an integer. The method still throws an ArgumentException if the argument is not valid.

So there you have it: always use exceptions. And if you're not going to listen to me, use error flags to indicate just one type of error.

Injection with Unity: Mind Your Lifetime Dependency

2011-05-03T20:24:00.007-07:00

For the past few months we've had this bug (PROD only, and unreproducible in any of our other environments) where every now and then we could not get data from our DB. The DB was up and running just fine, but the SqlClient on the web server just could not read the data (we'd get 2 errors: one about trying to read data where a reader was already open, and another about not being able to cast the data from the reader to a certain type). In case you're interested, the exact errors were:

System.InvalidCasException: Specified cast is no valid. At System.Data.SqlClient.SqlBuffer.get_Int32()
Invalid attempt to call Read when reader is closed. Somewhere in System.Data.SqlClient as well.

For a while we just lived with the error; it didn't happen very often and it only affected a few customers. Then the problem got worse and so we immediately blamed it on some environment change... we recycled the app pool on a consistent basis while trying to figure out what the problem was.

Finally, when the problem got bad enough, I did a little bit of research and concluded that we were seeing the problem because we were not disposing our LINQ contexts correctly. After trying to prove that the problem was in someone else's code (and failing), I quickly concluded that it must be Unity that was not disposing the contexts.

You see, I had assumed that Unity would new up an instance of our db context on every call to resolve, but it turns out that by default the lifetime of the dependencies Unity resolves is the same as the lifetime of the IoC container. In our case, the IoC container stayed alive throughout the entire app (because we needed to resolve stuff in our MVC controllers) and so the longer the app went without a restart, and the more requests it handled, the more we saw the error.

Fortunately, because we were using Unity to resolve all of our dependencies, and because the Unity code was all in one place and well isolated from the rest of the app, it was really easy to fix. All I had to do was write a per HTTP request lifetime manager and plug it into our container. I'll be posting the code for that container soon... but now I gotta go to bed. Looks like we'll have quite a few bugs to fight tomorrow. :)

Edit:

As promised, here's teh code for the "per HTTP request" Unity lifetime manager:

public class HttpRequestLifetimeManager : LifetimeManager
{
   private string _key = Guid.NewGuid().ToString();

   public override object GetValue()
   {
       if (HttpContext.Current != null && HttpContext.Current.Items.Contains(_key))
           return HttpContext.Current.Items[_key];
       else
           return null;
   }

   public override void RemoveValue()
   {
        if (HttpContext.Current != null)
             HttpContext.Current.Items.Remove(_key);
   }

   public override void SetValue(object newValue)
   {
        if (HttpContext.Current != null)
             HttpContext.Current.Items[_key] = newValue;
   }
}

write less

2011-01-20T22:12:00.005-07:00

honoring the title and idea of this post i'll just say this to get the point across:

the quality of your solution is inversely proportional to the amount of code you write to solve the problem.

unfortunately, i have to say a bit more to justify my thoughts.

i've always wondered about how to write good code; and i'm not the only one apparently:

well, i've finally figured it out: good code is written by writing as little code as possible. this is true because the when you write the bare minimum you will almost naturally end up with code that is easy to read, easy to test, easy to extend, and therefore, easy to maintain (to see more about the attributes of good code, take a look at the pyramid of code quality).

i understand that there are guidelines and principles that you must still follow when writing "just enough" code, but things really deteriorate fast when you don't KISS.

so next time you're inclined to do solve a problem with code, first think about if there's a way to solve it without any code. then, and only if you absolutely must, write as little code as possible.

Simplicity

2010-10-26T19:14:00.002-07:00

A couple of days ago while on my way to work, I was listening to Erno Rubik on an interview on NPR (yeah, I know, leftist media).

At one point the host (Doug Fabrizio, I believe) asked: "Why did you choose a cube? Why 9 squares on each side of the cube?" To which Mr. Rubik answered something to the effect of:

Well, I was trying to come up with the simplest solution possible. And that actually turned out to be very hard; it's always very hard to define something in it's simplest terms... (and a few more minutes of explanation about why reaching simplicity is so hard) Anyhow, I wanted to find the simple form to teach my students how to describe movement in mathematical terms; a cube with nine squares was the simplest way I could do that.

I just loved that answer. I've long had a feeling that when writing software the simplest solution will always be the most maintainable solution. Simplicity is the core principle on which YAGNI, DRY, and so many other software engineering ideas are built. As DaVinci said:

Simplicity is the ultimate sophistication.

So here's this post in honor of writing simple, sophisticated code. Long live simplicity!

Visual Studio 2010 Debugger Not Attaching Problem

2010-05-11T08:37:00.005-07:00

Since we moved to Visual Studio 2010, I've had 3 people ask me: "Is there some sort of magic you have to do to get VS2010 to attach to the IIS worker process (w3wp)"?

So, in the interest of (hopefully) saving you some time, I figure I'd write a little bit about what the problem with the VS2010 debugger is. Actually, the debugger is fine, it doesn't have any problems attaching; it's actually VS2010 itself that has a minor bug(?) while attaching.

As you can see in the image below, under normal circumstances Visual Studio automatically detects what type of code the process is running and automatically chooses the correct debugger:

Unfortunately, however, I've found that sometimes VS2010 cannot detect the correct code type in the worker process and attaches to the process using the .NET 4.0 debugger (I don't know why this happens nor do I know if this happens when attaching to other processes).

If you attach using the wrong debugger, you'll attach, but you'll notice that none of your symbols ever load and therefore none of your breakpoints ever hit. It's actually somewhat frustrating because everything looks like it should be working but doesn't.

But enough talk about the problem; here's all you need to do to solve the problem if you notice you're not using the right debugger:

Hit the "Select" button the in "Attach to Process" window.
Select the "Debug these code types:" radio button
Check all the code types you want to debug.

You're shooting for something like this:

I've found that after you choose the code types you want to debug, VS2010 remembers your selection, so you should never have problems attaching in the future.

Now, if I could just get rid of that stupid dialog asking me to confirm if I really want to attach to the "potentially dangerous" w3wp process, I'd be set. :)

3 Simple Rules To Good Object Oriented Code

2010-01-13T17:28:00.004-07:00

Mike and I had a long discussion about BDD last night. After he complained about how BDD didn't lead him to the solution he wanted, I said something to the effect of "You know, the only reason we have all these design methodologies is because we don't really know how to think OOP and write OOP".

At this point, before I continue, I need make something clear: I don't know what I'm talking about; all I know about BDD is what Wikipedia says about it. But I do know that writing good OO code doesn't require 3 different methodologies. All it requires if for programmers to think like objects.

I like to think that good OO comes down to 3 simple rules:

Objects are all about behavior; not attributes. If attributes defined objects, a cat and a dog would pretty much be the same thing since they both have eyes, noses, legs, etc.
Objects are discrete things from which you can build other things. They're a lot like LEGO blocks. Incidentally, code reuse is a byproduct of good OO code, not the purpose of it.
Objects like to be autonomous self directed entities. Objects don't like "managers"; if your code has lots of managers, you're probably doing something wrong. Objects like to coordinate their work through events; they don't like managers telling them exactly what to do at every point.

A bonus rule: Composition is usually better than inheritance; it gives you a lot more flexibility when having to change behavior.

To be honest: I'm mostly writing this because I'd like to hear from others what guiding principles you should have in your head when coding. Do we really need a lot of process if we just in our guts know how to write OO code?

The hardest thing about programming.

2009-10-28T19:49:00.003-07:00

When asked what the hardest thing about programming is, Bill Gates answered:

The hardest part is deciding what the algorithms are, and then simplifying them as much as you can. It’s difficult to get things down to their simplest forms. You have to simulate in your mind how the program’s going to work, and you have to have a complete grasp of how the various pieces of the program work together. The finest pieces of software are those where one individual has a complete sense of exactly how the program works. To have that, you have to really love the program and concentrate on keeping it simple, to an incredible degree.

What's interesting, however, is that the hardest thing about programming is also the best thing about programming.

I love programming because learning it is relatively easy, but mastering the art is extremely difficult. Like Gates says, reducing things to their simplest form is difficult - but fortunately, it's also beautiful. It's also true that the finest software is written by those who love the program and make a deliberate effort to keep it simple - and again, although difficult, being able to model an entire system in your head is pretty awesome!

Simple Events In ASP.NET MVC

2009-10-06T10:54:00.003-07:00

We've been busy improving our customer facing applications. Our new marketing VP noticed some significant improvements that could be made in the workflow of our customer acquisition funnel.

Making the changes that our marketing VP recommended required some major code changes so we took the opportunity to move our web apps into ASP.NET MVC.

One of the early decisions we made was to put as little business and application logic on our controllers as possible. Finding a place for business logic is straightforward most of the time; it usually belongs in your (domain) Models.

Placing application logic in the right place gets a little more complicated. We don't really have services for stuff like sending out emails, etc. so we have to write classes to do all that work.

Invoking email code, authentication code, etc. right in the controller is an ugly solution. To solve our problem, I came up with a simple eventing mechanism that allows us to move all application code out of our controllers.

Let's first start with what the code in one of the controllers would look like:


[AcceptVerbs(HttpVerbs.Post), ValidateAntiForgeryToken]
public ActionResult Index(IndexModel model)
{
     //code to interact with your model goes here.
     
     //And now the event code:
     AcmeEvents.Raise(new IndexRanEvent { IndexModel = model });

}

The great thing about the code above is that all sort of things could now happen once your index runs; there may be 1 or N subscribers to the event above. Furthermore, you could (at run time) change the code that runs when the event triggers... but I'm getting ahead of my self; we'll look at how that could happen in just a second.

Let's now look at the "raise event code". To do this, I decided to just use Unity's IoC container to resolve the code should trigger when the event get's raised:


public static void Raise<T>(T args) where T : IAcmeEvent
{
     foreach(var handler in unityContainer.ResolveAll<Handles<T>>())
     {
          handler.Handle(args);
     }
}

Some assumptions the code above makes: (1) IAcmeEvent has a Hanlde method; (2) by the time the Raise method runs, the IoC container must have been initialized to map the different "event types" to the specific implementations that will execute.

How you initialize the container is really not that important: you could do it in code, through a config, or any other method of your choice. What is important, however, is that when you initialize your container, you do it in such a way that code from other assemblies can execute. If you do this, you could literally just drop assemblies in your application and add to or change the behavior of your app.

So there you have it; a simple eventing mechanism to keep your MVC controllers unpolluted from application logic. I'm really happy about coming up with this solution; I hope you find it useful too.

Barricading Your Code

2009-08-05T13:37:00.010-07:00

You've heard it a million times: "Never trust your inputs". Or maybe you've heard its corollary: "Always sanitize your inputs". Easier said then done, right?

I'd like to show you a way to barricade your code so that you can always sanitize your inputs without littering all of your code with sanitation code.

The key to writing good sanitation code is controlling where and howyou accept external input. Just like the barricade in the image above, traffic can only flow on the sidewalks and not on the street.

Here's how to accomplish the same with code:


public class ClassThatUsesBarricade
{
    public void DoWork(string inputOne, int inputTwo, ...)
    {
         //sanitize inputs
         RealDoWork(inputOne, inputTwo, ...);
    }

    private void RealDoWork(string inputOne, int inputTwo, ...)
    {
         //don't worry about inputs because no external class could have called this.
         //do work
    }
}

By writing code like the code above, you've barricaded your class: data can only flow into your class through the public method. The private method can absolutely, 100% or your money back, trust its inputs without having to worry about sanitizing them. This way, your real work method can be short, and concise - after all, a method should only do one thing, and do it well.

If your class has a lot of methods with very similar signatures that do similar things, you might want to reverse the code I showed: you could have a private sanitation code that all public methods that actually do work call. However, that's not really barricading your important methods... at some point you could forget to call the private sanitation method (ie. set up your barricade), and you'd be screwed.

And there you have it, a code barricade.

When to use Class Inheritance in Programming

2009-07-07T20:35:00.008-07:00

Last week I listened to a friend of mine complain about some straight-out-of-college programmers he had recently hired: "All of their code has 10 layers of inheritance. It's crazy trying to maintain their code!".

Unfortunately my friend is right: inheritance, although powerful - when used correctly, naturally tends to increase complexity.

Since your primary purpose as a programmer is to manage and reduce complexity, you should favors solutions that don't use inheritance.

At this point you're probably asking yourself: When am I supposed to use inheritance? How do I know if I'm using it correctly?Fortunately, the great Steve McConnel, has written 4 clear cut rules on when to use inheritance and when to use containment:

1. If multiple classes share common data but not behavior, create a common object that those classes contain.

2. If multiple classes share common behavior but not data, derive them from a common bases class that defines the common routines.

3. If multiple clses share commond data and behavior, inherit from a common base class that defines the common data and routines.

4. Inherit when you want the base class to control your itnerface; contain when you want to control the interface.

In summary, only inherit if the new class truly is-a more specialized version of the base class.

Oh, and one more piece of advice on inheritance: avoid multiple inheritance like the plague.

How To Write Reusable Software: A Lesson From LEGO Blocks

2009-06-15T19:12:00.007-07:00

I don't know about the places where you have worked, but where I've worked we haven't had a very good track record for writing code that actually gets re-used. In fact, at one place we even gave up: "Software reuse is overrated" a boss muttered once.

I think the reason we fail so miserably is that we focus too much on the end goal (reusable code), and too little on the qualities that make software reusable. What, then, are the properties of reusable code? Well, that's where the LEGO blocks part comes in.

The LEGO block analogy comes from David West's Object Thinking, but I'd like to highlight a few ideas that make this metaphor so applicable to software development:

The interface is simple and intuitive. I've seen my 3 year old daughter build ships with LEGOs. If you want your code to be reused, it should be so simple and so intuitive that others could use it without having to refer to external documentation.
There's a finite and small number of LEGO block types. If you find that your library has a high number of classes, think about what you are doing. After all, everything on this planet is made from the same 92 (naturally occurring) elements.
LEGOs are cool because you can build stuff. The reason everyone loves LEGO blocks is because they can build whatever they want with them, not just what the box tells them to build. Your software should be the same way; you should be surprised by what end users build with your code. If your code only gets used for what you envisioned, you've failed to write reusable code: people are not able to compose software with your code.

Finally, don't forget that people don't really care about the technical details behind LEGO blocks; no one cares how the plastic is molded, dyed, etc. LEGO blocks aren't reused because of how well they're manufactured; they're reused because they let end users accomplish their goals.

Once people are able to accomplish their goals with your code, they'll be using it over and over gain. Guaranteed!

Custom Thousand Separator In C#

2009-06-09T08:03:00.000-07:00

I guess the NFA requires us to use a space (" ") instead of a comma (",") as our thousands separator. I dunno why the have to regulate something like that, but I had to figure out an elegant way to format our numbers the way the powers that be require them to be formatted.

After struggling with different inelegant solutions for a little while, I finally realized that different countries use different symbols as thousands separators; I knew then that my problem was really a localization problem. A quick search in MSDN lead me to the NumberFormatInfo class, which "defines how numeric values are formatted and displayed, depending on the culture" (MSDN).

The NumberFormatInfo class, has a NumberGroupSeparator property that defines what character to use to separate thousands.

So, here's how I ended up solving my problem:


public void FormatDouble(double doubleToFormat)
{
     NumberFormatInfo nfi = new CultureInfo("en-US", false).NumberFormat;
     nfi.NumberGroupSeparator = " ";
     Write(doubleToFormat.ToString("n", nfi);
}

Neat, huh?

Highly Parallel LINQ To Lucene Indexer

2009-03-24T17:22:00.014-07:00

I've been wanting to post about this for a while, but haven't been able because I was away vacationing in Hawaii. :)

I was recently given the assignment to index a couple of our tables that hold large amounts of data. Each tables has well over 3 million records, and each record is fairly large as it represents "emails" with our clients.

I knew from the get-go I'd have to write a highly parallel process in order for the task to finish promptly - just bringing the data over the wire from the database takes about 45 minutes.

I have to show a lot of code now, so please bear with me for just a second. The key here is to use several RamDirectory(s) for the small data chunks and merge them into a FSDirectory index that will hold all the data:


namespace Indexer
{ public class BatchIndexer[T] : ParallelIndexer[T]
 {
  IQueryable[T] items;
  bool isMergingFinished = false;
  private static object lockObject = new object();

  public override bool Stopped
  {
   get
   {
    var pstopped = base.Stopped;
    if (pstopped == true)
    {
     return isMergingFinished;
    }
    else
    {
     return false;
    }
   }
  }

  //stop setting the temp directory here and force the client to pass it in
  public BatchIndexer(string path, IQueryable[T] items)
   : base(new Index[T](path))
  {
   this.items = items;
   QueueChildren();
  }

  private void QueueChildren()
  {
   if (null != items && items.Count() > 0)
   {
    const int pageSize = 100;
    int pageIndex = 0;

    var currentPage = items.Take(pageSize);
    int pageCount = 1 + items.Count() / pageSize;
    while (pageIndex < pageCount)
    {
     ThreadPool.QueueUserWorkItem(new WaitCallback(QueueChild), currentPage);
     pageIndex++;
     currentPage = items.Skip(pageIndex * pageSize).Take(pageSize);
    }

    //wait for all children to be added to parent
    while (Children.Count() < pageCount)
    {
     Thread.Sleep(2000);
    }
   }
  }

  private void QueueChild(object stateInfo)
  {
   var currentPage = stateInfo as IQueryable[T];
   lock (lockObject)
   {
    AddChild(new BatchIndexerChild[T](new Index[T](), currentPage.ToList()));
   }
  }

  public override void Run()
  {
   //start indexing the children
   foreach (var item in this.Children)
   {
    var child = item as BatchIndexerChild[T];
    ThreadPool.QueueUserWorkItem(new WaitCallback(child.AsyncRun));
   }

   //wait for all of the children to finish indexing
   while (!IsChildrenStopped)
   {
    Thread.Sleep(1000);
   }

   //merge all indeces into main index
   this.Merge((from c in this.Children select c.Index.Directory).ToArray());
   isMergingFinished = true;
  }

  protected override void Dispose(bool disposing)
  {
   base.Dispose(disposing);
   if (disposing)
   {
    //release managed resource
    if (this.Index != null)
    {
     this.Index.Dispose();
    }
   }
  }
 }
}

I particularly like the approach I used because it threads the chunking of the data into smaller pieces AND is also threads the actual indexing of each chunk.

If you're going to index as much data as I had to index, however, you still have to pre-chunk the data and feed into several instances BatchIndexer; I found that 100,000 records was optima. You'll also want to make sure you release all database resources ASAP: you don't to be hogging a whole bunch of memory!

Anyhow, I hope this code is useful... it should at least give you start in the right direction. Let me know if you have any questions.

Factory Pattern

2009-02-22T19:54:00.003-07:00

After reading this article, a reader recently requested that I show some sample code for the "factory" mentioned in the article.

Before I give you some sample code, however, let me just write a few words about the Factory Pattern: As the pattern's name suggests, the Factory Pattern creates stuff. Specifically, "factories" create classes without coupling you to a specific instance of the classes it creates.

Here is some sample code:

public class CreditCardBase
{
  //this is the base class that all credit cards implement:
  //it could be an abstract class, or it could have some functionality;
  //it could also be an interface... whatever floats your boat.
}

public static class CreditCardFactory
{
  public static CreditCardBase GetCreditCard(object someObject)
  {
       //business logic that returns correct credit cart type goes in here.
       if(someObject == "something")
            return new Visa(); //visa of course implements CreditCardBase
       else
            return new Amex(); //also implements CreditCardBase
  }
}

public class CreditClassClient
{
   //this class consumes a credit card
   public void DoWork()
   {
        CreditCardBase myCreditCard = CreditCardFactory.GetCreditCard(new bizObject());
        //do stuff with myCreditCard
        myCreditCard.Run();
   }
}

And there you have it. That's pretty much all there's to it, and the code is pretty self explanatory.

There's one more thing that's worth mentioning: usually, classes returned by factories have private constructors - to avoid people instantiating the class without calling the factory. This, however is problematic for several reasons:

You may have clients that already instantiate your class.
People usually make private constructors singletons (please don't).
How do you extend a class that has a private constructor? :)

Finally, to the anonymous reader that asked for the factory code: let me know if this was useful.

How Not To Lose Money In The Stock Market

2009-02-10T21:50:00.001-07:00

One of the most important rules to remember when investing in the financial markets is to Preserve Your Capital! Not losing money is the first step before you can make any money.

And although I haven't figure out a technique that guarantees 10% yearly gains, I have figured out a pretty good technique for not losing money in major market draw downs. Allow me to show you a chart of one of the ETFs I like to use to trade:

The chart to above shows the daily price for QLD, along with the 50-day exponential moving average (in red) and the 10-day exponential moving average (in blue). At the bottom of the chart, you can also see the daily trading volume.

What's interesting in the QLD chart is that during the past year every major market drop has been preceded by a clear signal: the 10-day EMA crosses the 50-day EMA (heading down, of course). And this, even when the market has faced major volatility and huge daily drops.

You can go back as far as you like, and you'll find that the signal above always hold true. And this is not because I'm so awesome and I've figured out some magic formula - no, it holds true because of the very definition of EMA. The EMA lines just show you the averages of the stock price for the last 50 and 10 days respectively.

Of course this technique isn't perfect: you will lose some money before the 10-day EMA crosses the 50-day EMA; but at least you won't lose more that 50% of your money, like several people have in the past year.

Disclaimer: Don't take financial advice from a guy that writes software for a living (i.e. me). Also, don't take advice that makes a living selling financial advice (i.e. your financial advisor). Those guys don't make money in the markets; they make money making you feel good about losing 50% of your capital.

ASP.NET MVC Release Candidate Now Available!

2009-02-05T19:24:00.003-07:00

The release candidate for Microsoft's ASP.NET is finally here.

Sure, this is just another "me-too" Microsoft product: we've had Rails, Cake, and Grails - but this is great news nonetheless! The ASP.NET web forms model is one that I never really understood: Why would you try to imitate a windows form on the web? Anyhow, web forms have been long over-due for a replacement, and now MVC provides a great architectural pattern for building web apps with the .NET framework.

Scott Gu, has written and in-depth article describing some of the most important improvements on this RC. And, from what Phil Haack said on his blog about this release, I get the feeling there's just a few minor bugs that need to be addressed before the RTM. You could probably start using the RC now without any concerns. In fact, I have a co-worker that has been writing a site on MVC since beta with no problems at all. Another co-worker mentioned to me today how much more productive he was under the MVC model than the WebForms model.

If you're still unsure about ASP.NET MVC, go check out stackoveflow.com. StackOverflow was written on the beta release of the MVC stack, and it's an awesome site.

Now, if you're still unsure, just trust me on this one: put your WebForms on the ground, back away slowly! I don't want anyone getting injured here. :)

Software Engineering: There's nothing like it!

2009-01-26T16:33:00.006-07:00

Intro

Seriously, there's no other type of engineering like software engineering. And that, may or may not be a good thing: it just depends on what you decide to focus on.

I went to school in the same building where the mechanical engineers, chemical engineers, and civil engineers went to school - yes, I spent almost every waking hour of my college life in the infamous Clyde Building:

Yes, for those of you that are wondering, that building has no windows. I'm pretty sure they designed it that way so that your soul would be crushed within the first semester of your career. But that's beside the point: You want to know what I learned in 4 years as an EE at BYU? Well, for the most part, I learned a whole bunch of math... oh yea, and I also learned that writing software isn't like making bridges, roads or "stuff".

And here's why software enginnering is so different: software is cheap to make (Yeah, that's right; even though you think you're making big bucks by writing software, you're still cheap compared to what goes into creating a new car design). Oh, and there's something else: writing software is much harder than building bridges.

Software is hard

See, we can't really prove software will "work". Proving that a program will work according to the specs is pretty much a halting problem - that is, given a program and a finite input set, we can't even prove whether the program will even terminate!

Civil engineers, on the other hand have it easy: given a bridge and environment conditions (load, winds, etc.), they can tell you whether the bridge will stand or not. In fact, it's so easy to "model" bridges that we software engineers have written programs to tell you whether you've built a good bridge or not.

Software is also harder because as long as the program "works", it doesn't matter what the source code may look like. Sure, some code may be harder to maintain than other, but how do you tell? There's no metrics for code design; we only have smells.

Software is cheap

Let's not move on to why software is cheap. If you write a program that doesn't work according to specs, it's not a big deal: you just go back to Notepad, change a little bit of text and (perhaps) recompile. Even software distribution is cheap: when was the last time you got software in a CD? We could spend a lot of time and effort finding bugs and/or proving our program works correctly, but why would we do that? After some point, it really doesn't make sense to spend money finding and solving bugs that may never surface in normal use.

On the other hand, if you screw the pooch on a bridge you're toast: it's hard to fix, and it was really expensive to build (not to mention how much time it actually take to build). This is why civil engineers, plan, model, test and then finally, after a really long time, build. We just build because it's fast to build. How awesome is that?!

Conclusion

Now you know why you're much better than all those other engineers out there. So go rub it in their faces whenever you feel like it. :p

What do programmers do?

2009-01-23T09:42:00.002-07:00

Have you been ever asked what you do for work? I have, and it's always hard for me to answer: if I say I'm a computer programmer people immediately think I can fix their printer - or worse, they want me to build them a dating website for five hundred bucks.

The truth is that what we "computer scientists" do is more along the lines of what sorcerers do. Hal Abelson describes it really well:

Computational processes are abstract beings that inhabit computers. As they evolve, processes manipulate other abstract things called data. The evolution of a process is directed by a pattern of rules called a program. People create programs to direct processes. In effect, we conjure the spirits of the computer with our spells.

A computational process is indeed much like a sorcerer's idea of a spirit. It cannot be seen or touched. It is not composed of matter at all. However, it is very real. It can perform intellectual work. It can answer questions. It can affect the world by disbursing money at a bank or by controlling a robot arm in a factory. The programs we use to conjure processes are like a sorcerer's spells. They are carefully composed from symbolic expressions in arcane and esoteric programming languages that prescribe the tasks we want our processes to perform.

A computational process, in a correctly working computer, executes programs precisely and accurately. Thus, like the sorcerer's apprentice, novice programmers must learn to understand and to anticipate the consequences of their conjuring. Even small errors (usually called bugs or glitches) in programs can have complex and unanticipated consequences.

Fortunately, learning to program is considerably less dangerous than learning sorcery, because the spirits we deal with are conveniently contained in a secure way. Real-world programming, however, requires care, expertise, and wisdom. A small bug in a computer-aided design program, for example, can lead to the catastrophic collapse of an airplane or a dam or the self-destruction of an industrial robot.

Master software engineers have the ability to organize programs so that they can be reasonably sure that the resulting processes will perform the tasks intended. They can visualize the behavior of their systems in advance. They know how to structure programs so that unanticipated problems do not lead to catastrophic consequences, and when problems do arise, they can debug their programs. Well-designed computational systems, like well-designed automobiles or nuclear reactors, are designed in a modular manner, so that the parts can be constructed, replaced, and debugged separately.

Although I'm attributing the quote to Abelson, the text above comes from the book Structure and Interpretation of Computer Programs, written by Albeson, Sussman, and Sussman. The reason I'm attributing the quote to Albeson is because I first came across this by watching one of his MIT lectures on the same topic.

If you're interested in watching MIT's introductory "Structure and Interpretation of Computer Programs" class, you can find the videos here.

And now, go conjure some spirits!

The Perils of Sushi Bars

2009-01-15T19:26:00.005-07:00

This is the story of how my credit card was almost stolen by a Samurai in a sushi bar in Salt Lake City. If you're interested, read on.

About 2 months ago I went to a sushi restaurant here in Salt Lake City. After paying our bills and while we were walking out, Kevin (one of my co-workers) noticed that his receipt had his full credit card number printed on it. Rightly concerned, he immediately talked with the restaurant's manager and asked to see the restaurant's copy of his receipt - sure enough, the restaurant was storing hundreds of credit card numbers in their little dinky cash register.

The whole incident got me thinking about credit card transactions. Every time I buy something with a credit card it's like I'm handing my wallet over to the clerk and saying: "Here's my wallet. Take whatever you need and just hand it back when you're done." Who knows what the POS machine is doing with my credit card number! And that's the least of my concerns: What about the lame payment gateway that stores millions of credit card numbers on servers that will likely be hacked in the next 5 years?

Interestingly enough, however, the whole credit card thing seems to work remarkably well. I mean, there's little risk in the sushi bar overcharging; after all, I know where they're at, and I also have the the credit card company on my side: one call to the them and the restaurant we'll have to pay the bill. As far as the payment gateway is concerned, even thought there's the PCI standard, when they get hacked we all pay the cost (either thorough higher interest rates, more taxes, etc)... that's why I'm really concerned about them - I worked at a place where we were PCI compliant, but there were still gaping holes in our security model.

This particular sushi story, however, has a happy ending. We recently went back to the restaurant and were pleasantly surprised to see that our full numbers were no longer being printed on receipts. It's good to see that non-IT folks are finally starting to get the importance of securing information.

Steve Yegge Is My Hero

2009-01-08T18:38:00.000-07:00

I regularly read Joel Spolky's blog, Jeff Atwood's blog, and Steve Yegge's blog. I really like them all, but Steve's is probably the most thought provoking blog out of the three.

Recently, for example, he had a post where he discussed (at length), a very intersting topic:

Imagine Mario, and his brother Luigi, having a conversation in which Mario asks Luigi what's on the other side of the boundary to their Mario Kart
universe. Interesting, huh?

It's interesting because to Mario & Luigi, stuff on the "other" side of their universe boundary is "undefined"; it's undfined in the sense that anything could be there. In fact, to some extent, it doesn't even make sense for them to ask what's on the other side; they simply cannot define what's on the other side as long as they reside inside the Mario Kart universe.

On the other hand, programmers living outside the Mario Kart universe can easily define what's outside the Mario Kart universe: all they have to do is inspect the memory contents of the system running Mario Kart, and take a peek at what's in there, on the "other" side.

I enjoyed Steve's post on embedded systemsbecause it helped me make sense of the embedded system I live in. I now understand that anything could be out there; it's truly undefined.

Lucene.net UnAuthorizedAccessException

2009-01-08T16:32:00.006-07:00

Unless you specifically tell Lucene.net where to put its LOCK files (for the index readers and writers), it will put them in your OS's default temp directory (normally C:\Windows\TEMP\). This choice of location may be problematic.

As you can imagine, there are plenty of scenarios under which Lucene.net may be running under an identity that does not have enough to access to your temp directory. Under such scenarios, you'll get an UnAuthorizedAccessException thrown at you; and your code may not even be around to handle it.

To solution for this problem is easy; add the following to the section of your application's config file:

After doing this, make sure "Everyone" has full control of the folder you've chosen for Lucene.net to use as its LOCK file folder.

You should also consider properly disposing your indices when your process finishes so that all locks are release on a timely basis. Unfortunately, I can't tell exactly how to do that - you'll just have to figure it out.

Finally, if you're reading this for some other reason than having the exception thrown at you, you should really consider using Lucene. It's great!

2008 Reflections

2009-01-06T17:20:00.003-07:00

Man, it feels good to be back to blogging! It was an awesome Christmas break, but I must admit I kept coming up with ideas I wanted to blog about.

Anyhow, '08 was a great year for me: I moved to a great company and started blogging regularly. I also learned a lot of LINQ, some Ruby, and had some great personal moments.

As a recap of 2008, I wanted to share my favorite posts so far:
The Secret Behind SQL
Easy Background Tasks in ASP.NET
Why Partial Classes Are Wrong!

I also wanted to thank all of you that regularly read my blog. There's a few of you out there that are loyal readers, checking often whether I have new content or not; knowing that you're reading makes me much more critical of what I write. Thanks for your time!

Finally, here's and interesting graph of traffic on this blog:

You see that spike right around Nov. 17th, 2008? That, my friends, comes from onesmall comment buried among hundred of comment's in Jeff Atwood's blog. Isn't that crazy? That blog is the black hole of developer effort; I swear every developer on planet earth goes there to waste their time... I can't find any other explanation for getting so much traffic from just one comment. Thanks Jeff! And thanks to everyone that followed the link and took time to read what I had to say.

May 2009 be a great year for you.

Why You Should Be Controversial

2008-12-19T15:17:00.003-07:00

I have to tell you a story before any of this makes sense, so bear with me for a minute.

Michael Murray, a good buddy of mine, IMed me today telling me he had been invited to write a blog post for the the main page of tech.lds.org. I was really happy for him; he deserves it as a tribute to his tireless contributions to the LDS tech community.

Unfortunately, for reasons that I can understand, Mike was having a hard time coming up with what to say. I mean, there's tons you could say, but you have to be somewhat careful... after all some may mistake criticism of the software the LDS Church employees write with criticism of the LDS Church period. And then there's also the fact that Mike really cares about the tech.lds community; he's an active moderator in it.

Unfortunately, the tech.lds community suffers from the same problem any extremely homegenous community does: everyone says the same freaking thing all of the freaking time! Everyone talks about how IT is a means to an end; everyone talks about how we, as IT folk, should seek for way to further the Kingdom and be more wise servants. Please don't get me wrong: I know that what the tech.lds community preaches is true, but haven't we heard enough of that already?

Why don't we talk about some of the real issues with IT at the Church?

Here's a few ideas off the top of my head:

How come LDS software is not open sourced? I still remember when I helped build my chapel back in Ecuador; truly a memorable experience - it was great to be able to contribute beyond tithing money.
How come phone numbers are tied to households in MLS? What's up with that?!? And even more important, how come that wasn't fixed 10 releases ago?
How come the IT department used to pay terrible salaries to IT employees? I recognize this is changing, but what kind of people was the IT department expecting to hire?
Why did it take 10 years for meetinghouses to get broadband internet access? Did IT managers really just learn of white lists?
Why does it take days to transfer membership records into a ward? Isn't it just a simple look up? And if it's not just a simple look up, how come it's not?
Why is the Church developing on the Java stack? When was the last time you saw MLS running on a Linux box?

I've been deliberately controversial on this post because I wanted to make point: if you're going to say something, you might as well say something that's worth thinking about. I suspect there are good answers to all of the questions I've listed above; furthermore, I realize in the grand scheme of things I know nothing compared to what the fine folks at the LDS Church do. But at least, you read this far and know you're thinking about software at the Church too.

Disclaimer: I cannot say it more clearly than this: I'm criticizing IT at the LDS Church, I am NOT criticizing the Church of Jesus Christ of Latter Day Saints. To some extent, I'm not trying to criticize the folks the work in IT at the Church; I realize that were I in charge of IT things would probably be even worse than they're now.

The Secret Behind LINQ To SQL

2008-12-11T20:41:00.009-07:00

I finally get it! It's all about expression trees!

It all finally clicked for me when I saw the declaration of IQueryable:


public interface IQueryable : IEnumerable
{
  Type ElementType { get; }
  Expression Expression { get; }
  IQueryProvider Provider { get; }
}

See that? It's right there! How could I have been so blind for so long? But I'm getting ahead of myself. Let me tell you about expression trees.

What are expression trees? Well, they're a trees with expressions as nodes. What do they do? They provide a mechanism to convert code into data (expressions). Why is that useful? Because you may want to examine and/or modify you code before execution. In particular, it would be very useful, if we wanted to take say C# code and convert it to, oh... i dunno.., SQL statements.

Say then you created the following Expression:


Expression<Func<int, int, int>> expression = 
     (a,b) => a + b;

(There's a little magic going on in the above sample: the compiler knows how to take the lambda "(a, b) => a + b" and make a delegate out of it).

The above code would literally translate to a "+" plus node with two children nodes: "a" and "b". Obviously, I've simplified the tree for clarity. If you really want to see the tree structure, fire up VS2008 and take a look at the expression tree with the ExpressionTreeVisualizer plugin.

Anyhow, now that we have our C# code in a tree, we can easily parse that tree and convert the data to a string that SQL can understand. We can then take that string, send it across the wire to our SQL server and voila: we have LINQ to SQL. Cool, huh?

Now, here's something else to think about: IEnumerable offers most of the same methods IQueryable offers, yet the declaration is completely different:


public interface IEnumerable<T> : IEnumerable
{
   IEnumerator<T> GetEnumerator();
}

Notice how IEnumerable does not have an Expression tree? That's a fundamental difference between the two interfaces. This means that IEnumerable won't do anything with your code but execute it. Therefore, if you order, filter, or project an IEnumerable collection, the action will execute in the process where the collection lives; it will not be sent to a beefy SQL box that can handle ordering large sets easily (Yes, I've made that mistake. In fact, that's what inspired this post).

Turns out my CS professors were right: trees really are useful data structures.

Why Programmers Should Have Private Offices.

2008-12-09T09:48:00.005-07:00

Non-programmers don't get it: interruptions are a huge deal; it takes a lot of mental effort to load the program you're working on onto your brain.

Once you're in "the zone", you're effective and you write good code. Naturally then, having some dude from marketing stop by to ask a "quick question" is a total disaster; you have to literally unload a bunch of important information from your brain, just to listen to the marketing clown ask you for the 23rd time when the product will ship. Not cool. Not cool at all.

What non-programmers don't get is how hard it is to resume what you were working on. They don't understand that, to some extent, you have to load all the classes, variables, functions, etc. in your code right onto your memory; you have to enter the freaking Matrix, ant that's no easy task.

I'm guessing non-programmers think going back to writing code is as easy as going back to writing an email: you just read the last sentence you wrote and pick up from there. Well, it's not that simple.

Joel Spolsky has been advocating for private offices for developers well for almost a decade now. Yet there's few companies out there that give their developers such luxury. So, if you ever come across such perk, there's a good chance that's a company you want to work for.

From Recursion To Dynamic Programming

2008-12-05T21:40:00.001-07:00

Dynamic programming is a really useful technique. When used appropriately, it saves processor time and memory space.

Shoot! After writing all the code for this entry, I just noticed that the Wikipedia entry for dynamic programming uses the same example I thought about using in this post. Oh well, let me just show you some code and explain just a little.


private int recursiveFib(int n)
{
  if (n == 0)
   return 0;
  if (n == 1)
   return 1;
  return recursiveFib(n - 1) + recursiveFib(n - 2);
}

The above code calculates the Fibonacci number for any integer n. The code is easy to read and effective. However, there's one problem: we repeat calls to recursiveFib even though we may have already computed a value. For example for n=4, we'll call recursiveFib(2) twice; this may not seem like a big problem but for larger Ns the number of unique calls to recursiveFib is very small. What a waste of processor cycles!

Now that we've realized we're wasting resources because the number of unique calls to our recursive methods is small, we're ready to move on to dynamic programming. Here's the dynamic code:


private int dynamicFib(int n)
{
  int[] dynamicArray = new int[n];
  dynamicArray[0] = 0;
  dynamicArray[1] = 1;
  for (int i = 2; i < n; i++)
  {
   dynamicArray[i] = dynamicArray[i - 1] + dynamicArray[i - 2];
  }
  return dynamicArray[n];
}

Even though this code may not be as succinct as the recursive code, it saves resources. By taking a bottom up approach (rather than the top to bottom approach of the recursive method) and saving our previous calculations, we never have to recompute a value again! Although, with oil being as cheap as it is nowadays, you may not really care.

In summary:
If you want to solve a problem with dynamic programming, I recommend you first solve it recursively. This may seem like a waste of time, but you need the recursive solution to spot where the improvement to the algorithm needs to occur. Once you've figured that out, all you have to do is compute and store you partial solutions as you go.

Think About It!

2008-11-25T20:28:00.002-07:00

Disclaimer: My apologies to the fine people who, unlike me, actually write source control systems.

If I asked you to write a source control system, you might say: "Pff. That's easy dude! Just store a copy of the document every time make a change. When you need a particular revision, you just access that copy of the document". And although your system would work, I'd say: Think about it!

Storing a full copy of the document for every change is wasteful; revision changes are small: a line of text here and a line of text there. Why would you store a full copy of the document just for a small change?

At this point you may be tempted to say: "Well, then just store the original document, and store the changes as small delta files. We can then apply the deltas to get you any revision you want.". This, of course, is a much better solution as far as storage is concerned; however, I'd still say: Think about it!

Under normal circumstances, users access the most recent revision of a document far more often than any other revision. Furthermore, storing the original document and applying all of its changes is computationally expensive. It would then seem that having to apply all these deltas for our most common operation is a bad idea. Seeing this, how can we further improve our syste?

Well, how about we store the most recent version of the document instead of the original? This would mean we would have to store deltas to take us all the way back to the original version, but that's OK - it's not much different than what we were thinking about doing before. However, with this change, we can now perform our most common operation (return the current version of a document) in constant time. Also, the expensive operation (returning an older version of the document) now occurs on rare occasions. Better, huh?

And now that I'm out of ideas for our source control system, I'm going to go back and "Think about it!", a little bit more, because I'm sure there's still lots of room for improvement.

Spell Checking The Right Way

2008-11-20T21:37:00.004-07:00

In my fun-as-a-rock database class I recently got an assignment to correct misspellings in a file full with city names.

Now, there's two ways to do spell checking: the Microsoft way, and the Google way. Care to guess which is the wrong way to do it? Yup, you got it: the Microsoft way sucks! Ok, maybe it didn't suck back in the 17th century when Spanish Monks were doing all the spell checking known to mankind (which I think consisted of 3 or 4 individuals that actually knew how to read, or cared about spelling for that matter).

So, if the Microsoft spell checker and the Google spell checker could talk, what would they say?

Microsoft would say: Listen buster! My dictionary contains all the correct words in the universe; either you comply or you don't. Got it?

Google would say: What do I know about spelling? I'm just trying to figure out a way to make more money from all this content I just indexed. Oh, and by the way, that word you just typed, it look awful close to this other word I see a lot in my index. Is that what you meant?

The problem with the Microsoft approach should be obvious, but it's important to point out that the Google approach is not without faults either.

The biggest problem with the Google approach is that to some extent it's a form of crowdsourcing. If your crowd can't spell, then you're toast.

Last, but not least, I'd just like to show you some pseudo code on how I implemented my spell checker:

Read all the city names in the file while keeping track of every variation we've seen and how many times we'v seen it (in a hash, dictionary, etc). Take the most popular spelling for each city, and call that the correct spelling.
To correct word X, calculate its edit distance to all the correct spellings. Chances are word X is really the "correct spelling" it mostly resembles.
Figure out what do if you've never seen X before.

And that concludes today's post. Now if I could just get Google to write grammatically correct sentences for me, I'd never have to worry about proof reading my posts ever again.

Disclaimer: I would just like it to be known that I'm in no way a MS hater; in fact, I'm somewhat of a MS fan. I'd also like it to be known that I'm not a Google fan boy; in fact, I'm a little afraid of them - they read my email, and I'm sure they're the new federal agency that's in charge of spying on citizens.

How To Be A Better Programmer

2008-11-18T18:08:00.005-07:00

Glen Wagley, after reading my post on pair programming said to me: "I understand what you mean about cutting corners. But I don't do it anymore; it's not worth it".

That was a slap in the face. Here I am blogging about ways to be a better programmer, yet I still cut corners myself.

All of this lead me to think that the number one thing you can do to become a better programmer is to have courage and integrity: if you see code that needs to be refactored, refactor it; if you know you need to throw away some of your code, don't be hesitant and throw it away; write your unit tests first; write your documentation... you get the idea: do those things you know you should do even though you don't you don't always want to.

I guess what' I'm trying to say, if you're Mormon, CTR. If you're not Mormon, please contact your local LDS missionaries; they'll be glad to teach you what CTR means. Seriously, however, it's sad that we have sites like The Daily WTF were we laugh about the crap we, so called "pros", write. I know that the profession is new, but that should be no excuse; I don't see a site where surgeons laugh about all the times they've left scissors inside their patients (OK, I really haven't searched and there's probably one out there). I realize we all make honest mistakes, but we should draw the line somewhere.

Now that I'm done ranting, and while I'm on the topic, here's a list of other things you could do to become a better programmer (in no particular order): learn other languages & platforms other than the one you currently use; learn to write good prose (in more than one language?); learn to touch type (as Steve Yegge suggests here); read tons of technical books and even more non-technical books; learn how to market your ideas; be humble and learn from others (regardless of their title/position); use a text editor effectively...

I have a ton of these, but I'd rather hear from you now. What do you do to become a better programmer?

Update: Since writing this article, a good buddy of mine wrote a rather inspirational story on standing up for what's right. If you so desire, you can find said story here.

Brevity In Code

2008-11-15T16:42:00.001-07:00

Yes, I'm going to try to convince you to write less code. But, this should be an easy feat; I have Shakespeare also advocating my cause:

Brevity is the soul of wit.

Shakespeare's truism is readily apparent in good code: code that expresses complex concepts in succinct statements is beautiful and worthy of admiration. For several reasons, all other things being equal, smaller code is better code.

Fewer lines of code mean less bugs and lower maintenance cost. There are, however, other less apparent reasons for which you should try to write as little as possible:

You won't fall pray to the temptation of writing code that's not immediately necessary. In other words, you'll be YAGNI compliant. :)
It is better to be thought a fool than to write code and remove all doubt. Joking aside, however, the more you write, the more likely you're to make a mistake.
You won't get locked into poor decisions. I was watching Abrams & Cwalina speak at PDC today. One of their comments really struck me: they said, and I'm paraphrasing, that refactoring and correcting design mistakes in frameworks is harder when you have more code than what's absolutely necessary. Specifically, they regretted adding a public constructor to the System.Exception base class. That's it! Just one public constructor too many! And although they wish they could change it, they simply can't. If you make a poor decision, you'll have to maintain it.
You're less likely to repeat yourself. Or, to phrase this positively, you'll be DRY compliant.
Finally, by writing less code, you'll avoid the temptation to over engineer your solutions.

Learning to be concise in code is hard; it takes effort and patience. You'll have to refactor ruthlessly and mercilessly, but it will be worth the effort. Even though you won't have much code to show off, you'll be proud of it.

Character Encoding Utility

2008-11-13T17:26:00.002-07:00

Have you ever heard of tracer ammunition? Well, this post is kind of a tracer post. In a minute you'll see why.

I recently wrote a small utility to change the character encoding of very large files. I'm thinking about writing a GUI for my utility and making it freely available. Yup, that's right for free.

Except, before I go through the trouble of writing the GUI, I'd thought I'd find out if there's any interest in such tool or not. If you're here, reading this, that's enough to tell me you're interested. Now, if you really need it right now, email me and I'll be happy to send you the command line tool.

And now you know why this is a tracer post. See? I did learn something from The Pragmatic Programmer! Or from The 4 Hour Workweek. Take your pick; they're both excellent books.

Benefits (And Costs) of Pair Programming

2008-11-06T19:23:00.011-07:00

I really enjoy pair programming, and so I thought I'd write about some of the benefits I've seen from pair programming.

i.
Pair programming increases job satisfaction. Believe it or not, I crave the interaction with other geeks; I need the mental stimulus that comes from talking with my peers.

ii.
Pair programing increases code quality and decreases bug counts. I take pride in my craft; I like to write good code. Unfortunately, under pressure I cut corners all over the place (telling myself I'll refactor later). However, when I have someone watching over my shoulder, it's a lot harder for me to write hacky code.

But having someone watch what I'm doing isn't just about the guilt trip. I also appreciate having someone immediately available to discuss ideas and to steer me away from potential problems. This alone literally saves me hours in wasted effort.

iii.
Pair programming makes better programmers. I can't even begin to tell you how much I've learned from sitting next to someone while they code. Now, I must admit, probably 90% of what I've learned has little or nothing to do with programming. But it doesn't matter! Believe it or not, programming is a social activity; to put it simply, good code cannot be developed in isolation.

iv.
Pair programming makes programmers "faster". I honestly feel I'm 3 times more effective when I'm pair programming than when I'm sitting by myself in my cube. This is probably because when I'm stuck, or I have a question, I have someone immediately available to help. Also, there's a bit of added pressure to be faster since every hour at the keyboard is really 2 man hours at the keyboard.

v.
Now, what if I told you that all of these benefits come with a fairly low cost? Well, the good news is that they do: according to this study, the cost is only about 15% increase in development time. Not bad, huh?

So, if you're still not pair programming, go talk to your boss and start practicing now. You won't regret it!

Easy Background Tasks in ASP.NET

2008-11-04T18:09:00.006-07:00

Disclaimer: Admittedly, I got this idea from Jeff Atwood, but I think I've improved on it quite a bit.

At work I have an ASP.NET application that needs to check whether the Department of Treasury has published a new OFAC list (a list of people with whom we can't do business). If there's a new list, we need to parse it, store it, and make sure that none of our customers have popped up on the new list.

In phase 2, we plan to move this functionality into a windows service, but for now this is how I made this all happen in the background:

First, we start with the interface for our worker objects:


public interface IAsyncWorker
{
    //the name of the worker object
    string Name { get; }
    //return the next time the object should run
    DateTime AbsoluteExpirationTime { get; }
    //does the actual work the worker needs to do
    void DoWork();
}

Now, all we have to do is simply:

Add an implementation of our IAsyncWorker to the HttpRuntime.Cache.
When the cache item expires, call the worker's DoWork() method.
Add your worker item to the cache again so that it runs again.

If you follow the above steps you should end up with something like:


private static CacheItemRemovedCallback OnCacheRemoved = null;

protected void Application_Start(object sender, EventArgs e)
{
    AddAsyncTask(new BlacklistWorker()); //BlacklistWorker implements IAsyncWorker, of course
}

private void AddAsyncTask(IAsyncWorker worker)
{
    OnCacheRemove = new CacheItemRemovedCallback(CacheItemRemoved);
    HttpRuntime.Cache.Insert(worker.Name, worker, null,
        worker.AbsoluteExpirationTime, Cache.NoSlidingExpiration,
        CacheItemPriority.NotRemovable, OnCacheRemoved);
}

public void CacheItemRemoved(string workerName, object worker, CacheItemRemovedReason r)
{
    IAsyncWorker asyncWorker = worker as IAsyncWorker;
    if(null != asyncWorker)
    {
         asyncWorker.DoWork();
         AddAsyncTask(asyncWorker);
    }
}

I like this version better than Jeff's because it removes all conditional statements the CacheItemRemoved() method would have had if we had not created and IAsyncWorker interface.

This has been working great in our initial tests, but we still plan to move this to an external windows service at some point.

We're not worried about running out of threads (the thread does come out of the AppPool), since our task only needs to run every 24 hours. However, you might run into issues if you need your code to execute under a different identity than the threads in the AppPool.

This is a great technique: it gives you the ability to do async tasks with very little overhead.

Linq To Sql Debugger Visualizer

2008-11-01T20:54:00.000-07:00

I usually try not to just post links to someone else's content, but the LINQ To SQL Debug Visualizer deserves an exception.

The LINQ debugger is great; it saves you the trouble of having to use something like LINQPad - another great product I wish I had written, and lets you see what your LINQ query will look like in SQL and what it will return.

If you're like me, you'll be shocked you hadn't heard about this VS plugin before.

Note: A big thanks to Alex Greenfield for showing me this tool.

Book Review: An Introduction To Bioinformatics Algorithms

2008-10-29T21:41:00.000-07:00

Disclaimer: I'm no where nearly important enough for anyone to pay me to write this review. I have no conflicting interest in writing this review... unless you clicked on my amazon.com link and bought the book I guees.

So we're using An Introduction to Bioinfomratics (by Jones & Pevzner) in (surprise, surprise!) my bioinformatics class. In 7 years of higher education, this is the first time I can say "this textbook rocks!". Ok, of course it doesn't "rock"; it's a freaking textbook - not Larry Mullen on the "40" solo. Anyhow, it's a good book, and I'd get it if I was you; and I'd also make sure you click on my amazon.com widget there on the right before you purchase it.

Anyhow, let me tell you two things about this book:

Chapter 3 is a brilliant molecular biology primer. I've taken at least 3 semesters of bio and chem classes, and this book explained all of what I learned in those classes brilliantly and concisely (granted, I never got much out of college). In fact, after reading this book I felt like I finally understood molecular biology and could explain it to my grandma (which is harder than you think seeing how she's been dead for like 5 years now).
One thing that annoys me about algorithm books is that they always have lame sample of how the algorithm could be used. Well, not this book my friend! This book begins every chapter with a real molecular biology problem and then applies classic computer science college algorithms to the problems. This is what really makes me learn and understand the algorithms.

Honestly, if you're considering buying an algorithms books, this is one worth looking at. It's not comprehensive in the number of algorithms it covers, but the ones it does cover, it covers well.

Why partial classes are WRONG! (And why you shouldn't use them.)

2008-10-27T16:14:00.006-07:00

I'm absolutely positive the fine folks at Microsoft, particularly the brilliant Anders Hejlsberg (C#'s architect), know of a good reason for partial classes. But I've thought about this for a while, and I'm not convinced there's a good use case for partial classes.

To be intellectually honest I must admit I can actually think of a few scenarios where partial classes make sense; let me tell just lay those out before we continue:

You should use partial classes when some tool (such as Visual Studio), not a developer, writes something like an ASPX page.
You should sometimes use partial classes when you write code that will be auto generated (with CodeSmith or such tools). The sometimes I'm thinking of is DAL code.
You should use partial classes when... ummm, when you, ummm..

I guess that was it; there's only 1.5 good use cases I can think of. I know there must be other reasons whey C# allows for partial classes, so if you know of one, don't hate me for being stupid and leave a comment instead.

The most prevalent argument for partial classes I've heard of is the one that says partial classes are great for organizing code. To this I respond: bull. Partial classes are not great for organizing code; if your class needs to be spread across multiple files just to manage its complexity, there's something fundamentally wrong with your class. Furthermore, when was the last time that having to look for something in multiples files made it easier to find? Finally, and this is my biggest complaint about partial classes, the compiler will never know if all the files present make the entire definition for your class - I'm sure you can see the problems this could create for you.

Mike Murray (a friend and ex-coworker at TGN) and I had a discussion about this very topic, based on his post about partial methods. While we were talking about partial classes, he mentioned that at TGN he had seen fulfillment code that used partial classes. The developer that wrote the code thought it would be a good idea to separate the SKU specific code into partial classes. At first this may look like a good idea, but again I respond: bull. There's a much better way to implement SKU specific behavior for fulfillment code; it involves interfaces and polymorphism, two fundamental principles of OOD.

And the list goes on and on. But for just about every argument in favor of partial classes, there's an OO principle that will do the same job in a much better way.

So, in summary, let me tell you why you shouldn't use partial classes: partial classes will only help you write crappy code. I don't know about you, but I'm already pretty good at writing crappy code; I need tools and frameworks that do the opposite.

Just as a footnote: I read an interview where Anders was asked about C#'s support for tooling. He mentioned that yes, he always keeps in mind tool support when designing C#. I'm guessing this is the true reason for partial classes, and why you should steer off them.

The Cost of Throwing & Catching Exceptions

2008-10-23T19:55:00.006-07:00

Seeing how I recenlty wrote about how throwing exceptions is not a bad idea for, uhmm - you know, communication errors (believe it or not this is a topic of discussion), I thought I'd follow up with short follow up on the cost of throwing exceptions.

Those that argue against throwing exceptions for means of communicating errors, normally cite the "high" cost of catching exception: unwinding the stack looking for a handler, executing finally blocks, and finally returning control to the right place is allegedly a costly operation.

However, because of the way .NET exception handlers are stored in bytecode and because the code is optimized for the case in which an exception is not thrown (which makes sense considering that we shouldn't see TOO many exceptions):

The overall cost of a try...catch block that never handles an exception is a few bytes of memory - or at worst a few words - for the entry in the protected regions table. The only possible runtime penalty is the extra time to load those extra few bytes into memory. Since they are stored way away from the JITted bytecode stream, it's highly unlikely you're going to incur any additional cache-misses at runtime as a result of the handler too. Thus, the cost is essentially nothing.

However, the cost of not handling an exception is quite large:

The cost of not handling an exception that you should have may well be that your program crashes. This results in unhappy customers, a hit to your reputation and development time to go and do a bug-fix, which will almost certainly be much greater than if you had put it in there in the first place. Obviously, protecting code that can not throw an exception under any circumstances is a waste of your development time. But otherwise, it's best to be safe rather than sorry, safe in the knowledge that even if an exception never does occur in that bit of code, it's not really costing anything anyway.

(Both quotes from The Official Programmer's Heaven Blog on SEH performance)

One Important Consideration

Although the performance cost of structured exception handling is almost always minimal, there is one important consideration that you need to make when writing code inside try blocks: the compiler cannot optimize code inside try blocks.

Take Peter Ritchie's classical example; the following code


  int count = 1;
  SomeMethod();
  count++;
  SomeOtherMethod();
  count++;
  Console.WriteLine(count);

will get optimized to something that will effctively look like


  SomeMethod();
  SomeOtherMethod();
  Console.WriteLine(3);

The same code as in Figure 1, inside a try block, however, will not get optimized. The compiler will actually write code that will increment the "count" variable 2 times.

So, when writing exception handling code, try to be brief inside your try blocks. Other than that, you don't have too much to worry about.

StackOverflow's Dirty Little Secret

2008-10-21T16:35:00.006-07:00

Seeing how you probably got here from my link on SO, I won't bore you with the details about what SO is and how awesome it is. For everyone else, however, let me just mention that StackOverflow is a community where developers go to ask and answer questions - that's it, but it's AWESUM!

Here's a list of ideas on how to maximize your StackOverflow experience. In it, you'll find SO's dirty little secret; take a look:

Do:

Put your foot in your mouth (often). Every time I answer incorrectly at SO the community let's me know it promptly and pointedly. It's kind of embarrassing to be wrong online, so you really learn from your mistakes. Not only does the shock factor help you remember the correct answer but, as an added benefit, you will no longer carry with you the baggage of incorrect knowledge in your head.

I strongly recommend goofing up (and correcting your self); for a good sample of how I've goofed up, see this question. I can guarantee you, I'll never get that one wrong again.

Ask subjective questions. Even though by definitions subjective quesitons do not have one correct answer, there's still value in asking these kinds of questions. I'd say about 50% of the SO population is smarter than I am (yes, I realize why about 50% of the population is smarter than me), so there's a lot of value in seeing what other developers think.

If you care to see one of the subjective questions I've asked, take a lookie here.

Don't:

Think that people know what they're talking about just because they have a high reputation score. And this is SO's dirty little secret: you can get a high reputation (which allegedly represents expertise) by just gaming the system. I'm not saying that the system is broken - there's a lot of people out there that reserve the high rep; there are also many that don't. The other side of this coin is also true: there's a lot of sharp people with low reputation scores (probably because they have better things to do than try to get a high SO rep score).

So, again, don't judge people just because of their rep score.

Trust answers just because they have a lot of up-votes. Even though there's a lot of smart people at SO, there's also just a lot of people, and sometimes the herd mentality takes over and answers get up-voted for no other reason than the fact that others have up-voted it.

If you're a developer and you haven't joined StackOverflow yet, I strongly recommed you to head over there and start asking and answering questions.

How To Write Self-Documenting Code

2008-10-16T20:57:00.010-07:00

If I got promoted every time I spotted code like the one that follows, I'd probably be the President of the United States by now. Take a look at Joe's code here:


public static void main(String[] args) {
        String db = args[0];
        String uid = args[1];
        String pwd = args[2];
        
        String url = "jdbc:postgresql://localhost/" + db;
        Connection conn;
        try
        {
            conn = DriverManager.getConnection(url, uid, pwd);
            String query =  "SELECT A.city, A.zip, st_distance(A.longlat_point_meter, " + "B.longlat_point_meter) FROM utzipcode A, utzipcode B WHERE B.city='MENDON' " + "AND A.city<>'MENDON' AND st_distance(A.longlat_point_meter, B.longlat_point_meter) " + "< 20000 ORDER BY 3;";
            
            Statement stmt = conn.createStatement();
            ResultSet rs = stmt.executeQuery(query);

            ResultSetMetaData resultMetaData = rs.getMetaData();
            System.out.println(resultMetaData.getColumnName(1) + ", " + resultMetaData.getColumnName(2) + ", " +
                    resultMetaData.getColumnName(3));
            while(rs.next()){
                System.out.println(rs.getString(1) + ", " + rs.getString(2) + ", " + rs.getString(3));
            }
            rs.close();
        }
        catch(SQLException sqle){
            System.out.println(sqle.getMessage());
        }   
    }

Can you tell what's going on in the code above? You probably can, with some effort - but why should code so simple be hard to understand? How come the code doesn't tell me what it is doing? Plus, when I'm maintaining this beauty 10 years from now, I'll be hunting Joe down and pulling his fingernails every time I have to debug it - seriously, this code sucks.

Even though just about every programmer I've worked with knows better than to write code that's hard to read, under pressure we all get sloppy. In fact, I was just looking at some code I just wrote a couple of weeks ago as inspiration for this post.

So, now that I've gotten that off my chest, let's see if we can clean this up some. The n00b fix would be to pepper in some comments like this:


public static void main(String[] args) {
        String db = args[0];
        String uid = args[1];
        String pwd = args[2];
        
        String url = "jdbc:postgresql://localhost/" + db;
        Connection conn;
        try
        {
            //get the connection to the db
            conn = DriverManager.getConnection(url, uid, pwd);
            String query =  "SELECT A.city, A.zip, st_distance(A.longlat_point_meter, " + "B.longlat_point_meter) FROM utzipcode A, utzipcode B WHERE B.city='MENDON' " + "AND A.city<>'MENDON' AND st_distance(A.longlat_point_meter, B.longlat_point_meter) " + "< 20000 ORDER BY 3;";
            
            Statement stmt = conn.createStatement();
            //execute the query defined above
            ResultSet rs = stmt.executeQuery(query);

            ResultSetMetaData resultMetaData = rs.getMetaData();
            
            //print column names
            System.out.println(resultMetaData.getColumnName(1) + ", " + resultMetaData.getColumnName(2) + ", " +
                    resultMetaData.getColumnName(3));
            
            //print every tuple
            while(rs.next()){
                System.out.println(rs.getString(1) + ", " + rs.getString(2) + ", " + rs.getString(3));
            }
            rs.close();
        }
        catch(SQLException sqle){
            System.out.println(sqle.getMessage());
        }   
    }

That still looks terrible, however. Not only that, the comments aren't really that useful; they don't tell me anything I couldn't have figured out by reading the code. In fact, I think these types of comments make the code harder to read since they interrupt what's really important - the code. But that's kind of a side note: the real problem here is that comment's don't give me any insight or explain the intent of the code.

Alright then... What do we do now? Well, how about something like this:


public static void main(String[] args) {
        String db = args[0];
        String uid = args[1];
        String pwd = args[2];
        
        String url = "jdbc:postgresql://localhost/" + db;
        Connection conn;
        try
        {
            conn = DriverManager.getConnection(url, uid, pwd);
            
            //we're hard coding the query here because the db is out of SPs - joe
            String query =  "SELECT A.city, A.zip, st_distance(A.longlat_point_meter, " + 
                            "B.longlat_point_meter) FROM utzipcode A, utzipcode B WHERE B.city='MENDON' " + 
                            "AND A.city<>'MENDON' AND st_distance(A.longlat_point_meter, B.longlat_point_meter) " + 
                            "< 20000 ORDER BY 3;";
            
            ResultSet rs = ExecuteQuery(query, conn);
            PrintQueryResults(rs);
            rs.close();
        }
        catch(SQLException sqle){
            System.out.println(sqle.getMessage());
        }
        
    }
    
    private static ResultSet ExecuteQuery(String query, Connection conn) {
        try{
            Statement stmt = conn.createStatement();
            return stmt.executeQuery(query);
        }
        catch(SQLException sqle){
            return null;
        }
    }
    
    private static void PrintQueryResults(ResultSet rs){
        try{
            ResultSetMetaData resultMetaData = rs.getMetaData();
            System.out.println(resultMetaData.getColumnName(1) + ", " + resultMetaData.getColumnName(2) + ", " +
                    resultMetaData.getColumnName(3));
            while(rs.next()){
                System.out.println(rs.getString(1) + ", " + rs.getString(2) + ", " + rs.getString(3));
            }
        }
        catch(SQLException sqle){ }
    }

Ahhh... much better. I can tell what the code is doing in one brief overview - and most importantly, the code itself is telling me what it's doing!

Also, the code is now easier to debug and maintain because each method is only a few lines long; this means I'll know exactly where the code may be having problems next time it breaks by just looking at the stack trace. Finally, look at that comment about the query - that really tells me something I could have never known unless Joe had told me.

So there you go, that's self documenting code in 3 easy steps. Please let Joe know how much you appreciate learning from his mistake by dropping a comment. It'll make him feel better.

when to throw exceptions

2008-10-11T11:53:00.006-07:00

have you ever heard the saying "exceptions are for exceptional circumstances"? well, i have, and until recently, i was a firm believer of such thoughts. i'd normally code for every "exceptional" condition and try to do everything i could to avoid having to throw an exception.

however, just a few days ago, i came across this statement from Krzysztof Cwalina (program manager for the CLR team at MS):

One of the biggest misconceptions about exceptions is that they are for “exceptional conditions.” The reality is that they are for communicating error conditions. From a framework design perspective, there is no such thing as an “exceptional condition”. Whether a condition is exceptional or not depends on the context of usage, --- but reusable libraries rarely know how they will be used. For example, OutOfMemoryException might be exceptional for a simple data entry application; it’s not so exceptional for applications doing their own memory management (e.g. SQL server). In other words, one man’s exceptional condition is another man’s chronic condition.

Cwalina then goes on to say that exceptions should be used for common errors such as (1) usage errors, (2) program errors, and finally (3) system errors. it seems to me this in direct opposition of the "exceptional exceptions" mantra.

to be honest, i never understood why exceptions should only be used in "exceptional" circumstances. it's hard to define "exceptional", and that's exactly the point Cwalina makes in his quote.

i realize, however, there's circumstances when it doesn't make sense to throw exceptions; for example, why throw a DivideByZeroException when you can easily check for the condition an appropriately terminate?

but for the most part, exceptions provide an objected oriented way to communicate errors to clients. i think i'm switching from the "exceptional exceptions" camp to the "let's use exceptions to communicate errors" camp. what about you?

Update: I have a new article on the cost of throwing exceptions. If you now think that throwing more exceptions is a good idea, you might want to check this article too.

everything is an object

2008-09-07T12:54:00.003-07:00

over at stackoverflow someone asked "how do i thinki in OO?"

saint_groceon answered:

Some conceptual advice:

"has a": A dog has a tail; therefore the Dog class should have a member "tail"

"is a": A poodle is a dog; therefore Poodle should either be an instance or derived class of Dog

Thinking this way really sped up my ability to design object oriented structures. Otherwise, starting out it's easy to get twisted up and start adding members to classes that should actually just be instances, or vice versa.

to which i responded:

saint_groceon's beginner advice although correct, can lead to trouble:

Objects are not just a collection of attributes and behaviors. If that was the case, dogs and cats would be indistinguishable from each other, as they both have eyes, mouths and legs, and they both eat, sleep and play. In fact, this kind of "object" is almost no different than a C struct.

Furthermore, the type of thinking described in saint_groceon's post also leads to other problems: If your "Duck" class has a "quack" method, what happens when you need to implement a rubber duck that does not quack? (you may recognize this example from Head First Design Patterns)

I agree with the fact that objects are usually nouns from the domain's language. However an object is more than that: an object is anything capable of providing a limited set of useful services. And then, with these objects "we decompose the complex world around, and assemble those objects in various ways so that they can perform useful tasks on our behalf" (David West).

West nails the definition of an objcet; the most important characteristic of an object is it's ability to be composed with other objects to create useful things, much like all of our body parts (properly assembled) make us useful creatures. So, in some respect, everything is an object!

Edit: I realize that I said little about How To Think OO, and a lot more about what OO is not. I apologize for this, but I'm no OO master, and I'm barely at the stage where I have a feeling what OO IS NOT... I'm not at the point where I can instruct on OO.

This post was mostly an exercise to help me gather my thoughts about what I've learned so far. I hope you'll find it useful nonetheless.

delegation in a load balanced enivornment

2008-08-29T18:33:00.003-07:00

last week i tried to get impersonation and delegation working for a couple of wcf services and a website. unfortunately, getting this to work took me a lot longer than it should have, so i thought i'd share some of the things i learned.

in iis 6.0 and earlier, you can only get delegation to work in a load balanced environment, if you do the following:

make your app run under a domain account

set the proper SPNs (service principal name) on the account:

HTTP/appserver domainName\accountName
HTTP/appserver.fullyqualified.name domainName\accountName

trust the account (and the machine) for delegation in AD
run aspnet_regiis with the -ga flag and the domain qualified user name
finally add your domain account to IIS_WPG

i'm sure most systme engineers know this already, but as a developer, it's the first time i've come across this.

it's also worth mentioning that under iis 7.0 you do not need to do any of this: you can run your apps under NetworkService, and still have them load balanced.

what's funny is that after we finally figured out how to get delegation to work for our load balanced applications, we decided that it was way too much setup and we would move to iis 7.0.
in a way, a lot of the time i put into this was wasted, but i learned a ton about security and delegation/impersonation while doing the research to get this to work. i hope it'll save you some time. if it does, or if you have any other questions, please let me know.

the stateless web: it will always bite you!

2008-07-17T19:29:00.000-07:00

i spent good part of the day building a little control to upload documents for our crm tool today. because i wanted to give the customer the ability to decide how many files she wants to upload at runtime, i put a little "add another file button" on the control, which would dynamically let the user add another file.

as soon as i had this dynamic functionality built, i threw it on the control where it was supposed to go (yes, my control was nested inside another control, which was then placed on the page where the upload functionality was needed). unbeknown to me, the person who had written this page was also dynamically loading his control; unfortunately, he was only loading the control on the pageload, and not on the postback.

for about an hour i kept trying to figure out why the button onmy control wasn't firing its events. finally, in frustration, i decided to step through all of the code: as soon as I saw the pageload i realized what was going on: my control wasn't firing events on the postback because it wasn't even there when the page submitted!

most developers know that web-apps are stateless, but i think we fail to internalize what this really means. we assume that once we declare a variable, it will remain there as long as it's in scope. i'm not trying to blame my co-worker; in fact, i've made this same mistake several times (which is probably why i recognized it almost immediately). i think the problem is we first learn to program in statefull environments, and then make the switch to the web without fully understanding the implication of moving to this new environment.

anyhow, i'm somewhat upset over the fact that i didn't get to finish the control today. i'm planning on getting up early tomorrow, coming in before anyone can bug me and getting the control done - that is until i run into the next nuisance from the stateless web, i guess.

google analytics statistics

2008-07-13T20:04:00.003-07:00

i was just reviewing some stats from google analytics, and just want to summarize some things i found interesting.

i'm getting about 30 unique visitors every month; i have to admit, however, that i'm probably 3 of those. most of my traffic is organic and comes through google search (surprise, surprise!).

my most visited blog entry, is the one i had on particle swarm optimization in c#. my guess is that a whole bunch of lazy college students are looking for code so that they won't have to think in order to do their assignment. the traffic that this post gets is so much grater than anything else i've written, that i'm thinking about catering to lazy C.S college students ;)

the other blog entries that get quite a bit of traffic, are the ones on biztalk tips; i can see why this is since there aren't actually that many people that blog about biztalk.

from looking at the reports, however, i'm starting to learn the importance of writing so that you'll come up in search queries. i have an entry, for example, titled "do i really need an orchestration?"; a much better title would have been "when to use an orhestration", or something like that.

i'm excited about the little traffic i'm getting: there's some dude in sweeden that spent 15+ minutes on my site - i didn't even know that i had 15+ minutes worth of material on my blog! even though it's hard to find time to blog, i'll increase my efforts to do so. like jeff atwood, i've found that it truly is almost as much fun to blog about coding as actually writing code is.

usu's ezportal security assessment

2008-06-02T20:05:00.004-07:00

about 3 weeks ago i finished a security assessment for a new application the university of utah is planning on rolling out so that students, teachers, and staff to can manage all of their usu data.

our analysis included looking at the overall application architecture, looking at the coding, and a test to try to exploit vulnerabilities.

i thought it was interesting that every major hole in the application, came from the developers trusting the libraries and subcomponents they were using. for example, the developers were using an open source rich text editor (so that users could upload nice looking content without having to know html), that could easily be exploited to upload malicious code, or render content from any other site (yes, as in a xss attack).

so, the mantra of "find the dependencies -- and eliminate them", turns out to be true for security problems too.

particle swarm optimization (pso) in c#

2008-05-06T22:21:00.005-07:00

i was going to give a presentation at "code camp" on particle swarm optimization (pso), but unfortunately was not able to do so because of circumstances beyond my control.

so, i've decided to post my c# implementation here so you can take look at it and play with it. the code graphs the movement of the particle swarm, so it's cool to see how the different pso parameters affect the movement of the swarm.

the swarm is not divided into neighborhoods and each member only knows about their local best and the swarm's best value to date. if you want to change the code so that i does neighborhoods, be my guest.

the code is really rough, especially in the way the UI runs. also, there's no way to dynamically change anything; if you want to make any sort of change you have to find the pertinent code, recompile it and rerun it (ie. there are no config files, no input paramenters, nothing).

so, if you think my code sucks, please let me know!

biztalk database problems

2008-04-09T12:01:00.003-07:00

when we started implementing biztalk at tgn there was hardly any documentation on it, and so we had to resolve all of our problems through trial and error, or through help from blogs.

we had two database problems early on that made us wish someone would have warned us about potential problems:

by default, the biztalk messagebox is set to autogrow by 1mb. this is a problem, because if you have to autogrow your db, it's likely that you'll need to grow it by more than 1mb and so the 1mb is consumed promplty and the db has to autogrow once again. we had problems with this twice: the first time we just saw the cpu usage spike every time we autogrew; the second time the effects were worst as we we were running out of space on the disk and the server became stuck autogrowing and rolling back once it realized there wasn't enough space. so, if you're gonna autogrow your BizTalkMessageBoxDb database, autogrow it in big enough chunks to avoid performance problmes.
the PurgeSubscriptionJob_BizTalkMsgBoxDb and TrackedMessages_Copy_BizTalkMsgBoxDB jobs need to run every minute to keep biztalk running in good shape, but unfortunately they are not setup to run by default. we only found out about this when our applications started running absurdly slow and all of our clients complained about time outs.

perhaps if we had had a dedicated dba, and admin for our system, we would have noticed these problems before they caused issues. hopefully, you won't have to suffer through these yourself.

refactoring code with too many conditional statements

2008-04-04T21:53:00.010-07:00

i was once talking with the IS director of a certain company about the tell tale signs of bad code. i mentioned a few of the items that raise flags for me: code duplication, low cohesion in classes, etc.

the director, which happens to be a really good coder even though he's been in management for several years, mentioned that when he sees code with too many if statements (or any conditional branching for that matter), he knows the code could be cleaned up. after hearing his statement, i started considering on how to clean code that has too many if statements.

last week i had the opportunity to help a coworker refactor some code that looked something like this:


public class CreditCardService()
{
  public void DoWork()
  {
       if(creditCardCode = "VISA")
       {
            //about 10 or so lines of code here
       }
       else if (creditCardCode = "MC")
       {
            //about 10 or so lines of code here
            //the code is very similar in all branches
       }
       //a few more branches for the other credit cards we support
       //...
  }
}

the obvious problem with the code above is that it's not making good use of a basic programming principle: polymorphism.

we could clean up the above code by:

writing a base class that implements the common functionality across all credit cards
writing children credit card classes that inherit from the base class and implement what's different in each credit card
writing a credit card factory that return the right implementation to the CreditCardService class

so, we should end up with code looking something like this:


public class BaseCreditCard
{
     //all common fields go here

     public void DoWork()
     {
          //all common functionality goes here
     }
}

public class VisaCreditCard : BaseCreditCard
{
     //all fields pertaining to VISA go here
     
    public void DoWork()
    {
         base.DoWork();
         //visa specific functionality goes here
    }
}

public class CrediCardFactory()
{
     public static BaseCreditCard GetCard(string cardType)
     {
          //return appropriate credit card child class
          if(cardType.Equals("VISA")
               return new VisaCreditCard();
          //more code like the one above
     }
}
public class CreditCardService()
{
     private CreditCard card;
     public CreditCardService(string cardType)
     {
          card = CreditCardFactory.GetCard(cardType)
     }

     public void DoWork()
     {
          card.DoWork();
     }    
}

ideally, our factory will be really smart about picking the right credit card type, so that all of the if statements necessary to pick the right credit card will all be contained in it.

using polymorphism in this case makes the code much easier to read, and way easier to maintain as we now have all decision logic in one place, all common logic in another place, and type specific logic in its own place.

simple dependency injection

2008-03-28T22:10:00.028-07:00

the idea behind dependency injection has been around for a while, and i believe the term "poor man's dependency injection" is popular as well, but i thought i'd share the poor man's method anyhow.

it's common to find that n-tiered applications (especially applications with a persistence layer and a "business logic layer") although layered, are usually tightly coupled.

one nuisance that this coupling creates is the inability to write unit tests that only test the business layer. even though you can write tests for the business layer, these cannot run without calling the persistence layer, and thus you end up with slow tests (because of db calls) and redundant tests (assuming you have tests for your persistence layer).

a very simple method (the poor man's method, of course) to solve this coupling problem is to:

code to an interface (in the case of the service layer, make sure you're not calling a specific implementation of your persistence layer, but an interface).
overload the constructor(s) for your business logic class so that they also take a specific implementation of the interfaces it depends on.

if you do what i've described above, your business layer should end up looking something like this:



public interface IPersistenceLayer
{
    void DoWork();
}

public class BusinessLogicLayer
{
    IPersitenceLayer myDataStore;
    public BusinessLogicLayer(IPersistenceLayer someImplementation)
    {
         myDataStore = someImplementation;
    }
   
    public void DoWork()
    {
         //business logic here
         myDataStore.DoWork();
    }
}

with the above code you can easily write tests for your business layer that just take a dummy IPersistenceLayer and return mock objects. and so now we have fast tests that don't require any database setup and/or maintenance.

there is however, one obvious problem with the code presented: why should the clients to the business layer have to know about the layer's dependency? the answer is they shouldn't and that's why we keep all the default constructors, and just have those set the IPersistenceLayer reference to the commonly use implementation of the interface.

although almost trivial, this method of dependency injection is appropriate for simple scenarios and provides much flexibility.

linq and lambda expressions

2008-01-24T19:55:00.000-07:00

i'm aware that i'm about a year (or more, probably) behind on this, but the new c# lambda and linq features are awesome!

i wrote my first dlinq query today, and i'm now of the opinion that linq will literally change the way we code. i'm aware that i don't recognize all of the repercussions that come from linq, but just the very basic queries i did today, literally changed some more of the fundamental programming paradigms i held.

as for lambda expressions, i must say that even though the concept is not new (in fact, anonymous delegates and types are almost as old as programming), there's something to be said for how elegant lambda expression are: what would be ugly (or sometimes even impossible constructs) in c# are extremely simple and easy to read statements thanks to lambda expressions.

i'll keep posting on linq and lambda expressions as i learn more. but if you haven't had a chance to learn about them, i strongly recommend you to.

a4host.net

2007-12-11T20:11:00.000-07:00

i'm excited to announce that syscoders lc (the consulting company i started) has finally started a web hosting branch. please take a minute to visit our website at a4host.net!

some problems with biztalk's ftp adapter

2007-12-06T20:09:00.000-07:00

there are two issues you need to be aware of when using the ftp as a receive location:

ftp does not have a way to lock files while you're reading them; under some conditions, you could have more than one machine pick up the file and you could end up with duplicate messages. obviously then, you cannot have the host running the ftp adapter live on more than one machine. the receive function of the ftp adapter MUST run under a singleton host.
the biztalk adapter closes the ftp connection after 3 minutes of inactivity. this means that you must process your files promptly. if you're processing a large file, and it takes more than 3 minutes for your process to commit the message(s) to the message box, you will not be able to delete the file from the ftp receive location once your process finishes. this means that next time your receive location runs again, you'll pick up the same file!

we've encountered both of these issues recently, and both are easy to solve:

like i've already mention, have your ftp receive location run under a singleton host.
just bring the file down (without processing it) through ftp, and put it on some file folder. have the process that would normally work on the file pick the file up from this new location.

the "guaranteed delivery" nature is a really nice feature, but you have to be aware of the problems it can create. hopefully this will help you in your development; i know i was scratching my head for a while trying to figure out why the ftp adapter wouldn't delete the files it had processed.

on throwing errors

2007-11-16T15:19:00.000-07:00

not too long ago i spent quite a few days trying to solve what was a really easy bug.

after moving a class from one assembly to another, one of my biztalk orchestrations started throwing a runtime error saying that it couldn't serialize an instance of a XmlDocument.

because the runtime error did not return a stack trace, i was very confused: even though XmlDocument truly is not serialiazable, biztalk is able to dehydrate the contents of a XmlDocument instance by getting the XmlDocument's outerxml and storing it; upon rehydration, the XmlDocument is loaded with the saved string.

after a few days of thinking about this problem, I realized that the error was not coming from the orchestration's instance of XmlDocument, but instead from a class that held and instance of XmlDocument and which the orchestration was using.

i simply marked the XmlDocument variable declaration with [NonSerializable] and all the problems went away.

i mention all of this because if had just gotten the stack trace, all the hassle I went through could have been avoided.

i've noticed that the errors i throw are usually not very descriptive, because i understand the code and i know what's going on. however, i've also noticed that when co-workers get exception from my code, it's not clear to them what the error means.

thus the importance of throwing very specific and very clear error messages; even if they're a little verbose. and outsider needs to understand the context of what's going on in order to be able to understand and solve problems.

two important biztalk tips

2007-09-21T15:08:00.003-07:00

without further ado, here are the two tips (in question form):

do you really need an orchestration?
do you really need an xsd/xml message?

most developers making the switch to biztalk are most comfortable with procedural code, and thus writing orchestrations feels natural and is a lot more intuitive. however, i've learned that messaging based solutions are extremely powerful, easier to maintain, and incredibly flexible. i'm not saying that you can't build a messaging based solution if you are using orhcestrations... all i'm saying is that most of the time, whatever you're doing in an orchestration, really doesn't need to be done in an orchestration; this is specially true if all you're doing is message manipulation.

also, most projects don't need and xsd/xml message. a .net class as a message is, for most solutions, the better option. there's one big con in using a .net class however. if you need to publish your message externally, the you have to go with xsd, because a .net class offers nothing as far as publication goes. nonetheless, if you don't need an externally published message, go with a .net class... it'll save you tons of times and headaches ;)

i'm on slashdot!!!

2007-06-28T20:48:00.000-07:00

ok, so i'm really not on slashdot, but the project i'm working on was recently reviwed there.

since i don't particularly like touting my own horn (not to mention that no one reads this blog anyhow), i'll just say that i'm writing all of the back-end communication with sorenson genomics and internal systems.

i was excited about this project from the get-go, but the "national" attention it's gotten really makes this one something else.

communication!

2007-06-07T18:58:00.000-07:00

i've always been acutely aware of the fact that a lot of problems in life arise out of the lack of proper communication - and IT is no exception to this rule.

the oracle guys at place i work, are all (with one exception) non-developers, and have decided to start writing some web-services in order to expose some features that haven't been available before.

one of the services they wrote gets the purchase order for any given sales order. as i tried to consume the service today, i noticed that the purchase order had a flag for whether the purchase order had been canceled - but here's the thing: the flag was not a boolean, it was a string and it returned either 'Yes' or null.

so i called them and explained that it might be better to use a boolean. they seemed a little reluctant, so i went over to their desks to try to persuade them. and when i got there i realized that i could not explain what i wanted to explain without technical terms, and in a way that would make sense to them. i guess that happens when you only communicate with other techies on a regular basis, but that is really no excuse.

everything ended up ok; i had to explain a few concepts and terms like coupling, cohesion, etc. but in a way that meant something to them and would show them the values of their code.

communication is hard, especially when we can't speak the same language. i'm looking forward to more opportunities to interact with the oracle guys, so i can get out of my comfort zone and expand my vocabulary.

how to be an expert

2007-01-18T09:22:00.001-07:00

kathy sierra has an excellent article on how to become an expert. here's a quote, that captures the essence of her post:

For the superior performer the goal isn't just repeating the same thing again and again but achieving higher levels of control over every aspect of their performance. That's why they don't find practice boring. Each practice session they are working on doing something better than they did the last time.

i've blogged about becoming a pro too; you can find the post here.

self correlating ports (and parallel processing)

2007-01-08T16:31:00.001-07:00

the image above belongs to an orchestration i wrote to update a table with a list of currencies. the process is rather simple: all we have to do, is get the list of currencies we support (from the database), and then update the exchange rate.

to do this, i chose to use and extremely parallel method, in which every currency spins off its own thread, calls the service that contains current exchange quotes, and updates its value in the db.

what you don't see in the image above is the code that gets the list from the database and then instantiates a new orchestration for each currency. what you do see, is the part where the orchestrations (when they have finished their work) call back to signal that they have finished. after this, we send out an email notifying that we've updated all currencies.

doing this in biztalk is rather simple, since the functionality to asynchronously start orchestrations comes out of the box, and call backs are easily implement with self correlating ports. i like how biztalk makes this parallel process extremely simple.

if you wish to learn more about how to setup self corrlating ports, visit Stephen W. Thomas BizTalk Blog here for detailed instructions.

who's on your bus?

2007-01-03T19:51:00.000-07:00

i once read an interview with jim collins (author of from good to great and built to last) where he said the most important factor in a succesful company is whther or not "the right people are on the bus". if you have the right people, everything else will fall into place... at least that is the theory. and my gut feeling is that the theory is right; people are not everything, but they are important enough where if you have to just pick one thing, you pick the right people.

i mention this because of an experience i had at work today:
i found a problem trying to begin a transaction in one of our live servers. before calling the dbas, i made sure i had a pretty good idea of what was going on... the problem only surfaced when trying to start a transaction from a client that was on a different domain.

i called in one of the dbas and described what was happening and asked for his help to solve the problem. i was polite when i brought the issue up, but our dba immediately became aggresive and blamed the problem on the client. i was confident that the problem was on the server, so i insisted he should at least take a look at it.

we went back and forth, arguing where the problem was, for at least 10 minutes; i finally had to show our dba i wasn't kidding, and in a somewhat threatening (firm, more than anything) tone told him that (1)what he was saying made no sense and he knew, and (2) what he needed to do to get proof that it was the server that wasn't working.

after going through the steps i asked to go through, he finally admitted it was the server that wasn't working. he also told me he had no idea on how to fix it (i don't blame him on this... from the research i did before i called him, it was apparent to me that the problem wasn't trivial) and would get back to me whenever he had found a solution. i immedately thanked him for his time and for committing to solving the issue.

after i hung up, i started thinking: why did it have to come to this? why is it so hard to work with the dbas? (it's not the first time i've had a experience like this, and i'm not the only one that has problems working with them). i thought: "we need the right people on that job, it's a key job. we nee the right people on the bus".

my manager congratulated me on how i handled the situation (he actually sat there listening the whole time because this was a high priority issue for us). after a few comments i told him i thought the hardest part about being a "techie" is not the technology, but working rather working and communicating effectively with others; i also told him most problems in software arise because of problems in these areas... and thus the importance of getting the right people on the bus.

learning to drive

2006-12-06T16:28:00.000-07:00

in his book extreme programming explained, kent beck compares software development to driving a car, and makes the following insight:

The driver of a software project is the customer. If the software doesn't do what they want it to do, you have failed. Of course, they don't know exactly what the software should do. That's why software development is like steering, not like getting the car pointed straight down the road. Our job as programmers is to give the customer a steering wheel and give them feedback about exactly where we are on the road.

talk about a useful mindset to have! on a similar note, eric (my current manager at work) always tells me about how in one of his most succesful projects, they presented the product to the customer on a daily basis to provide the feedback beck talks about.

wtf: sonic cineplay

2006-12-05T22:28:00.000-07:00

i'm well aware of what wtf stands for; however, i choose to ignore that and have it mean Curious Pervasions in Information Technology.

here the is my first wtf:
on friday december 1st i rented a movie. for some unknown reason the dvd would not play correctly on our desktop... after about an hour of trying to make it work, we decided to to just watch the movie on our laptop.

a few days later, my wife tells me: "something is wrong with the desktop. it won't play the 'signing time' dvds either. i take a quick look, and sure enough, the dvds play but there is no sound.

after trying everything i could think of (checking codecs, checking settings on the player, checking audio drivers, etc.) for four days, i'm about to give up and roll back to a previous back up. before doing this however, i try one more google search... i find the following on one of roxio's customer forums:

All,

There is an issue with Cineplayer not working. That could be no video, no audio or corrupted video. We’re aware of the issue and are working with Sonic on getting this resolved shortly.

The problem only manifests when the clock is set to the month of December. So simply change the date to another month and it will work.

This is only a workaround. We are currently working with Sonic to resolve this issue. We’ve been told that a patch should be available by Wednesday, December 6th.

We apologize for the inconvenience and assure you we are working to get this resolved as quickly as possible.

this is a wtf on so many levels... i just can't even begin to comprehend why a codec does not work on the month of december. it would be great to look at the code that's causing this bug and then posting it for learning purposes.

i'm glad i didn't roll back to a previous back up; that would have failed and then i would have probably re-installed everything; after that i would have probably shipped my computer to dell... and pulled my hairs out :)

taking notes (and a little on UML)

2006-11-22T16:49:00.000-07:00

i've written about this on my personal blog, but i'll mention it here again... taking notes helps you remember stuff, so i like to do it even if i never look at my notes again.

not too long ago i interviewed with bill, a manager in the company i work for. while we were talking he took copious notes. while he was doing this he told me not worry, that he always does this even though he normally just throws his notes away.

under the guidance of this principle, i'd then like to write down the following idea from Fowler's UML Distilled:
use cases help you determine requirements; and actors help you determine use cases.

i like the idea because it's simple, easy to remember, and it helps keep the end in mind.

asp 1.1 and 2.0 on the same box

2006-11-21T22:21:00.000-07:00

the compnay i work for, and a lot of guys on my team have added the .net 2.0 framework to their boxes recently.

a lot of people have come to ask me either why their 1.1 web projects stopped working or why they can't get a 2.0 asp project to work. 9/10 times the problems arises because the box is trying to run asp 1.1 and 2.0 under the same app pool.

so, the answer to the problem is to make sure that at the very minimum all your 1.1 stuff is running under one app pool, and all of your 2.0 stuff is running under a different pool.

a lot of people have asked me why this is, and honestly, i don't know yet (but i'm looking)... i'll post a follow up as soon as i find out.

but there's an important lesson to learn here, not just some obscure fact about asp:
solving this issues is actually easier than what one would think; all one has to do is look in the eventviewer, where the following error would be logged:

It is not possible to run two different versions of ASP.NET in the same IIS process. Please use the IIS Administration Tool to reconfigure your server
to run the application in a separate process.

it's been my experience that most devs don't look in the eventviewer, iis logs, etc, to find clues to their problems. as i've worked with BizTalk (and thus have been forced to look there) i've learned what great tools the eventviwer, the performance counters, and others are.

a place holder

2006-04-03T21:06:00.000-07:00

just an entry fot this new blog of mine.