The hardest thing about programming.

When asked what the hardest thing about programming is, Bill Gates answered:

The hardest part is deciding what the algorithms are, and then simplifying them as much as you can. It’s difficult to get things down to their simplest forms. You have to simulate in your mind how the program’s going to work, and you have to have a complete grasp of how the various pieces of the program work together. The finest pieces of software are those where one individual has a complete sense of exactly how the program works. To have that, you have to really love the program and concentrate on keeping it simple, to an incredible degree.

What's interesting, however, is that the hardest thing about programming is also the best thing about programming.

I love programming because learning it is relatively easy, but mastering the art is extremely difficult. Like Gates says, reducing things to their simplest form is difficult - but fortunately, it's also beautiful. It's also true that the finest software is written by those who love the program and make a deliberate effort to keep it simple - and again, although difficult, being able to model an entire system in your head is pretty awesome!

Simple Events In ASP.NET MVC

We've been busy improving our customer facing applications. Our new marketing VP noticed some significant improvements that could be made in the workflow of our customer acquisition funnel.

Making the changes that our marketing VP recommended required some major code changes so we took the opportunity to move our web apps into ASP.NET MVC.

One of the early decisions we made was to put as little business and application logic on our controllers as possible. Finding a place for business logic is straightforward most of the time; it usually belongs in your (domain) Models.

Placing application logic in the right place gets a little more complicated. We don't really have services for stuff like sending out emails, etc. so we have to write classes to do all that work.

Invoking email code, authentication code, etc. right in the controller is an ugly solution. To solve our problem, I came up with a simple eventing mechanism that allows us to move all application code out of our controllers.

Let's first start with what the code in one of the controllers would look like:

[AcceptVerbs(HttpVerbs.Post), ValidateAntiForgeryToken]
public ActionResult Index(IndexModel model)
{
     //code to interact with your model goes here.
     
     //And now the event code:
     AcmeEvents.Raise(new IndexRanEvent { IndexModel = model });

}

The great thing about the code above is that all sort of things could now happen once your index runs; there may be 1 or N subscribers to the event above. Furthermore, you could (at run time) change the code that runs when the event triggers... but I'm getting ahead of my self; we'll look at how that could happen in just a second.

Let's now look at the "raise event code". To do this, I decided to just use Unity's IoC container to resolve the code should trigger when the event get's raised:

public static void Raise<T>(T args) where T : IAcmeEvent
{
     foreach(var handler in unityContainer.ResolveAll<Handles<T>>())
     {
          handler.Handle(args);
     }
}

Some assumptions the code above makes: (1) IAcmeEvent has a Hanlde method; (2) by the time the Raise method runs, the IoC container must have been initialized to map the different "event types" to the specific implementations that will execute.

How you initialize the container is really not that important: you could do it in code, through a config, or any other method of your choice. What is important, however, is that when you initialize your container, you do it in such a way that code from other assemblies can execute. If you do this, you could literally just drop assemblies in your application and add to or change the behavior of your app.

So there you have it; a simple eventing mechanism to keep your MVC controllers unpolluted from application logic. I'm really happy about coming up with this solution; I hope you find it useful too.

Barricading Your Code

You've heard it a million times: "Never trust your inputs". Or maybe you've heard its corollary: "Always sanitize your inputs". Easier said then done, right?

I'd like to show you a way to barricade your code so that you can always sanitize your inputs without littering all of your code with sanitation code.


The key to writing good sanitation code is controlling where and howyou accept external input. Just like the barricade in the image above, traffic can only flow on the sidewalks and not on the street.

Here's how to accomplish the same with code:

public class ClassThatUsesBarricade
{
    public void DoWork(string inputOne, int inputTwo, ...)
    {
         //sanitize inputs
         RealDoWork(inputOne, inputTwo, ...);
    }

    private void RealDoWork(string inputOne, int inputTwo, ...)
    {
         //don't worry about inputs because no external class could have called this.
         //do work
    }
}

By writing code like the code above, you've barricaded your class: data can only flow into your class through the public method. The private method can absolutely, 100% or your money back, trust its inputs without having to worry about sanitizing them. This way, your real work method can be short, and concise - after all, a method should only do one thing, and do it well.

If your class has a lot of methods with very similar signatures that do similar things, you might want to reverse the code I showed: you could have a private sanitation code that all public methods that actually do work call. However, that's not really barricading your important methods... at some point you could forget to call the private sanitation method (ie. set up your barricade), and you'd be screwed.

And there you have it, a code barricade.

When to use Class Inheritance in Programming

Last week I listened to a friend of mine complain about some straight-out-of-college programmers he had recently hired: "All of their code has 10 layers of inheritance. It's crazy trying to maintain their code!".

Unfortunately my friend is right: inheritance, although powerful - when used correctly, naturally tends to increase complexity.

Since your primary purpose as a programmer is to manage and reduce complexity, you should favors solutions that don't use inheritance.

At this point you're probably asking yourself: When am I supposed to use inheritance? How do I know if I'm using it correctly?Fortunately, the great Steve McConnel, has written 4 clear cut rules on when to use inheritance and when to use containment:

1. If multiple classes share common data but not behavior, create a common object that those classes contain.

2. If multiple classes share common behavior but not data, derive them from a common bases class that defines the common routines.

3. If multiple clses share commond data and behavior, inherit from a common base class that defines the common data and routines.

4. Inherit when you want the base class to control your itnerface; contain when you want to control the interface.

In summary, only inherit if the new class truly is-a more specialized version of the base class.

Oh, and one more piece of advice on inheritance: avoid multiple inheritance like the plague.

How To Write Reusable Software: A Lesson From LEGO Blocks

I don't know about the places where you have worked, but where I've worked we haven't had a very good track record for writing code that actually gets re-used. In fact, at one place we even gave up: "Software reuse is overrated" a boss muttered once.

I think the reason we fail so miserably is that we focus too much on the end goal (reusable code), and too little on the qualities that make software reusable. What, then, are the properties of reusable code? Well, that's where the LEGO blocks part comes in.


The LEGO block analogy comes from David West's Object Thinking, but I'd like to highlight a few ideas that make this metaphor so applicable to software development:

• The interface is simple and intuitive. I've seen my 3 year old daughter build ships with LEGOs. If you want your code to be reused, it should be so simple and so intuitive that others could use it without having to refer to external documentation.
• There's a finite and small number of LEGO block types. If you find that your library has a high number of classes, think about what you are doing. After all, everything on this planet is made from the same 92 (naturally occurring) elements.
• LEGOs are cool because you can build stuff. The reason everyone loves LEGO blocks is because they can build whatever they want with them, not just what the box tells them to build. Your software should be the same way; you should be surprised by what end users build with your code. If your code only gets used for what you envisioned, you've failed to write reusable code: people are not able to compose software with your code.

Finally, don't forget that people don't really care about the technical details behind LEGO blocks; no one cares how the plastic is molded, dyed, etc. LEGO blocks aren't reused because of how well they're manufactured; they're reused because they let end users accomplish their goals.

Once people are able to accomplish their goals with your code, they'll be using it over and over gain. Guaranteed!

Custom Thousand Separator In C#

I guess the NFA requires us to use a space (" ") instead of a comma (",") as our thousands separator. I dunno why the have to regulate something like that, but I had to figure out an elegant way to format our numbers the way the powers that be require them to be formatted.

After struggling with different inelegant solutions for a little while, I finally realized that different countries use different symbols as thousands separators; I knew then that my problem was really a localization problem. A quick search in MSDN lead me to the NumberFormatInfo class, which "defines how numeric values are formatted and displayed, depending on the culture" (MSDN).

The NumberFormatInfo class, has a NumberGroupSeparator property that defines what character to use to separate thousands.

So, here's how I ended up solving my problem:

public void FormatDouble(double doubleToFormat)
{
     NumberFormatInfo nfi = new CultureInfo("en-US", false).NumberFormat;
     nfi.NumberGroupSeparator = " ";
     Write(doubleToFormat.ToString("n", nfi);
}

Neat, huh?

Highly Parallel LINQ To Lucene Indexer

I've been wanting to post about this for a while, but haven't been able because I was away vacationing in Hawaii. :)

I was recently given the assignment to index a couple of our tables that hold large amounts of data. Each tables has well over 3 million records, and each record is fairly large as it represents "emails" with our clients.

I knew from the get-go I'd have to write a highly parallel process in order for the task to finish promptly - just bringing the data over the wire from the database takes about 45 minutes.

I have to show a lot of code now, so please bear with me for just a second. The key here is to use several RamDirectory(s) for the small data chunks and merge them into a FSDirectory index that will hold all the data:

namespace Indexer
{ public class BatchIndexer[T] : ParallelIndexer[T]
 {
  IQueryable[T] items;
  bool isMergingFinished = false;
  private static object lockObject = new object();

  public override bool Stopped
  {
   get
   {
    var pstopped = base.Stopped;
    if (pstopped == true)
    {
     return isMergingFinished;
    }
    else
    {
     return false;
    }
   }
  }

  //stop setting the temp directory here and force the client to pass it in
  public BatchIndexer(string path, IQueryable[T] items)
   : base(new Index[T](path))
  {
   this.items = items;
   QueueChildren();
  }

  private void QueueChildren()
  {
   if (null != items && items.Count() > 0)
   {
    const int pageSize = 100;
    int pageIndex = 0;

    var currentPage = items.Take(pageSize);
    int pageCount = 1 + items.Count() / pageSize;
    while (pageIndex < pageCount)
    {
     ThreadPool.QueueUserWorkItem(new WaitCallback(QueueChild), currentPage);
     pageIndex++;
     currentPage = items.Skip(pageIndex * pageSize).Take(pageSize);
    }

    //wait for all children to be added to parent
    while (Children.Count() < pageCount)
    {
     Thread.Sleep(2000);
    }
   }
  }

  private void QueueChild(object stateInfo)
  {
   var currentPage = stateInfo as IQueryable[T];
   lock (lockObject)
   {
    AddChild(new BatchIndexerChild[T](new Index[T](), currentPage.ToList()));
   }
  }

  public override void Run()
  {
   //start indexing the children
   foreach (var item in this.Children)
   {
    var child = item as BatchIndexerChild[T];
    ThreadPool.QueueUserWorkItem(new WaitCallback(child.AsyncRun));
   }

   //wait for all of the children to finish indexing
   while (!IsChildrenStopped)
   {
    Thread.Sleep(1000);
   }

   //merge all indeces into main index
   this.Merge((from c in this.Children select c.Index.Directory).ToArray());
   isMergingFinished = true;
  }

  protected override void Dispose(bool disposing)
  {
   base.Dispose(disposing);
   if (disposing)
   {
    //release managed resource
    if (this.Index != null)
    {
     this.Index.Dispose();
    }
   }
  }
 }
}

I particularly like the approach I used because it threads the chunking of the data into smaller pieces AND is also threads the actual indexing of each chunk.

If you're going to index as much data as I had to index, however, you still have to pre-chunk the data and feed into several instances BatchIndexer; I found that 100,000 records was optima. You'll also want to make sure you release all database resources ASAP: you don't to be hogging a whole bunch of memory!

Anyhow, I hope this code is useful... it should at least give you start in the right direction. Let me know if you have any questions.

Factory Pattern

After reading this article, a reader recently requested that I show some sample code for the "factory" mentioned in the article.

Before I give you some sample code, however, let me just write a few words about the Factory Pattern: As the pattern's name suggests, the Factory Pattern creates stuff. Specifically, "factories" create classes without coupling you to a specific instance of the classes it creates.

Here is some sample code:

public class CreditCardBase
{
  //this is the base class that all credit cards implement:
  //it could be an abstract class, or it could have some functionality;
  //it could also be an interface... whatever floats your boat.
}

public static class CreditCardFactory
{
  public static CreditCardBase GetCreditCard(object someObject)
  {
       //business logic that returns correct credit cart type goes in here.
       if(someObject == "something")
            return new Visa(); //visa of course implements CreditCardBase
       else
            return new Amex(); //also implements CreditCardBase
  }
}

public class CreditClassClient
{
   //this class consumes a credit card
   public void DoWork()
   {
        CreditCardBase myCreditCard = CreditCardFactory.GetCreditCard(new bizObject());
        //do stuff with myCreditCard
        myCreditCard.Run();
   }
}

And there you have it. That's pretty much all there's to it, and the code is pretty self explanatory.

There's one more thing that's worth mentioning: usually, classes returned by factories have private constructors - to avoid people instantiating the class without calling the factory. This, however is problematic for several reasons:

1. You may have clients that already instantiate your class.
2. People usually make private constructors singletons (please don't).
3. How do you extend a class that has a private constructor? :)

Finally, to the anonymous reader that asked for the factory code: let me know if this was useful.

How Not To Lose Money In The Stock Market

One of the most important rules to remember when investing in the financial markets is to Preserve Your Capital! Not losing money is the first step before you can make any money.

And although I haven't figure out a technique that guarantees 10% yearly gains, I have figured out a pretty good technique for not losing money in major market draw downs. Allow me to show you a chart of one of the ETFs I like to use to trade:


The chart to above shows the daily price for QLD, along with the 50-day exponential moving average (in red) and the 10-day exponential moving average (in blue). At the bottom of the chart, you can also see the daily trading volume.

What's interesting in the QLD chart is that during the past year every major market drop has been preceded by a clear signal: the 10-day EMA crosses the 50-day EMA (heading down, of course). And this, even when the market has faced major volatility and huge daily drops.

You can go back as far as you like, and you'll find that the signal above always hold true. And this is not because I'm so awesome and I've figured out some magic formula - no, it holds true because of the very definition of EMA. The EMA lines just show you the averages of the stock price for the last 50 and 10 days respectively.

Of course this technique isn't perfect: you will lose some money before the 10-day EMA crosses the 50-day EMA; but at least you won't lose more that 50% of your money, like several people have in the past year.

Disclaimer: Don't take financial advice from a guy that writes software for a living (i.e. me). Also, don't take advice that makes a living selling financial advice (i.e. your financial advisor). Those guys don't make money in the markets; they make money making you feel good about losing 50% of your capital.

ASP.NET MVC Release Candidate Now Available!

The release candidate for Microsoft's ASP.NET is finally here.

Sure, this is just another "me-too" Microsoft product: we've had Rails, Cake, and Grails - but this is great news nonetheless! The ASP.NET web forms model is one that I never really understood: Why would you try to imitate a windows form on the web? Anyhow, web forms have been long over-due for a replacement, and now MVC provides a great architectural pattern for building web apps with the .NET framework.

Scott Gu, has written and in-depth article describing some of the most important improvements on this RC. And, from what Phil Haack said on his blog about this release, I get the feeling there's just a few minor bugs that need to be addressed before the RTM. You could probably start using the RC now without any concerns. In fact, I have a co-worker that has been writing a site on MVC since beta with no problems at all. Another co-worker mentioned to me today how much more productive he was under the MVC model than the WebForms model.

If you're still unsure about ASP.NET MVC, go check out stackoveflow.com. StackOverflow was written on the beta release of the MVC stack, and it's an awesome site.

Now, if you're still unsure, just trust me on this one: put your WebForms on the ground, back away slowly! I don't want anyone getting injured here. :)

Software Engineering: There's nothing like it!

Intro

Seriously, there's no other type of engineering like software engineering. And that, may or may not be a good thing: it just depends on what you decide to focus on.

I went to school in the same building where the mechanical engineers, chemical engineers, and civil engineers went to school - yes, I spent almost every waking hour of my college life in the infamous Clyde Building:


Yes, for those of you that are wondering, that building has no windows. I'm pretty sure they designed it that way so that your soul would be crushed within the first semester of your career. But that's beside the point: You want to know what I learned in 4 years as an EE at BYU? Well, for the most part, I learned a whole bunch of math... oh yea, and I also learned that writing software isn't like making bridges, roads or "stuff".

And here's why software enginnering is so different: software is cheap to make (Yeah, that's right; even though you think you're making big bucks by writing software, you're still cheap compared to what goes into creating a new car design). Oh, and there's something else: writing software is much harder than building bridges.

Software is hard

See, we can't really prove software will "work". Proving that a program will work according to the specs is pretty much a halting problem - that is, given a program and a finite input set, we can't even prove whether the program will even terminate!

Civil engineers, on the other hand have it easy: given a bridge and environment conditions (load, winds, etc.), they can tell you whether the bridge will stand or not. In fact, it's so easy to "model" bridges that we software engineers have written programs to tell you whether you've built a good bridge or not.

Software is also harder because as long as the program "works", it doesn't matter what the source code may look like. Sure, some code may be harder to maintain than other, but how do you tell? There's no metrics for code design; we only have smells.

Software is cheap

Let's not move on to why software is cheap. If you write a program that doesn't work according to specs, it's not a big deal: you just go back to Notepad, change a little bit of text and (perhaps) recompile. Even software distribution is cheap: when was the last time you got software in a CD? We could spend a lot of time and effort finding bugs and/or proving our program works correctly, but why would we do that? After some point, it really doesn't make sense to spend money finding and solving bugs that may never surface in normal use.

On the other hand, if you screw the pooch on a bridge you're toast: it's hard to fix, and it was really expensive to build (not to mention how much time it actually take to build). This is why civil engineers, plan, model, test and then finally, after a really long time, build. We just build because it's fast to build. How awesome is that?!

Conclusion

Now you know why you're much better than all those other engineers out there. So go rub it in their faces whenever you feel like it. :p

What do programmers do?

Have you been ever asked what you do for work? I have, and it's always hard for me to answer: if I say I'm a computer programmer people immediately think I can fix their printer - or worse, they want me to build them a dating website for five hundred bucks.

The truth is that what we "computer scientists" do is more along the lines of what sorcerers do. Hal Abelson describes it really well:

Computational processes are abstract beings that inhabit computers. As they evolve, processes manipulate other abstract things called data. The evolution of a process is directed by a pattern of rules called a program. People create programs to direct processes. In effect, we conjure the spirits of the computer with our spells.

A computational process is indeed much like a sorcerer's idea of a spirit. It cannot be seen or touched. It is not composed of matter at all. However, it is very real. It can perform intellectual work. It can answer questions. It can affect the world by disbursing money at a bank or by controlling a robot arm in a factory. The programs we use to conjure processes are like a sorcerer's spells. They are carefully composed from symbolic expressions in arcane and esoteric programming languages that prescribe the tasks we want our processes to perform.

A computational process, in a correctly working computer, executes programs precisely and accurately. Thus, like the sorcerer's apprentice, novice programmers must learn to understand and to anticipate the consequences of their conjuring. Even small errors (usually called bugs or glitches) in programs can have complex and unanticipated consequences.

Fortunately, learning to program is considerably less dangerous than learning sorcery, because the spirits we deal with are conveniently contained in a secure way. Real-world programming, however, requires care, expertise, and wisdom. A small bug in a computer-aided design program, for example, can lead to the catastrophic collapse of an airplane or a dam or the self-destruction of an industrial robot.

Master software engineers have the ability to organize programs so that they can be reasonably sure that the resulting processes will perform the tasks intended. They can visualize the behavior of their systems in advance. They know how to structure programs so that unanticipated problems do not lead to catastrophic consequences, and when problems do arise, they can debug their programs. Well-designed computational systems, like well-designed automobiles or nuclear reactors, are designed in a modular manner, so that the parts can be constructed, replaced, and debugged separately.

Although I'm attributing the quote to Abelson, the text above comes from the book Structure and Interpretation of Computer Programs, written by Albeson, Sussman, and Sussman. The reason I'm attributing the quote to Albeson is because I first came across this by watching one of his MIT lectures on the same topic.

If you're interested in watching MIT's introductory "Structure and Interpretation of Computer Programs" class, you can find the videos here.

And now, go conjure some spirits!

The Perils of Sushi Bars

This is the story of how my credit card was almost stolen by a Samurai in a sushi bar in Salt Lake City. If you're interested, read on.

About 2 months ago I went to a sushi restaurant here in Salt Lake City. After paying our bills and while we were walking out, Kevin (one of my co-workers) noticed that his receipt had his full credit card number printed on it. Rightly concerned, he immediately talked with the restaurant's manager and asked to see the restaurant's copy of his receipt - sure enough, the restaurant was storing hundreds of credit card numbers in their little dinky cash register.

The whole incident got me thinking about credit card transactions. Every time I buy something with a credit card it's like I'm handing my wallet over to the clerk and saying: "Here's my wallet. Take whatever you need and just hand it back when you're done." Who knows what the POS machine is doing with my credit card number! And that's the least of my concerns: What about the lame payment gateway that stores millions of credit card numbers on servers that will likely be hacked in the next 5 years?

Interestingly enough, however, the whole credit card thing seems to work remarkably well. I mean, there's little risk in the sushi bar overcharging; after all, I know where they're at, and I also have the the credit card company on my side: one call to the them and the restaurant we'll have to pay the bill. As far as the payment gateway is concerned, even thought there's the PCI standard, when they get hacked we all pay the cost (either thorough higher interest rates, more taxes, etc)... that's why I'm really concerned about them - I worked at a place where we were PCI compliant, but there were still gaping holes in our security model.

This particular sushi story, however, has a happy ending. We recently went back to the restaurant and were pleasantly surprised to see that our full numbers were no longer being printed on receipts. It's good to see that non-IT folks are finally starting to get the importance of securing information.

Steve Yegge Is My Hero

I regularly read Joel Spolky's blog, Jeff Atwood's blog, and Steve Yegge's blog. I really like them all, but Steve's is probably the most thought provoking blog out of the three.

Recently, for example, he had a post where he discussed (at length), a very intersting topic:

Imagine Mario, and his brother Luigi, having a conversation in which Mario asks Luigi what's on the other side of the boundary to their Mario Kart
universe. Interesting, huh?

It's interesting because to Mario & Luigi, stuff on the "other" side of their universe boundary is "undefined"; it's undfined in the sense that anything could be there. In fact, to some extent, it doesn't even make sense for them to ask what's on the other side; they simply cannot define what's on the other side as long as they reside inside the Mario Kart universe.

On the other hand, programmers living outside the Mario Kart universe can easily define what's outside the Mario Kart universe: all they have to do is inspect the memory contents of the system running Mario Kart, and take a peek at what's in there, on the "other" side.

I enjoyed Steve's post on embedded systemsbecause it helped me make sense of the embedded system I live in. I now understand that anything could be out there; it's truly undefined.

Lucene.net UnAuthorizedAccessException

Unless you specifically tell Lucene.net where to put its LOCK files (for the index readers and writers), it will put them in your OS's default temp directory (normally C:\Windows\TEMP\). This choice of location may be problematic.

As you can imagine, there are plenty of scenarios under which Lucene.net may be running under an identity that does not have enough to access to your temp directory. Under such scenarios, you'll get an UnAuthorizedAccessException thrown at you; and your code may not even be around to handle it.

To solution for this problem is easy; add the following to the section of your application's config file:

After doing this, make sure "Everyone" has full control of the folder you've chosen for Lucene.net to use as its LOCK file folder.

You should also consider properly disposing your indices when your process finishes so that all locks are release on a timely basis. Unfortunately, I can't tell exactly how to do that - you'll just have to figure it out.

Finally, if you're reading this for some other reason than having the exception thrown at you, you should really consider using Lucene. It's great!

2008 Reflections

Man, it feels good to be back to blogging! It was an awesome Christmas break, but I must admit I kept coming up with ideas I wanted to blog about.

Anyhow, '08 was a great year for me: I moved to a great company and started blogging regularly. I also learned a lot of LINQ, some Ruby, and had some great personal moments.

As a recap of 2008, I wanted to share my favorite posts so far:
The Secret Behind SQL
Easy Background Tasks in ASP.NET
Why Partial Classes Are Wrong!

I also wanted to thank all of you that regularly read my blog. There's a few of you out there that are loyal readers, checking often whether I have new content or not; knowing that you're reading makes me much more critical of what I write. Thanks for your time!

Finally, here's and interesting graph of traffic on this blog:



You see that spike right around Nov. 17th, 2008? That, my friends, comes from onesmall comment buried among hundred of comment's in Jeff Atwood's blog. Isn't that crazy? That blog is the black hole of developer effort; I swear every developer on planet earth goes there to waste their time... I can't find any other explanation for getting so much traffic from just one comment. Thanks Jeff! And thanks to everyone that followed the link and took time to read what I had to say.

May 2009 be a great year for you.

Why You Should Be Controversial

I have to tell you a story before any of this makes sense, so bear with me for a minute.

Michael Murray, a good buddy of mine, IMed me today telling me he had been invited to write a blog post for the the main page of tech.lds.org. I was really happy for him; he deserves it as a tribute to his tireless contributions to the LDS tech community.

Unfortunately, for reasons that I can understand, Mike was having a hard time coming up with what to say. I mean, there's tons you could say, but you have to be somewhat careful... after all some may mistake criticism of the software the LDS Church employees write with criticism of the LDS Church period. And then there's also the fact that Mike really cares about the tech.lds community; he's an active moderator in it.

Unfortunately, the tech.lds community suffers from the same problem any extremely homegenous community does: everyone says the same freaking thing all of the freaking time! Everyone talks about how IT is a means to an end; everyone talks about how we, as IT folk, should seek for way to further the Kingdom and be more wise servants. Please don't get me wrong: I know that what the tech.lds community preaches is true, but haven't we heard enough of that already?

Why don't we talk about some of the real issues with IT at the Church?

Here's a few ideas off the top of my head:

• How come LDS software is not open sourced? I still remember when I helped build my chapel back in Ecuador; truly a memorable experience - it was great to be able to contribute beyond tithing money.
• How come phone numbers are tied to households in MLS? What's up with that?!? And even more important, how come that wasn't fixed 10 releases ago?
• How come the IT department used to pay terrible salaries to IT employees? I recognize this is changing, but what kind of people was the IT department expecting to hire?
• Why did it take 10 years for meetinghouses to get broadband internet access? Did IT managers really just learn of white lists?
  
• Why does it take days to transfer membership records into a ward? Isn't it just a simple look up? And if it's not just a simple look up, how come it's not?
• Why is the Church developing on the Java stack? When was the last time you saw MLS running on a Linux box?
  

I've been deliberately controversial on this post because I wanted to make point: if you're going to say something, you might as well say something that's worth thinking about. I suspect there are good answers to all of the questions I've listed above; furthermore, I realize in the grand scheme of things I know nothing compared to what the fine folks at the LDS Church do. But at least, you read this far and know you're thinking about software at the Church too.

Disclaimer: I cannot say it more clearly than this: I'm criticizing IT at the LDS Church, I am NOT criticizing the Church of Jesus Christ of Latter Day Saints. To some extent, I'm not trying to criticize the folks the work in IT at the Church; I realize that were I in charge of IT things would probably be even worse than they're now.

The Secret Behind LINQ To SQL

I finally get it! It's all about expression trees!

It all finally clicked for me when I saw the declaration of IQueryable:

public interface IQueryable : IEnumerable
{
  Type ElementType { get; }
  Expression Expression { get; }
  IQueryProvider Provider { get; }
}

See that? It's right there! How could I have been so blind for so long? But I'm getting ahead of myself. Let me tell you about expression trees.

What are expression trees? Well, they're a trees with expressions as nodes. What do they do? They provide a mechanism to convert code into data (expressions). Why is that useful? Because you may want to examine and/or modify you code before execution. In particular, it would be very useful, if we wanted to take say C# code and convert it to, oh... i dunno.., SQL statements.

Say then you created the following Expression:

Expression<Func<int, int, int>> expression = 
     (a,b) => a + b;

(There's a little magic going on in the above sample: the compiler knows how to take the lambda "(a, b) => a + b" and make a delegate out of it).

The above code would literally translate to a "+" plus node with two children nodes: "a" and "b". Obviously, I've simplified the tree for clarity. If you really want to see the tree structure, fire up VS2008 and take a look at the expression tree with the ExpressionTreeVisualizer plugin.

Anyhow, now that we have our C# code in a tree, we can easily parse that tree and convert the data to a string that SQL can understand. We can then take that string, send it across the wire to our SQL server and voila: we have LINQ to SQL. Cool, huh?

Now, here's something else to think about: IEnumerable offers most of the same methods IQueryable offers, yet the declaration is completely different:

public interface IEnumerable<T> : IEnumerable
{
   IEnumerator<T> GetEnumerator();
}

Notice how IEnumerable does not have an Expression tree? That's a fundamental difference between the two interfaces. This means that IEnumerable won't do anything with your code but execute it. Therefore, if you order, filter, or project an IEnumerable collection, the action will execute in the process where the collection lives; it will not be sent to a beefy SQL box that can handle ordering large sets easily (Yes, I've made that mistake. In fact, that's what inspired this post).

Turns out my CS professors were right: trees really are useful data structures.

Why Programmers Should Have Private Offices.

Non-programmers don't get it: interruptions are a huge deal; it takes a lot of mental effort to load the program you're working on onto your brain.

Once you're in "the zone", you're effective and you write good code. Naturally then, having some dude from marketing stop by to ask a "quick question" is a total disaster; you have to literally unload a bunch of important information from your brain, just to listen to the marketing clown ask you for the 23rd time when the product will ship. Not cool. Not cool at all.

What non-programmers don't get is how hard it is to resume what you were working on. They don't understand that, to some extent, you have to load all the classes, variables, functions, etc. in your code right onto your memory; you have to enter the freaking Matrix, ant that's no easy task.

I'm guessing non-programmers think going back to writing code is as easy as going back to writing an email: you just read the last sentence you wrote and pick up from there. Well, it's not that simple.

Joel Spolsky has been advocating for private offices for developers well for almost a decade now. Yet there's few companies out there that give their developers such luxury. So, if you ever come across such perk, there's a good chance that's a company you want to work for.

From Recursion To Dynamic Programming

Dynamic programming is a really useful technique. When used appropriately, it saves processor time and memory space.

Shoot! After writing all the code for this entry, I just noticed that the Wikipedia entry for dynamic programming uses the same example I thought about using in this post. Oh well, let me just show you some code and explain just a little.

private int recursiveFib(int n)
{
  if (n == 0)
   return 0;
  if (n == 1)
   return 1;
  return recursiveFib(n - 1) + recursiveFib(n - 2);
}

The above code calculates the Fibonacci number for any integer n. The code is easy to read and effective. However, there's one problem: we repeat calls to recursiveFib even though we may have already computed a value. For example for n=4, we'll call recursiveFib(2) twice; this may not seem like a big problem but for larger Ns the number of unique calls to recursiveFib is very small. What a waste of processor cycles!

Now that we've realized we're wasting resources because the number of unique calls to our recursive methods is small, we're ready to move on to dynamic programming. Here's the dynamic code:

private int dynamicFib(int n)
{
  int[] dynamicArray = new int[n];
  dynamicArray[0] = 0;
  dynamicArray[1] = 1;
  for (int i = 2; i < n; i++)
  {
   dynamicArray[i] = dynamicArray[i - 1] + dynamicArray[i - 2];
  }
  return dynamicArray[n];
}

Even though this code may not be as succinct as the recursive code, it saves resources. By taking a bottom up approach (rather than the top to bottom approach of the recursive method) and saving our previous calculations, we never have to recompute a value again! Although, with oil being as cheap as it is nowadays, you may not really care.

In summary:
If you want to solve a problem with dynamic programming, I recommend you first solve it recursively. This may seem like a waste of time, but you need the recursive solution to spot where the improvement to the algorithm needs to occur. Once you've figured that out, all you have to do is compute and store you partial solutions as you go.