The ByteBaker

If you’re a computer science student like me, then you probably write a fair amount of code. Depending on your coding style you probably write a fair amount of comments too. But do you write about your software or do you just write your software? Chances are you don’t. College computer science courses aren’t exactly writing courses (no matter how much typing you do). While part of a project may involve documenting the concepts behind your program, or why you made a particular design decision, you hardly ever write something like a manual for your software. However, if you’re planning to write production software at some point, especially if you’re thinking about creating something independently or as part of an open source project, then writing a manual might be more helpful than you think.

Writing a manual is more about than just putting together instructions on how to use your software. It’s important to put yourself in the place of someone who will be using your software (and is probably not a programmer). For you your program logic makes perfect sense (if it doesn’t, why is it even there?) but sometimes what makes sense from a programming point of view isn’t quite so simple from a user point of view. Looking at your program from a user’s point of view let’s you think in terms of what’s really useful and necessary rather than what’s easiest to program. Writing a manual is a good way to put yourself in the user’s seat.

The first things that come to mind whiel writing manual is the user interface. How is easy is it to do simple tasks? Where are the most important parts of your UI located? Is their a general cohesiveness in the way things are put together? Are there some things that you know your program is capable of, but are almost impossible to do due to the UI? Theses are all questions that directly affect how your users’ experience. And so they are top priority. If you find yourself taking up two pages describing something that the user will be doing dozens of time each day, then something needs to be changed.

A critical review of the interface will lead to deeper program logic. Are things which appear close together and related in the UI similarly close together in the program? Are the UI and the program logic equally clear and intuitive? Are you unintentionally exposing internal structure to your user? While writing the manual you should keep an eye out for any bad designs, you should also be thinking about missing or broken features. While that doesn’t mean you should implement everything that comes to your mind, sometimes some features are useless without supporting features. After all, your clients aren’t interested in the features themselves, they want to get their work done. If you find yourself writing about how not to do things or what can’t be done in your manual, rather than how to do them, then your program probably needs more work.

Having a written manual for a small art program, I’ve realized that there’s a fairly direct correlation between ease of use and ease of description. If you can explain a function and how to use it in a few lines, it’s probably mcuh better implemented than a description that takes up half a page. The first version of my program was a fairly tedious text-based program which went out of style around the same time as Apple BASIC stopped shipping. The descriptions for each option and how to get to them were equally complicated. Putting together a simple GUI not only made interaction easier, it also choppped down the interaction part of the manual to a few bullet points.

However, when you’re writing a manual it’s fairly easy to write from a developer’s perspective. After all, you have the guts of your program in your head, and you probably already know half a dozen unobvious little tricks and workarounds. You’ll either forget to put them in the manual or you’ll remember and think that they’ll make as much sense to others as they do to you. Chances are they don’t and you really don’t want to be putting workarounds in your manual if they’re hiding a bug. Fix the bug and then write the rest of the manual.

I’m starting to think that all students should at least once have a project that requires writing a user manual for non-programmers. Years of writing code and tearing through algorithms can make you think very differently from the average office worker using your program. It’s necessary to get in touch with your users and how they’ll be using what you give them. So the next time you write your great new world changing program, take a few hours off and write a manual for it.

As my research project continues our software is gradually growing more and more features, and we’re adding new functions almost everyday. While it’s certainly a very good feeling to be adding functionality, it can also get ungainly very quickly. About two weeks ago, Slashdot posted a very interesting query by a person interested in doing a PhD on software design principles. Since then, I’ve been giving more and more thought to the role that industry-style software engineering principles play (or should play) in computer science research.

Research in science has been mostly held to be separate from engineering. Chemistry research is not performed in quite the same way that chemical engineering is. Even in cases where there is significant amounts of high technology equipment is used (the LHC is a good recent example), the engineering teams are separate from the scientists. However, computer science is not really a typical science at all. Research in computer science faces many of the same problems that software development faces:

• Multiple people working simultaneously on large and growing codebases (team management).
• Adding or removing features without requiring major rewrites (extensibility).
• A way to collect large amounts of data and detect failures (testing and error logging)
• Having multiple copies of the experiment running, with different parameters (scalability and version control).
• Last, but not least, the experiment should actually work and produce results (in corporate terms, the software should ship).

Keeping the similarities in mind, what lessons can we as researchers learn from industry techniques and practices (by industry, I mean open source as well)? I’m not sure what other researchers do, or if there is some sort of standard procedure, so I’ll just talk about what my small team has done to make our work easier.

Divide and conquer:

The first thing we did was to carve up the various parts of the project. While my two teammates took up the core engine and the output system, I chose to deal with the human interaction part. At the same time. we had regular meetings where we planned out how our parts should interact as well making high-level descriptions of what each part of the code should do. This meant we could each develop our code largely independent of the others while still having a fair idea of where the whole project was going.

Throwing away version 1:

Though our first version was a working program, we made it keeping in mind that we would probably have to change a lot of things, possible even rewrite it. As a result, we developed it quickly, and focused on making it “just work” without adding too many features. This allowed to get a good feel for what the rest of the project would be like and let us make important decisions before we had invested a lot of time into it. In research you often don’t know what you’re looking for until you start looking, and this was the case for us. Our throwaway version gave us a better idea of which direction our research was heading, what concepts had been explored before, what was new and how difficult various aspects were.

Strict Version control:

Right after we started working on our post-throwaway system, we moved all our code to a Subversion repository on our college’s research cluster. We do are own work in our working commits and perform daily commits and updaes. Our repository is also divided into trunk, branch and tag directories to keep separate the most recent copy, older releases and other milestones. This way we can see the evolution of our project at a glance more easily than having to check out older versions. This comes in particularly handy, since being a research project we tend to be looking at what we did before and what results we got rather regularly.

Coding conventions:

Another post-throwaway decision was regarding coding styles. We fixed general naming and commenting styles, decided on tabs vs. spaces (4-space indent, in case you’re interested) as well as a number of minor things that weren’t project-threatening, but could have led to unnessecary confusion later on. We also keep detailed records of how we’ve changed our codes and any bugs we’ve introduced/fixed on a common wiki meaning that everyone is always up-to-date on changes.

Personal freedom:

By standardizing some aspects of our project, we’ve been able to have greater freedom as individual researchers. All our code is in platform-independent Python. Besides that our only other common tool is Subversion, which is also freely available and easy to use. By choosing open source and platform-independence, we have been able to have three developers work on three different platforms using different editors, IDEs and SVN clients. We can work equally well on our own machines sitting in our rooms, on the computers in our CS department’s labs, or even from the Mac Pro computers in the Arts buildings via SSH. It’s hard to quantify how much easier this independence has made our work. This has meant that we didn’t have to rush to the CS Lab’s whenever we had a thought or problem and we could whenever we wanted, however we wanted as long as we stuck to some simple rules. Scientists and programmers both need a certain amount of freedom and we’ve been able to secure this freedom by a proper choice of tools and techniques.

Things to do:

It’s been almost four weeks since we started using software engineering practices for our research work. We don’t know how things would have turned out had we not implemented the practices that we have. However we have come to rely heavily on the framework and guidelines that we have built for ourselves. At the same time, we’ve come to realize that there are parts of our project that could benefit from adopting some more industry-standard practices. I’ll end with a brief list of the things we plan to implement in the next few weeks:

Robust error handling: We’re currently using some very ad-hoc methods to handle errors and track down bugs. But we’re in the process of building a flexible, yet robust error handling and reporting system into our project. Since we intend our tool to be used by non-programmers, we need to implement a system that won’t mean looking at the code everytime something goes wrong.

Flexibility via a strong framework: Our work has turned out to have more applications than what we originally intended for it. As a result, we were considering forking our code into two separate branches. Though we still haven’t reached a proper decision regarding this, I’ve personally been trying to redesign our project into a “framework + dynamic modules” system. This is purely a software engineering effort, since it will mean a lot of refactoring and restructuring, but will not have much of an external effect rather than making our job easier.

Unit testing and other automation: Till now, an important part of our work has been getting user feedback, partly because our system is very visual intensive and also because we weren’t quite sure what we were looking for. Now that we have a better idea of where our project should be headed and what we should be looking for, we can begin to automate some parts of our work. However we won’t just be checking for bugs, but rather generating a large number of results and then further improve our system to get what we want.

I’d love to hear from anyone who’s been involved in computer science research projects and has opinions of how to use software engineering principles to make things easier/more effiicient.

For the past few days, I’ve been looking into Apple’s Cocoa infrastructure in some detail. The reason once again stems from my research projects. All our code is in platform-independent Python meaning that we can develop on any of the 3 popular operating systems and then run on any others. This has been a great advantage since the three members of our team each run different OSes. For most of the time our project was purely command-line and text input based, using graphics only for the final display. However we have now come to the point where we need to fork our system into two different systems: one for architects and one for artists. How each program will develop is something that we will work on over the next few weeks, but one thing is clear: for the artists at least, a purely textual interface will simply not do.

Till now our system had been accepting an instruction file as input. While this method of using a instruction was very powerful (we could essentially embed our own programming language), this not the direction we want to go with the artists. And besides, not every artist wants to sit down and right pages of instruction to draw a picture, we might as well ask them to write the program themselves. It would be best to create a GUI for them to use. Even before this had been clear, I had written a simple interface that was essentially a text editor containing a hook to our Python backend. It was written in Tk, was exceptionally ugly, but got the job done.

Now I think the Tk toolkit is capable of letting me create the entire GUI interface that I have in mind and it will be cross platform as well, but there’s one problem: it’s ugly, especially on the Mac. I think this is because the Python Tkinter version doesn’t use Aqua (like Tcl/Tk), but rather uses X11. Though I spend a lot of my time in Vim and at the command line, I’m still a fan of good looking user interfaces (see my last post). As a result I would feel rather guilty about making something that I know would look bad on the platform it would be used most. So like a good programmer off I went to find a solution.

I thought I would be able to use Cocoa to create a front end, especially since Xcode now has support for the PyObjC bridge. Unfortunately this has turned out to be much harder than it at first appeared. Cocoa is Apple’s very powerful object oriented API. It is almost everything you will ever need to create a Mac application. While it allows to create native applications by leveraging the powerful infrastructure Apple provides you, it also doesn’t allow you to perform the sort of rapid duct-tape styling programming that Python and UNIX encourages. In Python it’s very easy to write your core application in Python with any number of GUI front-ends that are easily interchangeable. Cocoa, on the other hand, is not another GUI toolkit that you can simply add to the mix. If you want to use, you’ll have to invest a certain amount of time in learning your way around.

Python can’t communicate with Cocoa directly. To  have your Python programs leverage Cocoa, you need to go through the PyObjC bridge. Though the bridge is certainly effective and I could use it, I’ve been having troubles finding decent documentation. Apple’s own tutorial is the sort of document I’m looking for, but it is dangerously out of date. Lacking a proper tutorial, I would have to figure out a lot of things on my own. Once again, a large time investment.

Considering that I have only two weeks left to make real progress on my project and that learning Cocoa or Objective-C was not my original goal, I’ve decided not to pursue that avenue at the moment. I would certainly love to learn Cocoa at some point in the future and PyObjC seems like it could become a very powerful tool. But I simply don’t have the time at the moment. I am going to start working on a proper GUI interface in Tkinter. Perhaps sometime later this year, I’ll get back to some Cocoa.

I installed Firefox 3 yesterday on both my old G4 Mac and my Windows laptop. Arch linux, which I normally use doesn’t have a package for it yet. Firefox 3 has a number of new features, which are being talked about on almost any random technology site, so I won’t bore you with them. I’ll just focus on one thing that I like: the changes to the user-interface.

Firefox 1 and 2 sported a UI that seemed to be uniform across platforms. However this uniformness across platforms also meant that Firefox didn’t quite look like the other applications for the platform. This isn’t really a big thing, especially considering the horrible UI inconsistencies introduced in Windows Vista. But at the same time, it’s little touches like this that can make or break an app. Sure there were skins, but you really want something like that to builtin. One of my favorite pet peeves was regarding Firefox 2 on OS X. The buttons for various web apps like Gmail or the Google search page’s search button would show up as ugly Windows 95-ish rectangular boxes rather the sleek Mac buttons. It was something that really irritated me, since I spent a lot of time in Gmail. I’ve been glad to see that the new Firefox fixes that and adopts a very native look and feel.

I also upgraded my G4 Mac to OS X Leopard today. I had been putting it off since it has only a 1.25GHz processor and less than a gigabyte of RAM. However, it turns out that my fears were mostly unfounded. It took about 20 minutes to install and it runs as fast as Tiger did. The only noticeable difference is that the boot and login times are somewhat longer. But considering that I rarely logout or shutdown, that isn’t really an issue at all. I’m surprised by how Apple’s development takes into account older machines and actively works to extend their lifetimes. This stands in sharp contrast to the effect that Windows has as the Great Moore’s Law Compensator. In fact I will go as far as to say, that some things actually run better on Leopard than on Tiger, the Leopard-Firefox 3 combo in fact seems to very capable.

On Tiger I always experienced Flash videos being somewhat choppy irrespective of what browser I used. That problem no longer exists on Leopard. Furthermore, the UI changes to Firefox fit right in with the overall darker look that Leopard sports. I’m really looking forward to using Spaces, which should come in really handy as I start doing Cocoa application development. I was also impressed by how little impact Spotlight scanning the disk had on performance. I would never have known it was going on if I hadn’t tried to use it. There are some features of Leopard which I won’t be using such as Time Machine ( I have a homegrown SVN backup system), and that will probably mean my system won’t be under a full load anytime. If it turns out that my system can’t handle (considering that I’ll be using Xcode fairly heavily), adding a gigabyte of RAM might help speed things up. I had been considering buying a new Macbook to last me until the end of college, but it seems like my current machine will be all the Mac I’ll need for a few more years (or at least until Snow Leopard hits the stage).

I’ve just returned from meeting one of our clients for my summer projects: a professor in my college’s art department. What we have developed over the last three weeks is essentially a way to stitch together fixed images into probability-controlled tilings. However, it turns out that what our artist friends really need is more fine grained image control. They want to be able to create much more interesting patterns combining both images and geometric patterns. As I was explaining how our program worked, I realized that our instruction language model was rather complicated and not really suited for what the artists needed. So it seems, that our program is really not a very good match to what the artists need.

At this point, there are a number of things we could. We could go back to the drawing board, start from scratch and rewrite our system according to what the artists need. We could also try to build in top of our existing program and try to add and modify it to bring its functionality close to what we need. However, we need to keep in mind, that we had also planned to use this system to create a automated urban planner. We still haven’t met with the architect yet, but it seems like our system might be a better fit for what she needs. But that doesn’t answer the question: what do we do?

Now if this were a real software project with a delivery deadline, this state of affairs would probably be a very bad thing. Luckily this is still very much a research project so there’s quite a lot of leeway. Unfortunately, we’re still not quite sure of what exactly we have to do. It seems that our general architecture is still valid and will work, however the details will have to be remade. We’ll need a new way to generate the patterns from the instructions, simply stitching together images won’t do. It is possible that we will need multiple drawing systems working on top and parallel to each other. The instruction language also needs to be changed to make it more suitable to the artists’ need. We currently have a system that is geared towards placement of individual large scale elements, and nothing that deals with simple pattern generation. Ideally we would need a blend of both.

At this point, it’s easy to get disheartened. After all, we’ve spent almost 3 weeks creating something that we just learned is pretty close to useless. On the flip side, we are certainly not going to get bored. From a personal standpoint, this has still been a success, I’ve learned about parsing methods and language design. The experience that I’ve gained will make it easier for me to help develop a language that is powerful enough to express everything that the artists want to, while not requiring them to spend days learning it. It also means that the next version will also hopefully be developed much faster and more smoothly than the previous two versions. I am going to make it my personal project to see how fast and efficiently we can develop subsequent versions.

is not very easy. As a programmer, you learn to use various pieces of software, like text-editors, compilers, debuggers, profilers, so on and so forth. All these tools are very useful, some are essential, and many of them have a learning curve, you have to invest time into learning the nuances of the particular tool if you want to use it to the maximum potential. Consequently when you are writing software that you will be used by other programmers (even if it is not a programming tool), making the software easy to use is not always the first thing that comes to mind. Let’s solve the problem and then we’ll worry about making it ‘user-friendly’. User interface gets pushed onto the backburner. The UNIX software user space is filled with various command line programs which are very effective in their own right, each with its own host of little options toggled by specifying combinations of letters and words on the command line. Not a problem for us coders, after all, text is what we live in. Not so easy for non-programmers who find the very idea of typing in commands some sort of deep black magic. Hence the rise of “graphical frontends”: point-and-click interfaces that put interaction with these text utilities in the background.

For a while i was of the idea that the GUI was for wimps. If you wanted to harness the real power inside your computer, you had to get down and dirty with the command line. However I’ve since come to realize that the real problem is not one of interface, but of thought style. I’m currently a program that (I hope) will be used by artists and architects. It is text-based and the user needs to type in configuration files. No point and click GUI (yet). I chose to take on the job of writing the front-end: the part of the program that the user would actually interact with. This meant creating a configuration language that would be easy for non-programmers to write. The language needed to describe certain elements and how those elements could be combined. My first thought was to create some sort of CSS clone for element description and something akin to Prolog for combination rules. Now, if I was writing this for programmers, I wouldn’t have given it a second thought. But I was writing software for programmers, I was writing for artists. Since then I’ve changed my language to more like Basic than any other programming language. I’ve tried to make it as close to plain English as possible, with no special symbols except for parentheses. The reason for this is simple: any configuration language, no matter how simple would mean that the artists would have to learn it’s uses. I would rather have them up and running and actually using the tool than learning my arbitrary syntax. By keeping the language as close to English as possible, I’ve tried to minimize the learning curve.

Error messages are another area where I’ve had to keep in mind that my users won’t be programmers. Compilers are notorious for having hard to read error messages. I couldn’t have my config file parser give equally esoteric error messages. Most of my parser is a simple recursive descent parser that just does simple syntactical analysis, however a part of it involves solving a system of linear equations. Once again it was very tempting to have the error messages speak in terms of the underlying logic and math, which to a programmer at least, isn’t very complicated. However, I made the conscious to have the error messages be plain English as well. The user gets a line numbers and a plain English description of what went wrong. By leaving out allusions to the math and logic, I may have made it harder for the user to fix an error in the configuration file, but that’s a risk I’m willing to take for the time being.

Programmers are instinctive tinkerers. We like playing around with things. Even when things are working right, we have a need to get in there and try to make it work faster, better, or just in a different fun way. We adore customization and love extensibility. Simply getting the job done is not enough, we want to find the most elegant, the fastest, the most efficient, the craziest way to do it. Living in our world of programmable text editors, modular IDEs and extensible browsers it’s easy to forget that often people just want software that gets the work done and stays out of their way. They don’t want unlimited power, they just want the damn thing to work.
Exercising this sort of restraint is not easy and it requires a lot of thought. I don’t want to give the user something that will overwhelm them, but at them same time, I don’t want to give them a crippled half-hearted piece of work either. Unfortunately there’s is no way to tell what a user wants or needs before you actually give them something to play with. Our software is at a tipping point: it has a small set of core features which work well, there is room to add much more, but we don’t know what will be essential and what will be extra cruft that no one ever needs. And so we are heading to meet our “clients” next week and hopefully at the end of the meeting we will have a new goal in mind.

Also known as “eat your own dog food”, this is the concept behind one of the most successful software engineering projects of modern times: the Windows NT kernel. The Windows NT kernel was written by a highly-talented team led by a man who is arguably one the best software engineers of all time: Dave Cutler. Dave Cutler was also the lead developer for another groundbreaking operating system: Digital’s VMS. However there was more to this project than talented developers: the whole team actually used Windows NT everyday as soon as possible. This meant that the developers exposed themselves to problems that would be encountered by the average user and could then fix those problems before actually shipping the finished product.

Let’s face it: software is buggy. We still have no clue about how to reliably write bug-free software. So we’re stuck with the situation of writing buggy software and then wrangling the bugs out of them. A lot of bugs are removed through the process of just writing a working piece of software. Automated testing also gets rid of a fair amount of bugs. However, there are some things that no amount of debugging or automated testing can get rid. Modern software systems are large and complicated and it’s hard to tell exactly how all the different parts will interact until you actually start using it all.

Even if you’re certain that your code is relatively bug-free, it’s still important to use software that you’ve written. There are a lot of things about software like look-and-feel, intuitiveness, ease of use, which can’t be determined automatically. The only way to see if your program has an elegant and smooth interface or is powerful, but clunky is to use it repeatedly. When you start using software that you’ve written on a regular basis, you start to think about how your software can be improved, what are the bottlenecks and hard-to-use features, what features are missing and what are unnecessary. This constant evaluation is a key ingredient of making better software.

Unfortunately, most of the software that is being created is made by developers who really don’t know how that piece of software is going to be used in the real world. After all, Photoshop wasn’t made my a team of artists and most users of Microsoft Word have never written a line of working code. So how do programmers go about creating software that they might never use? Enter the beta tester. Beta testers are given pre-release versions of software to evaluate and their suggestions are then folded back into the next release. The best beta testers are the very people for whom you’re writing the software in the first place. If you’re writing a software package for a specific client, then it is essential to have a continuous dialog open. Test versions should routinely be given out to get feedback and then that feedback should be incorporated into the next edition. If your software is for a mass market, then your user community will be your pool of beta testers, encourage them to give feedback and then take those opinions into account. Eating your own dog food is a good idea, but it’s not a disadvantage if you can give it to a bunch of other dogs and see what they think of it.

One of the best books on software engineering is Fred Brooks’ The Mythical Man Month. Even though it’s over thirty years old, most of the principles and ideas laid down are still applicable, some even more so now than thirty years ago. One of the ideas that I feel deserve special mention is that of the pilot system.

Like many great ideas, this one is very simple in essence. The first working system created as part of a software project is one that will be thrown away (intentionally or not), in the same way that chemical engineers will first build a pilot plant to test out their process before investing in a full scale plant. The reason for this is that it’s hard to get a grasp for what the real problems of a software system will be before some of it is actually implemented. Keeping this in mind, it’s best to start projects with the idea that version 1 will be thrown away and schedule the project accordingly.

The benefits of starting with a “throw away version 1″ philosophy are many and can help the development of any software project, large or small. Knowing that the first working version doesn’t need to be perfect lets the development team quickly create a prototype to explore the different aspects of the project. It allows experiments to try out various solutions to a problem before deciding which one to use. If you’re making software for customers (as most people are) having a working model early in the project’s schedule allows you get valuable user feedback and make changes accordingly. It’s often said that software users don’t really know what they want, but it makes everyone’s life a little bit easier if there is something tangible that can be pointed to and specific aspects of it praised or criticized.

Of course, knowing that you’re going to throw away the first version doesn’t mean that this version should be made sloppily or inconsistently. The same technical rigor should be applied to the prototype as to the final delivered product. In fact, making version 1 in a half-hearted fashion defeats the entire purpose of having such a version in the first place. Problems in the design of the product will only become apparent if version 1 reflects fairly accurately what the final is supposed to be. Engineering models are made to the same specifications and requirements as what the final should be (albeit on a smaller scale) and there is no reason why software prototypes should be any different.

I’ve had the opportunity to see the benefit of this approach first hand. We recently completed version 1 of our summer research project. Though we had known that the final version would probably be very different, we worked under the assumption that version 1 would be the basis for the later version and that if we screwed up now, we would have to pay for it later. The result is that version 1 really was a proper working system, which works (from the user perspective) in a manner very close to how we want the final to work. Of course, we are still throwing away version 1. Even though it works fine, we’ve seen that there are a number of fundamental flaws to our approach. The ad-hoc parser that I wrote for our configuration language works, but is ugly and inflexible. I’ve already replaced it with a more robust and flexible recursive descent parser. The output mechanism is currently hard-coded into the rest of the system, but it should be pluggable; we now have an idea of what sort of an architecture we want. In fact, the very core of our system will probably have to be replaced or at least changed substantially, because we just realized that what we had written wasn’t really what we had set out to do. All this might sound like a disaster, and it would have been if version 1 wasn’t designed with the idea of eventually throwing it away. We used it as a test bed and learned valuable lessons from it which we’ll now apply to the creation of version 2.

Unless your project is something very trivial, it’s best to start with the idea of throwing away version 1 and learning from the lessons that it will inevitably each. There is another software engineering principle that runs complimentary to “throw away version 1″ that can be best described as “eat your own dog food”, but that’s something for another post, maybe tomorrow. Till then, have fun with version 1.

For the past week I’ve been working on my first multi-person, long term software project. As part of my summer research project we’re building an urban planning system which automatically builds sample cities from a set of design conditions. I’ve been working with two other people and my professor, and though it isn’t a large scale project like writing an operating system, it’s still quite challenging. One of the major challenges that I’ve come to face is that it’s a very hard to stick to any designs that may have been made before actual writing of code. Furthermore, in a research project such as the one I’m working on, the problems aren’t well known and we’re discovering them as we go along. This results in the problem that I’m continually rewriting parts of my code and adding parts as I go along.
In class we were always taught to plan out our programs in a engineering fashion, and then implement. I’ve tried to follow this plan-then-implement methodology, but I’ve to improvise quite a lot. So I’ve been wondering, what’s the best way to make large pieces of software: grow them or build them? Building software is what I’ve been taught in class: where we make a  plan and then follow it. Now anyone who has written a sizeable amount of code will now that it’s impossible to plan for all eventualities. That’s what debugging is all about: dealing with things that you didn’t plan for. However debugging deals with errors, not with the basic structure of the architecture. I’ve previously advocated carefully planning out algorithms before implementing, and i still to that. But when it comes to actually building large software systems, it’s important not to try to plan everything out. I’ve come to realize that it’s best to combine a moderate amount of planning with a technique of organically growing your software.
What do I mean by growing? It’s simple, start with something small, that you know works (and that you’ve tested thoroughly) and keep adding on to that. You keep adding functionality one bit at a time, getting closer to what you finally need. This method has a two fold benefit, firstly it makes sure that you have a working program before you starting adding more functionality. That lets you rollback to a working program whenever things get out of control. Secondly, it leaves you free to add functionality and deal with complications without having to worry about fitting things into a preconceived plan.
A well conceived plan is good and essential to a successful software project, however it’s good to leave room for the eventualities that will inevitably occur, Growing software and building things on top of an already working base is a way that allows creation of software in a structured manner that leaves room for dealing eventualities.