Tuesday December 21, 2004

The recent colo move that I worked on all last weekend with my cohorts at Technorati was a good demonstration where rigor pays off: operations. Physically moving an entire network requires shared knowledge of complex network systems, detailed resource allocation and troubleshooting contingencies amidst a maze of dependencies requires rigorous documentation, planning and coordination. My hats off to Matt, David, Aaron, Bill and Adam for rising to the occasion and kicking butt. It's been an honor and a pleasure, gentlemen.

I'm in New Mexico for a spell, hoping to take a side trip up to Ski Santa Fe to get a lil bit o' frozen bliss.

Wednesday September 08, 2004

Software engineers engage in a collaborative game of intellectual creation that can only be won if they can easily converse with and overhear each other. That doesn't mean disrupt each other or sit on top of each other. That simply means that the barriers to communication need to be as low as possible.

The topic has come up a lot recently as a thread spun from Joel-on-software's Bionic Office fetish for private offices. Hey, I dislike interruptions as much as anybody else but in my experience, fixed, permanent private offices for individual team members are communication barriers. Shared workspaces where team members can't turn to face each other for impromptu meetings are also a problem. A JoS discussant Brad Hill gets it

...offices are not always the way to go - sometimes the nature of the product means that a much more collaborative and shared workspace is best.
The standard-issue cube farm is the worst of both worlds, offering neither the privacy and freedom from distractions of a real office or the collaborative flow and facilities of a "war room".

Your business success will depend on the extent to which programmers essentially live at your office. For this to be a common choice, your office had better be nicer than the average programmer's home. There are two ways to achieve this result. One is to hire programmers who live in extremely shabby apartments. The other is to create a nice office.

We will create an open workspace for our team, with small private spaces around the periphery and a common programming area in the middle.

Maybe Joel had a bad experience with The Bobs and their cube farm but his big kick for putting everybody in sequestered little spaces is a recipe for incommunicado.

Monday April 05, 2004

Anyway, I believe we are entering an age of non-intrusive frameworks (unlike traditional J2EE) that serve as lightweight application servers, or lightweight containers, if you will. AOP isn't a piece of software or a container, it's a different and complementary approach to ye olde Object Oriented Programming. AOP is about layers. It's about being able to layer on supporting functionality and weave it into your core functionality. AFAIC, Sun is put on notice: your big overdone API's for J2EE aren't necessarily the best way to develop all of the components for a scalable OO architecture.

A lot of the things that J2EE provides (such as security and persistence) might be better served as application aspects, layered onto the core OO model. And technologies such as Hibernate are here to help as well.

Layers involve complexity, but that's not a bad thing. Why, there are even bits of wisdom on the matter to be found in everybody's favorite DreamWorks feature film:

SHREK: For your information, there's a lot more to ogres than people think.
DONKEY: Example?
SHREK: Example? Okay, um, ogres are like onions.
DONKEY: [Sniffs] They stink?
SHREK: Yes. No!
DONKEY: They make you cry?
SHREK: No!
DONKEY: You leave them out in the sun, they get all brown, start sprouting' little white hairs.
SHREK: No! Layers! Onions have layers! Ogres have layers! Onions have layers. You get it? We both have layers. [Sighs]
DONKEY: Oh, you both have layers. Oh. [Sniffs] You know, not everybody likes onions. Cake! Everybody loves cakes! Cakes have layers.
SHREK: I don't care... what everyone likes. Ogres are not like cakes.

Tuesday March 30, 2004

Here's my list:

source control with commit notifications

It might seem obvious but when changes are happening all engineers who have any interaction (direct or indirect) with the altered component may need to know about it. Thankfully, getting emails out of CVS is a piece of cake. Other source control systems usually have hooks for this as well.

bug tracking

Identifying issues and keeping tracking track of them so that narrative descriptions, conversations and related media (screen shots, log files, etc) can be associated with the issues is crucial. Bugzilla is E-Z 'nuf.

reproducable builds

If a lot of steps that have to be manually triggered is required to build a software project then development is going to be intrinsically slow. The build cycles should generate reproducable results and do so in a scripted framework that can ultimately kicked off with one command.

unit testing

I'm a big fan an keeping as much business logic as possible out of the web runtime environment. If a web server,servlet container, EJB container and/or database is required to be running with your code deployed to run the tests, then you've been writing integration tests, not unit tests.

integration tests

Of course, your code has to work in the runtime environment of your application, so "in-container" tests such as Cactus are crucial.

functional tests

Ultimately, an end-to-end test that goes from HTML and HTTP, through the business logic and frameworks and down to the database is important to validate the system as a whole. This is where things like HTTPUnit come in.

automation

If you use something like Cruise Control to take care of updating from source control, kicking off builds, running unit tests, integration tests and functional tests with as high a frequency as possible, you've got a pretty robust development environment.

test coverage calculation

Not everything requires a test but there whenever there are semantics, the business logic should be tested and whenever there are environments to deploy to, there should be a level of integration testing -- I don't know what tools will effectively give you that stuff (send it to me if you know!).

a wiki

It's useful to have a central repository of facts about the project. Best practices, HOWTO do weird things type documents and stuff like that should be kept as authoritative documents.

a blog

A lot of software design decisions are made in face to face conversations and email threads. It's difficult to retrieve the original intentions and the pro's/con's discussed in that way. I've seen email threads pasted into bugzilla reports and wiki pages as a work around. However it seems to me that if the wiki pages and bugzilla reports instead linked to the blog entries where these conversations took place, they would be easier to keep track of, search for and so forth. Of course, a lot of decisions are still going to be the outcome of face to face and email conversations but they can still be synopsized and made into blog entries.

• How many software commits has each engineer made?
• How many different files were they commiting changes to?
• How many bugs were created?
• How many bugs were closed?
• How complete (percentage wise) is our test coverage?
• How many broken code artifacts were checked in (i.e. put the code base in a state when it wouldn't even compile)?
• Who did it (are there any grossly flagrant offenders)?
• How many builds with all tests run have completed successfully?
• How many failed?
• For the failures, what was the average time to resolution?

In the end, I think getting all of these parts working together will render a whole greater than the sum its parts. Getting a collaborative environment functioning isn't easy but a lot of these tools are readily available off the shelf to help it along. ( Mar 30 2004, 03:04:12 PM PST ) Permalink

Wednesday March 24, 2004

The funny thing was, the code in question sound like the kind of stuff that you'd typically think of as refactoring candidates. A lot of it is apparently old CGI's (run as Plain Old CGI's or Apache::Registry scripts) that atrophied into speghetti of gobbledy gook. So the problem there isn't one of taking software objects and restructuring them for better design, the first refactoring, if you will, is to take a bunch of procedural code and re-compose it as an object system.

Frankly, this could sound really boring. But on the other hand, there can be something very satisfying in something like this. Have you ever set aside an afternoon or a day to clean out the garage? You sort through a bunch of junk, shelve things that were set down on the floor, throw out a bunch of garbage, fire up the shop-vac and vacuum out all of the cob webs and organize the toolchest. In the end, you haven't produced anything tangible with any value. But you've created something that allows for creating value. I would never go into the garage and build a planter box if it's all cluttered and dusty. It's just too grotty.

Code gets the same way. It gets cluttered and dusty. Refactoring is explicitly not about creating tangible value (i.e. you don't add new features in a refactoring), it's about creating an environment within which tangible value can be created. Fowler sez that refactoring is more than just "cleaning up code" - it's a more methodical, tested and controlled fashion of cleaning up. That's all good. Unfortunately, it seemed as though I drew a blank stare from these folks when I talked about testing. I mean, I understand that test harnesses in Perl such as plain old Test, Test::Unit and Test::Harness are hardly standardized or well entrenched in Perl culture but it was kinda disheartening to see only vague recognition of the value of writing tests. When you're done cleaning the garage, there's that satisfying feeling knowing that you've brought a little order to a chaotic corner of the world. I get that same feeling when I'm working on code, breaking it and fixing it and when it's all said and done, running the test harness; see "100% passed" is another way of bringing order to the chaos.

Anyway, so the basic starting points for refactoring were absent: a system of objects and a test harness but if the goal of refactoring is to "make the software easier to understand and modify" (quoting Fowler, again) then perhaps just getting things to that starting point is the first important refactoring to undertake. Somehow, I don't think the connection between the "fire fighting" mode that these folks were perpetually in and absence of tests was immediately clear to them. They know that "things" need to be fixed (which is a good start) but IMO those fixes need to come in little increments of refactoring.

Now, I've never unit tested a garage cleaning effort but today's conversation made me consider how that'd be done. ( Mar 24 2004, 10:43:06 PM PST ) Permalink

Monday March 22, 2004

Martin Fowler recently wrote an article about this. In discussing the use of offshore programmers with his company, one of the things he mentions is using Cruise Control to aid continuous integration of multiply located collaborators. This sounds good. I've seen lots of problems with Cruise Control grinding down with OutOfMemory errors but I'm more suspicious of running in-container tests like Cactus than I am of Cruise Control per se.

An article I read in the paper the other week mentioned that the help-line calls for the welfare and food stamp programs in California are routed to offshore call centers in Mexico (for Spanish speakers) and India (for English speakers). What would be grimly humorous is if the State of California's Employment Development Department started offshoring its labor. ( Mar 22 2004, 09:35:16 PM PST ) Permalink

Friday March 12, 2004

There are a lot of different flavors of agile development. Some of them work best, ideally anyway, if all of the practices in that particular flavor's agile "recipe" are adhered to. It's been my experience that most situations are less than ideal. People come to work with a lot of preconceptions about the right way to do things or with their own ego issues that get in the way of buying into an agile methodology lock, stock and barrel. However, I found the book Agile Software Development Ecosystems (by Jim Highsmith) very thought provoking and led me to conclude that depending on the context of the workgroup, there may be no perfect fit for a particular methodology but not to despair. If you can get people to agree upon principles and values about what they want to accomplish, values that are open to new ideas and not doctrinaire about waterfall processes, it is possible (and in fact likely) to create an agile environment borrowing what is needed from one or many of the established agile development flavors. A properly established environment should be generative of agile practices and that's what I'm really looking for in my work place - instead of rigid rules, agree upon principles and communication norms and the appropriate practices will become clear as everyone's work styles come into play.

At least, that's my current thinking on the matter; I'm open to new ideas. ( Mar 12 2004, 10:53:22 AM PST ) Permalink

Wednesday February 25, 2004

"If your job involves sitting in a cubicle and not talking to anyone else, you are at risk. Get yourself into a role that is more attached to the customers or suppliers and is on the cutting edge of innovation."

Milton