I signed an oath

Alistair Cockburn is a well-known programmer who is fed up with
infighting in software development. So he came up with an Oath of Non-Allegiance.

"I promise not to exclude from consideration any idea based on its source, but to consider ideas across schools and heritages in order to find the ones that best suit the current situation."

Makes sense to me, and for more than software development.

Hidden depths, and Things That Will Never Be

A favorite xkcd cartoon and some things that will never be.

• A Unicode book that is both concise and useful
• A protocol with "Simple" in its title that has remained simple - SNMP, SMTP, SDLIP are just a few that have not.
• Revision control merges with absolutely no conflicts

~Matt

Action, Expectation, Observation

I remember writing some parts of Practical Development Environments vividly, despite the sleep deprivation that writing a book and having a new child will produce. One such section summarizes what goes into a good bug report:

The three key points to bear in mind when creating a bug report should be:

• How to reproduce the bug, as precisely as possible, and how often this will make the bug appear


• What should have happened, at least in your opinion


• What actually happened, or at least as much information as you have recorded

I still like what I wrote. And now I think it's good advice for other situations. For example,

• Action: I was taking my brother's toy from him. It was his new toy.


• Expectation: He should have let me take it because he wasn't using it anymore


• Observation: He hit me!

Maybe I'll get it printed on my next batch of business cards:
Action, Expectation, Observation.

~Matt

A fine and confusing error message from Java

public String myFirstMethod() {
    String result = "Hello world";
    return result;;
}

A straightforward piece of code that rudely produces a compile-time error "unreachable statement at line 3". "What's so unreachable about that line?" you might ask. Those experienced in the way of Java compilers simply smile and point out the double semi-colon. Accordingly to the compiler, there is an (empty) Java statement between the two semi-colons and it is never reached because the methods always returns before it reaches it.

If I had written writing that error messages, I might have made it say "double semi-colon on line 3".

How irritated customers obey a power law

There are over 4000 open issues about JIRA that people have voted for. The most popular vote has 592 votes but most issues have zero votes. I found a suprisingly good line to fit the data with a correlation coefficient 0.9882, if Excel is to be believed. I'm not sure that I actually believe in predicting what people vote for, but I guess this is one way of measuring your progress is how the coefficients of the predictor change over time. The flatter the line, the better job you are doing of evenly spreading out customers' irritation with your product.

As of July 2010, the coefficient of irriration for JIRA is -0.65.

Drawing a line in the sand

Joel Spolsky writing on his latest project concludes with

"Area 51 is not for everyone. If you don’t know what it’s for, or why it’s going to work, or you can’t figure it out, it’s not, actually for you."

Snark programmer, snark!

I'll not spoil the punchline

Too true.



From Geek and Poke

One Bug, Multiple Branches

I enjoyed presenting and discussing some new ideas about how to use JIRA to track bugs in different branches along with their associated builds. This was all at the April San Francisco Atlassian User Group last night, kindly hosted by Atlassian.

My idea is to create a new issue type named "Build" and then create a new build issue for every build. Ordinary bugs in JIRA can then have regular JIRA links named "Present in Build" and "Absent from Build" connecting them to the specific builds in which they are found or fixed. The interesting part is that if you link the Build issues to other Build issues, you get a tree of related builds such as the diagram above that you can then use to deduce which bugs are present in any given release, and also which releases a given bug is likely to be in. Useful information that is rarely available in current issue trackers.

My presentation is available via Google Docs. Feedback welcome, as are beta testers for the ideas in JIRA.

Verification code for Empire Avenue: EAVB_KMCAYMVSLW

Software Engineering and Academia

Bertrand Meyer has an excellent article about why academic computer science has a smaller impact than other academic disciplines do in their respective fields. As he puts it, "academic research has had its part, honorable but limited."

"Researchers in experimental physics or mechanical engineering employ technicians: often highly qualified personnel who help researchers set up the experiments and process results. In software engineering the equivalent would be programmers, software engineers, testers, technical writers; in the environments that I have seen, getting financing for such positions from a research agency is impossible."

"The only software we can produce, if we limit ourselves to official guidelines, is demo software. ... many of us work around the restrictions ... by spending considerable time away from research on programming and maintenance tasks that would be far more effectively handled by specialized personnel. The question indeed is efficiency."

For me, that's what a software toolsmith does.

How to change your OSX terminal color for ssh sessions

Sometimes I have lots of terminal windows open for ssh sessions to multiple machines. It can all be a bit confusing even if I use the remote machine name in the shell prompt. So I got terminal colorization working for me with help from here and here.
There is a default color for all screens that are using ssh and you can also customize the colors for different remote machines. When you end the ssh session, the terminal returns to black on white.

This is for OS X because it uses the osascript command to change the colors but the rest of the shell script should be just fine for Unix.

$ cat /usr/local/bin/ssh
#!/bin/sh

# Color the OSX Terminal app according the hostname used in ssh
# Assumes you log in with a simple "ssh myuserid@remotehost"
# Place in a directory such as /usr/local/bin that is before the real ssh location

# The value of whatever was after the @ in the command
HOSTNAME=`echo $@ | sed s/.*@//`

set_bg () {
  osascript -e "tell application \"Terminal\" to set background color of window 1 to $1"
}

set_text () {
  osascript -e "tell application \"Terminal\" to set normal text color of window 1 to $1"
}

# Return to black text on white background
on_exit () {
    # echo exited
  set_bg "{65535,65535,65535}"
  set_text "{0, 0, 0}"
}
trap on_exit EXIT

# My color names
#  green   10000,40000,40000 
#  blue    10000,20000,60000 
#  orange  60000,40000,00000 
#  red     60000,00000,00000 
#  yellow  60000,60000,40000 

case $HOSTNAME in
  fisheye) set_bg "{60000,60000,40000}" ; set_text  "{0,0,0}" ;;
  jira|jira2) set_bg "{10000,40000,40000}" ; set_text  "{0,0,0}" ;;
# unused colors follow
#   blue) set_bg "{10000,20000,60000}" ; set_text  "{65535,65535,65535}" ;;
#  orange) set_bg "{60000,40000,00000}" ; set_text  "{0,0,0}" ;;
#  red) set_bg "{60000,00000,00000}" ; set_text  "{0,0,0}" ;;
#  yellow) set_bg "{60000,60000,40000}" ; set_text  "{0,0,0}" ;;
#  gray) set_bg "{60000,57000,55000}" ; set_text  "{0,0,0}" ;;
  *) set_bg "{60000,57000,55000}" ; set_text  "{0,0,0}" ;;
esac

/usr/bin/ssh "$@"

97 Things Every Programmer Should Know

Another useful book in the "97 Things" series from O'Reilly has been published, this time for programmers. My contribution is "How To Use A Bug Tracker". There are more details at O'Reilly and Amazon.

This book was created by each person adding their contributions to a Wiki, where it was edited and discussed. The whole process worked really well due to the editor, Kevlin Henney.

p.s. My photo on the cover is four rows down, third from the left, just in case you're looking.

BitBake and reusing existing ipkg files

I've spent a few days helping a customer with BitBake, a python-based build tool for embedded systems. One of the distributions that uses BitBake is OpenEmbedded, a "build-every-single-thing-from-source" Linux distro.

The customer had a number of existing ipkg packages and wanted to add them to the BitBake'd image. That turns out to be an undocumented use case, so here's the information to save other people the time we spent.

For a target architecture of armv5te, the recipe file for a package with an ipkg file named mypackage-1.2.0.3-1_armv5te.ipk looks like the code below.

DESCRIPTION = "My Package"
DEPENDS = ""
LICENSE = "GPL"

PN = "mypackage"
PV = "1.2.0.3"
PR = "1"
PACKAGES = "${PN}"

# Just fetch and place the ipkg in the correct place to build the rootfs.

# As used in openembedded/classes/image.bbclass via rootfs_ipk.bbclass
PACKAGE_INSTALL_append = ${DEPLOY_DIR_IPK}/armv5te/${P}_armv5te.ipk

SRC_URI = "\
  http://ipkg-repository.example.com/subdir/${P}_armv5te.ipk \
  "

do_unpack() {
  echo "Null unpack method"
}

do_patch() {
  echo "Null patch method"
}

do_configure() {
  echo "Null configure method"
}

do_compile() {
  echo "Null compile method"
}

do_stage() {
  echo "Null stage method"
}

do_install() {
  oenote Creating directory ${WORKDIR}/install/${PN}
  # A directory is needed for each member of PACKAGES
  mkdir -p ${WORKDIR}/install/${PN}
}

do_package() {
  echo "Null package method"
}

do_package_stage() {
  echo "Null package_stage method"
}

do_distribute_sources() {
  echo "Null distribute_sources method"
}

do_package_write() {
  oenote Copying the downloaded ipkg package from ${DL_DIR} to\
 ${DEPLOY_DIR_IPK}/armv5te
  install ${DL_DIR}/${P}_armv5te.ipk ${DEPLOY_DIR_IPK}/armv5te
}

Version: BitBake 1.8.3

It doesn't look like much by itself, but the work behind it was, er, significant. BitBake and OpenEmbedded have a good number of layers, moderate documentation and great power.


Other tips:

Don't execute BitBake in a subdirectory or it will rebuild the world in a new tmp directory there.

OpenEmbedded apparently doesn't check the return code from the ipkg install commands, so if your files aren't appearing in your image (rootfs) directory, then check their dependencies or run the explicit ipkg command that can be found in the relevant run* command file.

If ipkg asks for user input about overwriting a configuration file, then the OpenEmbedded do_rootfs() function will hang and you will need to kill both the ipkg and bitbake commands manually.

See http://bec-systems.com/web/content/view/79/9/ for more information about adding new packages to images.

Remedy - free the attachments!

I've inspected a lot of issue tracking databases in the last few months - JIRA, Bugzilla, TeamTrack, ClearQuest, Trac, GNATS and Remedy¹ - but the one I want to consider in this post is Remedy (a.k.a. the "Remedy Action Request System", part of BMC Remedy Service Management from BMC).

Remedy stores attached files in the database and compresses them to save space. That's fine, and they provide a compiled API library to talk to the running server application and extract and decompress the files. But if you're trying to get the data straight from the database, you're out of luck. Searching suggests that the file compression is some proprietary form of zlib but that's it. No source, no algorithm, nothing!

If you're providing an API to extract the data, why not show the customers the source? I suppose they might want to hide changes in the compression algorithm from users, but I can handle seeing that. I'm all for using carefully constructed APIs but not at the cost of the data being locked into one database.

¹ I know, I know, you're thinking "wow, those toolsmiths sure do live life in the large!"

Can you see what I see?

Can you see anything in the pattern of this image?

If you choose colors for a software application, then make sure you print out a screen-shot of every screen on a regular black and white printer. If you can't understand something from looking at the printout, then you've used only colors to indicate something, and this will be a problem for up to 10% of your target audience.

The image above is part of a reverse colorblindness test. The image embedded in it is supposedly only visible to color blind people. I can see a circle very clearly but it would obviously be foolish to rely on color-typical people being able to see it.

Preprocessor Warning Signs

Using #if 0 makes searching hard

Some symbol is not defined so you grep for it and you find it in a file. But you've forgotten that you commented that part of the file out. So the symbol exists in the source but not in the .o file, leading to puzzled head-scratching. A better idea is to delete the unwanted lines since you can find them in your version control whenever you want them, and perhaps even leave a message about what they were.

Try this to see how many times you've done this:

[mdoar]$ find . -name \*.[ch] | xargs grep '#if 0' | wc

#ifdefs for unit tests mean that the code tested differs from the code used in the product


If #ifdef has to be used to make a function testable, then perhaps the function needs refactoring? If possible, use the build tool to create the tests, not the preprocessor. Some sample quotes at random from the web:

"I don't ever use ifdefs for unit testing because code should never know it's being unit tested."

"Tests must be non-invasive. I don't want to have to add #ifdef UNIT_TEST declarations into my production codebase as it'll end up making a mess and worse, could actually change behaviours. The framework and test code will be in an externally compiled project"

AtlasCamp

So I've booked my place at AtlasCamp, Atlassian's first customer/developer love-in happening this November up in Marin County, CA. I'm not really sure what to expect except lots of product discussions, and I suspect a drink or two. Maybe we'll write some code, I don't know. I'm looking forward to it.

Anyway, if you're also going and would like to carpool from around San Jose, I'll have a couple of places in my car.

~Matt

More bugs than revisions

(Debian bug count from http://master.debian.org/~ajt)

Heard as an aside: "that project's got more bugs than revisions!"

Which made me think what kind of project might reasonably expect that statement to be true. The space shuttle software as a whole, perhaps? Lots of testing implies lots of bugs, and scrupulous code reviews means fewer commits.

At the other end of things, many small projects might get a few hundred revisions and then a few hundred bugs over time. Of course, most bugs will end up closed in one way or another.

Anybody able to point to a real project with more bugs than revisions?

Choosing Project Names

The discussion What' in a Project Name? over at Coding Horror reminded me of section 3.6.1 of my book Practical Development Environments:

Project names are usually chosen by engineering groups, with one name for each significantly different version of the products that they are working on. There should be no need to change a project's name once it has been chosen. Product names, on the other hand, are the names that customers see, and these names are usually chosen to help a product sell or to become popular. Product names can change at the whim of a market research poll or a new VP of Sales.

Some general guidelines for choosing names for projects are:

Keep it short

Since project names may appear in filenames or source code, shorter project names are preferable; four to six characters is common. Longer names will only be abbreviated anyway, and usually in two different ways.

Use distinctive sounds

Project names should sound different from each other when spoken aloud by people whose native language is not the one used by the rest of the group. Even if everyone speaks English, having two projects named "ctest" and "seebest" is too close for comfort.

Use low-frequency letters

It's much easier to be confident that all references to a project name can be found if the name contains characters that are less common in the local language. This is a good argument for choosing project names that use unusual characters, such as the letters q and z for English.

Apocryphal aside: a few years ago there was a project named IDS that apparently had a function named IDSConnect. Then the project was renamed DIS and all its functions were renamed accordingly, which led to their function for creating connections being renamed to DISConnect. The letters d, i, and s are too common in English to simply reuse them in
such an anagram.

Make it unmarketable

Sometimes a project name will be reused as a product name, but not if it is already trademarked, or if you make it odd or crude enough! Project names don't have to have a theme, though that can be fun. They don't even have to be meaningful, just memorable with an obvious way of pronouncing the word. You can choose a number of suitable names once and then let people decide which one they want to use next. Names of stars, types of sushi, rare diseases, and characters from comic books are some ideas to start with for project names.

Unexepected side effects in shell scripts

Shell scripts that fail to preserve the expected environment are a waste of everyone's time. For instance, you call a function which changes the current directory to somewhere else for its own purposes but fails to change back on all exit paths from the function. Trying to remember which directory you might be in later on becomes combinatorially difficult as more conditional statements are added over time. To make it more concrete:

BAD:

function MyFunction() {
   cd some_random_directory
   # Do something useful
}

GOOD:

function MyFunction() {
   # Change to the necessary directory
   pushd some_random_directory

   # Do something useful

   # Now pop the directory off the stack    
   cd -
}

If you have conditional statements, then make sure you catch them too:

BAD:

function MyFunction() {
   # Change to the necessary directory
   pushd some_random_directory

   # Do something useful and test something
   if [ "${test}" == "value" ]; then
       return # Oops, this function left us somewhere unexpected
   fi

   # Now pop the directory off the stack    
   cd -
}

GOOD:

function MyFunction() {
   # Change to the necessary directory
   pushd some_random_directory

   # Do something useful and test something
   if [ "${test}" == "value" ]; then
       # Don't forget to pop the directory off the stack    
       cd -
       return
   fi

   # Now pop the directory off the stack    
   cd -
}

Basically, preserve an expected and known state between function calls, just like any modern other programming language.

Why tags are rarely used as you would expect

As a recent introduction to version control explains, a tag is a way to record a snapshot of the state of some files at a moment in time. When I ask clients why they want to tag their source code, one reason is always "so we can reproduce a build later on". And we generally nod our heads and move on.

The funny thing is that I've never actually seen a tag used for that purpose. Tags do get used to decide whether a particular bug fix was in a certain build, or as references for released builds. But as a developer if I need an earlier build, it's probably to debug a customer problem, which means that the software was already released. Since my build process will have carefully preserved any build artifacts such as debug symbol files, I already have the files that I need.

The other observation is that build systems are designed to be able to use different branches. So if I really needed to recreate a build, I could always branch from the tag and then build using the head of the new branch. Of course, this assumes that nothing else in the environment has changed.

In summary, tags are necessary, but maybe it's not for the reasons that we usually assume.