Dynamic Assets: Part I - Definition

I am currently working on a feature of a project that allows users to track assets that exist at a location. The interesting part of the feature is that an asset can be almost anything, and that the properties of a single instance of an asset may have little or no overlap with the properties of a different asset.

Properties in this context refer not just to the attributes of an asset, but also to how those attributes are displayed, entered by the user when adding/editing an asset, and validated when the user is saving an asset. Assets can have sub-assets, forming a tree. And the most interesting wrinkle: users should be able to define their own assets, with properties' data-types, display and validation all user-controlled.

I'm not going to talk about how the assets are stored (until a few days ago it was a toss-up between schema-less MySQL or MongoDB.) Instead, what follows is series of posts describing our solution for defining an asset, displaying the add and edit forms and validating an asset. This first post covers how asset types are defined.

An asset is basically a bag of properties, and each property has attributes that define what values it can hold and hints about how it should be presented to the user when displaying or editing.

For example: the plumbing system in your home may have properties like "installation date" or "water source" (where water source may be either "city" or "well".) The system may also have sub-assets, like "water heater", which has properties like "type" (gas, electric, solar) and "last maintenance date". There might also be a list of "showers", which are also sub-assets, and have properties like "location", "size" and "needs to be cleaned".

Our first pass was to define a class for each asset type, which would include the type's properties and how those properties should be displayed. The problem with this solution was two-fold: first, we don't currently know what properties each of our assets will or won't need to have (so an asset type's schema will be in flux); and second, we don't want our users to have to write code in order to define their own asset types. In fact, we don't know all the assets we will need to track for our own purposes yet.

Our solution was to store asset definitions as data instead of code. Then, all we need is a mechanism that reads a definition and builds a dynamic "class-less" asset object. The definition syntax is JSON, though the code that actually constructs concrete Asset instances doesn't care how the definitions are stored.

Here is some of the definition for the above plumbing system (I've intentionally added some constraints to demonstrate other bits of the definition syntax):

{
  "plumbing" : {
    "type" : "plumbing",
    "display" : "Home Plumbing",
    "instance_name" : "Installed %installation_date%",
    "fields" : {
      "water_source" : {
        "type" : "string",
        "options" : ["city", "well"],
        "other" : "Where does the water come from",
        "default" : "city"
      },
      "installation_date" : {
        "type" : "date",
        "required" : true
      },
      "water_heater" : {
        "type" : "subasset",
        "options" : ["gas_heater", "electric_heater"],
      },
      "showers" : {
        "type" : "subasset",
        "options" : ["shower"],
        "collection" : true,
        "max" : 5
      }
    }
  },

  "another_type" : {
    ...
  },
  ...
}

All assets must have a "type", which must be unique among all asset definitions. "display" is an optional field; if not specified, the display is the "type" string upper camel-cased and with underscores replaced with spaces (e. g. "plumbing_system" becomes "Plumbing System"). The display is used to identify an asset instance's type to the user, mainly on entry/edit forms and reporting.

The "instance_name" is the formatting string used to present a specific asset to the user. Wrapping a field name in %'s will substitute the actual value of that field when displaying the asset to the user. In the above example, if a plumbing asset has an installation date of "6/12/2009", then the asset would be displayed to the user as "Installed 6/12/2009". If no instance_name is given, the instance name defaults to the asset's id.

The rest of the asset is a list of fields, indexed by the internal field name, and each having some combination of descriptive attributes. The "type" attribute is the only required attribute for a field, and must have one of the values "string", "int", "boolean", "datetime", "date", "float" or "subasset". The difference between date and datetime is purely in the way that they are presented to the user (and how granularly the value is stored: dates are always rounded to midnight.)

"required" is a boolean flag indicating whether the asset can be saved without a value in that field or not. "hidden" is another boolean flag, specifying if the field should be presented to the user. We have to be careful that if a field is required and hidden the application must provide a value. A good way to do that is with the "default" attribute, which specifies the default value for that field. If the field is not hidden, the default value will be pre-entered or selected for the user on an "Add Asset" form.

The "options" attribute lists the valid values for a field. The exception is for "subasset" fields, in which case options is a list of valid subasset types (in our plumbing example, gas heaters and electric heaters could be their own asset types with their own set of fields, and an instance of either would be a valid value for the "water_heater" field.) If no type options are listed for a subasset field, any asset type is assumed to be valid.

If a user should be allowed to enter their own value in addition to the, then the "other" attribute can be used. If "other" is a boolean true, then an "Other" option will be added to the list of options, and the user will be able to enter whatever value they like. "other" can also be a string, in which case it is the text to display as the "Other" option, and the label on whatever form field is used to capture the user's input.

Some fields hold multiple values. In these cases, the "collection" attribute is set to boolean true. By default, the same value can appear in the collection multiple times. If that is not desired, the "unique" attribute can be used to validate that each value appears only once. Using combinations of "collection", "unique" and "options" can drive some pretty complex form behaviors (which will be shown in a future post.)

There are also "max" and "min" attributes. On a collection, the value of the attributes are integers specifying how many values are allowed to be in that collection. Otherwise, they represent upper and lower bounds on integer, float, date and datetime values.

The definition syntax so far gives us all the flexibility we need to define our project's known asset types, and we believe it will be useful in the future for easily adding new types and allowing our users to define their own assets types (probably through a form that translates into the JSON definition.)

Part II will be an overview of we actually use these definitions to form "class-less" asset objects on-the-fly.

Undead Wedding

Gave a talk last night with Norm Santos about full-stack web application testing using Node.js, Zombie.js and Vows. We presented it to TriangleJS meetup. It was pretty well-received.

My team's usage of Zombie/Vows has petered off a little as we've moved into other projects, but it's definitely a concept I'd like to revisit when we get a little time. As our app solidifies more, we're going to need more of this type of testing, in addition to our typical unit- and functional-testing.

Bash sockets

Here's a neat trick with the Bash shell that I learned today: Bash can open and read sockets natively.

When I say "natively" I mean, having been compiled with the `--enable-net-redirections` flag. But if that part is true (and it's at least true in Xubuntu 10.10) then you can read and write to sockets this way:

exec 3<>/dev/tcp/host/port

# Write to the socket as with any file descriptor
echo "Write this to the socket" >&3

# Read from the socket as with any file descriptor
cat <&3

So you could do something like pull down Google's home page like so:

exec 3<>/dev/tcp/www.google.com/80
echo -e "GET / HTTP/1.1\n\n" >&3
cat <&3

For my team's project, we have some scripts that need to pull down logging and monitoring data and then push it to the monitoring server's daemon. We set up each script to output its data in the right format, then call the whole thing from a wrapper script that redirects the output to the monitoring daemon. Here's what it looks like:

#!/bin/bash
#
# Takes a command and pipes the output to graphite server
# running on the local machine
#
# Usage: ./graphite-push.sh 
#

CMD=$@
OUTFILE=/dev/tcp/localhost/2003

exec 3>$OUTFILE
$CMD >&3

And it's called like so:

graphite-push.sh php ./collect-stats.php --flag --arg value

The neat part is that `collect-stats.php` can be run on its own and will output to stdout. The `graphite-push.sh` script just runs whatever command it is given and redirects stdout to the monitoring daemon.

Many thanks to Dave Smith for this useful bit of shell magic.

Jaded available on GitHub

Jaded, the PHP library I've been working on is now available on GitHub.

I talked about using Jaded as a lightweight model layer back in November. I've since decided to rip out the controller and view stuff. It wasn't coming along as I'd hoped, and I've taken to using Zend to get things up and running quickly.

Besides the model library, there's also a pretty handy autoloader and a database wrapper around PHP's PDO library. Over time I will be adding some other useful classes, including some Zend plugins.

The Goal

I have pretty strong feelings about Agile. That is to say, I can't stand it. When someone mentions Agile to me all I think about are overpriced books, puffed up consultants and "coaches" and a whole slew of project management software tools that give Pro[du|je]ct Managers way too much noise vs. signal about how a project is really going. And all of it gets in the way of what I really love to do: sling great code.

But man, oh man, do I dig being agile.

There's already a pretty awesome article about the difference between "Agile" and "agile". It says just about everything I could write on the subject. Rather than rehash ideologies, I'm going to introduce a concept that has helped me towards gaining an agile mindset: the goal.

I've had the goal in my head for a while now. I never bothered writing it down anywhere until a presentation that I threw together in about 30 minutes to help my new team with their agile adoption. That presentation is the first time I've seen the goal presented and worded in that form (I don't take credit for the goal; it is the product of a lot of reading, attending other people's presentations, some experiences good and bad, and a healthy dose of my own biased worldview.) Here is the goal:

Deliver working, usable, high-quality software as quickly as possible.

The goal answers a simple question: why are we adopting agile? This isn't about the project or product. The business has already defined the rationale for that. The goal is about the rationale for wanting to be agile while accomplishing the project.

Nothing in the goal (or the Agile Manifesto, for that matter), speaks to a specific process or method. And that's where the power of the goal comes from. It speaks to what the team is trying to accomplish, while making the how a secondary concern. Focusing on the goal allows a team to easily see where their agility is being compromised:

• Is our software working? Is it ready to be put in the hands of our users? Are we making continual improvements to it?
• Is it usable? Does it meet our users' needs and goals? Does it provide value to the stakeholders?
• Does it have high-quality? Is the code clean, as bug-free as we can make it, and easy to fix when the bugs we missed inevitably appear?
• Are we delivering fast enough? Are we meeting the current needs of the business? Are we able to act on ideas and opportunities before our competition does?

If the answer to any of those questions becomes "no" at any time, then the process is broken. Fix it. You don't have to follow Scrum or any other Agile methodology to the letter. Process is highly context dependent. Only you and your team know what will get you to the goal.

If everyone can agree on the goal, the rest is downhill. The team will work out it's own way of getting there. I suggest Scrum as a starting point because, as a method, it's easy to understand. But I believe that a team that is really focused on the goal will probably leave the guardrails of Scrum and start building their own process fairly quickly.

There's a concept that I first heard from Alistair Cockburn but originally came from Japanese martial arts: Shuhari 守破離. The basic premise is that there are 3 stages to attaining mastery of a skill:

1. Shu — You imitate the form exactly. You do not deviate from the specified process, and you follow it to the letter.
2. Ha — You begin to innovate for yourself. You tweak the process to take advantage of your own special circumstances and strengths.
3. Ri — You no longer think in terms of forms or processes. There is a constant, smooth and uninterrupted flow of thought into action.

Everyone starts at Shu. Scrum is Shu, as is any other Agile method. Moving towards the goal helps a team move out of Shu and towards Ha. This is where agile beats out Agile. When you stop thinking about Agile or agile, and you simply act in the direction of the goal, you have attained Ri.

Git, TargetProcess and Hashtags

At $job, we're using TargetProcess for our user story and task management. One of the neat things you can do in TP is tag commits to a code repository in such a way that they will link to the task associated with that commit:

#123 This commit is linked to task number 123

In order to get this to work, the hashtag has to be the first thing on the first line of the commit message. Anyone using `git` as their source control should immediately see the problem with this...

...In git commit messages, any line starting with # is considered a comment and is stripped out of the commit message. Quite annoyingly, there's no way to change git's default comment delimiter. Since $job is using Subversion, this isn't a problem except for myself, the lowly git user hobbling along with git-svn.

The solution I've come up with is a two-pronged approach involving changing a default and adding a git-hook.

Firstly, the default --cleanup option on a `git-commit` command strips trailing and leading whitespace lines and all commentary (lines starting with #). So I aliased `git commit` to only worry about the whitespace lines, by adding this to my .gitconfig:

[alias]
    cmt = "commit --cleanup=whitespace""

So now git will leave lines starting with "#" alone. This means I have strip them out myself. This is accomplished with a git commit-msg hook that looks like this:

# Use TargetProcess hashtags to link commits to tasks
# use this in conjunction with --cleanup=whitespace
sed -e 's/^#.*//' -e 's/^@\(#.*\)/\1/' -i $1

The first sed expression strips out comment lines. The second one translates any line that starts with "@#" into one starting with "#". The trick is that it does it after the comment line stripping. So now I can do commits that look like this:

git cmt

# The editor opens and I can type:

@#123 This commit is linked to task number 123
# This is a comment line that will get stripped out
#  So it this

When the commit goes through, voila! A commit message starting with a hashtag that can be picked up by TargetProcess.

Edit 2010-12-16: After experimentation and verification, it turns out the TP can pick up the task id hashtag anywhere in the commit message. So all the above is not really necessary. It's interesting from an educational perspective, though.

Private Git repository with Dropbox

I recently began working on a project that needs a remotely-accessible private git repository. The project needs to be done on the cheap, and we don't have the budget to pay for a private github or assembla repository. We're already using Dropbox to handle project documentation and such, so it became a natural fit for sharing code as well.

I already have the Dropbox client installed and set up on my development machine. Here is how to set up the repo:

cd /path/to/Dropbox
mkdir myproject
cd myproject
git --bare init

This initializes a "bare" repo; a repo without a working directory. In other words, you can't edit and commit to it directly. The repo is now your "remote", hosted on Dropbox, and accessible through the local filesystem.

Create a working clone from the remote repo:

cd /path/to/myproject-working
git clone /path/to/Dropbox/myproject .

This will complain about cloning an empty repo, which is fine. Now follow the normal git development cycle:

echo "Read this!" > README
git add README
git commit -m "Initial commit"

When you're done making changes and committing, it's time to push to the remote repo in Dropbox:

git push /path/to/Dropbox/myproject --all

The `--all` flag is necessary on the first commit to the remote. It can be left off of all subsequent commits.

That's it! Now the repo is available on any computer linked to the Dropbox account, and can be checked out using the normal `git clone`, work, commit, `git push` cycle.

If you want other developers to have access, simply share the repo directory with their Dropbox accounts as you would any other directory. They can clone it and push to it just like any other git repository.

Here's another trick if you don't have the Dropbox daemon (dropboxd) running all the time on your development machine. Put the following script in "/path/to/Dropbox/myproject/hooks". Name the script "dropbox-push"

#!/bin/sh

dropbox running
if [ $? -eq 1 ]; then
 echo "Dropbox started by another process."
 exit 0
fi

dropbox start
until [ "`dropbox status`" = "Idle" ]; do
 sleep 1
done
dropbox stop

Now make the post-receive hook runnable:

mv post-receive.sample post-receive

Edit the "post-receive" file, and add the following lines:

cd `dirname $0`
./dropbox-push

Save the file. Now whenever you (or another developer) pushes to the repo, it will check to see if the Dropbox daemon is running, and if not, will start it and let it run until the code is synced, then stop the daemon. Since the hook itself is in under Dropbox, it will be synced and will be run for all other developers pushing to the Dropbox repo as well.

So there it is, a free, private git repository; at least, until the repository gets larger than the Dropbox free 2GB limit.