2011-04-07

Dynamic Assets: Part III - Entry Forms

Now that we have instantiated our dynamically defined objects, we can get to the meat of our project: displaying a form based on the dynamic object.

My team uses Zend to build our forms, but the technique can easily be adapted for other frameworks. Since Zend_Form affords adding fields one at a time, we can easily build a form by looping through an Asset's field list:
$asset = $assetFactory->build('some_type');

$form = new Zend_Form();
// ... Set up our form action, method, etc. ...

$typeField = new Zend_Form_Element_Hidden('__asset_type');
$typeField->setValue($asset->getType())
    ->setRequired(true);
$form->addElement($typeField);

foreach ($asset->listFields() as $fieldName => $field) {
    if (!$field->isSubAsset()) {
        $fieldElement = new AssetFieldFormElement($field, $asset->$fieldName);
        $form->addElement($fieldElement);
    }
}

// ... Add a submit button, other form housekeeping ...
There are two interesting bits to the above snippet. Firstly, we intentionally skip rendering sub-asset fields as part of the form. We found it was easier to make the user create a parent asset before allowing them to assign sub-assets to it. This simplifies the form generation, display, validation and data persistence.

The second interesting bit is the introduction of a new class: AssetFieldFormElement. We give this class the field object for which to build form element(s), and the value of the field.

This class is where the heavy-lifting is done. The basic idea is to read the field information, instantiate a proper Zend_Form_Element object (or set of objects) and set the properties of that element. The rest of the class is mainly methods to pass through calls to things like "addValidator()", "render()", etc.

Using this class, we can turn any given AssetField object into a wide variety of HTML form elements. The simplest is a basic text box:
$field = new AssetField();
$field->setName("first_name");
$element = new AssetFieldFormElement($field, "Zaphod");
Rendered HTML:
<label>First Name</label> <input name="first_name" type="text" value="Zaphod" />

 
If the field has a list of options, it renders as a drop-down. For things like the "date" type, we have a form element which will automatically attach a date-picker to the text box.

In a lot of cases, a single field will render as multiple form elements. For instance, sometimes a field's value can be an array of strings:
$field = new AssetField();
$field->setName("favorite_restaurants")
    ->setCollection(true);
$element = new AssetFieldFormElement($field, array("Milliways", "Stavro Mueller Beta"));
Rendered HTML:
<label>Favorite Restaurants</label>
<input name="favorite_restaurants[]" type="text" value="Milliways" />
<input name="favorite_restaurants[]" type="text" value="Stavro Mueller Beta" />
<input name="favorite_restaurants[]" type="text" value="" /><a href="#" class="adder" rel="favorite_restaurants">[+]</a>




[+]
A bit of Javascript on the "[+]" link allows the user to add more text boxes as needed (we can use the AssetField's "max" property to limit that if need be.)

Hang on, what if we we have a list of options for the user to choose from, but they can only choose each option once, and they need to be able to add their own if necessary? Not a problem:
$field = new AssetField();
$field->setName("drinks")
    ->setCollection(true)
    ->setUnique(true)
    ->setOptions(array("jynnan tonnyx", "jinond-o-nicks", "Ouisghian Zodah"))
    ->setOther("Something else?");
$element = new AssetFieldFormElement($field, array("jinond-o-nicks", "Pan Galactic Gargleblaster"));
Rendered HTML:
<label>Drinks</label>
<input name="drinks[]" type="checkbox" value="jynnan tonnyx" /> jynnan tonnyx
<input name="drinks[]" type="checkbox" value="jinond-o-nicks" checked="true" /> jinond-o-nicks
<input name="drinks[]" type="checkbox" value="Ouisghian Zodah" /> Ouisghian Zodah
<label>Something else?</label> <input name="drinks[]" type="text" value="Pan Galactic Gargleblaster" />


 jynnan tonnyx
 jinond-o-nicks
 Ouisghian Zodah
 
A wide variety of form effects can be achieved through combining different asset field flags.

The other function that AssetFieldFormElement performs is adding appropriate validators to each form element. This allows us to use `$form->isValid($params)` just as we do with our application's static forms.

This is the last of the posts on dynamic class definition. I hope it was helpful.

2011-04-04

Dynamic Assets: Part II - Construction

In the last post I introduced a feature in my team's current project that will allow our users (and us!) to dynamically define "assets", and I explained the syntax for defining an asset.

Now that we have our definition, we need to actually construct an asset instance. There are several classes involved here: the AssetField, which stores the display and validation information contained in the definition for a single field; the Asset which is basically a container wrapped around a list of fields and field values; and the AssetFactory, which reads a definition and constructs an Asset by hanging fields on it.

In the code below, I'm intentionally leaving out most of the gory details because they're boring and I'm focusing more on the design than the implementation.

AssetField is essentially a code translation of the JSON definition. Here's some of what it looks like:
class AssetField
{
    const TypeString   = 'string';
    const TypeInteger  = 'int';
    // etc. for each type
    
    protected $name = null;
    protected $display = null;
    protected $type = self::TypeString;
    // etc. for each attribute that can exist in a field definition
    
    public function getName()
    {
        return $this->name;
    }
    
    // All setters return $this to provide a fluent interface
    public function setName($name)
    {
        $this->name = $name;
        return $this;
    }

    // ... other getters and setters for each property
    // listed in the definition attributes ...
}

// Create a field named "power_sources" whose value is a unique array
// of one or more of the values "Gas", "Electric", "Solar"
// which defaults to having Gas and Solar turned on
// and allows the user to enter their own value if they need to.
$powerSourceField = new AssetField();
$powerSourceField->setName("power_sources")
    ->setUnique(true)
    ->setCollection(true)
    ->setOptions(array("Gas", "Electric", "Solar"))
    ->setDefault(array("Gas", "Solar"));
    ->setOther(true);

Every Asset object will carry around its AssetField objects, so that it can provide information about how it is built. This allows our form construction and validation code to be very generic.

Since Assets don't know what fields will be hung on them, we use PHP's magic `__get` and `__set` methods to set and return field values. However, during implementation we realized that the times we want to know the value of a field are very rare; more often, we want to know the properties of a field. So we also utilize the magic `__call` to give our code access to the underlying field object.

Here is what most of the Asset class looks like:
class Asset
{
    protected $type = null;
    protected $display = null;

    protected $fields = array();
    protected $values = array();

    // Return the field object when its name is called as a class method
    public function __call($fieldName, $args)
    {
        return isset($this->fields[$fieldName]) ? $this->fields[$fieldName] : null;
    }

    // Get the value of the named field
    public function __get($fieldName)
    {
        return isset($this->fields[$fieldName]) ? $this->values[$fieldName] : null;
    }

    // Set the value of the named field
    public function __set($fieldName, $value)
    {
        if (isset($this->fields[$fieldName])) {
           $this->values[$fieldName] = value;
        }
    }

    // Hang a new field on this asset
    public function addField(AssetField $field)
    {
        $name = $field->getName();
        $this->fields[$name] = $field;
        $this->values[$name] = $field->getDefault();
        return $this;
    }

    // ... Helper methods for returning the list of all fields
    // setting the Asset type, display string and instance name format ...
}

// Get an instantiation of our plumbing system example
$plumbing = new Asset();
$plumbing->setType("plumbing")
    ->setDisplay("Home Plumbing")
    ->setInstanceNameFormat("Installed %installation_date%");

$waterSource = new AssetField();
$waterSource->setName("water_source")
    ->setOptions(array("city", "well"))
    ->setOther("Where does the water come from")
    ->setDefault("city");

$installationDate = new AssetField();
$installationDate->setName("installation_date")
    ->setType(AssetField::TypeDate)
    ->setRequired(true);

$waterHeater = new AssetField();
$waterHeater->setName("water_heater")
    ->setType(AssetField::TypeSub)
    ->setOptions(array("gas_heater", "electric_heater"));

$showers = new AssetField();
$showers->setName("showers")
    ->setType(AssetField::TypeSub)
    ->setOptions(array("shower"))
    ->setCollection(true)
    ->setMax(5)
    ->setDefault(array());

$plumbing->addField($waterSource)
    ->addField($installationDate)
    ->addField($waterHeater)
    ->addField($showers);

// Use the asset 
$plumbing->water_source = "well";
$plumbing->installation_date = "06/05/2004";

echo $plumbing->getDisplay() . ": $plumbing";
// "Home Plumbing: Installed 06/05/2004"  <--- comes from an overloaded __toString method

echo "My water comes from a {$plumbing->water_source}"
// "My water comes from a well"

// Instantiate a new shower asset
$shower = new Asset();
// ... set up the asset ...

// add the shower to the plumbing system
$plumbing->showers[] = $shower;

// What is the default value for the water source?
$field = $plumbing->water_source();
$default = $field->getDefault();
Calling a field as a method will return the AssetField object for that property. The field object can then be used in forms and validation. Another benefit of dynamically constructing our Assets in this way is that we can customize any asset on the fly without affecting any other asset of that type. From our plumbing example, let's suppose one user wants to track the serial number of their water heater, but no one else does. We just make sure that any instantiation of a water_heater asset for that user gets an additional "serial_number" field:
// Continuing from above
if ($userId == $customAssetUserId) {
    $serialNumber = new AssetField();
    $serialNumber->setName("serial_number");
    $plumbing->addField($serialNumber);
}
Any form generation and validation code will automatically pick that field up and display it for that user.

The last important class for constructing Assets is the AssetFactory. All Asset instances are constructed through this factory. It takes a type definition, which in the plumbing example is a JSON string. The factory doesn't actually care where the definitions come from or how they are stored, as long as if receives a properly formatted array. AssetFactory is given definitions and then uses the definitions to construct Assets on demand:
class AssetFactory
{
    // List of asset recipes this factory knows how to bake
    protected $definitions = array();

    public function define($definition)
    {
        // ... Validate proper formed-ness of the definition ...
        // ... Set some reasonable defaults for non-specified field attributes ...

        // All assets get an id field
        $definition['fields']['id'] = array(
            'type' => DW_Asset_Field::TypeString,
            'hidden' => true,
        );

        $this->definitions[$definition['type']] = $definition;
    }

    // Construct an asset of the given type
    public function build($type)
    {
        $definition = $this->definitions[$type];

        $asset = new Asset();
        $asset->setType($definition['type'])
            ->setDisplay($definition['display'])
            ->setInstanceNameFormat($format);

        foreach ($definition['fields'] as $name => $fieldDef) {
            $field = new AssetField();
            $field->setName($name)
                ->setType($fieldDef['type'])
                ->setDisplay($fieldDef['display'])
                ->setHidden($fieldDef['hidden'])
                ->setUnique($fieldDef['unique'])
                ->setRequired($fieldDef['required'])
                ->setMin($fieldDef['min'])
                ->setMax($fieldDef['max'])
                ->setOptions($fieldDef['options'])
                ->setOther($fieldDef['other'])
                ->setDefault($fieldDef['default']);

            $asset->addField($field);
            $asset->$name = $fieldDef['default'];
        }

        return $asset;
    }
}

$factory = new AssetFactory();
$factory->define(json_decode('{
    "type" : "plumbing",
    "display" : "Home Plumbing",
    "instance_name" : "Installed %installation_date%",
    "fields" : {
      "water_source" : {
        "type" : "string",
        "options" : ["city", "well"],
        "other" : "Where does the water come from",
        "default" : "city"
      },
      "installation_date" : {
        "type" : "date",
        "required" : true
      },
      "water_heater" : {
        "type" : "subasset",
        "options" : ["gas_heater", "electric_heater"],
      },
      "showers" : {
        "type" : "subasset",
        "options" : ["shower"],
        "collection" : true,
        "max" : 5
      }
    }
  }'));

$plumbing = $factory->build("plumbing");
If we have any customizations, like our serial_number from above, we can call a `customize()` method on the factory from within the `build()` method, or pass the result of the `build()` to some other customization class. At the moment, we don't have any requirements like that. The important thing is that we now have an Asset object that we can pass around to our generic Asset handling code.

Next up, a description of how we generate a form from a generic Asset object.

2011-04-01

Dynamic Assets: Part I - Definition

I am currently working on a feature of a project that allows users to track assets that exist at a location. The interesting part of the feature is that an asset can be almost anything, and that the properties of a single instance of an asset may have little or no overlap with the properties of a different asset.

Properties in this context refer not just to the attributes of an asset, but also to how those attributes are displayed, entered by the user when adding/editing an asset, and validated when the user is saving an asset. Assets can have sub-assets, forming a tree. And the most interesting wrinkle: users should be able to define their own assets, with properties' data-types, display and validation all user-controlled.

I'm not going to talk about how the assets are stored (until a few days ago it was a toss-up between schema-less MySQL or MongoDB.) Instead, what follows is series of posts describing our solution for defining an asset, displaying the add and edit forms and validating an asset. This first post covers how asset types are defined.

An asset is basically a bag of properties, and each property has attributes that define what values it can hold and hints about how it should be presented to the user when displaying or editing.

For example: the plumbing system in your home may have properties like "installation date" or "water source" (where water source may be either "city" or "well".) The system may also have sub-assets, like "water heater", which has properties like "type" (gas, electric, solar) and "last maintenance date". There might also be a list of "showers", which are also sub-assets, and have properties like "location", "size" and "needs to be cleaned".

Our first pass was to define a class for each asset type, which would include the type's properties and how those properties should be displayed. The problem with this solution was two-fold: first, we don't currently know what properties each of our assets will or won't need to have (so an asset type's schema will be in flux); and second, we don't want our users to have to write code in order to define their own asset types. In fact, we don't know all the assets we will need to track for our own purposes yet.

Our solution was to store asset definitions as data instead of code. Then, all we need is a mechanism that reads a definition and builds a dynamic "class-less" asset object. The definition syntax is JSON, though the code that actually constructs concrete Asset instances doesn't care how the definitions are stored.

Here is some of the definition for the above plumbing system (I've intentionally added some constraints to demonstrate other bits of the definition syntax):
{
  "plumbing" : {
    "type" : "plumbing",
    "display" : "Home Plumbing",
    "instance_name" : "Installed %installation_date%",
    "fields" : {
      "water_source" : {
        "type" : "string",
        "options" : ["city", "well"],
        "other" : "Where does the water come from",
        "default" : "city"
      },
      "installation_date" : {
        "type" : "date",
        "required" : true
      },
      "water_heater" : {
        "type" : "subasset",
        "options" : ["gas_heater", "electric_heater"],
      },
      "showers" : {
        "type" : "subasset",
        "options" : ["shower"],
        "collection" : true,
        "max" : 5
      }
    }
  },

  "another_type" : {
    ...
  },
  ...
}
All assets must have a "type", which must be unique among all asset definitions. "display" is an optional field; if not specified, the display is the "type" string upper camel-cased and with underscores replaced with spaces (e. g. "plumbing_system" becomes "Plumbing System"). The display is used to identify an asset instance's type to the user, mainly on entry/edit forms and reporting.

The "instance_name" is the formatting string used to present a specific asset to the user. Wrapping a field name in %'s will substitute the actual value of that field when displaying the asset to the user. In the above example, if a plumbing asset has an installation date of "6/12/2009", then the asset would be displayed to the user as "Installed 6/12/2009". If no instance_name is given, the instance name defaults to the asset's id.

The rest of the asset is a list of fields, indexed by the internal field name, and each having some combination of descriptive attributes. The "type" attribute is the only required attribute for a field, and must have one of the values "string", "int", "boolean", "datetime", "date", "float" or "subasset". The difference between date and datetime is purely in the way that they are presented to the user (and how granularly the value is stored: dates are always rounded to midnight.)

"required" is a boolean flag indicating whether the asset can be saved without a value in that field or not. "hidden" is another boolean flag, specifying if the field should be presented to the user. We have to be careful that if a field is required and hidden the application must provide a value. A good way to do that is with the "default" attribute, which specifies the default value for that field. If the field is not hidden, the default value will be pre-entered or selected for the user on an "Add Asset" form.

The "options" attribute lists the valid values for a field. The exception is for "subasset" fields, in which case options is a list of valid subasset types (in our plumbing example, gas heaters and electric heaters could be their own asset types with their own set of fields, and an instance of either would be a valid value for the "water_heater" field.) If no type options are listed for a subasset field, any asset type is assumed to be valid.

If a user should be allowed to enter their own value in addition to the, then the "other" attribute can be used. If "other" is a boolean true, then an "Other" option will be added to the list of options, and the user will be able to enter whatever value they like. "other" can also be a string, in which case it is the text to display as the "Other" option, and the label on whatever form field is used to capture the user's input.

Some fields hold multiple values. In these cases, the "collection" attribute is set to boolean true. By default, the same value can appear in the collection multiple times. If that is not desired, the "unique" attribute can be used to validate that each value appears only once. Using combinations of "collection", "unique" and "options" can drive some pretty complex form behaviors (which will be shown in a future post.)

There are also "max" and "min" attributes. On a collection, the value of the attributes are integers specifying how many values are allowed to be in that collection. Otherwise, they represent upper and lower bounds on integer, float, date and datetime values.

The definition syntax so far gives us all the flexibility we need to define our project's known asset types, and we believe it will be useful in the future for easily adding new types and allowing our users to define their own assets types (probably through a form that translates into the JSON definition.)

Part II will be an overview of we actually use these definitions to form "class-less" asset objects on-the-fly.

2011-03-23

Undead Wedding

Gave a talk last night with Norm Santos about full-stack web application testing using Node.js, Zombie.js and Vows. We presented it to TriangleJS meetup. It was pretty well-received.

My team's usage of Zombie/Vows has petered off a little as we've moved into other projects, but it's definitely a concept I'd like to revisit when we get a little time. As our app solidifies more, we're going to need more of this type of testing, in addition to our typical unit- and functional-testing.

2011-03-15

Bash sockets

Here's a neat trick with the Bash shell that I learned today: Bash can open and read sockets natively.

When I say "natively" I mean, having been compiled with the `--enable-net-redirections` flag. But if that part is true (and it's at least true in Xubuntu 10.10) then you can read and write to sockets this way:
exec 3<>/dev/tcp/host/port

# Write to the socket as with any file descriptor
echo "Write this to the socket" >&3

# Read from the socket as with any file descriptor
cat <&3
So you could do something like pull down Google's home page like so:
exec 3<>/dev/tcp/www.google.com/80
echo -e "GET / HTTP/1.1\n\n" >&3
cat <&3
For my team's project, we have some scripts that need to pull down logging and monitoring data and then push it to the monitoring server's daemon. We set up each script to output its data in the right format, then call the whole thing from a wrapper script that redirects the output to the monitoring daemon. Here's what it looks like:
#!/bin/bash
#
# Takes a command and pipes the output to graphite server
# running on the local machine
#
# Usage: ./graphite-push.sh 
#

CMD=$@
OUTFILE=/dev/tcp/localhost/2003

exec 3>$OUTFILE
$CMD >&3
And it's called like so:
graphite-push.sh php ./collect-stats.php --flag --arg value
The neat part is that `collect-stats.php` can be run on its own and will output to stdout. The `graphite-push.sh` script just runs whatever command it is given and redirects stdout to the monitoring daemon.

Many thanks to Dave Smith for this useful bit of shell magic.

2011-02-12

Jaded available on GitHub

Jaded, the PHP library I've been working on is now available on GitHub.

I talked about using Jaded as a lightweight model layer back in November. I've since decided to rip out the controller and view stuff. It wasn't coming along as I'd hoped, and I've taken to using Zend to get things up and running quickly.

Besides the model library, there's also a pretty handy autoloader and a database wrapper around PHP's PDO library. Over time I will be adding some other useful classes, including some Zend plugins.

2010-12-22

The Goal

I have pretty strong feelings about Agile. That is to say, I can't stand it. When someone mentions Agile to me all I think about are overpriced books, puffed up consultants and "coaches" and a whole slew of project management software tools that give Pro[du|je]ct Managers way too much noise vs. signal about how a project is really going. And all of it gets in the way of what I really love to do: sling great code.

But man, oh man, do I dig being agile.

There's already a pretty awesome article about the difference between "Agile" and "agile". It says just about everything I could write on the subject. Rather than rehash ideologies, I'm going to introduce a concept that has helped me towards gaining an agile mindset: the goal.

I've had the goal in my head for a while now. I never bothered writing it down anywhere until a presentation that I threw together in about 30 minutes to help my new team with their agile adoption. That presentation is the first time I've seen the goal presented and worded in that form (I don't take credit for the goal; it is the product of a lot of reading, attending other people's presentations, some experiences good and bad, and a healthy dose of my own biased worldview.) Here is the goal:

Deliver working, usable, high-quality software as quickly as possible.

The goal answers a simple question: why are we adopting agile? This isn't about the project or product. The business has already defined the rationale for that. The goal is about the rationale for wanting to be agile while accomplishing the project.

Nothing in the goal (or the Agile Manifesto, for that matter), speaks to a specific process or method. And that's where the power of the goal comes from. It speaks to what the team is trying to accomplish, while making the how a secondary concern. Focusing on the goal allows a team to easily see where their agility is being compromised:
  • Is our software working? Is it ready to be put in the hands of our users? Are we making continual improvements to it?
  • Is it usable? Does it meet our users' needs and goals? Does it provide value to the stakeholders?
  • Does it have high-quality? Is the code clean, as bug-free as we can make it, and easy to fix when the bugs we missed inevitably appear?
  • Are we delivering fast enough? Are we meeting the current needs of the business? Are we able to act on ideas and opportunities before our competition does?
If the answer to any of those questions becomes "no" at any time, then the process is broken. Fix it. You don't have to follow Scrum or any other Agile methodology to the letter. Process is highly context dependent. Only you and your team know what will get you to the goal.

If everyone can agree on the goal, the rest is downhill. The team will work out it's own way of getting there. I suggest Scrum as a starting point because, as a method, it's easy to understand. But I believe that a team that is really focused on the goal will probably leave the guardrails of Scrum and start building their own process fairly quickly.

There's a concept that I first heard from Alistair Cockburn but originally came from Japanese martial arts: Shuhari 守破離. The basic premise is that there are 3 stages to attaining mastery of a skill:
  1. Shu — You imitate the form exactly. You do not deviate from the specified process, and you follow it to the letter.
  2. Ha — You begin to innovate for yourself. You tweak the process to take advantage of your own special circumstances and strengths.
  3. Ri — You no longer think in terms of forms or processes. There is a constant, smooth and uninterrupted flow of thought into action.
Everyone starts at Shu. Scrum is Shu, as is any other Agile method. Moving towards the goal helps a team move out of Shu and towards Ha. This is where agile beats out Agile. When you stop thinking about Agile or agile, and you simply act in the direction of the goal, you have attained Ri.