Agile Ditch Digging

I'm a strong believer in agile software development. Every project I've worked on evolved into something every different from what it started as. Agile believes in trapping that type of change; it believes in understanding that you will know more as you build; agile deals with "managing ignorance".

On our current project we are migrating from a VW fat client to a Seaside application with short GS transactions that uses the legacy meta model, UI layouts and data structures. Our design isolates the UI layer in the VW Seaside server and the domain layer in GS.  We parse the VW window spec into Seaside components and communicate with GS using RESTful data calls (nested arrays of strings and oop numbers), and get XML back.

All of that works well: it's quick, looks nice, scales and is much easier to wrap SUnit tests around. And it was built using agile development techniques. Mostly.

I find that agile works best in the 'construction' side of the work, where you can define the user stories and measure the pace of delivery (the ditch digging). There is, however, another flavour of software development, the R&D or 'creative' side, like designing the framework and tools that the application code rests on. It's not something the user sees; it's just part of the application's fabric.

Recently we thought about how we were dealing with widget level feedback. That's where you enter 'abc' in the 'name' field and get feedback when you move on to another field. If 'abc' is an invalid value, it would be useful to see that right then ('on blur'; when the widget loses focus), instead of waiting until the 'save' button is pressed. The same goes for updating depending values: if the 'comment' defaults to 'name', having it change when 'name' is entered is useful.

Our original approach was to do this behaviour in the web code, since it had access to the display components. We soon realized that is was more important that the code have access to the domain and be able to reuse the legacy validation code, so we moved the logic to GS. Seemed simple enough.

Turned out that shifting that one responsibility triggered a lot of framework redesign. Originally, the web component built up the set of changes, and passed them to GS on 'save'. It was simple and worked. With the field level code on GS, each 'on blur' event had to trigger a GS call and, more importantly, had to package the full view update state into call so the domain code would see the current displayed state . Performance is not an issue, since each call takes from 10 to 50 ms, but the code change was more complex than it originally seemed.

I could find no good way to communicate the status of this 'big bang' change. The problem was complex, then things got delayed due to other work, and we made some critical design changes as we understood the technical issues better. None of that was well communicated. From the outside looking in, the project just stalled. Precisely the kind of optics you don't want, and the kind of problem that agile techniques are supposed to deal with.

I simply do not know how to measure 'thinking time', especially my own.  Finding a solution may take me five minutes, or five hours.

It was interesting to feel the pendulum swing from 'creative' to 'construction' as the work progressed; the construction phase is so much easier. Easier to do, easier to measure and easier to manage. Everyone is more at ease when you can show that you're 80% done, vs. just telling them that "you're close".

Digging a ditch is easy, assuming you know how.


Simple things should be simple. Complex things should be possible.

Inspecting nested Seaside components

I'm spending most of my time working on the tools to support a web port of a VW fat client application. We read windowSpecs and use them to build our own web 'widget' instances that know how to render themselves in Seaside using absolute positioning. This, combined with widget attribute meta data, has allowed us to automate most of the VW to Seaside port.

A challenge with this approach is that the Seaside views can get complex, with deeply nested subcanvas components. When debugging a button or input field widget, it's nice to not have to spend time getting to the specific instance. Seaside's Halos work well for that, but not when the view has a lot of components; things get lost is the noise.

A few fields...

...is all it takes to make things messy...

To make things easier we added a hidden 'inspect' icon to each VW based component that is only rendered if 'WAAdmin developmentToolsEnabled' is true. It's toggled by a WAToolPlugin subclass icon. Here is how the same view looks with the inspect anchors visible (the title for each inspect icon is the display string of the objedt that would be inspected when clicked)...

Each 'widget' is contained in a div for absolute positioning. Inside this div we added the inspect render method...

renderInspectOn: html
Seaside.WAAdmin developmentToolsEnabled ifFalse: [^self].
html anchor
class: 'subcanvasInspector'; 
style: 'display: none; position: absolute; '; 
title: self displayString; 
onClick: (html jQuery ajax 
    script: [:s | s << (html jQuery ajax callback: [self inspect])]);
with: [html image style: 'width: 12px; height: 12px; '; url: RepWebFileLibrary / #inspect16Gif].

...and then we toggle the display with...

renderInspectWidgetsToggleOn: html
    html anchor 
      onClick: (html jQuery class: 'subcanvasInspector') toggle;
      onClick: (html jQuery class: 'subcanvasInspectorPlus') toggle;
      with: [
        html image 
  class: 'subcanvasInspectorPlus'; 
     url: Portal.RepWebFileLibrary / #inspectPlus24Gif.
html image 
  class: 'subcanvasInspectorPlus'; 
     style: 'display: none; '; 
  url: Portal.RepWebFileLibrary / #inspectMinus24Gif].


Off...

On...

The other icons are for root component inspect, matching VW component inspect and a button to open the VW view (the deployed Seaside image has no VW domain view classes; these are being used during the application port).

With this setup we're able to port a VW application with 4497 window spec methods and keep our manual code work manageable.

Simple things should be simple. Complex things should be possible.

RESTful GemStone with multiple sessions

The project I'm working on is moving to a full web deployment from a classic VW fat client + GS. Our layers are now Seaside on VW and GemStone. To allow for this migration, we've done interesting work with VW window specs and domain meta data, which I'll comment on in future posts. Here I'll go over how we are interfacing to GemStone from our Seaside server.

Our goal is to have a snappy application, so we view Seaside as a simple presentation layer. It contains no domain classes, only view components with widgets that are coded to know which attribute they display. Getting data from GS is done by packaging the list of displayed attributes, along with the oop of the displayed object, and sending it to GS. GS answers an XML string, which is parsed into 'domain node' objects (generic data holders) and then used by the display.

xmlDomainNodeOop:attributes:collectionOop: #(6944141057 #('comment') 656304641)


<domain>
<oop>6944141057</oop>
<objectClassName>BIDcustomer</objectClassName>
...
<domain>
<objectClassName>ByteString</objectClassName>
<text>comment</text>
... <value>a comment string</value>
</domain>

We develop in a full VW client, so debugging is done by sending the #xmlDomainNodeOop:attributes:collectionOop:  method in a workspace. Very handy when trying to recreate a user problem.

If you've ever worked with GemStone (and, if you're programming in Smalltalk, my sympathies if you have not), you would be familiar with the GBS interface, the magical code which keeps objects in sync between the client and the server. You can choose to have methods execute where they make most sense (like UI heavy methods on the client and big data footprint methods on the sever) and know that your objects are in the correct state in both environments. Very cool.

With our oop + attributes question and XML answer, we don't make use of that GBS feature. In fact, we'd prefer a way to turn it off. Since each request and answer pair is independent, there is no need for session state and we can run each Seaside server with multiple sessions, using round robin dispatching.

Doing that introduces a few technical curiosities (and a huge thanks to Martin McClure for his help in fixing them). First, you can't share a class connector between GS sessions. In the single session model, I had a thin 'client dispatcher' class that forwarded class messages to server class that interfaced with the domain model and provided tools for building the XML answer. For multiple sessions, I had to link to an instance, so both class became singletons and each session defined its own connector.

Next, we tripped over the fact that you can't share objects between sessions. That makes sense in a GBS model, since syncing the objects is a session responsibly. But we were not passing domain objects and each request and answer were done with new Array and String instances. Turns out the problem was a method with an attribute parameter coded as #( ('value') ). Our parameter copy was not deep enough, so each tried to connect the nested array and we ended up with the error...

Attempt to associate a Array with more than one GemStone session

Replacing the parameter with 'Array with: (Array with: 'value')) fixed the problem. We've since added our own deepCopy extensions to prevent these types of problems in the future.

The session round robin mechanism is done with our on GS session wrapper. A class collection of session instances is rotated through, with the 'next session' pointer incremented after each use. If a session semaphore is busy, we skip to the next one. If they're all busy, we wait on the one we started with. We've been testing with three sessions, and since most of our GS session access takes less than 50ms, it's very rare that we a delay due to a busy session. We still have the occasional outlier (like the five seconds in this sample), but most of those are due to GS faulting pages in our test environment. We expect our production server to have everything cached.

It will be interesting to see how well we scale. Our system uses a dispatcher image to direct Seaside sessions to GS images, so we can adjust to load by increasing the number of Seaside images, and the number of GS sessions each image can support.

So far this technology mix is working well for us, and we're getting very positive feedback from our users.


Simple things should be simple. Complex things should be possible.

Consistency tests

SUnit tests are a fundamental part of how I write code.  I always wonder how people that don't develop with tests know when their code is ready. Perhaps they leave that to their end users.

I'm most cases my SUnit tests check for specific values, your typical...

self assert: anObject value = 'what I expect'

...But there are a couple of projects that I've worked on where checking for a state of an entire object was helpful. These are not typical SUnit assertions, since creating the test ahead of time is not practical. Instead, they are consistency tests: making sure that the state of an object has not changed.

The first project was Report4PDF, a simple VW reporting framework that uses PDF4Smalltalk (mailing list)

Adding tests for simple actions didn't add much value, since the challenge of a report tool is getting all the layout definitions to work well together; it's the net result that mattered, not the individual outputs. Those were well tested in the PDF4Smalltalk SUnit tests.

For Report4PDF, after I manually checked a report I wanted to make sure that the output did not change. As more complex reports were added, the simple reports acted as the regression tests. Edge cases were the most interesting, and most of those were found in real world use. I simply did not have the imagination to create the strange scenarios found in the wild.  So, an anomaly would  surface in production, I'd build a test report that had the same problem, fix it, and add the corrected report check to the test suite.

Stored data consists of both a diagnostic display string and a byte array of the rendered PDF document. The diagnostic string represents the low level data sent to PDF4Smalltalk and rarely needs to be updated. The PDF byte array needs to be rebuilt each time a material change is made to PDF4Smalltalk.

Report4PDF tests are in the Report4PDF-test package and coded in R4PReportTest.  Reports methods are  prefixed with 'example', like...

exampleAlignCenter
" self new exampleAlignCenter saveAndShowAs: 'exampleAlignCenter.pdf' "

| report | 

report := R4PReport new.
report businessCard.
report traceToTranscript.  
report page grid section origin: 10 @ 10; width: 100; height: 100; border: 1; align: #center; string: 'center align'.
^report

...which produces the output...

...#createTestContentsPrintOutput: is used to create an output content method...
outputAlignCenter

"Generated on February 26, 2012 4:22:43 PM"

^'Report
page width: 252
page height: 144
margin: #(0 0 0 0)
layout: 0 252 144 0
font: #Helvetica font size: 10
page number pattern: ''<page>''
page total pattern: ''<total>''
layout pages: 1
---
page width: 252
page height: 144
maximum Y: 144 (page height - footer)
output parts: 85
0 @ 0 line: 252 @ 00.5
0 @ 10 line: 252 @ 100.5
...

9.5 @ 110 line: 110.5 @ 1101
10 @ 110 line: 10 @ 101
#(10 0 0 -10 34.155 18.215) center align'


...and #createTestMethodHexString: is used to create the byte array of the PDF document...

pdfAlignCenter
"Generated on April 20, 2012 7:35:33 AM"

^'255044462D312E330A25E2E3CFD30A312030206F626A0A3C3C092F50726F6475636572202850444634536D616C6C74616C6B20312E322E3529093E3E0A656E646F626A0A322030206F626A0A3C3C092F54797065202F436174616C6F670A092F...660A3937370A2525454F46'

...finally, #createTestMethodPrintOutput: is used to create the SUnit test method which builds the example output and checks the result.  First, the output string...

testOutputAlignCenter
"Generated on February 26, 2012 4:22:53 PM
(  self new createTestContentsPrintOutput: #exampleAlignCenter )

(  self new exampleAlignCenter saveAndShowAs: 'exampleAlignCenter.pdf' )  "  

| report |

report := self exampleAlignCenter.

report buildPDF.
self assert: report printOutput = self outputAlignCenter.


...and then the PDF array...

testPDFAlignCenter
"Generated on February 26, 2012 4:22:49 PM
(  self new createTestContentsHexString: #exampleAlignCenter )

(  self new exampleAlignCenter saveAndShowAs: 'exampleAlignCenter.pdf' )  "  

| report |

report := self exampleAlignCenter.
self assert: (report byteArraySUnitAs: 'testAlignCenter.pdf') asHexString = self pdfAlignCenter

Class side convenience methods are available to rebuild all the output and test methods. Handy when PDF4Smalltalk changes.

----
The other project is the domain model of the application we're building at work. It's the same idea: while developing we write SUnit test that check for specific values. Typically this requires us to build complex domain resources. Once these are built, and we've checked the model manually, we add a 'capture' (the word 'snapshot' was already used in domain code) of the domain object's state that records all the domain attributes in an array, and stores the array in a data method.

How the data is stored is not that important. Most large Smalltalk applications I've worked on had some kind of meta data for domain objects which can be used to generate a data string.

What was interesting was how used the capture data vs. the regular SUnit tests. Normally, we want the tests to stop when an assert fails, but for captures we wanted the test to continue and have it generate a 'capture report' of which values were different. That's because a simple change, like adding a new domain attribute, would cause almost every capture for that domain class to fail.

After some trial and error, we have this workflow...

• if the capture only contains new or deleted attributes, rebuild the capture array, since none of the old data changed
• if any capture data change, generate a capture report (stored as a 'report' prefixed method) and continue
• if, however, a capture report already exists, cause an assert to fail
• if a capture report is generated, open a browser on the method
• if any capture reports are created, a final #assertNotCaptureReports will cause a failed assert

Having the capture test stop if an existing capture report is found allows us to selectively diagnose data issues. We've also added a button to the SUnitTool toolbar which rebuilds all the data captures. Handy when attributes are changed, which is almost daily. 

On a side note: I've been at HTS now for four months, spending long hours learning and updating a 15 year old framework that was written with somewhat esoteric design patterns. I see now how lucky I've been over most of my Smalltalk career, mostly working on code that I either created myself, or developed with a team that shared common development ideals. James Robertson has a good podcast on the topic of Common Pitfalls. I think I have examples of everything he and Dave Buck talked about, plus some great ways to not interface with GemStone (and I now loathe lazy initialization, especially when deeply nested and combined with silent exception handling).

Simple things should be simple. Complex things should be possible.

Opening a Seaside view from VW

I spend a good chunk of my time working on adding web interfaces to legacy Smalltalk applications, both in VW and VA. The larger project is in VW and is built with a big in-house framework. A couple of years ago I wrote a VW windowSpec to Seaside component builder which used the framework metadata to bind Seaside components to domain objects. It worked, but required too much of an investment to fully deploy.

So, we took another look at what clients needed from a web interface and decided that a 'portal' model was a better fit: a limited access web site useful to a subset of users. It is implemented with a Seaside image that has no domain objects, just parsed XML data from a RESTful GS interface. Seaside sessions share one GemStone session and rely on the application framework for login and security. It works nicely.

One of the views is table display of competitors by project, showing who is bidding on which section of the project, their bid status (won / lost / undecided), the estimated bid amount, and so on. This particular display was a challenge to do in the VW framework because it only supports a fixed number of columns in a table, and does not allow for in-cell editing (there may one day be support for the dataset widget).

We still wanted to make this display available to the VW users and the Seaside table looked nice. The solution was to launch a browser showing the selected table from a VW button press. This hybrid user interface (VW + web browser) may allow for a smoother incremental deployment of a full web based interface vs. an expensive and disruptive big bang approach.

When the 'Show table' button is pressed, a session token is saved on the logged in 'user' object in GemStone (each user has their own 'user' instance which handles things like application login and security). The oop of the saved session token is passed as a URL parameter (ExternalWebBrowser open: '...?start=12345678), and the token contents (user oop and timestamp) are checked to see if it is valid: oop of the user object must match the user object that contains the token & the timestamp of the token must be within a few seconds. If it matches, the token is cleared and a Seaside session is established. Each token can only be used once, for a short time and to access an internal web site; seems reasonably safe.

The token also contains display information which the Seaside image uses to build the table; a user presses a button and a browser opens on the expected table. Changes are stored in GemStone, so both the browser and the VW client see the same data.

Flyover components are rendered to display attributes and allow for updates. Users can change the status of a bid by pressing 'won', 'lost' or 'unknown' buttons in the flyover component. This is a quick way to edit the bid state vs. the VW based multi-window, multi-click sequence. I tried to use Seaside's jQuery tools to build the onMouseOver and onMouseOut scripts, but I found it simpler to just write the few lines I needed.

This script, as passed to table data's #onMouseOver: , positions the hidden flyover component (aFlyoverId) to the left and top of the cell under the mouse (aCellId), and then shows it. I was able to do this with Seaside jQuery code, but I could not figure out how to add the cell width to the flyover's 'left' position.

onMouseOverFlyoverId: aFlyoverId cellId: aCellId
^'
$("#', aFlyoverId ,'").css("top",$("#', aCellId ,'").position().top);
$("#', aFlyoverId ,'").css("left",$("#', aCellId ,'").position().left + $("#', aCellId ,'").width() + 8);
$("#', aFlyoverId ,'").show();
$("#', aCellId ,'").css("background-color","#F2F7FA");
'

The flyover component has its own #onMouseOver: script to keep it visible when the mouse moves away from the cell and over the flyover component.

Views that show a consolidated view of objects, like the competitor table, are good candidates for the initial web interface. The XML based data gathering from GS is quick, since no domain objects are faulted to the client, and the display options are more flexible. Whereas VW fat client's detailed object level views are better for fine grain data.

The next step is to merge the windowSpec Seaside component builder with the RESTful web portal. Not hard to do, but we'll need to see if there is client interest.

Simple things should be simple. Complex things should be possible.

Who needs objects?

Dave Thomas has said that the object abstraction is too complex for the majority of programmers. Most business software is CRUD with a bit of business logic mixed in. And it can scale by building loosely coupled systems (works for the internet, eh). Dave is a giant; he sees far. I think he's right.

So what does this mean to an object evangelist like me?  Probably not much. That vacant look on most people's faces when you try to explain objects says it all. If it does not address an immediate need, object abstraction is noise.

At the Toronto Smalltalk User Group meetings we sometimes have one or two students from Ryerson University. By attending they've already indicated that they're interested in more than the generic C syntax procedural stuff they learn in school. Joshua Panar and Dave Mason, the two profs that sponsor our group and use Smalltalk in their OO course, have said that getting the regular students out is a challenge. They're not interested. They don't see it as improving their education or job prospects. Suggesting that they should broadening their horizons falls on deaf ears.

There are two types of programmers: the toolsmiths (abstractionists) and the tool users (constructionists). Smalltalk developers seem to all be abstractionists. It is natural of us to extending our environment. Want a framework? Build it. Need a new compiler behavior? Add it. It's easy; it's common for us, yet unheard of by others.

Most programmers are constructionists. They have a job building and maintaining business applications. As Smalltalkers we ask ourselves: how can we get these programmers to use Smalltalk, to see how much more productive and enjoyable our environment is? The answer, I believe, is to reduce barriers to entry.

How to do that? Here are my wishful thinking answers...

• Merge the dialects (ya, I know: unlikely). Selecting a dialect as the first step in exploring Smalltalk is a big problem. You need to know a lot to make a good decision, at the point where you know little. Yes, the VW & Digitalk merger was a bust. But that was another time. But I can dream...
• Use a common online forum.  The Balkanization of the Smalltalk community is a problem. Think of how hard it is for a Smalltalk curious person to find information. If we at least used a common forum, like Stack Overflow, it would be easier to find cross dialect posts, and it would be more visible to the larger developer community. I'll advocate for it again at the the upcoming STIC conference, but I must be turning into a cynical cranky old fart, because I don't think there will be a change.
• Support simple scripting. I know it's been done in various ways (S# was cool), but we should be able to point to a simple script tool for people to try. If there is a good option out there, consider this: I'm a Smalltalk cognoscenti, and I'm not aware of an option that does not require firing up an image. What does that say about how well we get the word out?
• Start with prototype objects. Self and javascript got it right. It is easier to explain objects if you can defer talking about classes, and where the value of classes is discovered as a useful pattern.
• Make Smalltalk IDEs rock. I know the Smalltalk vendors and volunteers have done a great job with the resources available, but VisualStudios and Eclipses of the world are slick by comparison.
• Examples. Lots of examples. It would be great if we could point to real applications that people could fire up, test and explore. And templates; wizard driven templates to help build new applications, like those found in MS Access. Need an application to track students? Here's an example and / or a tool to help you get started. If nothing else, make it easy to get started.

Yes, abstractions are hard. But abstraction allows you to do things that would be far too difficult and expensive otherwise. Knowing how to think in abstract terms is a powerful skill that will make you a better technologist. It is our job, as those that understand this, to make it self evident to others.

Simple things should be simple. Complex things should be possible.

VW Code Coverage

The past few days I've been working with the code coverage extensions to SUnitToo in VW (SUnitToo(verage) and SUnitToo(lsoverage). It is an impressive tool. And the metrics accurately reflected the way I developed some packages. Our in-house issue tracker and patch manager had tests added after the fact, with a resulting code coverage of 30%. The Report4PDF package, the PDF4Smalltalk based report writer I'm working on, was developed from the start with SUnit tests, with a code coverage of 65%.

At first I thought that 65% seemed a bit low, since all the code has associated tests. But the code coverage is smart: it does not just measure method hits, but which paths within a method are exercised. In many cases these were exception and error message branches; the paths less taken. But some cases should have been tested. These were very narrow special cases that I had simply not thought of. The code looks OK, and I'm confident that it will work, but I won't release it without an SUnit test.

If feels like I've been given a flashlight to see the dark nooks and crannies of my code.

I also like the idea of using the path count (the denominator in the code coverage number) as an application complexity metric. It could work well when used with number of classes, number of methods, and average method size. I wonder if 'number of paths per method' would be useful.

It is an impressive tool. Thanks to all who made it available.

Simple things should be simple. Complex things should be possible.