Friday 22 April 2011

Smalltalk trivia

I miss the days of The Smalltalk Report and discussions on comp.lang.smalltalk. It was a time of lively debate about syntax, naming conventions, coding patterns, dialects ... all the signs of an energetic community. It's not like that any more. I think it's due to the fragmentation of the Smalltalk forums. The threads are interesting, but specialised.

My coding is strongly influenced by Kent Beck and his coding patterns. I remember reading his column in The Smalltalk Report and seeing the evolving consensus of how to code Smalltalk. My iteration block still have :each or :each<something> as the binding variable.  In his "Fundamentals of Smalltalk Programming Technique",  Andres Valloud makes the point that code formatting is a pointless debate and that we should be comfortable reading any Smalltalk code. And he's right. But I'd still like to know what others find more readable. It's like learning to write gooder English.

For example, I trip over the extra white spaces inside blocks that most of the Seaside code has. Where I'd write a method like...

aBoolean

    ifTrue: [self doThis]
    ifFalse: [self doThat].
aList do: [:each | self doSomethingWith: each]

...I find the Seaside code is formatted as...

aBoolean

    ifTrue: [ self doThis ]
    ifFalse: [ self doThat ].
aList do: [ :each | self doSomethingWith: each ]

...I wonder why that convention was adopted. In my eyes the block is just not hugging the code enough. I don't find that style in other Smalltalk code. And it really does not matter; I'd just like to understand the though behind it. 

Another example: In our VW application,  the framework code uses abbreviations for local and parameter variables, so I see a lot of code like printOn: aStrm and | errMsg idx t v m s | ... drives me nuts. But it was adopted because of how easy it used to be to name a local variable the same as an instance variable. We do refactor code this kind of code now as we maintain it, but it's easier to accept when the thought behind it is understood. 

And here's a pattern question: when is it better to double-dispatch? In our VW code was have an #echo method which writes a printString of the receiver to the Transcript, like the newish VW #out method (unfortunate name conflict with an Opentalk method). We also have #echo: which writes both the receiver and the argument to the Transcript, so...
    'Time' echo: Time now
...shows in the Transcript as...
    Time 10:10:54 AM

I added the same code to our VA application. So what's the best way to write this method? We don't want strings to be printed with single quotes, like...

Transcript cr; show: 'this string' printString  

'this string'
vs.
Transcript cr; show: 'this string'
this string

...so we can implement #echo on Object as Transcript cr; show: self printString and on String as Transcript cr; show: self. But what about #echo: ? We could implement on Object as...

Object>>echo: anObject

    Transcript cr; show: self printString, ': ', anObject printString

...(I prefer the : delimiter to just a space) and we'd need...

String>>echo: anObject

    Transcript cr; show: self , ': ', anObject printString


...but what if anObject is a String? You would end up with superfluous single quotes. You could add a new method like #asEchoString with an Object and String implementation and send that instead of #printString, which is easy to read, or you could try double dispatching, which I decided to do (mostly out of curiosity). In this case I don't think it's a better pattern, but it's interesting (and very useful for more complex problems). And I like that they are all one line method.

Here is what my implementation (no value returned so that a := b echo answers b)...

Object>>echo
self printString echo

String>>echo

Transcript cr; show: self.

Object>>echo: anObject

anObject echoAfter: self printString

String>>echo: anObject

anObject echoAfter: self

Object>>echoAfter: aString

aString echoString: self printString

String>>echoAfter: aString


aString echoString: self

String>>echoString: aString

Transcript cr; show:  self , ': ', aString


'test' echo: 123    >> 'test: 123''test' echo: 'this' >> 'test: this'123 echo: 'this'    >> '123: this'123 echo: 456       >> '123: 456'  



Tuesday 19 April 2011

VXML server isolation

Our IVR VXML Seaside server needs to handle up to 700 calls per hour. Normally that's not a problem, but we do get a few timout errors per week, which is why we're moving to a three image load balanced deployment.

Recently we had another issue with our current setup: the server hosting the file share, to which call files are written, failed. Calls were still processed, but the final write failed. Write are done in a forked block with an exception handler, so the image dealt with the errors cleanly. And the call data was all written to a log, so no data was lost.

However, there was a performance hit on the server image and we started to get timeout errors on more than 10% of the calls. So we turned off the Seaside server which triggered an automatic failover to our AWS hosted server. It works, but it's a bit slow. Callers could wait several seconds for the call prompts (we'll be upgrading it soon).

This was the first time we had an error with the file share and it pointed out a dependency that we'd prefer to avoid. Ideally, the VXML servers would continue to process calls even if other servers are down; they should as isolated as possible.

Our strategy is to use the VW Opentalk image to image interface and have a data server gather call information from the three VXML servers. It will also send them updated validation files. This way, the VXML servers have no folder share dependencies,  the data server can show status across all three VXML server, and the VXML servers can function even if the entire back office is down. And we can host the data & VXML servers with different service providers.

We've been using Opentalk in our client patch system for a couple of years now. It's familiar and reliable. And ya gotta love the deployment flexibility you get with Smalltalk.

Friday 15 April 2011

IVR VXML from Seaside

We use Seaside to serve anything that takes data from an http call. One of these services is an IVR system hosted by Voxeo. It requires a VXML file that defines the audio script and records answers. We have a sublcass of WARequestHandler that answers static VXML content for the initial request:


handleRequest: aRequestContext
| response | 
response := aRequestContext response. 
response 
contentType: self contentType; 
nextPutAll: self document requestVXML.
aRequestContext respond.


...and builds contents for the response, which contains a simple 'Thank you' if all the data is valid...


handleResponse: aRequestContext
| response result request | 
request := aRequestContext request.
result := self processCall: request.
response := aRequestContext response. 
response 
contentType: self contentType; 
nextPutAll: result.
aRequestContext respond.


Our production system, which our client uses to record at-home service calls, peaks at about 600 calls per hour. And the same image that serves the VXML file also has a conventional Seaside interface for monitoring the system and for configuration. Very handy. For example, we have a Google graph to show one day's call distribution.



Recently we added more validation to the VXML script, so that shifts and employee numbers could be checked during the IVR dialog. The new interaction places more demands on the Seaside image, so we're testing a deployment with three images and simple round-robin load balancing (the current single image deployment gets about two timeout errors per week). Each VXML call is RESTful so we have no need for session affinity. Testing all went well, until something interesting came up.

Testing was all done on a Windows server and deployment is on a Linux box. No big deal for VW. But when we tried the new deployment, the IVR system said the VXML content had errors. Looking at the VXML file on our Seaside display showed extra blank lines, but the content was correct. Inspecting the file content as read showed that the <cr><lf>s were all replaced with <cr><cr> (you gotta love all the delimiter differences between Windows, Linux and Smalltalk). I didn't think that IVR server would have an issue with that, but to be sure I replaced the file read from...
    ^file contentsOfEntireFile
...to...
    ^file contentsOfEntireBinaryFile asString
...and it turns out that fixed the problem.

Makes we wonder what people do who work with tools that don't allow the deep diving that Smalltalk does. I suspect that when it all works their productivity is high, but there must be more frustration when things break.

Thursday 14 April 2011

VA Smalltalk and Seaside

I support a VA Smalltalk app for which we're adding a web interface, using Seaside. The first feature we've ported is a 'Site Map' view, a hierarchy of GIF images with configured 'hot point' rectangles for navigation. Users click to drill down to more other images or to display data in a table.

In VA the display is updated by replacing the 'image area' #labelPixmap and adjusting the window shell width and height. 'Hotpoints' are managed by mapping 'Pointer Motion' and 'Button Press' callbacks to a collection of objects that define the rectangle, which shows an action specific mouse pointer, and and triggers the action when the mouse button is pressed. All of this is user configurable using a view where they add image, draw rectangles and define actions.

A few years ago we added a feature which optionally displayed the current value of a measurement in the displayed image. It eliminated the need to click on a data 'hot point' and was handy for images with several data points. We've now added the same capability to the Seaside view.

In VA, the data is displayed by first defining instances of CwLabel in the imageArea (a WkImageWidget) at the correct position. The #labelString and #backgroundColor are then set based on user parameters, like the display date. Background color is used to show the 'quality' of value, so a grey background indicates incomplete data. To hide the data, the CwLabel's #visible property is set to false.

This is what it looks like in VA...


To do this in Seaside, we use a style 'position:absolute; left:10px; top:140px; z-index:-1;' for the GIF and then render the displayed data with...
 'position: absolute; top: %1px; left: %2px; z-index:2; border: 2px solid black; background-color: %3'
...using #bindWith:with:with: to set the top, left and background-color. The result looks like this...


I continue to be impressed with how easy it is to do interesting web work in Seaside.

Monday 11 April 2011

Dynamic Wiki

We use an internal MediaWiki web site for document and status reporting. Combining our static wiki text with dynamic output from our issue library and patch manager has worked very well for us. We use a MediaWiki webservice extension to add graphs, tables and issue lists.

For example, this code will get a list of my issues...

{{#webservice:<our internal web server>/Issues?wiki=user:Bob_N.| %//div%}}

...the output can be tested by using the URL directly in a browser, which answers wiki formatted output...
==Bob N. - current== 
{| ! ID ! System ! Client ! Title ! Stage ! Text ! P. ! Assigned ! Date ! Type |- | [http://<our internal web server>/Issues?issue=3353 3353] || BID || CS || HTS Customer Portal: changes to project display || [[image:Blueplay.png]] current || '' '' || 3 || Bob N. || 2011-04-11 || Modification |- |...etc.

...and, when parsed by MediaWiki, shows as a list of issues...


...with links back to the issue library. We combine the web service extension with Google charts to add live charts to our wiki, using an entry like...


{{#webservice:<our internal web server>/Issues?Issues?wikichart=backlog| %//div%}}

...which generates wiki formatted text that contains a Google Chart API call...


<websiteFrame> website=http://chart.apis.google.com/chart?&cht=bvo&chs=800x350&chg=0,10,4,0&chtt=Working+Issue+Backlog&chts=19293D,20&chbh=24,30&chco=00FFFF,00CCFF,0099FF,0066FF,0033FF,0000CC,000066&chdl=a: week|b: 2 weeks|c: month|d: 2 months|e: quarter|f: 2 quarters|g: 6 months&chd=t:0,0,0,0,0,0,0,0,0,0,0,0,795|0,0,0,0,0,0,0,0,0,0,0,766,767|0,0,0,0,0,0,0,0,0,0,0,730,699|0,0,0,0,0,0,0,0,0,0,660,661,639|0,0,0,0,0,0,0,0,0,540,543,491,474|0,0,0,0,0,0,290,465,456,398,381,344,335|141,182,214,254,282,289,256,235,76,72,69,54,54&chds=0,795&chxt=x,y&chxl=0:|Apr 2010|May 2010|Jun 2010|Jul 2010|Aug 2010|Sep 2010|Oct 2010|Nov 2010|Dec 2010|Jan 2011|Feb 2011|Mar 2011|Apr 2011|1:|0|79|158|237|316|395|474|553|632|711|790 name=Working Issue Backlog align=middle height=380 width=800 border=0 scroll=no longdescription=chart </websiteFrame>

...again, this can be tested directly in a browse to show the expected charted (the data shows a ramp up of using the issue library)...

Friday 8 April 2011

Patch Manager

We use a VW / Seaside based Patch Manager to send patches to our clients. Some of the design ideas may be of interest.

We cannot access our clients machines directly, but we can get permission for outbound ports through their firewalls. So our patch management process is based on polling: the remote patch clients connect to a patch server every minute, get a list of patch names and timestamps, and then request updates or send changes (for patches that are filed out at a client site). Commands, status, logs, exception walkback files are also passed along.

User access is though a Seaside interface to the patch server.  Developers select patches to send to a client from a table of published patches, filtered by applications and versions. Remotely filed-out patches can also be retrieved and published. If a patch is updated, the developer can choose to see the difference and can then send the updated patch. Our code file-out methods add a user and timestamp comment, which is extracted by the patch server.

In this example, = shows a patch that is published, > < shows an updated patch (current for one version, different for another), pressing an arrow either retrieves the updated patch or sends the older version (grey button for older). And + is for patches that have not been sent, pressing + sends the patch. Each patch is linked to an issue. Some developers prefer to prefix their patches with issues numbers, but it's not required.


Communication between the client and server images is by VW's Opentalk. As of version 7.7 it has been rock solid (version 7.6 has issues with brokers timing out).

Every ten minutes a 'retrieve client state' command is put on a 'pending commands' queue. At the next polling action, the command is retrieved by the patch client and a 'state file', generated by the client application, is send back to the server. If any of the client states indicate a problem an email is sent to the support team and a red notification icon is displayed on both the patch manager and issue library web page, as well as our in-house wiki. Pressing the icon shows a formatted view of the state file with errors in red. We are similarly notified if communication with the client fails, which is detected by an absence of polling activity.

A similar mechanism is used to retrieve client walkback files. The client image already emails the walkback file, so this is a backup process. It retrieves walkback files that were not able to be sent, and it builds a useful history of client errors (which, fortunately, continue to decline).

Not having to manage the patch files manually has proven to be a big productivity gain.

Thursday 7 April 2011

Graphing with Google Charts

We use Google charts to show various issue metrics. As is typical of a Smalltalk developer, I built my own class to manage the API calls. When I first looked into Google charts (two years ago, I think) I could not find community sourced options and building my own proved educational. Props to those that do build general purpose Seaside components, it's not trivial. My stuff is narrowly focused, and even that took some work.
My latest update added 'age' colours to an issue backlog bar graph. Something that may look like this...

Simple enough to do, but I tripped over an interesting API quirk. Originally, if you wanted to show a stacked bar graph, each value needed to have the lower value added to it (using cht=bvo, for chart type bar, vertical, overlap). So, if you wanted to show a bar with 10, 20, 15, 12 as stacked colours, the series was defined as 10, 30, 45, 57. Works great and it's easy to write the code to adjust the numbers. But, if one of the numbers is zero, the simple adjustment fails. If we want to graph 10, 20, 0, 12, the adjustment would be 10, 20, 20, 32.  But, that second 20 ends up overlaying the colour of the first 20.  What you really want is 10, 20, 0, 32.

At least that was true back then.  Using the Chart Wizard shows that you can use...

chxt=y
chbh=a
chs=300x225
cht=bvs
chco=00CCFF,0099FF,0066FF,0000FF
chd=t:10,10|20,20|15,0|12,12
chtt=Vertical+bar+chart
...to get as stacked chart like this in the Live Chart Playground...

It's stuff like this that makes the general purpose Seaside graph components impressive.  My original charts used cht=bvo instead of cht=bvs. Maybe it was supported, but I didn't see it (Actually, I still use the cht=bvo in order to keep the legend order the same as the graph colours). It takes time to keep up on all this, while still doing your regular job.  Thanks to those that do.

Wednesday 6 April 2011

Issue Library - how it started

A To Do list grows up. 

Some background on one of our projects: our in-house Issue Library, a Seaside application to track workflow. 

The Issue Library started off as a simple online 'to do' list, a way to track my own workload and to experiment with Seaside code. Others started to use it, and eventually it became a real application. A nice bit of guerilla marketing; a home grown issue tracking project would never have been approved. We did look at some open source options, but in order to make an issue tracker useful to a wide audience, it seems that you need to have a lot of options, with the app eventually getting really big and complicated.  Gjaller http://www.gjallar.se/ looked promising: I really liked the idea of working with a configurable Seaside code base. But I had no luck getting it to work.  

The design goals were to make it web based, simple, easy to update and more than a toy.  It needed to be real enough to learn the solid Seaside development details. Data is stored in XML files, one per issue.  Files are read at image startup and written when changed.  Since the Seaside image is the only thing that updates the XML files, the data in memory and on disk is consistent (and trivial to check).  Object are always in memory which give us excellent performance. Writes are fast, about 10ms. I've used this technique with my home 'Quicken' type app.  As long as the image size is reasonable, it works great. Right now the image, which also runs our Patch Manager, is about 225MB. As it grows, old issues will be archived.

Objects which map to XML files are subclassed from our an abstract class with all the XML support code. Read / write is done with #loadFile + #parseXML and #saveFile + #buildXML. 

Parsing is done with VW's XMLParser

[xml := [XML.XMLParser new
validate: false;
parseElements: stream]
on: XML.BadCharacterSignal
do: [:ex | ex resume]] 

Values are then extracted by interating over the XML elements.  Each element is sent as an argument to #parseElement: which subclasses implement.  As each element is found, the corresponding value is set and the next element processed. 

A utility method is used to find the tags. 

parseElement: anElement tag: aString doText: aBlock
anElement tag isNil ifTrue: [^false].
anElement tag type = aString ifFalse: [^false].
anElement elements isEmpty ifTrue: [^true].
aBlock value: anElement elements first text.
^true

For example, extracting the 'title' element is done with...
(self parseElement: anElement tag: 'title' doText: [:value | self title: value]) ifTrue: [^self].
...title is initilized to a new String, so if none in found in the XML file, or if it's empty, the title will stay as initialized.  Other string attributes are set to '<none>' so show that they were never set.

Collection attributes are extracted by iterating over the nested collection, using another utility method...

parseElement: anElement tag: aString doEach: aBlock
anElement tag isNil ifTrue: [^false].
anElement tag type = aString ifFalse: [^false].
anElement elements isEmpty ifTrue: [^true].
anElement elements do: [:each | aBlock value: each].
^true

To extract the list of comments, we use...

(self parseElement: anElement tag: 'comments' doEach: [:value | self parseCommentElement: value]) ifTrue: [^self].

...we create a new instance of a comment object and pass on the XML elements for it to parse...

parseCommentElement: anElement
| comment | 
((anElement tag type) = 'issuecomment') ifFalse: [^self error: 'XML parse error... expected ''issuecomment'' tag'].
comment := TSissueComment newForElement: anElement issueFile: self.
self comments add: comment.


Building the XML is done by adding XML nodes to an XML document.  This could probably be made cleaner with some utility methods, but it's a very static method.
buildXML
| document node | 
document := XML.Document new.
document addNode: (XML.PI name: 'xml' text: 'version="1.0"').
document addNode: (node := XML.Element tag: 'issue').
node addNode: (XML.Element tag: 'id' elements: (Array with: (XML.Text text: self id printString))).
node addNode: (XML.Element tag: 'title' elements: (Array with: (XML.Text text: self title asOneByteString))).
... 

#asOneByteString is a hack to eliminate any characters over an integer value of 255.  We have Mac users that are able to paste in characters that the XML.Document will not tolerate. 

Collections are just nested nodes...

buildCommentsXML: aNode
"allow for comments as strings, for initial conversion to comment objects"
| listNode | 
self comments notEmpty ifTrue: [
aNode addNode: (listNode := XML.Element tag: 'comments').
self comments do: [:each | 
listNode addNode: (each buildCommentsXML)]].

Where #buildCommentsXML adds text content to a new node...
  node := XML.Element tag: 'issuecomment'.

Simple, low tech and robust enough to handle our needs. 

Getting started

Hello.

Maybe it's time to start writing this stuff down. Over the years I've benefited from the public postings of others, and I've done a few of experience reports at conferences, usually with positive feedback. I'd like to do more, hence this blog. Let's see how this goes.

I work in Smalltalk.  Learned it in a York University night school AI class in 1990 and switched to it professionally in 1993. Something of an orthogonal move from my work as an MVS systems programmer at EDS. Lately I've spent a lot of time with Seaside.  I really like where browser technology is going: smart remotely connected clients, obviating the need to install remote code.  And on the Internet, no one knows you're writing Smalltalk.  I buy in to Avi Briant's view of Seaside 4.0: lots of Javascript talking JSON to the server (check out his talk at Smalltalk Solutions 2011 (hopefully the videos will be available soon)

Some of the stuff I've tripped over my be of value to others. I'll write about various Seaside projects, my work with automating SUnit tests and running the Toronto Smalltalk User Group.  All Smalltalk; no sports, travel or home renovation. As with anything, I believe this will get better with experience. Eventually this may even by useful.

For those in the Toronto area, check out the Toronto Smalltalk User Group at www.smalltalk.ca (or www.smalltalk.toronto.on.ca)