Cordys BPM Constructs 101a

I’m now really quite comfortable with Cordys process modelling and the models component constructs.  There are simple differences between certain constructs that I’ve used in other products such as MS Biztalk so I wanted to absorb the Cordys documentation and present a sort of quick and dirty intro to each construct and how Cordys uses them to execute your processes.  As there are several constructs and I wanted to go into some detail, I’ll present some of them here and the rest in a subsequent article.

If you don’t already have a copy of the Cordys community edition VM that runs C3 or BOP4 on a CentOS virtual hard disk, I’d suggest getting hold of it so you can ‘play’.

/*Note that most of the constructs can only be used if a user is granted the Business Analyst role, from the Cordys Classic Studio ISV package.*/

Start Event – Only one per process. Can have several trigger types:

– Message (a defined process specific message)
– Timer (by setting a time unit (e.g. days) and no of occurrence
– No Message or Timer

End Event – Process can have multiple end events. Error, Message and Rollback. You define an output message by dragging a ‘Document Type’ from the BPM Components repository to the end event. This then invokes a web service as the process ends. You can specify custom error XML in the ‘error details’ tab of the end event.

Activity – The activity is the construct that represents a generic step in the process and should be configured with an activity ‘type’. It can be of type ‘Application’ (web service calls) or ‘XForm’. When of type XForm, an additional XForm tab becomes available in the activity properties.  An activity can also be set to be a ‘dummy activity’ meaning it does nothing.  Each
activity can be conditionally executed based on conditional evaluation returning a boolean result.

Decision – The decision is a basic construct that allows multiple mutually exclusive outputs. It’s actual conditions are defined in the connectors that exit the decision construct and Xpath evaluations can use message map data for data driven decisions.

Intermediate Message Event – The intermediate message should have a process specific message defined and is invoked when a message is received to the process.  Normally this is used to ‘wake’ a waiting process instance. The gateway inmessage based on the process instance id (ensuring message correlation).  For asynchronous web service calls, an intermediate message can be used to receive the response message from a service operation call. Dragging the output message from message map will set this.  Therefore it is useful to know whether
the operations you are calling via your activities are sync or async.

Compensate Event – Used to rollback any effects of an activity or context or activities when an exception is caught.  A compensate event can be defined for an activity, sub-process,
context (embedded sub-process), for each, while and until loops. An activity or sub-process can only have one compensate event associated. Compensate is represented as two left pointing arrows, side by side in a circle.

Delay Event – An intermediate event that stalls the process for a configurable amount of waiting time. Delays can be of type:
– Fixed delay.  That is a set amount days, hours, minutes or seconds.
– Message read delay.  That is a delay value read from a process specific message using an Xpath reference. As with a standard activity, a delay may be conditionally executed based on a boolean return evaluation.

Exception Event – An exception is an event which is fired when a Cordys error occurs (e.g. a soap fault is detected via external service responses or via SOA grid messages).
As you would in programming code, you can specify a specific exception name to catch or just catch any exception.  Once an exception is thrown, process execution is diverted to the connected exception event where subsequent exception handling activities should follow.  There are several error types which can be caught and which are configured on the exception event construct:
– All. All exceptions are caught, regardless of the error code.
– Custom Error. You can set an error code you may expect from a service response indicating an error has occurred (e.g. a SOAP fault
– System Error. Simply put, a Cordys internal error where some part of Cordys may not be operational (a soap processor may be down for example) or syntax rules are broken.
– Communication’s Error. These errors are specific to SOAP faults returned from external service calls. Error code 11 is returned for comm’s failures.

There are error codes that are only raised when a sub-process is being invoked from a parent. These are:
– Process Loading Error
– Process Instantiation Error
– Process Model Not Found Error

… more to follow soon!


Metastorm BPM : It’s not an application development tool

After 2 years of designing a large operational system using Metastorm v7.6, I wanted to reflect on why it’s a bad idea to use Metastorm BPM to build big workflow based systems.

The problem with the finished system is not that it doesn’t satisfy the requirements or doesn’t perform well enough (considering), it’s that it is a maintenance nightmare.  I wrote an article this time last year whilst travelling back home to Holland from being snowed in and which concerned why maintainability is the most important design factor (over and above scalability and extensibility).  Coupled with a ‘big bang’ design approach (over an agile dev approach) and consistent requirement changes, it’s a surprise the system runs in its operational state.

I don’t wish to run the product down, because for small to medium workflow driven applications, it does the job. But, it’s clear lack of object orientation is the biggest single product flaw and when building a big system with Metastorm this cripples maintainability.   A solid design architecture is obviously of major importance.  Basic application architecture fundamentals such as breaking a system design down into cohesive functional ‘components’  that represent ‘concern’ area’s for the application can be difficult to implement.  This is down to the fact that process data is stored in the database per process and passing data between processes using flags can become messy, especially when certain characters are passed using those flags (that Metastorm classes as delimiters).  Sub-processes are then an option, but these also have inherent flaws.

Forms, which again are application components are process specific, so re-use is again suffering and so replication of forms has to be done, further disabling the idea of good maintainability.

Having data repeated in processes and having no code dependency features is bad enough, but because you have to remember where you have used process variables and keep in mind when and where values may change, the tool puts all the responsibility on the developer.  Once the system get’s very large, the event code panels (the ‘on start’, ‘on complete’ etc) get very complicated and tracking variables and when they may change etc becomes a struggle in itself.  Changing a variable value in a large process has the risk of making other parts of the process not work quite right because ‘you’ve forgotten that the variable is used in a conditional expression later on’.

This then begs the question, should you even use the Metastorm event/do this panels for ANY business logic.  I’d say no.  Only UI or process specific logic should be used and you should push ALL of your business logic to server-side scripts and a suite of external .NET assemblies.  You can then at least implement a fully swappable business logic layer.

So along comes v9.  This product is a great move towards stronger application architectures.  OOP design and ability to debug alone save a whole lot of system maintenance time.  So although this version takes us closer to being able to create solid, maintainable operational applications, it was released too early.  It is slow (halving development productivity against version 7), it had many broken features and grids, one of the most used visual components, especially for data driven applications (which is most business apps) were just terrible.  They continue to perform almost independently from the rest of the system and patch 9.1.1 is still addressing grid shortfalls.  Obvious shortfalls which should have been picked up by a thorough QC team @ (OpenText) Metastorm.

The new OOP approach means that designers and developers no longer have to use the ‘Line by line interpreted’ syntax of v7 and can design re-usable components.  So there is a greater case for using Metastorm BPM as an application development tool for fair-sized applications but whilst development productivity is still poor and the designer is still very buggy, it’s not quite there yet.

Cordys BOP4 : Messaging on the SOA Grid

I’ve spent a few weeks getting used to Cordys BOP 4 and as I usually try and do with a new product, I wanted to know more about what’s going on under the bonnet with it.  The central coordinating component of Cordys is its SOA grid, which takes care of messaging between all of the core Cordys services and other web services.  Based on the information provided in the Cordys offline documentation and because I’m a visual learner, I’ve drawn up the following image that should hopefully shed some light on how Cordys organises its internal services and how they communicate via the SOA grid. Click on the image to zoom to actual size.

What I’m trying to show here is how Cordys deals with an inbound service request.  The dark line represents the path of the message along the service bus.

To illustrate an example of how the above image can be used to understand what Cordys does is the request of an XForm from the Cordys client.  The client sends a request to display an XForm so sends a HTTP request to the web server for a resource of .caf file extension.  The web server, based on the .caf file extension hands the request over to the Cordys web gateway.  The web gateway contacts the LDAP service container and checks for the location of the XForms service container (the LDAP service must always be up and running for proper SOA grid functioning).  The LDAP service container has an LDAP application connector which talks to CARS.  Next the SOAP request is sent to the XForms service container and the XForms engine takes care of rendering the HTML response.  Not only that, but the XForms engine also validates controls against schemas and automatically contacts other web services required whilst rendering.  Once the HTML is generated, it is returned via the SOA grid to the Cordys web gateway, then back to the calling client.

I should mention at this point that web services on the SOA grid are called based on the service operation name and namespace in the SOAP request.

This is very high level and it’s always a good idea to read further into the Cordys documentation, but I hope this graphic helps to illustrate the architecture of services, service containers and service groups on the Cordys SOA Grid.

Understanding Multi-tenancy

I’m doing a lot of research and practical ‘playing’ with the Cordys BOP 4 environment at the moment.  It’s a relatively young product from a young company (founded in 2001) but from what I have understood about the product and its architecture it is a strong, versatile collaborative tool that really supports rapid application development and maintenance for business change thus reducing general cost of ownership.  I don’t want to talk about the product itself, I will be doing that soon enough, but I wanted to cover one of the software’s best features in my opinion and that is its ability to operate within the enterprise and in the cloud utilizing a fully multi-tenant architecture.

Multi-tenancy is an architectural principle which says that a single install of a software program / server application can service multiple client ‘organizations’, providing a customized isolated application environment with its own data.  This is in complete contrast to the idea of multiple instances for each organization (i.e. installing an instance of some server software and underlying database to serve one organization and store only that organizations data).  Each ‘organization’ (in the general sense of the word) is classed as a tenant, using the application, therefore if one single installed application can serve multiple tenants their customized view of the application, then it is said to have multi-tenancy (supports multiple tenants). Google apps is a perfect example of a multi-tenant architecture.  Multi-tenancy is the foundation architecture for most if not all Software as a Service applications, thus cloud applications support multi-tenancy.

How multi-tenancy is implemented in an application can vary from application to application.  The general principle however is to have the application virtually partition portions of its data and services to different consuming organizations.  A very simple example at the data level would be to have the application generate a set of related data tables under new database schemas as new organizations are set-up to use the application (so a schema per organization).  This separates off the data into logical groups that have their own security context on the database.  There are other ways to partition the data, but this is just to illustrate one potential method.

So multi-tenancy is a software architecture and one that is prevalent in cloud applications.  Cordys BOP 4 does this very well and I’m looking forward to investigating this product and its virtualization capabilities further.

Metastorm 9 : Ignores ‘Client Paging’ property

Grids were one of the biggest reasons I upgraded to 9.1 from an application written in  Mainly because the ability to wrap text in grids was removed in v9.0 and so columns with large amounts of text to display became unreadable.  Now although this has been fixed in 9.1, grids are still throwing up issues.   The first is noted in my previous post where you cannot assign a business object field to more than one column of a grid (worked in v9).  Second, the grid page size on many (if not all grids) appeared to reset to 50, regardless of what was previously set.  Lastly, it appears on deployment that 9.1 is completely ignoring the fact that I’ve unchecked the client paging check-box as I don’t want to deal with pages of data in the grid.  The below example shows what is set in the designer and the resulting form after deployment….

It is clear that the client paging property is not checked for this grid.

You can see from the above however that paging is still active on the grid (the pages are shown to the left of the screen grab, but the white space represents the paging bar). Clearing the browser cache doesn’t help in any way.  It really makes me wonder how much in-house quality control testing is done at Metastorm.

Metastorm 9.0 to 9.1 – Same key has been added

So I performed the relatively simple Metastorm 9.0 to 9.1 upgrade this week and for the most part, functionality for existing applications has remained.  There are however a few issues I experienced, the biggest being a new message appearing on a few of my forms. ‘An item with the same key has already been added’.

The error kept showing its face on grid heavy forms, especially when you select a row in a grid or where there are buttons on a form and the grids are dependant on other fields.  After some trial and error (and guess-work) it became apparent which grid was causing the error to occur and further to that I noticed that the grid had the same business object item assigned to 2 columns.  I killed one of these columns, redeployed and all was well. The annoying part is that this worked in v9.0 with no issues.  So it seems that in 9.1 you can only assign business object items to a single grid column only.

Metastorm announcing BPM 9.1

This morning at US Eastern Daylight Time, Metastorm are set to make some product launch announcements including Metastorm BPM 9.1.  The other product announcements include Smart Business Workspace 9.0 and Metastorm Business Performance Intelligence Dashboards and the predefined dashboards for Metastorm Business Process Management, Metastorm Integration Manager and Metastorm Knowledge Exchange.  These boosting the product(s) visibility to business users.

For me personally, the Metastorm BPM 9.1 release is the most welcome news.  I work with this product every day and if you’ve read my recent posts, I’ve shared my concerns about its general stability and poor performance.  It sounds like many new features in this release surround the client UI and building of forms with this version (hopefully!) introducing Panels (collapsible groupings of form controls), Field Anchoring (spacing your form content when window maximized), improved field visibility options, dynamically visible actions and I’m sure many others.

Many Metastorm BPM developers are looking for big improvements in this release and are hoping that Metastorm has made the development experience more efficient.  It has taken some getting used to the new way of design and it can be tedious at times, so any new features that boost development efficiency I can see being received with applause by the development community.

Let’s wait and see what comes to pass.

Metastorm 9 Designer Performance

Ok, so reading back over my last few posts, I appreciate that I’m on a bit of a ‘moaning’ streak about Metastorm 9.  I think before I talk about my experience with the BPM Designer IDE, I need to offer up a disclaimer of sorts.  I appreciate version 9 of the product and the Designer in particular have  had quite an overhaul and a platform transplant (now .NET based).  The way we now design processes does feel more solid and reliable and the use of C# in designing business services / login and the ability to deal with data as objects is far, let me repeat, FAR better than in 7.6.  So in terms of the efforts put into version 9 to release a piece of software that was albeit more developer friendly (less analyst) but was focused on a more maintainable design structure is appreciated.

The area I think that has suffered with the release of 9 is the performance. Especially of the Designer. In short, the Metastorm BPM 9 Designer is SLOW.

Performance Vs Visual Studio

So I’ll reveal my machine stats before I start.  This machine is only a development machine and runs 32-bit Windows 7 because of the problems encountered with Designer on 64-bit.  The processor is a 2.26GHz dual core and the machine runs on 3GB of RAM.  Not that fastest, but as I say, it’s for development only.  I run Visual Studio 2008 alongside the Metastorm designer and regardless of machine performance, Visual Studio, being a far bulkier product is lightning fast compared to the BPM Designer.  I can literally build (syntax check and compile) C# code into a managed module (wrapped by an assembly) in the time it takes me to select one of the input parameters of a custom visual script activity.  That is NOT good.

Load Time

So, the first and most widely reported performance issue with the designer is the start-up.  If you’re running SR3 then this is no longer such an issue compared to the initial release of 9 and even SR1 which had terrible start up times. It is still an issue however.  An obvious factor is the start page, which reads in the homepage from the Metastorm Community Central site (content and images).  This should not be a major issue as reading in a page with minimum graphics on even the most basic broadband connection is quick.  An ASP.NET server will accept a HTML page request, build an executable environment on the server with the page/required objects, then execute the code and respond to the client with the output HTML page in a matter of milliseconds, so I don’t believe this is a major hold up, but it does at add to the load time.  You can however turn this off via the firewall if you’re not interested in what the community is talking about.

As far as tests go, Metastorm Designer alone loads on this development machine in 22 seconds.  This is the time when the designer is in a usable state and the ‘timer’ has stopped spinning.  For visual studio 2008, I’m ready to work in 9 seconds, SQL Server Management Studio in 7 seconds and Excel within 3 seconds.  So whatever you can say about Microsoft products, they get the job done and are quick with it.  I found that running the Designer by clicking on a solution file within windows explorer adds approx another 20 seconds to the loading time, running at an average of 40 seconds across 3 separate tests.  Obviously the loading of the file and the validation of the .NET code adds to overhead here, but this is still uber slow.

General Operational Performance

In terms of general operation, I’ve never used an IDE or any design application (e.g. including any version of the Adode Creative Suite) that has to think as much about every action that takes place.  Whatever design action I seem to perform, there is a short lag in the designer responding to what I’ve done.  This can sometimes only be a second, but I don’t expect to have to wait for everything done, especially if there is a deadline looming for unit testing or client playback.  It feels very much like I am asking Designer to perform a task and it is deciding whether to grant me the honour.   Some examples:

– Sometimes selecting an item from the Toolbox and dragging onto the design surface (forms generally) can be completely ignored because the Designer is busy with something else.

– Creating roles is painfully slow.  Adding a name then tabbing out requires about 3 seconds of thinking time, what is being done here a low level call to the OS? (being sarcastic)

– Real time code validation on server-side scripts seems to take about 5 seconds.  When you change your code because of a syntax error, which you have to rely on your spidey C# senses for instead of visual studios neat ‘Errors List’ box, the Designer takes a good 5 to 8 seconds before re-validating your code and confirming all is well.  Why?

– Loading up the visual script for an on complete event takes about 7 seconds.

The worst for me however, that requires a paragraph all of its own is the absolute drag that is filling in the parameters / arguments of a custom visual script activity that has been promoted from a server side method.  Just after dropping the visual script activity, which now appears in the visual scripts toolbox you can complete the visual script arguments, which essentially make up the method signature.  As i click into each argument to fill in the data to be passed at run time, it literally takes about 4 seconds to place the text selector into the field.  Then I have to click the small expression builder icon, which takes another few seconds.  This stuff should be instant people, I’m not asking for any computation to be done, just Focus().

Overall, I’m happier to be designing Metastorm processes this new way, but it feels like such an uphill struggle to design what I could of designed in 7 in half the time.  I still believe that longer term, you will spend less time maintaining the application because of its greater stability, but trying to get a design off the ground by building the fundamental layers is just tedious.  There is talk that 9.1 will address many of the performance issues, let’s hope so, this application really is the slowest IDE I have ever used.

In terms of productivity, I could go as far as saying that building an ASP.NET application that utilized Windows Workflow would be a quicker development task that would be easier to debug and test.  The question arises – Why invest in a tool like Metastorm when you are not really gaining the benefits of having an integrated UI, Business Logic and Workflow environment?

Thoughts on this topic are welcome. Last rant for a while, honest 😉

Metastorm 9 : Call to SSPI has failed

Metastorm 9 is a great product. It’s a leap forward on its v7.x parents and despite its initial productivity slow downs, you generally feel like you’re building a more solid / stable application with v9.  This being the case I do have a small number of gripes to get off of my chest that focus on  (blame;-) the product testers over at Metastorm.  Did they actually try and deploy anything on a 64-bit system during testing?

1) Designer incompatibility

My last post covered this point, so I don’t plan to go into much detail here but releasing a product that fails to deploy your solutions on a 64-bit install just because the COM objects the application uses can’t be loaded into the executing process is poor design.  There are ways to load 32-bit assemblies without completely neglecting the fact you need to work with non 64-bit compatible COM objects for your product to work properly.

2) Remote Deployment Security Validation

So if you’ve gone with installing your engine to a 64-bit machine, realized that you cannot deploy from the Designer application on that machine and moved the Designer onto a 32-bit machine, you’ll need to adopt remote deployment.  To deploy to the 64-bit engine, you have to point the Designer install to the DeploymentServiceConfig.xml file being hosted in the escripts folder of the engine server.  This can be done during the Designer installation, via the registry or via the Designer options.  This xml file contains the tcp location of the deployment service(s).  The service is a Windows Communication Foundation service with a tcp endpoint (hence the need for net.tcp port sharing on the server).

What you may find if you are running Metastorm BPM v9.0 or v9.01 is that deployment fails and you are told that ‘A call to SSPI has failed’ and are advised to review the inner exception.  Interesting when a stack trace is not displayed to you via the Designer so you are not able to review the inner exception.  SSPI is the Security Support Provider Interface for windows operating systems and it’s job is to accept windows credentials from an application (via RCP, Named Pipes, DCOM, Winsocks2 etc) and authenticate those credentials against the operating systems security accounts manager (SAM) database. So in effect its the security middleman.  Clearly in this case, either the windows account used by the remote Designer or the account running the WCF service locally on the server is not able to be passed to SSPI (i.e. call failed).  Hotfix sorts this issue however, which is great, but come on Metastorm, how does your initial product ship with an inability to fully support remote deployment of solutions? That’s just sloppy on the part of the product testers and quality control staff, especially when the SR1 release fails to address the issue and a hot fix is needed?

3) Security

Why in v9 am I still finding that the service/database password(s) provided during the installation of the server are stored in the registry?  Maybe those who have not noticed this yet may think that sharing this information is a security risk in itself? It should be.  Metastorm need to change this. Passwords sucked into any application should NEVER have that password stored in clear text anywhere on the system, full stop.  Encryption is not hard to come by.  You can read a password into the application, encrypt it down to a persistent store, then decrypt it when it is read back into the application (even a simple 8-bit key DES encryption scheme is better than nothing).  Application security 101.

Rant over 😉

Metastorm Designer 9 on x64/64-bit

I’ve seen a few blog posts and have ran into this problem myself.  Metastorm v9 will run fine on an x64 system architecture, but the designer has one problem with this.  When it comes to deploying your Metastorm solution, the deployment fails and doesn’t tell you much about why it has.  For those developers who know anything about Windows, you know to go look in the Application Event Log and read the issue logged.  It probably says something along the lines of :

System.Runtime.InteropServices.COMException (0x80040154): Retrieving the COM class factory for component with CLSID {0E59F1D5-1FBE-11D0-8FF2-00A0D10038BC} failed due to the following error: 80040154

The Metastorm Designer itself runs within a 64-bit process and you can tell this my opening up task manager and noting that it does not have a *32 next to the Designer.exe process (which would indicate it was running in 32-bit).  When the Designer attempts to use a COM object that will only run within a 32-bit process, it fails to instantiate / create an instance of that COM object.  The CLSID (Class ID) shown in the event log error above is the unique ID of the COM component that is unable to be loaded (this ID also stored in the registry as COM components must be registered unlike .NET components).

The offending COM object, is the MSScriptHost / Script Control, which is used to interpret windows scripts written in JScript and VBScript.  .NET Interop is the set of services that allows .NET to easily wrap COM Servers exposing a simple .NET interface (aka a .NET Callable Wrapper) that can be used via .NET applications.  Metastorm v9 is written in .NET and communicates with the required COM components in this way.

You can see from the following stack trace that Metastorm attempts to validate the client side scripts before the issue occurs:

at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at Metastorm.Common.Scripting.Interop.ScriptHost.MSScriptHost..ctor()
at Metastorm.Model.Scripts.Validation.ClientScriptValidator.CompileClientScript()

The quick fix here is to install the designer on a 32-bit machine and reference the 64-bit engines repository service.