EAI

Transactional Isolation


Transactional isolation is usually implemented by locking whatever resource is being accessed (db table for example) during a transaction thus isolating the resource from other transactions.  There are two different types transactional locking: Pessimistic locking and optimistic locking:
Pessimistic Locking : With pessimistic locking, a resource being accessed is essentially locked from the time it is first accessed in a transaction until the transaction completes, making the resource unusable by any other transactions during that time. If competing transactions simply need to read the resourceonly (a SELECT of a data row for example) then an exclusive lock may be overkill. The lock exists until the transaction has either been committed or rolled back at which point, the resource is made available again for other transactions.
Optimistic Locking : Optimistic locking works a little differently.A resource being accessed isn’t locked when first used, but the state of the resource is noted.  This allows other transactions to concurrently access to the resource and the possibility of conflicting changes is possible. At commit time, when the resource is about to be updated to persistent storage, the state of the resource is read from storage again and compared to the state previously noted when the resource was first accessed. If the two states differ, a conflicting update was made, and the transaction will roll back.
Advertisements

Biztalk 2010 : Configuration Guide


Biztalk is Microsoft’s Integration and Process server (sometimes referred to as an Enterprise Service Bus), is now in its 2010 edition and seems to be going from strength to strength 10 years after it hit the market.  I’ve worked with versions 2004 and 2006 R2 of Biztalk in the past for EDI projects and it’s a great product with a lot to offer.  It has custom adapters for connecting LOB systems and is heavily integrated with the .NET Framework, especially WCF.  The great thing about this new version of Biztalk is that the installation is easier than ever. Installation has become simpler with each release since Biztalk 2006 – I remember Biztalk 2004 being a nightmare to install when it came to its prerequisites installations (SQL Server SP’s etc).

For Biztalk 2010, al the prerequisite components required are downloaded for you by the installer (if you don’t already have the .cab files available on the disk) and the once the installation has finished (it takes about 15 minutes), the basic server configuration option ensures you are up and designing Biztalk applications in a very short time by auto configuring single sign on and setting up all the databases and service accounts for you using the single account specified on the first page of the configuration screen.

In terms of the audience for this post, if you are a newbie to Biztalk and wish to follow along with the configuration steps, hopefully this is simple enough for you. I do reference certain Biztalk features without explaining them in any detail so the more experienced developers may feel more at home having setup previous installs of the server.  In terms of environment for this install, I am using a Windows 7 machine and will be setting up a single server as no pre-existing Biztalk server environment exists to join with on my network.  I already have .NET 4, Visual Studio 2010 and SQL Server 2008 R2 installed.  In terms of the software component pre-requisites for the install, here is the list:

Prerequisites:

– Microsoft SQL XML 4.0 with Service Pack 1

– Microsoft Office Web Components

– Microsoft ADO MD.Net 9.0

– Microsoft ADO MD.Net 10.0

– Setup runtime files

– Enterprise Single Sign-On Server

– Enterprise Single Sign-On Administration

– Microsoft Primary Interoperability Assemblies 2005

Features Being Installed:

– BizTalk EDI/AS2 Runtime

– Windows Communication Foundation Adapter

-Portal Components

– Business Activity Monitoring

– Windows Communication Foundation Administration Tools

– Developer Tools and SDK- Documentation

– Server Runtime

– Administration Tools And Monitoring

– Additional Software

– Business Rules Components

– BAM Alert Provider for SQL Notification Services

– BAM Client- BAM-Eventing

Installing Biztalk 2010

Now I’m assuming you are either installing a paid for version of Biztalk or are trying out the trial offered on the Microsoft Biztalk site and so have the legit zipped installer package.  Once you have the installer package extracted to your disk, select the setup.exe file from the BT Server folder.  The install should be pretty self explanatory. Select the features you wish to try out and select install. Grab a coffee whilst this screen does its thing:

Don’t worry about the installers lack of interaction with you. Biztalk installs the required server components first and allows you to configure the server databases and register the server components later, which is what we’ll walk through now using the Biztalk Server Configuration tool.  This can be found in your program folder in the start menu:

The initial configuration screen will ask what mode of configuration you wish to proceed with.  In this case, we are choosing ‘custom’ as the basic mode performs all of the configuration for you and that’s not much use when we’re trying to understand the configuration process.  Supply the screen with your SQL Server instance name and also provide a windows account for use by the Biztalk service.  If you use an account that is part of the local or domain administrators group, you will be warned of the security risk related to doing so, but you can continue.  It isn’t advised to be using an administration account for a production Biztalk installation.  Select the ‘Configure’ button to view the following screen (note – this image was taken post configuration so shows items already configured. Your screen will contain red circles illustrating non configured features):


Enterprise Single Sign-On

The first of the server features to configure is single sign-on as most of the other features require this for their configuration.  Select Enterprise SSO from the tree, enable single sign using the check box provided and create a new SSO system.  Doing this will create a new single sign on database (SSODB) onto the SQL Server specified. A SQL Server login for the SSO Administrators group is also created.  As well as the database, the single sign-on service will be set. I am using the same service account specified earlier.

Before leaving SSO, ensure you set a SSO backup password and a location for the backup.

Biztalk Server Group

Next we join or create a Biztalk Group.  Your Biztalk server needs to be part of a group enabling you to manage your Biztalk server infrastructure at the group level.  In this instance I’m installing to my development machine only so will be creating a new Biztalk Group which I will manage from this machine.  Enable the Biztalk Server Group and then create a new group. If you’re familiar with Biztalk you’ll see the configuration for the Management and Tracking databases and of course, the all important MessageBox database.  Again, specify the database names (I always stick to the defaults) and select your Biztalk role to windows group mappings for Biztalk Admins, Operators and B2B Operators.

Biztalk Runtime

Next up, we have the critical Biztalk runtime. This is the engine of Biztalk and deals with processing and routing of messages into the messagebox and out of the system. You may also be paying for your Biztalk server on a runtime basis (license per runtime).  Ensure the ‘Register the Biztalk Server runtime components’ checkbox is ticked.  Biztalk applications are hosted within a process host so whilst registering the server runtime, leave the ‘BiztalkServerApplication’ and ‘BizTalkServerIsolatedHost’ checkboxes ticked.  Both of these are in-process hosts, so will need a windows account to run on. I’m using the same account used for SSO.

Biztalk Rules Engine

If you’re going to creating rule policies / vocabularies etc, you need to activate the Biztalk Rules Engine.  The rules engine allows you to abstract the business rules from your orchestrations (processes) to enable easy management of rules away from the process itself.  Rules are such things as ‘if PO cost is greater than £500.00 and cost centre = ‘101ADM’ then do this’.  Ensure ‘Enable Business Rules Engine on this computer’ is checked and specify the business rules database store (again I would recommend leaving the default  ‘BizTalkRuleEngineDb’).

Business Activity Monitoring [Portal]

BAM is the business activity monitoring side of Biztalk. It allows you to monitor certain business KPI’s against your process instances so that business stakeholders have real time updates on how their business is doing.  Monitoring of course is one of the key elements in process management and is one of the core drivers for process improvement.  If you plan on using BAM, enable it and confirm the database names.  As this is my development server, I won’t be adding BAM Alerts at this time, even though I do have SQL Server
Notification Services installed.  Last up on BAM and the configuration in general, we will be enabling the BAM portal.   This is the window into the activity data produced by BAM Services and is hosted on the local IIS server.  This section deals with some of your web server settings and for now I will be adding the site to my Default Web Site on IIS (version 7). You can of course setup your own site / application pool for this.

EDI/AS2

In this instance I have chosen NOT to install the BizTalk EDI/AS2 runtime as I’m not planning on creating any X12 or EDIFACT documents anytime soon and unless you are familiar with these areas or are actively using Biztalk to deal with EDI files it’s best not to install this feature.  The AS2 part of this relates to HTTP security (AS1 – SMTP, AS3 – FTP).

Applying The Configuration

So… I’m all done with my servers configuration. I have configured my Biztalk installation and am ready to Apply my configuration preferences.  Select ‘Apply Configuration’ from the top left of the server configuration window.  Biztalk will present you with a summary of your configuration choices as shown below:

Once you proceed, the configuration will take a minute or two and once completed, you will be presented with a list of green ticks (hopefully) against your server features. Any problems will show up with the familiar red circle with cross.  Typical issues can involve database rights and authentication using the windows account credentials provided.

Once the server has been configured, we can check all is well by opening up the Biztalk Server Administration application.  This is the application for managing your deployed Biztalk Applications and all of their artifacts.  Expand the Biztalk group and if all is well, you should see the default Biztalk applications and artifacts list as shown (notice we connect to the Biztalk Management Database for this info):

Another useful check is to boot up Visual Studio 2010 and check that Biztalk project template is available to you:

Congratulations, you are ready to begin creating XML Schemas, WSDL, Orchestrations, Custom Pipelines and more.

Implementing a BPEL Process on GlassFish ESB


Note : Source XSD files for this project can be found here. Create 2 XSD files from the attached word document content.

I’ve been a .NET developer for a good few years now.  I’m familiar with OOP concepts and the .NET architecture and recently I’ve had the language / framework itch.  I’ve always wondered why Java remains such a popular language for rich client, web and embedded systems and decided, finally to jump into Java.

My initial thoughts where how similar in syntax C# is with Java.  There are of course small differences and Java appears to only have about 70% of the language features that C# / .NET offers, but for .NET developers, Java syntax and language features could really be picked up over a weekend.  The architecture of the JVM should be an initial read also.

Once I felt comfortable with the differences in the language, the differences in J2SE, J2EE and J2ME and how the component / container architecture of 2JEE works (JSPs, Servlets, EJBs etc), I was very keen to jump into the area I know and enjoy developing the most, integrating systems via messaging.  For this task I’m using OpenESB, a business integration platform similar to BizTalk (deploying OpenESB applications to the GlassFish Server instead of the Biztalk Server).

The key word here is OPEN.  The platform is entirely based on open standards such as XML, SOAP, WS-*, Java and Java’s SOA /  Business Integration approach called JBI (Java Business Integration).  Now the title of this blog is GlassFish ESB,  this is an edition released as part of the GlassFish Application Server that includes the core Open ESB engine and allows integrated and easy design with the NetBeans IDE.

I should stop for a moment and clarify what ESB is all about.  This acronym stands for Enterprise Service Bus.  The enterprise service bus concept is a kind of middleware architecture approach that provides a central messaging (including message brokering and routing), process orchestration /execution and message transformation (mediation) platform that allows software and systems of varying types to communicate across the enterprise using open standards, most importantly XML and SOAP.   Biztalk, Websphere, Business Integration Server (by Seeburger), MuleESB, JBossESB etc are all types of ESB.

Deploying a basic BPEL process.

The GlassFish server and Open ESB core components are installed with the NetBeans IDE install if you download the installer from the OpenESB website.  At the time of writing, Open ESB runs on GlassFish 2.2, but I’ve not managed to install to v3 of GlassFish.  To illustrate the ease of use and how quickly you can deploy a BPEL process to GlassFish, we will create a small BPEL module that receives a message from a SOAP endpoint, follows a simple BPEL process and returns a response message to the endpoint.  This is the standard first ‘example’ from the OpenESB site and forms a simple Loan Application process.

Create a new ESB Project

First, you need to launch the NetBeans GlassFish ESB IDE (integrated development environment) and create a new project. Select the BPEL Module project and select ‘next’.  Specify the project name as bpLoanApplication and leave ‘Set As Main Project’ ticked. Once the project has loaded, take a look at the Process Files directory. We will be adding the message schemas, WSDL and BPEL process file in here.

So, we need to define the request and response message structures using XSD files and then build a WSDL file using these message XSD’s to allow a service client consuming this process to understand the structure of the request message to send.  XSD files represent the XML schema of the message and includes important information such as element names and element data types. Here is our request message schema:

Import both XSD files included in the source files (Tutorial Files > jbiConcepts > Service Registry > LoanRequestMsgs) into the project.  These represent the request and response messages.

Create WSDL File

Next we need to create the WSDL file that the service consumer will read using the standard url?WSDL link.  WSDL is the web service description language and its mission is to describe to consuming clients what operations the web service offers so that the client can determine what it needs to do to interact with the service.  In the projects window, right click the ‘Process Files’ directory and select ‘Add’, choose ‘WSDL Document’.  The wizard will open and you can specify the WSDL file name (loanApplication.wsdl) and select ‘Next’ to move onto the main Configuration screen where you will define the Input and Output message parts.  You can select nodes from the already imported request and response XSD files and NetBeans will auto detect the data types of the message elements.  Once you  have specified the inbound and outbound message structures and specified a port name, you can select ‘Finish’ and the WSDL file is added to the process files folder.

Create and configure the BPEL Process

Now that we have our messages and WSDL contract, we need to create the BPEL process and associate the inbound and outbound messages with the process.  Again add a new file to the ‘Process Files’ directory and choose ‘BPEL Process’.  You will see the default process diagram appear in the design area.  Using the Palette on the left, drag Send, Receive and Assign shapes to the process diagram as illustrated below.  Receive and Send actions should be self explanatory, but the Assign action will map data values from the request message to the response message.

Next, drag a ‘Partner Link’ onto the left hand side as per the above and call it ‘plApplicant’.  This partner link will represent the logical message entry and exit point to the process.  This partner link will be associated with a SOAP endpoint later on.  Lets now configure our partner link and message send/receive actions.  Double click the partner link and a properties window will open up to allow you to associate the WSDL file created earlier.  Using the WSDL, the partner link type can be specified.  Once the partner link has been configured, we need to configure the Receive message action; double click the Receive action and in the properties window  you can now associate the recently created Partner Link name, the Operation name (the request operation specified in the WSDL) and the creation of a local variable that will represent the message within the process (ApplyForLoan). Leave ‘Create Instance’ ticked.

We need to also configure the Response action here.  Double click it and in the opened dialog again specify the single partner link and partner link operation as well as specifying a variable for the outbound message (ApplyForLoanOut, See below).  We do this as we will be mapping data from the inbound message variable to the outbound.

Assign values to the response message

Now that we have created our process structure, a logical partner link which will link the process to an endpoint of some kind and our Request and Response message handlers, we can now map the inbound message data elements to the outbound message.  This is done using the ‘Assign’ action placed in the centre of the process.  Double click it to open up the assignment mapper interface.  The mapper interface allows you to map values (message elements) from the request variable we created earlier, which contains the inbound message to the outbound message variable we specified, which again contains the outbound message schema.  In short, you create a relationship between source and target nodes (message elements) and if necessary can perform a group of ‘transformation logic’ actions on the pane in the centre of the mapper.  In this example, I am just create two static string values that will be mapped to the outbound message.  You do however have the ability to run some XSL transformations on the data, run some Java code on the data and using Operators and Filtering functions can perform decision based transformation.  Those familiar with the BizTalk Orchestration Designer will recognise these as Functoids.

Once you have completed your map, save it and close it.  You are now ready to build the ‘process’ part of the application also known as the BPEL Module.  Right click the ‘bpLoanApplication’ project and select ‘Build’. The build.xml output window will show you are the results of the build.  You may see a warning that warns that certain message elements are not used, this is fine as its only an example and we are not using all parts of the message.

Creating a host application for our BPEL module

Now that we have created our BPEL Module, we need to host it in a Composite Application.  We do this by creating a new composite application project that will host our initial bpLoanApplication module.  Select ‘New Project’, choose ‘SOA’ then select ‘Composite Application’ – call it ‘caLoanApplication’.  This will a new project to the project window, ensure this is now the main project. When the project loads, you will see an empty design surface that is awaiting WSDL ports and one or more JBI modules, which in our case is our BPEL Module.  Using the palette, select a SOAP binding from the binds list and drag it to the WSDL ports area on the left.  Note that will will not be able to configure the endpoint until the caLoanApplication project is built.

Next, we need to drag our BPEL module to the centre part of the design surface so that a message sent to the SOAP endpoint will be sent to the BPEL process we have designed.  The great thing here is that you don’t have to go out searching on the disk for the modules .jar file, just select the still open bpLoanApplication project and drag into the centre of the design surface – a nice new bpLoanApplication module will appear.   Now its time to build, right click caLoanApplication and select ‘Build’.  This build has now created a relationship with the JBI module bpLoanApplication and you may notice that under the caLoanAppication JBI modules folder, there is now a new JBI module called bpLoanApplication.jar.  Now that the project has a relationship with our JBI module, we can now configure the SOAP endpoint.  Right click the new SOAP endpoint and select properties, select the … next to interface name and select the only listed interface you see in the drop down (this is the logical port we created in the BPEL process earlier).  Provide an endpoint name and an endpoint path on your web server; this is the link between the logical WSDL port and an endpoint.  For this example I use http://localhost:${HttpDefaultPort}/loanApplication/loanApplicationEP as my endpoint address as I’m running this test locally (and no kids, clicking the link won’t work).

Almost done… Drag the connector from the now configured WSDL SOAP endpoint to the bpLoanApplication JBI Module.  We now have a way of receiving the message via SOAP, sending it to the JBI BPEL process and a way to get a response back to the calling SOAP endpoint.  Your composite application surface should look like this:

Great. Now all that is left to do is deploy it to your GlassFish server.  Right click the caLoanApplication and select ‘Deploy’.

Nice work…

Enterprise Architecture : An Introduction


As a budding Enterprise Architect, I’ve always been interested in the bigger picture when it comes to how an organization plans, executes and maintains its IT infrastructure.  Before we jump into what Enterprise Architecture is as a concept, we need to understand why IT is such an important function of any operating organization and reiterate why IT remains the back bone of a business.

Information Technology in the eyes of an organization is one of its Strategic Assets. Assets are a businesses economic resource (something owned that has material value) that for the most part adds value.  Organizations deploy this asset in supporting the business via new technology / development and the day to day task of running the IT ‘engine’.  An example of where IT supports the business would be the availability and advanced analyzing of sales or marketing data that can make or break a business decision and which could ultimatley effect the businesses position in the market.  What is important to remember here is that the goal of any business or organization or division or department is never fixed.  Goals of all sizes are moving targets and so the IT infrastructure that supports that business/organization must continue to improve and grow to support these goals.

A quick tour through the IT generations

IT didn’t just appear as a business function overnight, it developed through generations of supporting business processes. Here is a quick look at the story so far:

First Generation IT (G1) – Developed in the 70’s to support ‘batch based’ business processes. Batch based processing was a collection of automated series of programs (jobs) that had no human intervention. Batch based computing mainly occurred on main frame computing environments.

Second Generation IT (G2) – Emerging through the late 80’s to early 90’s was ‘Closed Multitask computing’ where programs could individually run threads of processing at the same time. This period saw the advent of Windows.

Third Generation IT (G3) – Emerging in the mid 90’s this period saw the ‘connected’, online accessibility era. The face of the business became the web site and new business models were born.

Fourth Generation IT (G4) – 2000 – The ubiquitous (everywhere), always connected service based architecture that IT departments strive for today.

Fifth Generation IT (G5) – 2015+ (who knows?)

As businesses move between these generations of change, massive IT infrastructure investments are made.  A good example to follow how business processes have changed throughout the generations is to look at the Banking sector.  In the 70’s you could only perform your banking duties between certain hours on certain days of the week and you had to go visit a physical branch.  Transactions where processed in batches on a day to day basis and statement printing was also carried out in batches so you would only know how much money you had on a monthly basis.  Today, banks offer global access to your money, statements and global online trading has changed the face of investments.  All of your banking information is available in one of two places, 1) via your personalised internet banking web portal and 2) via global ATM network.

Because of these continual improvements to banking IT infrastructure over the years, banks are thriving as a business and essentially run the cities of the world with their cash (except when the government / tax payer has to bail them out because of a global recession of course, but let’s not focus on that 😉

So What is Enterprise Architecture

Enterprise Architecture is one of those subjects that can easily invoke tongue twisting. The official ANSI/IEEE definition of EA is :

“the fundamental organization of a system, embodied in its component objects, their relationships to each other, the environment in which the objects reside and the principles governing the systems design and evolution”.

Makes sense right?  The Metaphor would be the design of a city (think Sim City).  The architectural blue print details the roads, utilities, open spaces (parks) and their locations / interactions.  Using this example utilities are the technologies and frameworks employed,  roads provide the communication and networks and the open spaces / parks are the applications and systems and how they interact with each other.

Whats the goal of enterprise architecture then?

It’s important not just to pass the definition of Enterprise Architecture without mentioning the ultimate goal of Enterprise Architecture.  The goal is to create a unified IT environment (so common hardware, platforms and technologies using open/common standards), create an alignment of the technology with the business, to make use of or ‘re-use’ existing IT assets and finally, promote the sharing of project management / software design approaches across the enterprise.  The end result at least in theory is that the Enterprise Architecture (blueprint, roadmap etc) will make IT investment cheaper, make IT more strategic, and ensure IT is more responsive to
business change.

Enterprise Architecture has 4 basic domains (according to most EA frameworks):

Business Architecture – Documentation that outlines the company’s most important / business critical processes (the most critical domain but also the most difficult to implement)

Information Architecture – Identifies where important blocks of information such as a customers data are kept and how that data is accessed.

Application System Architecture – A map of the relationships of software applications to one another.

Technology Infrastructure Architecture – The blueprint for the hardware, storage systems, and networks.

In terms of skill sets for the aspiring Enterprise Architect :

(1) Business Process Modelling, Design, Re-Engineering (BPMN, BPEL, BPM Servers)
(2) Database Administrator / Designer (SQL, LinQ, Database Design, RDBMS)
(3) Software Developer / Architect and Enterprise Integration techniques (Middleware, API, Software Dev Platforms / Frameworks)
(4) Network and Hardware specialist (Servers, Common Server Roles, Network Stack incl Protocols, Topology planning, Operating Systems, Hardware)

Breaking the Enterprise Architecture down

An enterprise is a whole corporation, a division of a corporation, a government agency, a single department, or a network of geographically distant organizations linked together by a common business goal / objective.
Enterprise architecture relates to understanding the collection of child business entities/objects that comprise one of the above enterprise definitions and how those elements interrelate.

Enterprise architecture =

Business Processes       + Applications       + Technologies        + Data

People                                          Systems                        Communications          Information
Processes                                    Software                       Services
Strategies
Business Drivers
Budgets
Functions
Stakeholders

An architecture is blue printed
A road map plans the delivery of that architecture

A quick note on Governance

The primary purpose of enterprise architecture governance is to ensure that an organization’s IT investments are closely aligned with business goals and processes, so that limited IT resources are allocated to areas of highest impact on organizational performance. Enterprise Architecture governance has its own framework but we won’t get into that now.

Enterprise Architecture Frameworks

Layered frameworks and models for enterprise architecture have proved useful because layering has the advantage of defining contained, non-overlapping partitions of the environment. There is a number of models/modeling techniques, the top 5 most common follow:

1. Zachman Enterprise Architecture Framework (ZIFA or just Zachman)
2. The Open Group Architecture Framework (TOGAF)
3. Extended Enterprise Architecture Framework (E2AF)
4. Enterprise Architecture Planning (EAP)
5. Federal Enterprise Architecture Framework (FEAF)

All of these models seek in one form or another to make use of the concept of a generic service/object-oriented architecture where sets of like functions are grouped into reusable services that can be described as ‘service objects’. More complex capabilities are then built from a combination of these basic service objects just as the elements are made of atoms. For now we’ll leave the discussion of Enterprise Architecture here and come back and discuss a couple of these Enterprise Architecture frameworks in separate articles.

Hopefully this has wet the taste buds for the bigger Enterprise Architecture picture.

Comparison of the high end BPM/SOA product offerings


After a long week of working excessing hours on a BPM project coming to the end of its development cycle, it was nice to spend a day just catching up on emails, drinking large amounts of coffee and reading a few new BPM/SOA articles.  One such article I found quite interesting and thought I’d share.  AMR Research have issued this report on SOA/BPM products in the higher end market share and offer a great comparison of features and architecture by asking each vendor to demo running a common process.  It’s a great birds eye view of the big vendors and how the BPM/SOA tools they offer integrate with their application servers, ESB’s and registry features of existing products.  My only issue was that it appears to be very non Micrsoft with Biztalk Server being left out of the picture… it has an solid foundation in IIS, MSMQ and .NET and is an SOA server afterall with its EAI/B2B/BPM capabilities? – The report itself is not exactly fresh from the shelves (issued in 2007) but still brings interesting and valid findings to the table.

Anyway, a good report by AMR (which I think may have been leaked as you’d normally have to log into AMR’s web site for report access.

check it out here.

Softshare ECS : An Overview


ECS Title

American sofware company Softshare have been around for a good 20 years and have offices in California and Holland. They specialize in B2B/EAI primarily focused on XML and EDI message transfer and document mapping. They offer a smaller web EDI tool for the lower end business customer called ‘Athena’ which enables small businesses to trade with the larger and more technologically advanced enterprises that demand electronic trading.  Softshare provides two flag ship products, Delta their data translation / mapping engine and their e-Commerce Server, ECS.  Both tools work well together to provide a communications and document transformation service for B2B and EAI strategies however our focus here is on ECS.

Features

Version 6.0 of their e-Commerce Server has not long been released and it comes in 3 flavours, ECS Lite, ECS Standard and ECS Enterprise. ECS Lite as its name suggests is a low feature version of the server and certain features such as certain pipeline processing, scheduled commands, scheduled reports and native AS1,2 and 3 support are not available. ECS is clearly the best option for heavy EDI users.  Not only does ECS offer improved features in this latest release but its core (routing) engine has been completely re-written in Java and so is no longer just a windows compatible server.  ECS operates as a systems connector so that EDI, XML and structured file messages can be passed to their Delta translation maps. Like the bigger EAI/B2B products, ECS Server uses Input and Output channels (once known as Output Queues) to retrieve messages or to post messages to different platforms / servers.  Input and Output channels support the major communications protocols (HTTP, POP3, FTP and SMTP) as well as security protocols AS1 (SMTP),2 (HTTP) and 3 (FTP).  Clustering is now supported in this latest release and fail-over features have massively improved (in that ECS 5.0 had none and fail-over was down to getting creative with the underlying SQL Server).  A master controller node can be configure with a backup controller which is able to take on operations 30 seconds (customizable) after the master controller is deemed non responsive.  Another super new feature to ECS 6.0 is the concept of pools for load balancing for inbound messages across different servers in the pool.

Input and Output Channels

As stated, the standard concept of inbound and outbound channels remains. An input channel is similar to Biztalk’s receive port (however with only one configurable receive location). The channel allows configuration of the uri or file location, security parameters, scheduling, content filtering and disposition (what happens to a message after pickup).  The Input/Output channel also offers a pipeline processing feature (similar to Biztalk’s pipeline albiet with a limited number of processing activities).  Pipeline processing allows the message to go through several processing steps  including ASCII to EBCDIC character set conversion, message header manipulation, character replacement, string replacement, unicode conversion and command line processing.  The command line pipeline feature allows passing the processed message to an external application (with parameters) for processing (e.g. special decryption processing or extraction of compressed file formats).  My only niggle with ECS channels  is the inability to plug in your own pipeline processing modules that can perform customized processing. Custom channels can’t be created either. A channel framework that programmers could tap into would be a massive selling point for this server.

Management Console

The management console application allows you to manage your ECS installs across your network and itself has seen improvements in this version.  Your nodes, pools, management (exception handling and notifications), channels and projects are all managed from within the console and now your trading partners are also available (trading partners where once only available through the Trading Partner Management utility).  This version of ECS (and Delta) take a project approach to your B2B maps and channels and make designing mapping projects in Delta and managing them (including importing and exporting) in ECS a breeze (not unlike a Biztalk application).

ECS Management Console

Data Administrator

Data Administrator is your window into your ECS servers traffic.  All messages that pass through the system are logged in the ECS database and the files (and file headers) are stored on the physical disk in a logical address based folder structure – Data Administrator allows you to filter data in order to locate specific messages.  The application allows you to see the full routing of a message from the channel it arrived on, the translation map that has modified the message and the output channel where the resulting message has left the system.  The most common use of Data Administrator is the ability to ‘re-stage’ or resend a message (or collection of messages called a batch), this feature can become super useful when message mapping errors occur or connected servers are down (e.g. sending to an FTP Server that is off-line for maintenance).

Summary

All in all, ECS Server is a strong server product as a result of years of development. It lacks in certain departments against the bigger players in EAI and B2B arena and even though it does provides event rule features it is far from offering full business process and orchestration creation tools.  This is essentially an advanced communications server that interoperates with a mapping engine to provide a translation and comms package (in conjunction with the Delta translation engine) for XML and EDI based B2B trading.  It is very good at what it can accomplish for the price tag and in my experience is completely under utilized in most organizations where it is installed.  It is capable of a lot more than I can cover in this article and the option to request a 30 day demo from Softshare direct is available.  Its latest release has come along way and along with Delta is an excellent choice for small to medium sized enterprises.  Lastly, Softshare offer a great free product called EDI Notepad and this is a must for anyone that deals with EDI messaging on a regular basis.

An Introduction to Middleware


Regardless whether you’re working on a BPM project or a fully fledged EAI project, it’s quite likely that you’ll need to get one system talking to another at some point. Middleware is an overused generic label for the grey area that is connecting systems (platforms and software).  It can mean a plethora of different things and tends to be a misunderstood area of technology (‘Frank: our middleware takes care of that… Nancy: What middleware do you have?… Frank: don’t know’).

I’ve just finished reading a book named ‘Inter-Enterprise Application Integration’ that I must of picked up at some point when working in europe (can’t remember when). It’s a pretty good book and goes into some depth regarding middleware and what that actually means across different technologies. What I’m attempting to do here is to condense that down into my own words and summarize the major technologies used in implementing middleware.

Middleware

So starting off with the main question… What is middleware? – The way I like to see Middleware is not as one particular technology, its more of ‘mechanism’ that allows one application (lets say software entity)  to communicate and trade data with one or more software entities (normally across disparate platforms that don’t easily speak the same language).  Middleware isn’t a server that you just install and applications can start chatting… it’s an approach. Ok so that’s quite a broad explanation… but hopefully by the end of this article the picture will be clearer.

Middleware comes in several flavours, most of which I’m going to cover here in limited detail. First the approaches followed by the communication types and finally middleware technologies.

Middleware Models (approaches)

Point to Point Middleware – Point to point as the name suggests emulates a sort of pipeline between 2 applications directly. Both applications communicate data down the pipe in most cases at the same time. Application A can shout down the pipe to application B and vice versa. Now this model of middleware is quite old now as it doesn’t quite fit in with the service based ‘loosely coupled’ approach that modern software developers take. Not only that, but there is no middle tier processing going on here, its direct only. An example of Point to Point middleware is some code in Application A that calls some code in Application B exclusively.  RPC is a type of point to point middleware.

Many to Many Middleware – This model allows many systems to talk to many other systems. This model covers more of the newer middleware technologies including TP Monitors and Message Brokers.  It is a more flexible approach to implementing middleware and for the most part is a more cost effective choice.

Middleware Communication Methods

Queued Communication – Queued communication generally requires some form of queue manager to place a message into a queue. The remote application is then able to connect to the queue and retrieve the message. If the sending system requires a response, the remote system will place that response in that (or another) queue for pickup. Most message orientated middleware use queued communications as it provides a way for applications to obtain messages even when the remote system is off-line.

Publish / Subscribe Communication – Pub/Sub as its known free’s an application from the need to understand anything about the system it is communicating with. All the sending system has to do is release a message to the pub/sub agent (not unlike marvel comics delivering their latest to the chap down at your local newsagent). The agent / broker then redistributes the message to all interested (subscribed) parties.

Request / Response – As it says on the tin, this communication method involves one application sending a request to a secondary system which would respond. Message brokers and application servers tend to utilize this approach.

Fire and Forget – This model deals with the sending system just firing off a message with no regard for a response.

Types of Middleware

Middleware like many other technology styles is evolving and these days its hard to differentiate between features that different middleware packages tend to employ.  Big middleware products like Biztalk connect systems in many ways and so blur the distinctions between middleware types.  What follows is a list of types of middleware that you will be able to identify with in your favourite middleware applications.

RPC Middleware – This type of middleware has been around for a good while and is probably the easiest to understand middleware. They provide developers with the ability to invoke a method with one application and have it execute on a remote program somewhere else. To the calling program user, the fact that the method is being executed on another machine is hidden. The problem with RPC’s however is that they can carry a lot of processing overhead so perform isn’t the best. DCE (Distributed Computing Environment) is a well known type of RPC as it provides a good collection of RPC services to deal with application integrity and security.

Message Orientated Middleware – MOM was created to address some of the short coming that came with using RPC’s. At its core MOM is just queuing software that uses messages (byte sized units of information)  to move data between applications (like mail between mail clients). This approach is loosely coupled, meaning that the communicating software doesn’t need to know much about the other. The big plus for using MOM is that it follows an async model unlike RPC which is synchronous.  The queue manager manages message delivery and neither the sending or receiving applications are blocked in any way from continuing processes (as occurs with RPC).  The messages being sent via the message queue are essentially a structure (schema) and content (data). With this model there is less of chance of data being lost when connections to applications go down.  MSMQ from Microsoft and MQ from IBM are both message orientated middleware.

Distributed Object Middleware – Middleware? you say?.  Indeed, distributed objects are classified as middleware because they facilitate inter-application communication. They are also mechanisms for application development for providing company wide method sharing. Distributed objects are just small chunks of code that expose interfaces that other chunks of code can call.  Two of the well known distributed object models are COM and CORBA.

Application Servers – Probably the fastest growing type of middleware the application server is nothing new. Most application servers are deployed as web enabled middleware that process transactions from web enabled clients. Application servers are also adopting the very latest languages like Java and .NET to allow further integration with back end systems to occur.  App servers provide not only a way to share and process application logic but also connect applications to back end resources such as ERP systems, databases and even old legacy applications (like traditional mainframes).

Message Brokers – Message brokers facilitate information movement between two or more applications and can account for differences in application semantics and platforms. For this reason message brokers have become the choice technology for Business To Business integration. Message brokers are a more advanced MOM as they implement business rule validation and advanced message routing. They are able to transform the structure and content of messages as they are routed via some kind of transformation service. Biztalk is a message broker based server application.

As new technologies emerge, the middleware arena becomes even more blurred. Application development is becoming easier by the day and connecting systems is becoming far easier than it used to be with the aid of technologies named above.

Biztalk Server 2006 : The Biztalk Message


Biztalk Message

First off, I’m going to talk about the Biztalk message from a developer perspective so an understanding of .NET or in fact any OOP fundamentals are a bonus.

The Biztalk message is the container that holds an instantiated object(s) from the Biztalk Message Class. The message object has public properties called context properties (message metadata that can be used by the Biztalk engine to filter messages).  Context properties are added by the receiving adaptor and new properties can be added.  Message fields can be promoted and become properties of the message also. These ‘promoted’ properties are stored in a separate property schema (property bag) that can be loaded into memory when loading the entire message by the message box to speed up message routing. These distinguished or context properties are accessed as properties of an object would be accessed :

Message.MessagePart.Property (e.g. Invoice.Footer.TotalAmount = 90)

Here Invoice is the message object and TotalAmount reflects a promoted field that corresponds to a field in the message schema. This of course makes the field value public so that orchestration components and the message box can ‘see’ the properties.  The other important member of the Biztalk message class (BTXMessage) is the loadfrom() method, which loads the message data into the message class.

Its important to remember that a Biztalk message is immutable and so is incapable of  ‘mutating’ / be changed.  The message is created by the receiving adaptor and can only be changed when ‘copied’ into a new message container using the ‘Construct Message’ shape within the orchestration designer. Construction of a message involves creating a new entry in the message box Database.  Modifications can only occur when you make a new instance of the message and copy the old to the new (OldMessage = NewMessage) via the expression editor. All message properties are copied.

Message Inheritence

Each message created to the message box has a GUID (globally unique identifier) which identifies the message in the message box.  Within an orchestration, biztalk uses .NET classes to represent individual messages (the orchestration itself essentially compiles down into a .NET assembly).  These classes act as a generic wrapper around the underlying message data.

Each message biztalk is derived from the :
Microsoft.BizTalk.XLANGs.BTXEngine.BTXMessage class which ultimatley inherits as follows:

System.Object (The base class for all .NET classes)
  Microsoft.XLANGs.BaseTypes.XLANGMessage
    Microsoft.XLANGs.Core.XMessage
      Microsoft.BizTalk.XLANGs.BTXEngine.BTXMessage

When you construct a message in Biztalk you are essentially instantiating a new BTX Message class (and copying the data from the old class to the new one if modifications are required).

Message Types

Biztalk messages types come in 4 flavors and how you assign values and construct messages differs slightly depending on which message type you have chosen to create.  These messages can be single or multipart and can hold Binary, XML or a serialized .NET Class data and the type of message is set in the message properties window within visual studio’s orchestration designer.

a) .NET Classes (single part message on which serialization can be set).

b) Multi-Part messages (each part of a multi part message is exposed via intelli sense  like so: MyMessage.PartA.LoadFrom(“Hello World”))

c) Schema Messages (a single part message so assigning value is done at the message level, not the property (or part) level as there is only one part).

d) Web Message (multi part message used in conjunction with SOAP based web services)

Biztalk Server 2006 : Free Audio Book Training


BizTalk Logo

Back in late 2005 when Biztalk 2006 first came to the market I began testing it at what is now Virgin Media in the UK. Over a period of time as I implemented a collection of Biztalk applications I started to collate information from various sources and compiled these notes into a logical flow that describes what Biztalk server does and what components are involved in creating a Biztalk application.  The information used is a mix of my own findings and extracts from various articles on Biztalk over the last few years, all combined into an organized audio book.  I almost called this article ‘Biztalk 2006 as told by an android’ however since the reader is nothing more than the Mac OS X text to voice technology. That said, he is very understandable and the information is still communicated in a clear way, not only that but it is very hard to find free ‘complete basics’ Biztalk training, especially in audio book format ANYWHERE on the web.  The only amusing pronunciation however is its version of the Biztalk mapper functoids (mapping functions) that he verbalizes as ‘Functo-oids’…

The file is downloadable as a rar archive and is around 79MB. Download the files here. Content :

01a – Intro – Introducing BizTalk Server 2006 R2 2:12
01b – Intro – What BizTalk Server Provides 2:37
01c – Intro – How BizTalk Server 2006 is used 4:08
02a – BizTalk Architecture – Engine 4:43
03a – Connecting Systems – Intro 0:59
03b – Connecting Systems – Adapters 4:43
03c – Connecting Systems – Pipelines 6:30
03d – Connecting Systems – Subscriptions 2:12
04a – Defining Business Process – Intro 0:44
04b – Defining Business Process – Using Orchestrations 2:47
04c – Defining Business Process – Schemas 1:11
04d – Defining Business Process – Mapping 4:22
04e – Defining Business Process – Orchestration Designer Part 1 3:22
04f – Defining Business Process –  Orchestration Designer Part 2 6:14
04g – Defining Business Process – Business Rules Engine Part 1 2:50
04h – Defining Business Process – Business Rules Engine Part 2 3:36
05a – Mgmt and Monitoring – Intro 0:27
05b – Mgmt and Monitoring – Installing BizTalk Server 2:32
05c – Mgmt and Monitoring – Scalable Configurations 4:07
05d – Mgmt and Monitoring – Application Management 3:11
05e – Mgmt and Monitoring – HAT 1:44
06a – SSO – Part 1 Single Sign On 2:13
06b – SSO – Part 2 Single Sign On 4:03
07a – Information Workers – Intro 0:54
07b – Information Workers – Part 1 BAM 2:57
07c – Information Workers – Part 2 BAM 3:50
07d – Information Workers – BAS Intro 1:42
07e – Information Workers – BAS Trading Partner Management 2:41
07f – Information Workers – BAS Business Process Configuration 1:48
08a – Other Technologies – Intro 0:40
08b – Other Technologies – Part 1 WWF 3:13
08c – Other Technologies – Part 2 WWF 3:40
08d – Other Technologies – WCF 2:14
09a – Wrap Up – Conclusion 1:51