MOSS 2007 - Workflows and it exposes to Enterprise Information Integration and Enterprise Application Integration

I am personally very excited about MOSS 2007 and it's brand new key futures, from last few weeks I am really involving my self to see MOSS's new key futures and it's innovative implementation(s) for those areas intersecting with Integration, rather it's Enterprise Information Integration (EII), Enterprise Application Integration (EAI) or Data Integration (ETL). I have worked in my past with SharePoint Portal Server 2003 and it was really complicated to integrate different user activities/events with across the enterprise through BizTalk and Web services. Always it dragged me to develop more custom components/web parts/web controls and calling web services, transferring information and events through BizTalk server. Looks really complicated but believe me it was much tougher then it looks...

Now at new mission I am trying to find out more flexible integration/Work flow base implementations, here I have developed one sample solution to support my view on the Information Integration within MOSS 2007 Tasks and announcement. I am primarily trying to develop some workflow which can pick announcement which is direct live integration with sample Loan Application Management (any custom application generate application details) through WF (which hosts on Windows Services and connected through WCF for my sample, it's open space for any integration tool provide direct integration with MOSS site like BizTalk ;-) etc...). Let me comeback to sample, now I have developed one pilot Workflow which pick each announcement and generate tasks on SharePoint as well as through WCF/Web services it integration with external applications too. Another Workflow application get invoked on completion event of those Task and automatically generate following to do tasks and assign automatically.

Now these workflow also provide me standards functionalities of Windows Workflow Foundation, like listen, event base routing (you can check my previous articles to know more about event base routing), it's really empower developer to go beyond the classic and traditional boundaries of SharePoint sites to next generation workflow and information oriented collaboration development.

Let's back to sample, workflow1 picks announcement and process it and generate proper Task and other artifices, too. In the process of generating automated task and artefacts it's utilize many EII services to authenticate credit and other historical information of an applicant.

You can see below snap, first activity shape will invoke the workflow on any new announcement arrival or creation. Then the workflow is trying to received data and contract through Get activity and parallel processing other code too. At end of successful processing of whole parallel shape it will create all tasks needed for the announcement and at end it will log whole history. Please have a look at figure-1 and figure-2.





After deployment on to your MOSS Site you can check the workflow status by going to Site Settings->Galleries->Workflows. You can see figure-3 and figure-4 for site navigation and Workflow Collection. Site Collection Workflow will show you the association of the workflow to different activities and In-progress workflows too.





Let's come to another sample workflow of the pilot solution, which watch on every complete event of Task and according to the event it flow the application further for consideration and human approval. Where it also involve it self into request response with Web services and EII Service contract through workflow service. And similarly create further to do events in MOSS site.
Have look to MOSS Site Snaps.

1. Creating announcement for sample Loan Application.


2. You can see below figure announcement has been added to list. It shows 11111 id for loan application.


3. After sometime it was picked by workflow and processed, in solution I have appended string called " Processed By...", you can see here it has updated title of announcement and also added new relevant task for the loan application. During the creating of new task it went though to WCF application and receive back some valuable credit information and rating.


4. Task's detail page, you can see here generated description and information.


5. Once Task has been completed another workflow2 processed it and generated siblings task and to-do also it updated the title of the task (You can track related artifacts by "11111" id).


Such futures of MOSS 2007 make life easy for Integration developer and we can also develop such infrastructure suits MOSS 2007 features which help more easy and quick development for complex definitions.

Previously I was using BizTalk adapter or custom .net code but it always want huge resource to implement such workflow and integration for SharePoint site also it increase high volume of data and security threats while such implementation like workflow really help to develop much abstracted information and service access layers and less resource consuming infrastructure artifacts. I analyze such implementing with workflow pattern and pilot application performance results, it has ability to shirk our transaction volume across BizTalk by 40% compare to previous and 50% better performance with easy to maintain requirement. These are not derived figures but these are my assumption on the standard scenario which I analyzed.

Hope you liked this article and please send your valuable feedback and ideas to nilayparikh@gmail.com

Consuming WCF Services in BizTalk Server 2006 R2 - Add generated items (WCF Services)

I was thinking a long back to come up with this article, and now just found time to write about WCF services and it's implementation with BizTalk Server 2006 R2. I have designed few small solutions in past few months to deal with scenarios like Information Integration and SOA with BizTalk Server R2, where I found the great chance to work upon the WCF adapter series as well as Web Service Factory : Modeling Edition. Both made me big fan of them, it's great to work on both, I am not focusing on WebService Factory modeling in this article but will discuss about WCF services consumption inside BizTalk and few videos might help people to work on them.

I would like to share few images below, by reading this images you can understand the service I have coded for sample, if you are interested in sample code please mail me, and I promise you will find that sample someday in you mail box... when that I don't know. :-) But will try send you soon as possible for me.

The best way I feel to consume WCF services inside BizTalk solution is by using WCF Services Consume Wizard which you can find in Visual Studio Template, this wizard will power you by generating schema and port binding configuration files for your solution based on service descriptions. Not going into deep how this work internally here but focusing on the practice I gone through.

Below image shows Generated Items and Visual Studio Template.

Please have a look to the below Video Presentation where you can find what steps I followed to generate my sample project. Hope it will surely help you in any manner while working with WCF and BizTalk together.

As result of Wizard you will get schema and binding files inside your solution, that schema will be generated on basis of the metadata of the consumed services and it could be used as multipart messages in case of request-response services. Below snaps will show you the BizTalk schema presentation against the .net message contracts and data contracts.

DataContract Source Code:


MessageContract Source Code:


Service Implementation Code:


BizTalk Schema:



BizTalk Test Message I used for my sample:

You can analyze your self the presentation of Inbound message and the Contract representation inside WCF Implementation upon the above snaps. You can consume the generated schema in routine BizTalk solutions and patterns, while deploying such solution you can use generated binding information for such implementation with Endpoint transportation and Behavior information regarding WCF hosting and services. It is very good feature provided by wizard which make our life very easy :-). Let's have a view to below video presentation on the WCF Http and Custom Adapter configuration inside through BizTalk administrator console.

Hope you liked this article and it will give you fair idea about the BizTalk Server 2006 R2 and WCF coupling in Solution, as a system/solution designer point of view I really enjoyed to work with this two made for each other technologies and it provide such ease to workout most complex scenarios regarding service orientation and integration. As well as I realize the solution's capability to work over Internet Integration where security is the major concern and we generally prefers to go with ws-HTTP or HTTP channels for integration, on such scale it can help us to give such secured integration system.

Please provide your feedback about the article at nilayparikh@gmail.com.

BizTalk 2006 R2 with PeopleSoft Integration & MicroSoft BizTalk PeopleSoft Adapter Overview

Recently, I come accords requirement to integrate PeopleSoft ERP implementation within existing ESB. I have worked on many PeopleSoft integration through BizTalk, so just taking opportunity to write something exciting about knowledge I have gathered.

MicroSoft is very much aware about importance of PeopleSoft ERP and it's huge existence in Industry, so BizTalk R2 LOB adapter set has include PeopleSoft adapter which is very useful artifact during our PeopleSoft integration, it facilitate developer in many manners like developing schemas, messages. There are few limitation also we are facing with the adapter but those limitation could be overseen with different kind of implementation patterns and of course we can archive most of requirement what we want to with the adapter. There are also many third party adapters are available for PeopleSoft integration but I have explored more, the only one third party adapter I have worked with is iWay last year, yes it is very impressive but while comparing overall prospective I would defiantly go with R2 LOB. But it purely depend on the requirement and what extend we are looking for integrate the systems.

Microsoft Line of Business PeopleSoft Adapter's Architecture.

LOB PeopleSoft adapter's Prerequisites:
# The Java 2 Platform must be installed on the BizTalk Server that the BizTalk Adapter for PeopleSoft Enterprise is running on.
# The PeopleSoft Java Object Adapter JAR file, psjoa.jar should be copied to a folder that is accessible to the BizTalk Server that the BizTalk Adapter for PeopleSoft Enterprise is running on.

Implementation & Design Patterns:
BizTalk PeopleSoft adapter facilitate your requirement to query and receive data from PeopleSoft ERP also you can do basic data operation like Insert, Update and Delete. But here one thing very clear, but it will not allow live integration from outbound direction (from PeopleSoft). To archive such requirement you can implement services inside PeopleSoft which can push live data feed into any queue or physical location where BizTalk or any external service can monitor feeds with very less latency, so how you can get such live implementation.

I would like to share the architecture, I designed for such live implementation landscape recently. You can view the picturing below.

The above architecture, I designed for the two-way integration with PeopleSoft for our client. Low latency outbound messaging integration can be archived by MSMQ or any other queuing implementation between BizTalk and PeopleSoft. Also many time in different scenario requirement we need information integration between different systems that could be archive through cutting edge technology like WCF / WSE 2.0 where BizTalk R2 provide functionality integrate such services, you can also give term like 'EII' to such implementation. More over by implementing such landscape we are moving towards service orientation and service bus. BizTalk provides a ease to integrate many systems within such implementation, also it leverage extensibility of ERP implementation.

You can also go through below links for further references at MSDN.

1. Tutorial: Using the BizTalk Adapter for PeopleSoft Enterprise to Retrieve Data from PeopleSoft Enterprise
2. Tutorial: Using the BizTalk Adapter for PeopleSoft Enterprise to Write Data to PeopleSoft Enterprise

Please provide your valuable feedback on nilayparikh@gmail.com.

Advance mechanism to handle exceptions for .net component used inside BizTalk solution

Finally after one month’s big vacation I am writing something technical, few days back my one of good friend came to me and asking me about the exception handling inside BizTalk for those scopes which contain calls .net classes and their methods. Well, I enthused about this article, and also a good chance to document some good practices I have worked for in some previous projects to improve the exception management and automating support routing. The article also includes some videos, hope it will help you to understand my views and idea about this mechanism I am presenting here.

In every system or solution exception handling stands with primary requirement, this article is not elaborating exception handling fundamentals in .net platform but I am focusing towards exception handling in .net component which we are adding as assembly inside BizTalk Solution/Project. Those exceptions handled inside .net assembly then the question comes how we can manage execution flow if exception occurs with orchestration logical flow, need to put flags, variables, calls and then with decision shape we can route files or log it within orchestration, correct. Hummm, it’s really odd solution, it can surely fulfill our purpose but it’s not recommended. This is very heavy on orchestration as well not recommended for runtime performance of BizTalk orchestration. Well, there is always an Generic Exception we can handle inside Orchestration for any scope but it will not give specialized block to handle such exceptions like if I want to handle DivideByZeroException and NullReferanceException in different block and then I want to divert the files at different Send Port so my technical support team can look and appropriately correct them individually and send those back for reprocessing, so I can give my customers better and fast experience with BizTalk support group.

An unhandled exception inside .net class which you can surely catch inside BizTalk Orchestration as Generic Exception. But If you have requirement like you need to execute few statements to compensate and then you also want catch that exception inside BizTalk Orchestration to route the logical execution path. Bit confusing requirement but believe me it’s normal one, there you will question me, why I shouldn’t catch inside BizTalk only and execute all compensate statement inside catch block, really possible but it will be not appropriate for performance, suppose if you handled it inside .net assembly then assembly call, refection will improve run time assembly performance inside BizTalk rather then calling multiple time. Yes then the question, what to do about the logical execution flow inside Orchestration, if the requirement want to route the message to any other branch or to any send port if exception occurs, possible? We can rethrow the exception inside .net assembly’s catch block after the compensation has been executed inside code, rethowed exception will be catch by BizTalk orchestration and we can route logical flow as per the requirement.

Below picture shows the implementation for handling the rethrowed exception inside .net assembly.

You can also go through the below videos shows the complete implementation about what I have discussed.

.net Assembly implementation video

BizTalk implementation video

Hope you liked my article and might it will help you, Your valuable feedback and suggestion are welcome to nilayparikh@gmail.com

BizTalk 2006 / R2 Publishing Throttling State #2 (imbalanced message publishing rate, input rate exceeds output rate)

Our next destination in BizTalk Throttling exploration drive is inbound Throttling states, publishing throttling state #2 Throttling due to imbalanced message publishing rate (input rate exceeds output rate).

Throttling state # 2, due to imbalanced message publishing rate, input rate exceeds output rate:

This is the most simple scenario to understand, here imbalanced message publishing rate means the ratio of the incoming messages by outgoing messages or vice versa.

The following is the condition of the Throttling state:
- (Message publishing incoming rate / Message publishing outgoing rate) * Rate overdrive factor in %

This condition belong to rate base throttling, For inbound (published) messages, BizTalk Server throttles publishing of messages if The Message publishing incoming rate for the host instance exceeds the Message publishing outgoing rate * the specified Rate overdrive factor (percent) value. The Rate overdrive factor (percent) parameter is configurable on the Message Publishing Throttling Settings dialog box. Rate based throttling for inbound messages is accomplished primarily by inducing a delay before publishing the batch of messages into the MessageBox database. No other action is taken to accomplish rate based throttling for inbound messages.

In following general scenario, I found the the throttling state can be possible in environment.

- very high demand to process messages and less available resources to process those messages.
- high use of co-related messages and self-correlated ports, and multiple subscribers for single received message.
- Slow outbound adapters.
- imbalance between host configured for inbound adapter and host configuration for XLANG / host configuration for outbound adapter.
- high processing complexity.



Understanding about Publishing throttling threshold parameter,

Minimum number of samples: Minimum number of messages BizTalk Server will sample for the Sampling window duration before considering rate-based throttling. If the actual number of samples in a sampling window fall below this value then the samples are discarded and throttling is not applied. This value should be consistent with a rate at which messages can be published under a medium load. For example, if your system is expected to handle 1,000 documents per second under a medium load, then this parameter should be set to 1,000 * Sample window duration in seconds (or more precisely, 1 * Sample window duration (milliseconds)). If the value is set too low, then the system may experience a throttling condition under low load. If the value is set too high, then there may not be enough samples for this technique to be effective.
Enter a value of zero to disable rate based inbound throttling.
The default value is 100.

Sampling window duration (milliseconds): The time-window measured in milliseconds, which is used to calculate the publishing rate based on the samples collected. The duration should be increased if the latency required for publishing a single message is high.
Enter a value of zero to disable rate based inbound throttling.
The default value is 15,000.

Rate overdrive factor (percent): This controls how much higher you allow the request rate to be than the completion rate before a throttling condition occurs. For example, if messages are being published at a rate of 200 per second and this parameter is set to 125, then the system will allow the publication of up to 250 messages per second (125% * 200 = 250) before applying throttling. Specifying too small a value for this parameter will cause the system to throttle more aggressively and could lead to over-throttling. Specifying too large a value for this parameter will cause under throttling and prevent the throttling mechanism from recognizing a legitimate throttling condition.
The default value is 125.

Maximum throttling delay (milliseconds): This is the maximum delay BizTalk Server will impose on a message instance due to throttling. The actual delay depends on the severity of the throttling condition.
Enter a value of zero to disable inbound throttling.
The default value is 300,000.

Triggering Mechanism of State #2 publishing throttling: In the sampling window duration, if the incoming/outgoing ratio excceds the configured parameter "Rate overdrive factor" value then BizTalk will be in throttling state.

Throttling Action: Block the publishing thread for a dynamically computed time period until the Message Publishing Incoming Rate is at par with the Message Publishing Outgoing Rate * the specified Rate overdrive factor (percent) value. By blocking the publishing thread, BizTalk can reduce the queued up messages, I have noticed many time the queued up inbound messages has very bad impact on the memory as well as the processing on the BizTalk artifacts. And the impact of the queued up messages is showing exponential memory & utilization which can also impact indirectly overall to every processing in BizTalk. So by blocking publishing thread, BizTalk is reducing incoming message flow and keeping flow healthy.

Sample collection of BizTalk Perfmon data and Graph: You can view in below graph that when the publishing throttling rate condition is setisfying the system in throttling mode #2, also you can analyze as per the sample window state through it.



If you would like to have the perfmon logs data, biztalk throttling configuration snap and graph in compress file for studying or analyzing it further for study which I have simulate for study, please write me to nilayparikh@gmail.com

Hope this article helped you to understand state#2 of throttling for you.

Related articles posted by me on the same blog:
BizTalk 2006 R2 - Throttling - Perfmon Parameters - My Experience (Tuesday, January 15, 2008)
BizTalk Server 2006 / R2 Throttling Mechanisms (Friday, January 4, 2008)

MSDN Resource Link:
http://msdn2.microsoft.com/en-us/library/aa559591.aspx
http://msdn2.microsoft.com/en-us/library/aa559893.aspx
http://msdn2.microsoft.com/en-us/library/aa559628.aspx
http://msdn2.microsoft.com/en-us/library/aa578302.aspx
http://msdn2.microsoft.com/en-us/library/aa547859.aspx

Thanks for BEAR me during the article. Your valuable feedback and suggestion are welcome to nilayparikh@gmail.com

BizTalk 2006 R2 - Throttling - Perfmon Parameters - My Experience

BREAKING NEWS:These days.. Dr. Time Pass is spending most of time to understand and implement best throttling scenarios :-).

Let me come back to official language. Few day back, I saw lots of throttling in our BizTalk environment. My enthusiasm drag me to the further exploration in throttling mechanism. Let me grab opportunity to document some cooked things.

Since last weeks, I saw my BizTalk environment is facing suddenly unusual throttling behaviors so I tried to find the root cause behind the throttling state in my BizTalk environment.

The environment is facing lots of message delivery throttling during pick processing hours, and due to the delivery throttling state, and delays our whole environment get stunned. After this incidence, I start taking this functionality very seriously and now digging lots of stuff related to it, here in this article I am going to share my experience as well as the understanding I got after going through many books, articles and MS-KBs.

As we all BizTalk professional aware about the inbound message processing and the internal BizTalk instruments takes care about the inbound message management. In short if I like to explain the word "inbound message" then I can stat "messages towards messagebox" as inbound messages, in BizTalk messagebox is the heart of any environment or implementation. We were facing problem due to unsymmetrical messages' incoming publishing rate and messages' outgoing publishing rate, I noticed in our environment incoming message rate is very high in compare to outgoing message (here outgoing means subscription to messagebox, we are talking about deliveries), in resultant the number was increasing for the messages in memory. BizTalk recognized and our environment moves towards throttling state but still throttling was also not able to make the system stable and put it again towards stable node. Due to the very heavy message rush and less outgoing rate very high number of the messages were waiting in queue for processing. We saw almost 300-700 seconds as delay time during the processing. And finally the nightmare came to true :-0, and counter called "Days without Servility 1" need to reset. The thing I would like to convey is the requirement of the optimized configuration parameter for the throttling while setting up any host. I will stat 'must' recommend for setting-up a set of well configured and planned host for the pick time processing for the critical interfaces. I don't want to drive my article into implementation designing, but I would like to come back to the throttling scenarios.



In this article I would like to focus around perfmon counters, few important ratios, throttling states.

Let's dig first to some state counters, state counters shows the state of the current situation in your environment under monitor. BizTalk 2006/R2 comes with very useful perfmon state counters.

some high (*) counters like High database session, High database size, High in-process message count, High message delivery rate, High process memory, High system memory, High thread count, these all counters represents only a state by showing value like 0 or 1, where 1 the respective focused area cross the configured or warning level. These shows very useful information which should be monitor in your Biztalk environment.

In-process message count, Active instance count are two message queue counters which show the Number of in-memory messages delivered to the XLANG engine or the outbound messaging engine that are not yet processed and instances are in active state in memory referring to the EPM or XLANG engine, respectively.

Specialized message delivery counters like Message delivery delay (ms), Message delivery incoming rate, Message delivery outgoing rate, Message delivery throttling state, Message delivery throttling state duration, Message delivery throttling user override. These all perfmon counters should be monitored regularly during the pick processing to understand the behavior of your environment and the coming out patterns to understand and prevent the failures.

Message delivery delay (ms): The current delay in ms imposed on each message publishing batch (applicable if the message publishing is being throttled and if the batch is not exempted from throttling).

Message publishing incoming rate: Number of messages per second that are being sent to the database for publishing in the given sample interval.

Message publishing outgoing rate: Number of messages per second that are actually published in the database in the given sample interval.

Message publishing throttling state:

• 0: Not throttling
  
• 2: Throttling due to imbalanced message publishing rate (input rate exceeds output rate)
  
• 4: Throttling due to process memory pressure
  
• 5: Throttling due to system memory pressure
  
• 6: Throttling due to database growth
  
• 8: Throttling due to high session count
  
• 9: Throttling due to high thread count
  
• 11: Throttling due to user override on publishing

Message publishing throttling state duration: Seconds since the system entered this state. If the host is throttling, how long it has been throttling; if it is not throttling, how long since throttling was applied.

I would like to place basic example of some perfmon data and counters in graph representation.



I will try to explore on Message publishing throttling states on my coming articles.

Thanks for BEAR me during the article. Your valuable feedback and suggestion are welcome to nilayparikh@gmail.com

Related articles posted by on the same blogs.
1. BizTalk Server 2006 / R2 Throttling Mechanisms

BizTalk Server R2 - WCF HTTP Binding through BizTalk WCF Adepter

A very good video demonstration by Stephen W. Thomas, the video demonstrate WCF HTTP Binding through BizTalk WCF Adepter. Worth spending 13 minutes on the video.

To download the video please Click Here

Downloading the related sample files might also be helpful. They can be found here:
Click Here

Live Play (no need to download the file) of this video is available here:
Click Here