Monday, 29 September 2014

SOA / API Development Tools Tips

Having been delivering SOA projects for over a decade, I’ve come across many tools some of which I hate and avoid using them (unless a client demands so)  and others that have made my life easier when delivering projects.

The below table is a recollection of tools I have used or come across which  I find useful and recommend SOA/API practitioners to use or at least evaluate. Enjoy :)

SDLC Tool Name Description + Links
Project/Scrum Management and Collaboration Jira Jira when used for scrum management is In my opinion is the the best tool I've used for managing scrums specially in an offshore / onshore distributed environment. It also has capabilities to manage GIT repositories which makes this an ideal tool for a complete cloud/based agile project with users all over the world.

https://www.atlassian.com/software/jira
  Agilo I used this tool some time back successfully in a large onsite SOA project and it worked very well for us. Product has come along way since then and It's more feature rich than it used to be. It has a 30 day trial.

http://agilosoftware.com/
  Coop App very clean UI supporting features such as share status updates, questions, links, and others.

http://coopapp.com/
  PBworks Suite of tools including in-app instant messaging, live notifications of changes to work spaces, live editing of documents, voice collaboration, wiki, and others. Can be used for full collaboration including project management.

http://www.pbworks.com/
Configuration Management GitHub I have been using this tool recently but found it amazing specially because of the GitHub client which is not intrusive to filesystem (like Tortoise and that create issues some times in the local file system) and very easy to use. It's social collaborative capabilities are awesome but also supports private projects however this latter feature it's not free. Also there are tutorials online.

https://github.com/

try.github.com
  Subversion This is the version control system I've used the most and still using the most in client side. Compare to any other traditional version control system (i.e. CVS, Clear Case, MSTFS, RTC) I think this outperforms them all, not only because of it’s simplicity, but because of the wide availability of subversion clients available and community support. My preferred client is Tortoise. Also there are clients available for JDeveloper. There is different installation options. The apache option can take longer as you have to install Apache HTTP s ever separately, for this reason I used collabnet which is very easy to use but has some licensing implications. There is also bridges available to link subversion with other repositories such as GitHub, or to use SVN clients against repositories other than subversion.

Servers:
www.collab.net

https://subversion.apache.org/

Clients:
http://tortoisesvn.net/downloads.html

Jdeveloper Subversion Extension (added from Jdev directly)

Bridges:
https://svnbridge.codeplex.com/

http://www.bioconductor.org/developers/how-to/git-svn/
  Jenkins The best continuous integration tool I have used so far (even when compared to proprietary tools like IBM BuildForge). There are so many plugins available like for example reporting, performance testing, Maven integration, which makes this a true gem when implementing large integration projects.

http://jenkins-ci.org/

https://wiki.jenkins-ci.org/display/JENKINS/Plugins
  Dockers In simple words, Dockers it's a new paradigm for creating portable applications that are available in the cloud and can be checked out and run in multiple platforms that have Dockers installed. Yes, it's that simple. Dockerized apps can be checked out from a central server, can be modified, re-packed, checked into the cloud again and so on... Reason I say it's a new paradigm is because Dockerized apps are basically running in fully segregated compute capacity of the underlying OS, so basically it's sort of like a virtualised environment, but it's not. It's just a container ! there is a great tutorial in the following link:

https://www.docker.com/

https://www.docker.com/tryit/
  Puppet Put simple, Puppet it's a framework to automate the installation and configuration of systems / applications in a variety of environments, i.e. from sandbox, to dev., SIT, UAT, NFR, Prep rod, to production. Puppet is very robust and it's very scalable. By creating Puppet manifests using a DSL based on groovy, you can define the installation steps of complex systems and complex topologies such as Oracle RACs and Oracle SOA Suite clusters. It comes in 2 flavours, an open source version which has several restrictions (i.e. no GUI), and a licensed version puppet enterprise) with is more feature rich. For puppet to work in a distributed environment, there is a Puppet central server which stores all the manifests (in a YAML store) and puppet agents which are deployed into the targets where software will be installed and configured. There is a new type of "masterless" configuration being popular now days which removes dependency on the master server.

http://puppetlabs.com/

http://puppetlabs.com/puppet/puppet-open-source
  Chef Conceptually it's a similar solution to Puppet. However instead of DSL based on groovy, it's DSL it's Ruby. Also instead of "manifests" you create "recipes". Some people from a sysadmin background don't like this and also the fact that it uses JSON data structures, and a Postgress data base. However if you come from a web development background (like me) this should not be an issue at all! in fact the opposite. Chef also comes in different flavours Chef Essentials (Free), Chef Subcription, and Site. Also there is a hosted options. To be frank, I found the flavours a bit complicated to undersand.... Chef also relies on a client/server architecture, where a server contains all recipes and the clients are deployed to the nodes where installations/configurations will take place. Note that although there is a free version, Chef it's in fact not open source....
  Ansible The work of Michael DeHaan (creator of Cobbler a popular linux install server), it's yet another CM tool but unlike Puppet and Chef it does not follows a client/server architecture meaning that is agentless. It follows a push model and it's transport it's SSH. Configuration files are store in YAML "playbooks" (equivalent to recipes or manifests). The playbooks can actually reside on any node and then as said configurations are pushed using SSH. This makes Ansible of all options the simplest one (in theory as in practice I haven't tried it yet but I willl as I find it very attractive).
API Management & Governance API Catalog 12c New product released at OOW! a light-weight version of OER, fantastic tool to create a catalogue of your available services and APIs without the hazard of implementing a full OER solution. Basically all you need to do is set install API Catalog (with same binaries as OER 12c) and harvest your OSB and SOA Domains or WSDL/WADL URL’s directly. It has a very clean and nice new UI so you don’t have to worry about all the complexity of OER.

To install download OER 12c and select API Catalog install option.
  API Manager 12c New product released at OOW! not out yet, but will be soon!. Basically built on top of OSB, it will allow you to define from OSB console or JDeveloper which OSB Proxy services are “managed” and then they will become available in API Manager. API Manager is basically an application which will provide a facility for API designers as well as API consumers to access and use/managed your published APIs. At present this product is not meant for DMZs but rather for internally published APIs. For DMZs, API Catalog is a better options.

Not available for install yet.
  Oracle Enterprise Repository (OER) 12c Will not say much about this as my blog is full of information about this tool plus I've written a book about too. One thing though, many companies are in a phase now to try and modernise and transform their services landscape, this tool can help gained proper visibility over existent assets therefore create a proper inventory of available assets which can then be used to re-design / transform the solution into a more mature state and also avoid ending up in more duplication as transformed services will be visible/searchable in the repository. There is a coming 12c release of the products which will provide new flavours of the product, one of which is light-weight and focused for API management.
Design MOGUPS If you also have a requirement to build a UI mockup for your services, this is a fantastic online tool to do all of this. I really found it easy to use and feature rich. It's not free but monthly fee are reasonable.
  INK Scape My preferred tool to create killer vector-based diagrams when doing design. The best of all, it's open source.
  Tools for generating UML like designs: Although InkScape it's great for graphics, when it comes to generating UML like diagrams, it really falls a bit short specially when it comes to creating a central-collaborative repository. For this purpose the tools I usually end up using are:
1) Sparx Enterprise Architect: The best tool out there in my view but not cheap.

www.sparxsystems.com/products/ea/

2) MS Visio (the obvious option): Good tool but lacks strong collaborative features like EA

www.microsoftstore.com/Microsoft_Visio

3) JDeveloper for Sequence and Class diagrams (although it's not really the best tool for this): Option if there is no EA or visio license

oss.oracle.com/jdeveloper.html
Build Oracle JDeveloper As an Oracle SOA expert, this is the IDE of choice for building SOA composites and OSB services and APIs. 11g version is very good however the 12c version truly is a gem and so much capabilities are now available in the tool (such as building OSB's proxy/business services and OEP streams).

https://oss.oracle.com/jdeveloper.html
  XML Spy I still believe this is the best XSD / XML editor there is. Specially when building complex canonical models with tons of inter-dependent schemas, this tool truly steps up. Unfortunately it's not cheap! But in large projects with tons of XML editing, purchasing it can be justified based on productivity gains

www.altova.com/xmlspy
  Online Json Editors There are some excellent online JSON editors available. Here some links of the ones I often use:

http://www.jsonschema.net/

http://www.jsoneditoronline.org/

http://json.parser.online.fr/
  Notepad ++ Many times we want to edit files quickly and that aren't necessarily related to a JDEV app or project (i.e. config files, shell scripts, etc). For this scenarios instead of using notepad, I always used Notepad ++. I think is the best editor out there for windows plus it has some very useful plugins

http://www.sunjw.us/jstoolnpp/
  JSToolNpp for Notepad ++ Excellent Notepad ++ JSON viewer plugin

http://www.sunjw.us/jstoolnpp/
  D3 Not really a tool but rather a JavaScript lib. The most powerful graphic generation library I've seen so far. Combining the graphic capabilities of this library with your solutions will deliver a greater impact to the consumer of the solution. At the end of the day "if you can't draw it you can't understand it" and with this tool, you can generate hundreds of different views to exploit your data.

http://d3js.org/

Free eBook available in following link: http://chimera.labs.oreilly.com/books/1230000000345/index.html
Testing Gen Rocket Generating robust and meaningful test data dramatically improves the quality of the code as test cases become more relevant and scenarios more realistic. However this is often an undermine task and it's usually developers that end up creating test data which not always covers all required scenarios or is meaningful enough. A colleague of mine shared this tool and although it's not open source, it does provide some very good features to create projects to generate data for defined data objects and automate the generation of test data at a massive scale. Data can be generated in several formats such as XML, csv, etc.

http://www.genrocket.com/
  GenerateData A very simple, web based, open source tool for generating test data. It all can be done online and data can be generated in different formats like csv, xml, JSON, SQL, LDIF, etc.

http://www.generatedata.com/
  Oracle SOA Suite Test Suite Great feature of SOA Suite available in Jdeveloper to create and automate unit test scenarios to test composite wires and components. It can also be integrated with Jenkins jobs for example for continuous integration / regression testing

http://docs.oracle.com/cd/E21764_01/integration.1111/e10224/bp_testsuite.htm
  SOAPUI This is now a well know tool in the industry for testing web services. However incredibly as it sounds not every knows that the tools can also be used to generate service stubs and to do performance and penetration tests although these last too require purchase of a license, but still an awesome tool.

http://www.soapui.org/
  Code Compliance Inspector (CCI) Comes in two flavours. As JDEV plugin and also with OER as a utility. I had issues with the former but the latter worked quite well for us and we also managed to incorporate CCI code check as part of our build/deployment/continuous integration process. So every time someone checks code into main branch, we actually automatically checkout the code, do a CCI code compliance test and then build and deploy. If code is not compliant we send a error notification to a distribution list.

http://docs.oracle.com/cd/E28280_01/doc.1111/e15754/codecomplianceinsp.htm

http://docs.oracle.com/cd/E28280_01/doc.1111/e17366/chapter20.htm
  Zed Attack Proxy (ZAP) Created by OWASP community, this is a fantastic open source tool to test security vulnerabilities in web applications and can also be used for testing REST and Web services

https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project
  Apache JMeter For me one of the best open source performance testing tools available, allows for all sort of performance tests to be created either for simple HTTP requests or for web service or REST service invocations. It supports distributed and multi-threading testing.

http://jmeter.apache.org/
  WSDigger It's an open source tool for testing security on web services. Haven't use this tool vastly but I used it against standard SOAP based services and it worked well. I however don't see that it has been updated recently which is a shame really!

http://sourceforge.net/projects/foundstone/files/WSDigger/
Production Support Xymon Really awesome, open source and lightweight tool to monitoring web services, networks, etc by using different extensions available. For web services, XYMON can be used to monitor health of HTTP endpoints for example and to ensure that services are up and running. When issues are identified, XYMON can be confirmed to send notification alerts via SMS and email amongst others.
  Oracle Business Transaction Management 12c BTM for transaction monitoring

http://www.oracle.com/technetwork/oem/btm-496775.html
  Oracle BAM 11g/12c BAM for activity monitoring and user/business facing dashboard:

http://www.oracle.com/technetwork/middleware/bam/overview/index.html
  Oracle Enterprise Manager 12c OEM for operational monitoring (App performance management, instance status, health checks, etc.)

www.oracle.com/EM

Monday, 30 June 2014

New in SOA Suite 12c and AIAFP/OER Backward Compatibility Table

Finally Oracle SOA Suite 12cR1 (12.1.3.0.0) is out (http://www.oracle.com/us/corporate/pressrelease/soa-suite-12c-062614).Having implemented all versions of SOA Suite (10.1.3.1-5, 11.1.1.1-7) I am particularly more exited about this new release as I was when its predecessors were launched. And this is because unlike the previous releases, these versions is releases in the middle of major trends such as cloud computing, mobile integration, Internet of Thins, Machine to Machine, are starting to become main-stream and I see SOA Suite 12c as a fundamental building block to supporting all of these new trends!
Listing below some of the new features of Oracle SOA 12c that I am more exited about:
Is about the last point two points  (AIA/OER) that I want to talk a bit more, specially because both of them are governance related features.
Although 12c introduces many features which will make it easier to govern a SOA project, if you are using AIA or OER in 11g or if you are about to start a 11g project, you must take note of the following:
  • AIA Foundation Pack will not be available 12c meaning there is no, and my understanding is that there will not be an AIA FP 12c release. This means that I don’t expect AIA pre-built integrations to be created explicitly for 12c . This makes sense to me as the original purpose of AIA FP was to provide a SOA framework specially aimed to help Oracle Apps practitioners kick start integrations between Oracle Applications (remember that at the time SOA was still immature specially for the Oracle Apps community in which SOA was still a big unknown). Note that some backward compatibility will be provided in SOA Suite 12c by enabling the SOA Core Extensions during the installation process.
  • AIA Object Library (EBOs, EBMs, WSDLs) are not installed automatically as part of Oracle SOA 12c Core Extensions. To make these available you will have to extract your 11g objects and package them in a JDeveloper project so they can be deployed into MDS. I personally don’t think this is an issue at all as this is the same way I used to do it in 11g anyway!
  • OER 12c (12.1.3) will be released soon. This new version will support harvesting of SOA Suite 12c assets. I will provide links on this topic later on.
Having said that, it is important that if you are starting a new 11g project (and you are evaluating using AIA or OER) or if you are already planning an 11g to 12c migration, you consider these factors and plan accordingly. In fact, in a recent 11g project I was faced with the dilemma on whether or not to make use of AIA and OER. I will not go into details on the decision made, however what I will share if the following table which helped me and also the customer make an educated decision:
SOA 12c AIA/OER Backward Compatibility Table
Feature
Requires AIA FP Installation?
SOA 12c AIA/OER Backward Compatibility
Service Constructor JDeveloper Plugin
Yes and No. Yes, It can be used independently of AIA as it is a JDeveloper Plug-in however it requires AIA FP to be installed on server as otherwise composites deployed which were developed with constructor will not run on server.
This feature is not available in 12c. Not even as part of 12c Core Extensions
Code Compliance Inspector (CCI)
No. It can be used independently of AIA as it is a Jdeveloper Plug-ing
Part of OER bundle. OER 12c (12.1.3) will be released soon and will support 12c SOA 12 harvesting
Lifecycle Workbench
Yes. It requires AIA FP
This feature is not available in 12c. Not even as part of 12c Core Extensions
Composite Application Validation System (CAVS)
Yes. It requires AIA FP
CAVS is available in 12c as part as SOA Core extensions
Setup Pages
Yes. It requires AIA FP
Exception handling features of AIA FP are supported on 12c as part of 12c Core Extensions
AIA Message Resubmission Utility
Yes. It requires AIA FP
Exception handling features of AIA FP will be supported on 12c as part of 12c Core Extensions
AIA Solution Pack
No. Comes as part of OER installation
Part of OER bundle. OER 12c (12.1.3) will be released soon and will support 12c SOA 12 harvesting
AIA Harvester
Yes. It requires AIA FP
Part of OER bundle. OER 12c (12.1.3) will be released soon and will support 12c SOA 12 harvesting
AIA Deployment Plan Generator
Yes. It requires AIA FP
This feature will is not available in 12c. Not even as part of 12c Core Extensions
AIA Installation Driver
Yes. It requires AIA FP
This feature is not available in 12c. Not even as part of 12c Core Extensions
AIA WSM Policies
Yes. It requires AIA FP
Yes, this will be available in 12c. Anyhow creation of Policy Sets in WSM which will be supported in 12c are not complex to create
AIA Composites
Yes. It requires AIA FP
Yes, exception handling related composites are available in 12c as part of SOA Core Extensions in 12c
AIA Prebuilt Integrations
Yes. It requires AIA FP
It is not my understanding that AIA pre-built integrations (i.e. PIPs) will be created for 12c. However some backward compatibility will be provided in 12c with SOA Core Extensions
AIA Error Handling Framework
Yes. It requires AIA FP
Exception handling features of AIA FP are supported on 12c as part of 12c Core Extensions
AIA Logging Framework
Yes. It requires AIA FP
Logging features of AIA FP are supported on 12c as part of 12c Core Extensions
AIA Metadata
No. They can be extracted from an environment that has AIA and then deployed as JDEV Metadata Projects
AIA Object Libraries (including EBOs) are not installed in 12c along with the Core Extensions. Only way to deploy these libraries to 12c will be packing them in a JDeveloper project and  deploying it to 12c MDS

Sunday, 18 May 2014

SOA Transformation through SOA Upgrade

Much has been said about Oracle SOA Suite 10g (or JCaps) upgrades to 11g and how features map between both versions. There is also plenty of information online about this topic both official and unofficial. It’s not news to many that for example SOA Suite 10g is currently in extended support and product will enter sustaining support by the end of 2014 (I will explain more about what extended and sustaining support means later in the blog). However one fact remains truth: There are still many companies out there running platforms that are (or soon will be) in sustaining support, and that don’t yet have an upgrade strategy. I say this based on my own experience as I am currently helping several customers do exactly this.

Having said that,  I wrote this blog in an attempt to give SOA experts, Integration Leads and Architects key pointers that can serve as inspiration to come up with a transformational approach when defining an upgrade strategy. Note that I am using the word “transformation” deliberately and I will explain why shortly.

Note that although this article is mainly related to the Oracle SOA 10g to 11g technology stacks, the approaches, tips and information provided in this blog should also be applicable when defining any technology upgrade. In fact, once 12c is more mature I will probably refresh this blog to cover 11g to 12c upgrades.

Following my key pointers to help you define your upgrade as a SOA Transformation:

1) Understand the product roadmaps and planning to move in advance
2) Take a SOA Transformation approach and not just a technology upgrade
3) Elaborate a SOA Transformation Roadmap
4) Understand current and future technology stacks and identify potential risks and challenges in advance
5) Define a service transformation methodology
6) SOA transformation also requires organisational changes and maturity

1) Understand the product roadmaps and planning to move in advance

This is one of the most important points and one that many have either failed to understand or have just ignored (hence why many companies still stuck in 10g and have no plan to upgrade yet). This is important because by understanding the product releases and features, release dates, and support lifeline you can plan in advance an upgrade approach and avoid having to do something tactically, in a rush and with limited budget.

Before getting further into this topic, it is critical that you understand the basics of Oracle product releases:

  • Oracle product releases: There are two main points that you need to be aware of when talking about releases: Major release and patch set release (also referred to as component release). Let’s take for example Oracle SOA Suite 11gPS6 which is the same as saying 11.1.1.7
    • Oracle talks about product versions in two different ways: 1) Product release name (i.e. 11gPS6) , 2) Product release number (11.1.1.7.x). Either can be used to refer to a specific product release, however I prefer the release number as it gives more detail.
    • 11 is the major release. so for example 10g to 11g actually means major release 10 to major release 11. A major release means that there is a significant change in technology platform. It usually also means that you can’t simply apply a patch but rather requires more planning and preparation as it might require having to install and configure separate infrastructure and more elaborated service/code migration
    • The “g” is for “grid”. This is more of a commercial/marketing thing. For example in version 12, it’s 12c where “c” is for cloud
    • PS standards for patch set. A patch set is a single patch that contains a series of patches. In the example discussed PS6 standards for patch set six
    • For details on how to read the product release number refer to: http://docs.oracle.com/cd/E28280_01/core.1111/e10105/release.htm
  • Oracle patching and upgrade terminology: Oracle has specific definitions for what patching, migration and upgrade means. Patching is basically  applying a patch to an existing installation. Migration is moving code from a third party product to an Oracle product. Upgrade means moving from one major version to another (i.e. 10g to 11g, or 11g to 12c).  For more info on this refer to following link:
  • Oracle support stages: There are 3 support stages:

1) Premier Support: This is where you want and should target to be as full support will be provided in terms of patching, security fixes,  better product certification with other products and less obvious but important there will be more people skilled in support teams to deal with products in this stage

2) Extended Support: Pretty much the same as premier support apart from the fact that you will have to pay an extra fee (on top of standard fee). Also extended support may not include certification with some new third-party products/versions

3) Sustaining Support: You should always plan to avoid going into sustaining support. Reason is simple: not only an extra fee will be require on top of standard fees, but Oracle will not create new patches or fixes for products in sustaining support. Moreover there will be less available Oracle support resources to help you with you issues as most of the engineers are already supporting latest version of the product (versions in premier and extended support)

Refer to the following documentation for more details on Oracle support stages and support policies:

Assuming that you have gone through the previous points, I created the following diagram to try and depict the evolution through time of the different Oracle SOA Suite releases and their patch sets along with their respective product support lifelines:

Product-Roadmap Based on this diagram and the information described in earlier points I would strongly recommend that:

  • Plan your technology upgrade to latest major release at least one year before extended support starts. This will ensure that you will always be in premier support
  • Try to keep up with latest patchsets. Note that there are constraints in extended and sustaining support around which patch set you are running (refer to previous links). Note that normally patchsets do not require re-coding and all migrations are usually assisted and can even be scripted. If you are worry about regression testing your code, look into implementing continuous integration or similar disciplines to automate execution of core test scripts (this is a best practice and you should try to do it anyway)
  • If you already too late for you meaning you are in extended support (i.e. running 10g) and have no upgrade planned or you are only planning upgrade at this point, then by any means try to do it properly just don’t go and rush it. Chances are that you may anyways end up having to run the platform in sustaining support, not ideal by any means, but also not the end of the world. So my suggestion would be don’t waste any more time and plan a transformation strategy (see my subsequent points)

2) Take a SOA Transformation approach and not just a technology upgrade

But what does it mean to take to take a SOA transformation approach?

Let’s say for example that you are looking at upgrading your SOA infrastructure to a new major release, let’s say 10g to 11g or 11g to 12c. A typical upgrade project would focus mainly on technology and would almost certainly aim at changing as less as possible the existing code and services. By simply doing a technology upgrade, one would not only miss the opportunity to leverage the new product features, product enhancements and latest best practices available  (because you would try migrate the code as-is whenever possible) but would also miss an opportunity to consolidate, rationalise and improve your existing services landscape therefore bringing benefits that would result in lower TCO.

For this reason, instead of focusing merely on the technology shift, take a step back and look at the bigger picture. Think about the potential benefits that you could bring to your organisation if you could also transform (not update or improve but transform) the processes and organisational/cultural aspect of the current solution that prevents realising such benefits.

How to know what benefits can be realised? There is always room for improvements.  There will surely be areas that could be improved and haven’t for different reasons. For example there is no budget or there is no sponsorship from above to make the necessary changes or simply there is no enough skills in-house to execute such changes. To identify such areas that could be improve best thing to do is to conduct an assessment of your current integration landscape as suggested in point #3.

Also because a transformation approach requires fundamental changes to the core processes, technology and organisational/cultural aspects of your SOA solution, it is imperative of success that you engage with different parts of your organisation as early as possible to sell your approach and get buy-in. The toughest part of a transformation project is nothing to do with technology, it’s in fact dealing the the organisation and cultural changes it brings. I elaborate further on this topic in point #6.

Last but not least, there is no project is there is no funding. So to secure funding you must get approval from CFO or the like (unless there is spare IT budget to spend on this, in which case you would still have to justify the budget needed to CIO or CTO). I always compare this to going to a bank and asking for a loan. The bank wont lend you any money unless they are sure that they will get the money back and with interests (benefits). The bank business is to lend you money, so they want to do it, but they qualify the risk and if it makes sense then they will do it. Is the same within an organisation. C-level and D-level executives want to fund projects that deliver tangible benefits. But they wont just give funding away without proper analysis that there is a return on any investment they make. So to prepare for this you usually need to:

  • Define a strong business case: The business case should express qualitatively and quantitatively why funding this initiative is the right thing to do. It should emphasize on the cost of doing nothing or doing it wrongly.  Furthermore the business case should not be based solely on the fact that your platform will run out of premier or sustaining support at some point. Yes this will certainly help, however to justify a transformation project funding you need a lot more substance. If you have some budget available and you don’t have the time to do this, you could get an expert to help you define the business case. Below some references that can serve as inspiration for this task:
  • To support your business case, define a SOA transformation roadmap which lists at a high level (but with enough substance so it looks thought through) the set of activities needed to get to where you want to be. Following point elaborates more on this topic.

3) Elaborate a SOA Transformation Roadmap

This may sound a lot more simple than it actually is in practice. In order to succeed in elaborating a roadmap you must understand:

  • Current integration landscape including architecture, maturity and service catalogues (see previous points)

Below a sample high level view of a roadmap.

Rodmap

From this sample high level roadmap I would like to highlight the following:

  • There are to major phases: Foundation Phased and Implementation Phase. Foundation Phase –as the name suggests, aims to deliver a strong foundation. Key design decisions, standards, reference architectures, enhanced software development lifecycles, methodologies and governance frameworks should be defined delivered in this stage. Implementation Phase focuses in the rollout of the new platform not only to existing projects that will have to be upgraded or migrated from other platforms but also by making the new platform available to incoming projects
  • Implementation phase starts with a pilot project. This is not a PoC but rather a real project with very limited scope but that use cases covers enough scenarios and test cases to ensure that the future architecture is robust enough to support known requirements
  • Once the pilot has been delivered a “lock down” should be defined to the as-is platform. For example if you are running 10g, you should not allow any more developments in the 10g platform once lock down has been put in place. Again this is easier said than done, and politics will kick in at this point specially as some influential and important projects will demand a solution
  • Transformation of  existing projects to the new 11g platform doesn’t have to be and should not be in a big bang. You should define a plan that allows you to run your 10g or other platforms in parallel to the new 11g platform so you can incrementally align projects which will –in a phased manner, implement the same services in your new 11g platform
  • New incoming projects will require support. Lots of support. Don’t just assume that providing  a link to some documentation will be enough for them to understand how to deliver according to the to-be reference architecture, guidelines and frameworks. One way of dealing with this is to assign some resources (ideally architects or designers) from your team into each of these incoming projects. These resources should act as mentors of the new solution and should coach on how to do things the new way
  • This won’t take a day and it will require planning and it will require proper sponsorship from the right level (I keep saying this and sometimes I feel that I sound like a broken record, however it’s so important that I don’t mind repeating it)
  • The following link provides some good tips when executing transformation programmes: http://www.itbusinessedge.com/slideshows/show.aspx?c=85374

4) Understand current and future technology stacks and identify potential risks and challenges in advance

This is a task that you will have to elaborate into detailed during your foundation phase. However even before you get to this phase, you should already have a good understanding of your as-is technology stack and how it maps to your to-be technology stack. Following an example of what I mean. The diagram shows how Oracle SOA Suite 10g and related technologies maps to the 11g suite and related technologies.

10gTo11g

There are several factors that you should understand before you even start putting together a plan. What you are looking for is factors that will drive higher risks and effort. For example you should be looking for understanding:

  • Upgrade paths: This no only applies to code but also to the code. For example, you should be aware in advance if there are specific restrictions in terms of minimum versions and patch sets that your platform and code should be on before doing an upgrade. You should also be aware of there are assisted ways of upgrading your core infrastructure as well as your code. In the previous diagram I depicted how some technology components of 10g maps nicely to 11g and how others don’t. For example, BPEL 10g and  ESB 10g maps directly to SOA Suite 11g BPEL and Mediator engines. It is also visible that there is an assisted / automated way of upgrading the technology and code. However it is not the same story for other components such as OBPM 10g, BPA 10g processes and WSM 10g. These will require manual re-coding (yes re-coding as there is no official wizard or tools am aware of to automate this). Some good information on this (specially on the topic of 10g to 11g):
  • Architectural and best practices : Even if there are guided upgraded paths for some technology components there are several other factors that should be consider if you truly want to embark on a SOA transformation and not just a technology upgrade. Don’t just use a wizard because they are available (also be warned about Wizards –see the following point). To truly transform, you should evaluate architectural factors as well as new best practices available. Ideally a foundation phase as previously described would target to set in stone your future reference architectures and best practices so this task become a lot easier. Based on the previous diagram I can provide several examples of what I mean (however you should come up with your own depending on your target architecture and best practices):
    • According to Oracle best practices, the strategic service bus in 11g is OSB and not Mediator. However if you have built services using ESB 10g the easiest way to migrate the code to 11g is by using the assisted wizards or tooling meaning that your ESB code will end up in SOA composites which will run in Mediator engine. This however doesn’t mean that you are making the best out of your to be architecture because in 11g for example, OSB has better routing capabilities. However in some scenarios migrating directly to Mediator might make sense. For example, if your service implements WS-Addressing (asynchronous services) as OSB does not supports this natively (there are work around thought but it’s not native).
  • Technology constraints: There could several changes in the technology stack that are less obvious and you will only find out after you start testing your code. You should try and identify this either by learning from other’s lessons learnt or engaging your vendor for detail feedback on this. Again using the previous diagram, I can list the following examples of what I mean:
    • Wizards don’t to it all: Even for the technology components where an upgrade wizard is available don’t just assume that everything will work once code has been upgraded. There are several things these wizards don’t take care off. I would recommend the following link for more details on this: http://www.slideshare.net/oracle_imc_team/oracle-soa-suite-upgrade-10g-to-11g
    • There is no MDS in 10g. Therefore if  in 10g you had all of your schemas deployed as web application and accessible via HTTP ideally you would like to move these schemas into MDS in 11g (you don’t have to, but this goes back to my point about best practices). This however would require some changes in the code (so mds references are used instead of HTTP references) and also the schema project itself (so it can be deployed to MDS via JDeveloper or ANT). Refer to following link for more info: http://redstack.wordpress.com/2011/02/28/deploying-your-canonical-data-model-to-mds/
    • If your 10g services consume other 10g services is by using concrete WSDL rather than abstract WSDL’s then you will run into issues. In case you don’t know the difference between the two, abstract WSDLs are usually generated by JDeveloper and do not contain any binding and/or port type information. Once a JDeveloper is deployed, Oracle SOA Suite automatically generates a concrete WSDL that actually refers to abstract WSDL but that defines the binding protocols and port types. In 10g during start up Oracle Application Server started each service in the same order that they were deployed. This ensured that if you had deployed your services in the right order, then they should all start correctly. However in 11g, Weblogic  Server starts the services randomly so if a particular composite depends in a concrete WSDL for a service that hasn’t yet started, then the service will fail to start. To fix this issue you should change your code so it refers to abstract WSDLs whenever calling another internal service. More on this: https://blogs.oracle.com/aia/entry/aia_11g_best_practices_for_dec
    • OBPM 10g (former ALBPM) doesn’t have assisted upgrade toolkit to 11g. So if you want to migrate OBPM 10g code to 11g you have to re-implement your processes. There is a chance that OBPM 10g  will have assisted upgrade tooling directly into BPM Suite 12c
    • WSM 10g also doesn’t have assisted upgrade toolkit to 11g. So if you want to upgrade WSM agents you should probably be looking at manually re-implementing them as 11g WSM policies and if you want to upgrade your WSM Gateways you should manually re-implement 10g policies into Oracle API Gateway (OAG)
    • Business Process Architect (BPA) Suite also doesn’t have assisted upgrade toolkit to11g. Some aspects of what BPA suite can do can be migrated to BPM Suite 11g. However more richer enterprise modelling features Value Added Chains, organisational modelling, and others will only be supported in BPM Suite 12c
    • Oracle Application Server 10g is no longer use in 11g and beyond. Weblogic Server becomes the core Oracle’s strategy JEE server

5) Define a service transformation methodology

It doesn’t have to be an overkill methodology. What you need is some concise steps which sets in stone the approach to follow when transforming a service from your existing platforms (i.e. 10g or other legacy) into your target solution. Purpose of the method is to ensure that all transformation activities are occurring following a consistent method and using the same set of guidelines and standards.

Method

6) SOA transformation also requires organisational changes and maturity

This is for me without a doubt the most complex aspect of a transformation initiative. This because you can change all tools and even upgrade most of your processes , however without having a behavioural and cultural change in your organisation chances are that the same issues you had in the past will repeat all over again. This is explained in detailed in my SOA Magazine article 9 Tips for Organizational Maturity in SOA also available in my blog www.soa4u.co.uk

Friday, 24 January 2014

SOA Governance FAQ

After more than decade implementing SOA, I’ve come to the conclusion that one of the reasons that prevent organisations from realising the benefits of SOA is the lack of some sort of governance in their SOA implementations. I have also come to the sad conclusion that even though this is well known fact, many (end customers and SIs alike) still lack understanding of what SOA Governance is and why is it needed. Having said that, I created this post not only for my own reference, but also share this knowledge for those wishing to improve their understanding on this amazing topic.

Hopefully this FAQ provides some concise answers to many of these questions.

R. In simplistic terms, SOA Governance is the alignment of processes, people and tools needed to ensure that a SOA implementation is successful. SOA Governance primary objective is to maximise the benefits that SOA can bring to the business and IT. One thing to bear in mind is that SOA Governance does not exist on its own; it is in fact an extension of IT governance and EA governance.



For technical communities, SOA Governance is usually seen as a discipline that defines the processes, the roles and responsibilities and the tools needed to successfully implement a sound SOA solution.

For the business community and also less technical people, SOA Governance means a discipline focused at ensuring that business benefits and also ROI is realised by adopting SOA.

2.     What SOA Governance isn’t?

R. SOA Governance as a discipline is often confused with other important disciplines and concepts of software engineering. This is mainly because the term governance as such as quite broad and covers several aspects of software engineering.

·          Standards: such as naming conventions, patterns, reference architectures, standards, amongst other, cannot be considered to be Governance.  Although these are important assets, if not enforce through a process they are often forgotten and out-dated.
·         Configuration Management, Version Control or Continuous Integration: Although these are very important disciplines within software engineering and contribute greatly towards adopting governance, without supporting processes, policy enforcements and the right tools, these disciplines can become an overhead and easily run out of control.
·         A Tool: Tools without structured process around it add little value and will likely end up not being used for its original purpose.
·         Review Gates: Having a panel (i.e. Design Authorities) responsible for approval or rejection of deliverables is an important aspect of Governance however without some sort of automation, robust traceability, policy enforcement, and control these can be time consuming, inaccurate and ultimately unmanageable.

3.      Why is SOA Governance needed?

R. Without some level of SOA governance chance of success when implementing SOA are minimum. This is because unlike other software architectural styles, SOA requires a good level of definition and analysis up-front before embarking into a SOA implementation. Costly lessons learnt have proved that.

Having said that, the following table lists the key problems that in my opinion could be solved by implementing SOA Governance.

Problem
Consequence to the Business
Lack of visibility over existing assets and its performance
Minimum asset reuse and duplication introducing extra costs (both in CAPEX and OPEX). Without a level of analytics it is not possible to determine ROI.
Tactical Projects over of Strategic Solutions
Projects have their own agendas which deliver short term benefits to the project but that add no long or mid term Enterprise value.
Poor decision making and Lack of accountability
No sense of ownership makes decision making, policy enforcement and accountability an impossible task.
Low quality of Assets which become difficult to maintain and change.
Higher complexity and cost of change introduces Risks to the Business preventing new and innovative solutions to be introduced.
Poor estimation techniques and inaccurate planning
Projects cost more than estimated mainly because of “unknowns”... (i.e. Rework, extra activities and deliverables, dependencies, complexity, and others)

4.      What are the objectives of SOA Governance?

R. The objectives of SOA Governance are should always be aimed at delivering tangible benefits to the business. By tangible it is meant that these benefits can be measured quantitatively and qualitatively and therefore there is no ambiguity and the positive results that SOA brings to the business.

·         Aligning the SOA strategy to the business objectives
·         Delivering a Framework  suited for Business Agility and Change
·         Aligning SOA to Enterprise Architecture
·         Providing visibility over existing Assets, its use and operational performance
·         Improving the quality of assets by enforcing policy and standards
·         Increasing the ROI by asset re-use
·         Reducing the cost of change and support
·         Reducing the risk of failure
·         Improving agility and promoting innovation

5.      What are the key components of SOA Governance?

R. At a conceptual level SOA governance consist of the following components:

·         A SOA Strategy: Define SOA objectives which are consistent with what the business vision and strategy is and aims at delivering business benefits
·         A SOA Governance Model: Defines the, governance processes, governance tools, governance roles and responsibilities, governance artefacts needed to implement governance.
·         SOA Design Time artefacts: All of the SOA governance related assets required to support the design-time aspects of governance (i.e. requirement elaboration, analysis, design and build). For example reference architectures, development standards and policies, programming standards and policies, tools such as repositories and/or agile management tools.
·         SOA Run Time artefacts: All of the SOA governance related assets required to support the runtime aspects of governance (i.e. test, deploy, monitor, maintain, improve, and retire). For example, deployment frameworks, testing automation frameworks and continuous integration, monitoring tools, runtime policy enforcement tools.

At a logical and physical level, we are talking about all of the assets such as processes, documents, tools and policies that have to be delivered as part of a SOA Governance implementation.

6.      What are the top challenges that prevent successful SOA Governance implementation?

R. From my experience the top 3 are:

·         Lack of pragmatism when implementing SOA Governance. SOA Governance can be define as complex as one wants. SOA is so broad and it covers so many aspects that if one do not apply the right level of pragmatism it is quite easy to lose sight of the ball and end up building a Ferrari when what was really needed was a BMW or worst a Ford.
·         Dealing with people and other organisational challenges. Because SOA spams and affects almost every aspect of the organisation and their systems, it becomes incredibly difficult having to deal with people from different departments, with different views and different priorities. This means that one might end up in the middle of a political nightmare just by trying to do the right thing….
·         Lack of governance in other areas of the organisation. For example, lack of IT governance or EA governance. Because SOA governance is an extension of these two, if these other types of governance are missing, it becomes incredibly difficult to sell the benefits that SOA Governance has to bring and therefore preventing securing any funding. Without senior sponsorship SOA governance adoption becomes almost impossible.

7.      What are the most common mistakes organizations make when implementing SOA Governance?

There are many common mistakes made when implementing SOA Governance however it is generally accepted that the most common ones are:

·         Misunderstanding of what SOA governance is and therefore lack of investment in the tools, processes and people needed to successfully implement it.

For example, many think that SOA Governance is just about writing a bunch of document standards (which get updated once a year) and expecting people and vendors to follow it.

Or for example implementing a tool but not defining the right process around it such as the tool in facts delivers value to the overall process.

·         Another typical mistake is implementing Governance purely at technical level without engaging the business. This means that the objectives of implementing SOA Governance are not focused at delivering business benefits. This means that investment made can’t ultimately be justified and thus converted to a benefits realised by the business.

·         Getting the wrong people to implement SOA Governance. So let’s say that the IT and EA department both have successfully implemented governance and they want to define and implement SOA Governance to ensure that the SOA implementation and its projects are successful. For this they hire a SOA Architect. However this SOA architect is a blue sky thinker understands the concepts and theory but has not practical experience in real live implementing SOA. The outcome will probably be an overkill solution that’s not pragmatic, delivers more complexity than was needed, no one understands it and ultimately the implementation has no buy-in from the key communities such as the developer’s communities.

8.      What Oracle tools do I need to successfully implement SOA Governance?

R.  The components that build up the Oracle SOA Governance Solution infrastructure are depicted in the following diagram:
·         Oracle Enterprise Repository (OER): Oracle Enterprise Repository implements a design-time governance toolset used to support service lifecycle and other key stages of a solution, such as service discovery, and to provide a framework to promote service reuse.
·         Oracle Service Registry (OSR): Oracle Service Registry is an UDDI version 3 compliant registry used to support runtime governance. It provides a runtime interface to the Enterprise Repository allowing service consumers to dynamically lookup service locations at runtime.
·         JDeveloper: Is Oracle's preferred integrated development environment (IDE) for the development of software solutions using Java, SOA Suite and other technologies such as SQL, PLSQL, XML, PHP, amongst others. JDeveloper offers a wide variety of plugins to integrate with other products such as OER and OSR.
·         Web Service Manager (WSM): This is a security policy manager that allows administrators to centrally define and manage security policies in a non- intrusive manner. Policies can be attached to services in order to enforce security and to enforce compliance to enterprise security policies. Policies can be attached during the design phases as well as at runtime.
·         Oracle API Gateway (OAG): Former Oracle Enterprise Gateway, OAG is a standalone for implementing robust security polices into services. OAG is typically deployed as a policy enforcement point (PeP) in demilitarized zones (DMZ) where services are consumed or exposed by applications located in untrusted networks.
·         Oracle Enterprise Manager (OEM): This is a web based application for managing and monitoring the Oracle based infrastructures on which services execute. The OEM SOA Management Pack is an add-on for Oracle Enterprise Manager that delivers a complete toolset for configuration management and monitoring of SOA infrastructures.
·         Business Transaction Management (BTM): Delivered as part of OEM (although requires separate installation), BTM as its name suggests is a tools that allows transactions to be monitored as they spammed across multiple systems. Unlike tools like OEM or Fusion Middleware Control that monitor transactions within their own containers, BTM uses observers to collect transaction information from different sources then using complex matching algorithms, it reconstructs the end to end transaction in a single graph. The sources can be anything from a J2EE application server, an MS Internet Information Server, to products such as OSB, SOA Suite and Webcenter.

Refer to the following Oracle website for more information on Oracle’s SOA Governance solution

9.      To what degree does successful SOA governance require organizational change?

R. In many ways. SOA Governance defines new processes that spam across different departments, it may require implementation of new tools and also processes around it, and also requires the creation of new roles and responsibilities.

It also implies that people from different aspects of the organisation need to interact and collaborate. One would think that this is not an issue, but in fact is one of the greatest challenges when implementing governance as previously mentioned.

In the following article I elaborate a bit more on this topic and also provide useful tips that can help reach organisational maturity.

10.      What impact, if any, trends such as Cloud, API development, Mobile Computing, Big Data, and the Internet of Things( IoT) have on SOA Governance?

R. All these new trends require and implement SOA one way or another. The fact is that all of these evolving technologies are in their core dependent on distributed computing as the systems and data that these new technologies depend and interact with are located all over the internet. For example, Big Data systems such as Hadoop, feed from structured and unstructured data sources located all over the internet. Mobile apps also require information to be made available to them preferably using light weight services such as RESTful services that could be located anywhere on the internet. IoT it's all about internet enabling devices that we use in our day to day life (watches, fridges, alarms, etc, etc, etc) and allowing them interact with intelligent systems which are located somewhere in the cloud. These systems would then help us become smarter and more efficient in our day to day activities by making sense of the data that is constantly being captured. Again, without some sort of efficient and distributed client/server communications this would simply be impossible

SOA as an architectural style and philosophy, was conceived to fundamentally support the concept of distributed computing whilst inheriting the best features from other paradigms such as Object Oriented Programming (OOP) and Enterprise Architecture Integration (EAI).

Moreover, all of these emerging technologies relay on Application Programming Interfaces (APIs) either to make their data available to other systems and/or apps or to consume data from other systems. Although initially the API movement evolved somehow independently of SOA architectures, APIs eventually evolved to become an integral part of SOA. This is because many of the problems that APIs were designed to solve are actually shared by SOA (i.e. support for a diverse of technologies and open standards, support for componentization and abstraction, deliver flexibility and promote reuse, etc). Even the tools use to create, monitor and managed API are the same or very similar to the tools used when implementing more traditional SOA architectures.

Having said that, SOA Governance, as a discipline, is aimed at ensuring that  people, processes and tools are all aligned towards achieving the same goal: delivering IT and Business benefits. The impact these trends have on SOA Governance? well, SOA Governance will have to adapt to support many new asset types (i.e. APIs) and support the emerging processes, tools and roles that are required to enable these new technologies to deliver benefits not anymore to the one company implementing SOA, but also to the many parties (businesses, people and machines) that benefit from these technologies. For example, in the case of APIs, there is another discipline known as API Management, but in reality this discipline is of an extension of SOA Governance however more focused on achieving similar goals but more specifically around APIs. The consumer of an API can be anyone on the internet: a person, a business or even a machine

11.      Are there any practical books on how to implement SOA Governance?

R. Although there are several books out there in the topic of SOA Governance, there isn’t in fact many in the topic of how to actually implement it in practice. For this same reason I decided to write a book on the topic.

If you are interested in knowing more about the book please have a look at the following link:

For regular updates on SOA Governance join the Oracle SOA Communities: