Thursday, 9 April 2015

A Word About Microservice Architectures and SOA

In this article I will talk about my first conclusions and my point of view regarding Microservice Architectures. As there is still quite a lot of confusion and debate out there on this topic, I will try to describe with my own words what Microservice Architecture is, how does it differ from typical Service Oriented Architectures (SOA) and what design principles and practices governs it.
What is a Microservice Architecture?
In the article http://martinfowler.com/articles/microservices.html written by Fowler and Lewis, Microservice Architecture is described as following::

Microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies

The article overall it’s a fantastic piece of work (really suggest you read it). The way Microservice Architectures it’s defined opens up a few pandora boxes (in a good way I think) which I will talk about subsequently.
First of all, if you are familiar with SOA and it’s guiding principles this will seem very familiar (read for example: http://en.wikipedia.org/wiki/Service-oriented_architecture or http://www.soa-manifesto.org/). Yet, if you noticed the highlighted texts, it’s not quite the same as what we are used to in traditional SOA. The truth is, wether we accept it or not, SOA architectures evolved around the adoption of certain design patterns (such as Enterprise Service Bus (ESB), canonical schemas, centralised contracts, -see http://www.soapatterns.org/ for more) and the use of SOA specific infrastructures to build and deploy services and APIs became the approach of choice (note that the service vs API topic it’s not discussed in this post. For my view on this read http://www.soa4u.co.uk/2013/09/restful-is-also-soa.html).
From my perspective, I would define Microservice Architecture’s as both 1) a design pattern and 2) a discipline for delivering services and APIs. To elaborate further based on my conclusions I can highlight the following guiding principles:

  • Delivering business focused and lightweight services/APIs that are truly design, built, deployed and executed independently of each other (meaning that in terms of infrastructure dependencies, they share very little)
  • Strong focus on people collaboration and communication as the main mechanism in the adoption of best practices and standards rather than common set of strict guidelines and standards that constraint the way services are define, built, deployed and maintained
  • DevOps (config management, deployment automation, CI, Continuous Delivery) as a fundamental building block rather than a value add
  • Scalability should be easy as services are very lightweight and stateless (The same service can run in many servers and DevOps makes the deployment process automatic and easy)
  • Doesn’t encourages the use of monoliths to deploy services (a monolith is for example an application server or an ESB). Services should run almost as demons
One can argue that SOA architectures can also satisfy the listed requirements as SOA it’s really an architecture paradigm that can be realised in different ways. I personally think this myself and I would regard Microservice Architecture as a SOA design pattern, however as per my previous point, comparing it with traditional SOA architecture’s there is a difference.

Microservices vs SOA
To make my point more clearer, I created the following diagram to compare the system layers in a traditional SOA infrastructure to a Microservice one:


Microservices Vs SOA System Layers
As it’s visible in the diagram, there are many similarities between the two, however there are three key differences, two of which are very obvious but one that is not  (there are more that you can deduce from the digram and all the links I’ve provided):

  • First of all, traditional SOA implementations have been built around the use of monoliths ( or as Microservices calls it: Monolithic Architecture pattern -read http://microservices.io/patterns/monolithic.html)
  • In a Microservice Architecture on the other hand, services and APIs are built and deployed without almost any dependency on an underlying application server or application. This means that services run as processes of the OS and share very little in terms of underpinning application infrastructure. This is because in Microservices (as per the highlighted text in Fowler’s / Lewis's definition) services and APIs should ideally run independent from each other. Another good article for this is http://microservices.io/patterns/microservices.html
  • The third one -not obvious, is around composition of services. Traditional SOA promotes the use of orchestration (i.e. BPEL) whereas Microservices encourages the use of choreography. The fundamental difference here is that in Orchestration you need an “orchestra director” that defines and manages the interaction logic between all services (basically the BPEL code that executes or even in SOA composites, the SCA file like composites.xml that dictates at runtime how components interact). Choreography is different. Is a design time mechanism to model service interactions and behaviours but then once a service is built and deployed they should “know what to do”. Think of it as a protocol. A good analogy: "Dancers dance following a global scenario without a single point of control" (http://en.wikipedia.org/wiki/Service_choreography)

The following forum thread also provides a good point of view on the commonalities and differences of the two http://stackoverflow.com/questions/25501098/difference-between-microservices-architecture-and-soa
Now that I’ve described some of the key differences, I want to make it more obvious by showing a piece of code for a service built following the Microservice Architecture pattern. The snipped is based on Node.JS (https://nodejs.org/)  Express (a lightweight app for Node -http://expressjs.com/). Do note that Microservices can b built in any technology (i.e. Java) so long that the guiding principles are adhered to. 

var express = require('express');
 
var app = express();
 
app.get('/wines', function(req, res) {
   res.send([{name:'wine1'}, {name:'wine2'}]);
});
app.get('/wines/:id', function(req, res) {
   res.send({id:req.params.id, name: "The Name", description: "description"});
});
 
app.listen(3000);
console.log('Listening on port 3000...');

(full example on http://coenraets.org/blog/2012/10/creating-a-rest-api-using-node-js-express-and-mongodb/)
As it’s visible, this simple REST API is fully self-contained. It not only defines the resources that the API will provide and the HTTP verbs to use, but also defines the port the API will listen to. This basically means that this API will be self-contained and running in it’s own process. Fascinating isn’t it?
In regards to people, Microservice Architectures encourages the creation of multi-disiplinary teams that are organised around the capabilities that specific micro services will support, rather than creating different teams based upon for example the different technologies and layers involved in the creation of a single microservice. As per Conway’s law, the systems you build will be a reflection the organisation and communications structures you followed ( I’ve about this in http://www.soa4u.co.uk/2013/11/9-tips-for-organizational-maturity-in.html). In addition it is encouraged to have strong communication and collaboration mechanisms as this becomes the preferred way to share best practices and standard documents written ages go and available in somewhere in a forgotten content management system.
Furthermore DevOps/Contionus Delivery in Microservices really is a must-do rather than a value add. At the end of the day, DevOps main objective is to bridge the gap between development and operations team therefore delivering high degree of agility into the main software development lifecycle (SDLC) stages. For automation aspects of DevOps, below a nice diagram that can serve as inspiration on what tools can be used for each phase.

DevOps Tools
Microservice Architecture is NOT a silver bullet! - ESBs are still needed
I personally think ESB’s still have a role to play to bridge application integration gaps in large enterprises as “choreography” alone all become streamy challenging in such large landscapes. The need for a robust tool to handle message routing, complex transformation, enrichment, throttling, and virtualisation tasks is still very valid and needed. The truth is that large organisations have many systems, many of which are commercial of the shelf (COTS), others are legacy, and others bespoke. Solving this integration challenge its not an easy tasks and a combination of technologies and architectural approaches is usually needed.

ESB based SOA
I can see this same issue occurring with Microservices as inevitably the number of microservices will grow exponentially and without a doubt governance will become challenging. This is the reason why SOA Governance and API management tools exist. it is one choice to define the right governance model (strict or lose). Also common problems such as visibility, lack or reusability, dependency management, amongst other will still be there in micro services. Therefore leveraging SOA governance and API management tools such as service and API catalogs and enterprise repositories will add a lot of value in they will help solve some of the common governance challenges.
I personally think that Microservice Architectures add most value when use for the right use case. For example, to build systems and applications with a very well define system boundary. Not necessarily to address all of the integration needs in an organisation. In other words, I don’t think Microservice Architecture is a silver bullet. I see it as another very good pattern that can be used when it applies, but as everything if overuse, challenges will come for sure.


Coexistance of SOA and MIcroservices
Another excellent use case where Microservice Architectures can apply nicely is around Internet of Things (IoT) and Machine to Machine (M2M). This is because by default devices are constrained by default on it’s hardware capacity and although at the moment focused is mainly in sensors, there is a lot of value in also having devices expose functionality and data via lightweight APIs (using protocols like MQTT or AMQP). This will allow devices to either interact amongst them (and this can be modelled using choreography languages like WS-CDL for example) or take part in a broader IoT ecosystem where the device can take part in a broader business process.
Microservices for IoT
Conclusions
I definitely see a lot of benefits and value adds in Microservice Architectures. I like the fact that there is very clear boundary between services not only logically but also physically. I never liked the fact that if an ESB goes down all services running in it also get affected…For this reason I think that major SOA and ESB’s vendors will embrace the Microservice Architectures way and might end up extending existing tools somehow to support or partly support this paradigm  or creating additional tools into their suites to support the creation of micro services.
Microservices Bus
Finally I think that some of the forgotten specifications such as WS-CDL (http://www.w3.org/TR/ws-cdl-10/) will get traction again as they do have a role to play in the Microservices world, or at least that is what I would like to think.


Hope you enjoyed the article :)

Sunday, 9 November 2014

Oracle API Catalog 12c Full Overview

Oracle has recently released Oracle Enterprise Repository (OER) 12c and Oracle API Catalog (OAC) 12c.I am particularly exited on the latter as it aims to simplify management of APIs and promises to be able to create catalog of your services and APIs rapidly. Furthermore, although OER 12c maintains the same richness as in OER 11g, this product is targeted to advance users that are in need to address more complex governance requirements whereas OAC it’s a light-weight version of OER however tailored to address specific use cases of API management such as cataloguing, discovering and rating APIs.
This blog is about my first impression when installing, configuring and using Oracle API Catalog. In the blog I provide all steps I followed to achieve these tasks and at the end I provide a summary of my experience and feedback of the product.

Installation Pre-requisites

  • Download Weblogic Server 12.1.3 and OER 12.1.3 software binaries from following link (note that OAC uses same OER installer)
http://www.oracle.com/technetwork/middleware/repository/downloads/index.html
  • You will need an Oracle Database installed (you may also download database from previous link). If you already have an environment with an Oracle Database available you can check supported versions from following link
http://www.oracle.com/technetwork/middleware/ias/downloads/fusion-certification-100350.html
  • Ensure you have installed JDK 7.0 Update 55 or later. You can download it from the following link:
http://www.oracle.com/technetwork/java/javase/downloads/index.html
  • Oracle API Catalog Home Page and Installation Guide:
https://docs.oracle.com/middleware/1213/oac/index.html
https://docs.oracle.com/middleware/1213/oac/install-cat/toc.htm

Installation

Firstly install Oracle Database and Weblogic Server as per instructions.
Then continue with installation of OAC 12c as per following instructions: https://docs.oracle.com/middleware/1213/oac/install-cat/intro.htm#CATIG146
Following key screenshots taken during installation of Weblogic Server and OAC and also some tips:
Installing Weblogic
- Launching Weblogic 12.1.3 installer
image
- Click Next and choose your FMW home (I’ve used /u01/fmw2 as I have SOA Suite 12c under /u01/fmw)
image
- Click Next and select “Fusion Middleware Infrastructure”
- Click Next and un-select option to get updates if this is a local or demo environment, then click Next and click Yes
- Click Install and then Finish if all goes well, otherwise check the logs for issues and retry
image
Installing OER binaries (OAC uses same installation binaries)
- Launch OER installer
image
- Click Next and select previously created FMW home (in my case /u01/fmw2) and click Next
image
- Click Next, Next again and then Install
- If all goes well click Finish, otherwise check the logs for issues and retry
Install Patches for RCU and Weblogic
- Instructions in following link https://docs.oracle.com/middleware/1213/oac/install-cat/install_gui.htm#CATIG394  , below summary:
- Go to Oracle Support and download patch 18791727 (RCU COMPONENT FOR OER OAC)
image 
image
- Run Opatch
image
- Check that it succeeds and if so apply Opatch for Weblogic
image
- Go to Oracle Support and download patch 18718889 (WEB SERVICES RUNNING ON WLS FAIL WHILE MAINTAINING SESSION)
image
image
- Run Opatch
image
- Check that it succeeds
image
- Run ./opatch lsinventory to ensure all patches were applied successfully
image
Run RCU
- Run <FMW_HOME>/oracle_common/bin/rcu.sh
image
- Click Next, then select “Create Repository” and “System Load and Product Load” and click Next
image
- Enter DB details and click Next and then Ok. Then select which schemas you wish to create. In my example I created schemas for both OER and OAC (ensure you use the right prefex. In my case I used “DEV” but you may used any other depending on what environment you are installing). Then click Next and Ok again
image 
- Enter password for the schemas and click Next, check that all schemas are correct and click Next and then Ok twice
image
- Click Create
image
- Click Close if all goes successfully, otherwise check logs and try again
image
Extend Domain
- <FMW_HOME>/oracle_common/bin/config.sh
image
- Select “Create a new domain” and ensure you provide a meaningful name to the domain, i.e. oac_domain (this will not cause installation to fail but in future it might create confusion around what the domain contains and purpose of it). Then click Next
image
- Select the template for “Oracle API Catalog" and click Next. Don’t try select OER and OAC template simultaneously as it will won’t allow you!! I tried this too!
image
- Enter the admin credentials for Weblogic server and click Next
- Select Production if you want more robust security or Development if you wish credentials to be loaded from boot.properties. Use default JDK (you may change this setting in a real production environment)
image
- Enter the DB details and credentials for the OAC schema and click Next
image
- Test connection and if successful click Next
- Select “Administration Server” and “Managed Servers, Clusters and Coherance” and click Next. Specially ensure you don’t miss Admin Server!!
image
- Select “All Local Addresses” so admin server listens to all IPs available in the server. Also ensure you change admin server port if you have other environments installed in your machine. Then click Next
image
- Do the same for oac_server1. Select “All Local Addresses” and ensure port it’s not already used then click Net
image
- Click Next twice (as am doing a demo environment I have no need set up a cluster and pinned servers to a machine)
- Click Create
image
- If Domain creation is successful click Finish
image
- Start Admin Server and then oer_server1
image
image
- Check the following link for best practices on stopping and starting Weblogic admin and manage servers: https://docs.oracle.com/middleware/1213/wls/START/overview.htm#START112
- Check post-installation steps for running OER diagnostic tool: https://docs.oracle.com/middleware/1213/oac/install-cat/post.htm#CATIG197

 

UI Look & Feel

Ok so finally the fun start!! we’ve done the installation and everything seemed to have gone smooth. Now let’s have a look at the UI:
- Once servers have  started open following url: http://localhost:8111/oac
image
- Enter default credentials (admin / weblogic1) then it will ask you to change credentials
- Home page looks very clean and nice. I can see that there aren’t as many options as OER and there is no “Asset Home Page” so I won’t be able to create/add assets from here. There isn’t an option either to create/modify Asset Types. OAC only support one asset type which is APIs.
image
- There are 2 other main pages: MyAPis and Admin
image
image
- The Admin page looks very similar to OER and the security model around users it’s similar to OER however there is no concept of “Projects” as there was in OER 11g.  I think this is a good thing as Projects could confuse non-experts users. If you are not familiar with this model, below is how OER 11g user security model looked like. In red entities that don’t apply for OAC so it should give a good idea of how users and departments can be configured
image

Harvesting API Assets

- Harverster is available under <FMW_HOME>/oer/tools. Unlike OER 11g, no need to unzip ! just configure HarvesterSettings.xml and start using it!
image
- Let’s configure HarvesterSettings.xml to connect to OER
image
image
- Run ./encrypt.sh as following to encrypt password in HarvesterSettings.xml
image
- Now let’s harvest a REST API by running “./harvest.sh –file “<REST WSDL URL>”
image
Note: Harvester has several features available including harvesting OSB services and SOA Composite applications. For more detail info read the following page:
https://docs.oracle.com/middleware/1213/oac/administer-cat/harvest_oac.htm#CATCG433

Discovering APIs

Now let’s go back to the OAC home page and let’s see if our API is registered.
- By default APIs harvested are not registered as “Published hence they don’t appeared in the home page. So to find the API select “All APIs” from dropdown menu as per below and click on search icon
image
- Now the harvested API (WADL file) appears in the search results. Click on the API to see more details
image
- On this page you can see all metadata associated with the API. Let’s edit the API by clicking the pencil icon
image
- Now from here we can change the status of API. From Active Status dropdown select “Published” and click save. Now API should show up in home page
image
- Yep, now is there.
image
- If you want to rate the API and add comments, you need to add the API to MyAPIs. To do this click again on API and then click on Folder icon
image
- Then go to the “My APIs” page, and from there click on the Comments icon. A popup will appear, from there rate your API and add your comments
image
- Go back to home page and you will see your 5 star API :)
image

Summary

  • Installation is comprehensive and I like the fact that RCU is part of Weblogic binaries. Although you had to apply 2 patches before you can run RCU and extend domain it’s pretty straight forward and well documented. In total it took me between 1hr and 1 1/2 hr to do core installation (bearing in mind that I used an environment with a DB already installed and I had already downloaded all binaries and patches needed)
  • Tools (in <FMW_HOME>/oer/tools) are already unzipped which makes it a bit easier to configure harvester
  • Encrypting password can be done directly against password in HarvesterSettings.xml which is very straight forward to do
  • Harvesters is the same as OER. So executing harvester is very straight forward and easy. Best of all a lot of the information available in my SOA Governance book for 11g will still be useful for OER 12c and OAC
  • Harvester supports REST APIs published with WADL files
  • OAC UI’s are very lean, user friendly and intuitive. There isn’t as many pages as OER and it is not possible to add assets or create/modify asset types from the console. However I think this is a good thing to keep the complexity low and focused at only cataloging APIs asset types
  • The “MyAPIs” screen I found very cool. Also the ability to rate the APIs and submit comments / reviews I think will be extremely useful is OAC is used for example as an external catalog for “customers of the APIs” to rate and comment on APIs
  • Admin page seems similar to OER however simplified as OAC has no concept of “Projects” (which I think is a very good thing). Also only basic access settings are available unlike OER where you can configure CAS (Custom Access Settings). Also other features such as Compliance Templates are also not available. Again this is not a bad thing as it adds to the point of keeping it simple
  • My recommendation to implementers and companies looking to adopt governance but that don’t yet have mature processes or tools, to start off by using OAC and as their SOA adoption and governance mature then incrementally start planning for a more complex governance implementation using OER 12c
  • All in all, OAC is the killer OER app. Installation process it’s smooth (even patching process is simple), UI is very clean, functionality is “fit for purpose” and harvester is as feature rich as one available in OER.