Implementing Oracle Integration Cloud Service

Understanding Oracle's Integration Cloud Service and how to get the most out of your implementation

Differentiating the ICS Agent Types – Definitive Tip #5

In our book we talked about the difference between the agents offered by ICS, namely the Connection  and Execution agents. Whilst we differentiated the two, we did focus on the connection agent as this is the type we expect to see used in most cases. However the execution agent still suffers from a level of confusion, and it has been helped by being called  ‘ICS on-premises’.

As part of a number of recent conversations the questions and confusion of what the execution agent is and how it works has come up. There is the well known saying ‘a picture is as good as a 1000 words’ which prompted us to develop the diagram below as a power point slide – in its power point form much of the detail is used as an animated build up.

ICS Agent Comparison
Differences

The diagram should give you the differences, but let’s call them out …

ICSAgentComprison

AspectConnectionExecution
Data communicated back to the cloud- Agent Status
- Data Gathered
- Status
Data retrieved from the cloud- Details for status reporting
- Connection configuration
- Details for status reporting
- Everything else is built using the local server
On-Premises Connectivity OptionsAll connectors including File connectivityLimited connectors available - File is not an option
Prerequisites- Java- Java
- Database
Components DeployedManaged Server - Managed Server
- Admin Server for configuration
- creation of integrations

When to use the Execution Agent

The execution agent is intended to help address the following scenarios:

  • the 2 systems being integrated are on-premises, therefore the ‘cost’ of sending the data to the cloud for processing makes little sense,
  • When ICS comes up against security considerations such as data residency restrictions. So processing of data can be achieved in a location considered acceptable.

What to be Aware of when using the execution agent

Whilst the execution agent addresses some challenges, it does introduce additional considerations, particularly …

  • Monitoring is only limited to a heartbeat between the cloud and the agent, so log gathering will need to be implemented on-premises by what ever monitoring tool is being used,
  • with only one managed server, the resilience of the deployment needs to be considered,
  • Need to ensure the deployment is patched and maintained. Otherwise you risk the possibility of the execution agent nolonger even able to heartbeat back to the ICs parent. Plus the cosnideration for remaining within support agreements,
  • An executrion agent can not be connected to a connection agent today.

Selecting which iPaaS Product(s) to use – ICS Definitive Tip #4

In the last few years Oracle’s iPaaS offerings have progressed significantly from one or two solutions such as Mobile Cloud Service to in excess of 35 offerings. This makes it challenging to decide which product(s) to use. The means for selection depends upon a range of factors including:

  • Most suitable licensing model (ICS provides a lower cost of entry than SOA CS as you’re paying by message volumes)
  • Development skills and desire to resilience or invest in people (SOA on-premises to SOA CS is easier than a technology change such as ICS)
  • The amount of control you have in an environment – JCS offers more control than ACCS for example

This is in addition the technical features that the different products offer. In terms of capability, Oracle iPaaS products can overlap, but better than having gaps. Not only is there overlap, you can achieve the same results with one or more products. Obviously, the permutation that provides the least amount of overlap is likely  to provide the best value in terms of investment.  To help see and understand this we have in the past used a visualisation such as the one shown below which helps evaluate these considerations.

Oracle PaaS options overlayed
It is worth saying that this visualisation has been subjectively constructed, and as products evolve over time it is only true for a limited period of time. Never the less it is helpful in reviewing options, and explaining product recommendations.

You can also see the animated view of this presentation version here.

Short Best Practices Presentation for Oracle Partner Event

The book’s authors presented a brief session on the best practices at a packed Oracle Partner event. The following presentation – shows the material that was presented.

Some of these points are expressed in the book already, others are our Definitive Tips and Guides posts.

Applying configuration management and related dev practises – ICS Definitive Guide #3

Good development practises such as  configuration management and naming conventions have been a topic of conversation a number of times recently for us when it comes to ICS. In this post we will look at what configuration management means to ICS and the application of naming standards.

Why Configuration Management?

So why do we need configuration management with a tool such as ICS? The question depends on what stage of the software lifecycle you are, but a few reasons:

  • As a developer there have been occasions where we have tried to optimise something and then decided to reverted back to the last secured state in configuration control.
  • Deleted something and then regretted it, whilst desktops have the concept of paper basket to recover from, not so in ICS.
  • If your providing packages of ICS functionality you may have different released versions – so need to keep master copies of every version available to refer to.
  • Sometimes we get ‘configuration drift’ – and therefore need to have a reference point to compare to. This allows us to determine the drift and if it is appropriate to reset an environment. Whilst this may sound less significant, it is enough of a problem for tools like Puppet to incorporate the ability.
  • Tracking what versions of integrations are running in a development vs production environment.

The following diagram shows the development lifecycle and where these scenarios fit.

Configuration management life cycle

Whilst it is possible to achieve a configuration control by managing assets through the use  of just a filing system; using a repository allows you to track who applied the changes, when and create tags to label significant versions. Not to mention that not many file systems are easy to  share or handle change conflict. With many of the cloud offerings, Oracle will provide Developer Cloud Service which includes a GIT repository.

Whilst neither GIT or SVN are recommended solutions for binary artefacts (more on this here) it is a solution within the Oracle ecosystem.

Artefacts In and Out of ICS

So how do you get the content of ICS to and from a configuration repository? ICS provides the means to export integrations or packages either manually or through scripting of the REST API. For example to retrieve an integration we can use a CURL command like the following:

curl -X GET -u jstein:welcome1 -H “Content-Type:application/json” -o ./TestProject.iar https://integration.us.oraclecloud.com/icsapis/v1/integrations/TEST/01.00.0000/export

As you can see we call the export operation on the TEST integration with a version of 01.00.000 and direct the output to a file called TestProject.iar.  This file can then be checked into GIT.  As explained in the book, .iar files are a form of zip file, so binary in nature.  Being able to use GIT or SVN to perform differences is not really an option directly.  However their are tools which can be used to compare the contents of ZIP files, for example:

Correspondingly the REST API also allows for imports (load into) ICS with commands like:

curl -X PUT -u jstein:welcome1 -H "Content-Type:application/json" \
-F file=@"TestProject.iar;type=application/octet-stream" \
https://integration.us.oraclecloud.com/icsapis/v1/integrations/import

This time we tell ICS to read the file as an octet stream using cURL (as it is binary) – note that we don’t address a specific integration in the target URL. The other thing to be aware of is whether the import is creating or replacing an integration. This is differentiated by the use of PUT (replace) vs POST (create).

Automating a full snapshot

Whilst the two examples show how you can retrieve (export) and upload (import) specific integrations.  You also need to get hold of all the integrations to automate the snapshot of an environment. Necessary as it is so easy to develop changes and forget to snapshot or baseline the integrations.  so you might want to periodically snapshot the entire set of integrations within your ICS environment.

This can be done coupling the REST API that allows us to retrieve a list of integrations, and then the results can be iterated over.  The following API will retrieve that list:

curl -X GET -u jstein:welcome1 -H "Content-Type:application/json" -o integrationList.json https://integration.us.oraclecloud.com/icsapis/v1/integrations

This will provide a json file called integrationList.json over which we then need to retrieve the self elements as they point to the URL of the integration to export. If you look carefully you can also see the lastupdated stamp in each integration description. This could be used to tune the snapshot to just retrieve those integrations that have changed. But be aware, this information maybe reporting in a different timezone to the timezone your script will be comparing with.

Tracking Between Development and Production

With the ability to import and export integrations. There are several options for tracking the versions of integrations between environments. the simplest, but also the most susceptible to error, is simply compare CRC values on the .iar file with you your GIT version. More elegant would be to actually annotate the contents of the .iar file (we would suggest the Description element in the PROJECT-INF/project.xml file). This does create the issue of opening up the .iar to apply the annotation and dependency .iar file structures not changing.

Coding Standards

Industrialization of development goes beyond configuration management, and should consider things like coding standards? The question may beg, why do we need such things when you do not code? It is true ICS does not involve coding in the traditional sense (although the introduction of JavaScript capabilities will start to change that); but coding standards include simple things like naming conventions – and we certainly name things in ICS.

Let’s step back, why do we have coding standards? We use such standards to drive consistency and ease of understanding across a team and over time, this holds true for ICS equally as much as a traditional development platform and even system configuration.

You’ll note through the book we adopted a camel case approach for readability and post fix in the names to indicate the associated chapter. This point is underlined by several posts on the SOAJavaPoint blog suggesting conventions.  The important thing is to agree with everyone involved the standard and comply to the agreed standard.

If you are creating JSON definitions, XML schemas, and SOAP endpoints for your integrations you should consider your guidelines for naming there as well.  We typically followed XML conventions in the book.

Checking on Naming Compliance

With the ability to export the integrations.  Compliance can be done by unpacking the .iar file and then running some basic parsing on the configuration files. With the .iar file unpacked applying XQuery to locate the relevant elements within a configuration file along with a REGEX path and it would be easy to validate whether people are applying the conventions or not. Within Chapter 13 of the book we actually looked at the breakdown of the .iar file.

Using iCal in Scheduling – ICS Definitive Tip #3

In ICS Definitive Guide #2 we looked at the scheduling of integrations and mentioned the possibility of using iCal. In this tip we shall explore advanced iCal facility for defining schedules within ICS.

icalThe iCal format is probably best known for sharing meeting events in tools such as outlook. The format is actually a standard defined by the IETF originally in 1998 (https://www.ietf.org/rfc/rfc2445.txt)  and updated several times the latest being IETF Logo - owner of iCal std 2015 (https://tools.ietf.org/html/rfc7529). The goal of the definition has been for the purpose of driving standardisation across PIM tools (Personal Information Management) tools which includes Outlook and Thunderbird etc. It is a substantial standard the original document ran to 150 pages, but the section supported by the schweduler’s advanced option purely relates to the reoccurence definition covered by section 4.3.10 of the original RFC, and only takes five pages. If you want to know lots about iCal and the varying revisions then you may find https://icalendar.org/ helpful.

It is worth mentioning that today ICS under the hood exploits Oracle’s Enterprise Scheduler Service (ESS). ESS documentation identifies that it works to the 2445 version of the specification (see here). ESS became a 1st class citizen component with the start of the 12c middleware product stack. As a result, if you see any errors processing an iCal configuration the error codes will reference ESS does.

So why suopport for this part of the iCal standard. Well unlike the UI provided, the FREQ definition with all the elements of the definition that can be provided mean some very sophisticated schedules can be defined.  For example you could create a definition that says the scheduler should only trigger 9.15, 9.30, 14.15, 14.30 on Wednesdays for week numbers 7, 8, 21 and 49.

Syntax

For ease of reference, the definition of how a FREQ element should be created is, according for RFC2445:

“FREQ”=freq *(

; either UNTIL or COUNT may appear in a ‘recur’,
; but UNTIL and COUNT MUST NOT occur in the same ‘recur’

( “;” “UNTIL” “=” enddate ) /
( “;” “COUNT” “=” 1*DIGIT ) /

; the rest of these keywords are optional,
; but MUST NOT occur more than once

( “;” “INTERVAL” “=” 1*DIGIT ) /
( “;” “BYSECOND” “=” byseclist ) /
( “;” “BYMINUTE” “=” byminlist ) /
( “;” “BYHOUR” “=” byhrlist ) /
( “;” “BYDAY” “=” bywdaylist ) /
( “;” “BYMONTHDAY” “=” bymodaylist ) /
( “;” “BYYEARDAY” “=” byyrdaylist ) /
( “;” “BYWEEKNO” “=” bywknolist ) /
( “;” “BYMONTH” “=” bymolist ) /
( “;” “BYSETPOS” “=” bysplist ) /
( “;” “WKST” “=” weekday ) /

( “;” x-name “=” text )
)

freq = “SECONDLY” / “MINUTELY” / “HOURLY” / “DAILY”/ “WEEKLY” / “MONTHLY” / “YEARLY”
enddate = date
enddate =/ date-time ;An UTC value
byseclist = seconds / ( seconds *(“,” seconds) )
seconds = 1DIGIT / 2DIGIT ;0 to 59
byminlist = minutes / ( minutes *(“,” minutes) )
minutes = 1DIGIT / 2DIGIT ;0 to 59
byhrlist = hour / ( hour *(“,” hour) )
hour = 1DIGIT / 2DIGIT ;0 to 23
bywdaylist = weekdaynum / ( weekdaynum *(“,” weekdaynum) )
weekdaynum = [([plus] ordwk / minus ordwk)] weekday
plus = “+”
minus = “-”
ordwk = 1DIGIT / 2DIGIT ;1 to 53
weekday = “SU” / “MO” / “TU” / “WE” / “TH” / “FR” / “SA”
;Corresponding to SUNDAY, MONDAY, TUESDAY, WEDNESDAY, THURSDAY,
;FRIDAY, SATURDAY and SUNDAY days of the week.

bymodaylist = monthdaynum / ( monthdaynum *(“,” monthdaynum) )
monthdaynum = ([plus] ordmoday) / (minus ordmoday)
ordmoday = 1DIGIT / 2DIGIT ;1 to 31
byyrdaylist = yeardaynum / ( yeardaynum *(“,” yeardaynum) )
yeardaynum = ([plus] ordyrday) / (minus ordyrday)
ordyrday = 1DIGIT / 2DIGIT / 3DIGIT ;1 to 366
bywknolist = weeknum / ( weeknum *(“,” weeknum) )
weeknum = ([plus] ordwk) / (minus ordwk)
bymolist = monthnum / ( monthnum *(“,” monthnum) )
monthnum = 1DIGIT / 2DIGIT ;1 to 12
bysplist = setposday / ( setposday *(“,” setposday) )
setposday = yeardaynum

Applying the syntax to get complex schedules

With all of that you have if careful all the information to create the most complex schedules you like.  To illustrate here is a validating example:

FREQ=YEARLY;INTERVAL=3;BYMONTH=5;BYDAY=2,7;

This translates to a schedule triggering every 3 years on May 2nd and 7th. As you can see trying to express this using calendars and menus would probably prove to be fairly complex. But pretty simple in text once the notation is understood.
We should address a couple of challenges that exist.

Avoiding the traps

Firstly (the current ICS documentation here) shows the example expression as:

FREQ=MONTHLY;BYMONTHDAY=1,10,15;BYHOUR=5,10,15,20;BYMINUTE=15

However the example omits a key terminating semi colon (essentially each part of the expression needs to be terminated by a semi colon). So, the correct representation is:

FREQ=MONTHLY;BYMONTHDAY=1,10,15;BYHOUR=5,10,15,20;BYMINUTE=15;

Whilst not too serious, an easy mistake to make. Unfortunately today the error messages from ESS back to ICS are rather uninformative with an error message and code of

ESS-09402 Failed to compile recurrence expression

So if you see this error, our advise is to simplify the expression or build it up a piece at a time if you can.
The last issues we have yet to understand relate to the definitions support for extension attributes shown as x-name in the specification. Well to be honest, this is not important as ICS and ESS do not appear to have any extension elements. The slightly more troubling question, which we have yet to get to the bottom of, is the support for COUNT in the expression which does not appear to be accepted.

Using Integration Versioning – ICS Definitive Tip #2

In this Definitive Tip, we are going to look at the ability to use Versioning on Integrations to seamlessly deploy update integrations is an important requirement of any enterprise integration platform.

ICS supports the model of semantic versioning, perhaps better recognised as versioning as Major, Minor and patch.  Which is best described at http://semver.org/.

This post will demo a minor and then a major version change. So we can see how change as the behaviours change between the two. A major change indicates an likely change in definition and minor one reflects logical change.

Seeing Versioning in Action

To demonstrate versioning we need an integration that is currently active. Ideally, you will have some sort of data feed or test framework, such as SoapUI or even a simple cURL script regularly triggering the integration. We need this script running to demonstrate the impact of the cutover.

In our example we’re going to use the service from Definitive Guide #1 (DG1), but given that we are not going to really modify the service itself, this will work with any service you want.

Applying A Minor Change

With everything ready, we can start by cloning or creating a draft of our sample integration. With the clone or draft action we are presented with the Create dialog. In the dialog we can modify the version number at the major or minor levels, for example 1.00.0000 to 1.10.0000. We can complete this step using the Create button.

Version Number changed

With a new integration with same name as an existing integration. With a higher version number we can now make some modifications to the integration. The changes should be easy to spot in the demo, for example adding or changing Tracking ids.

Once the changes have been made, you can save the integration returning to the list of Integrations. The original integration is unimpaired so far, and any active calls are still showing as being handled by the version 1.00.0000 of the integration, as you can see in the next screenshot.

2 Versions of an Integration

Now, we activate the new integration. When the new integration becomes active, the older version will be deactivated. Leaving us with only 1 active integration, and we should see our test harness now triggering the modified (version 1.10) integration without having changed anything.

Major Version change

Unlike minor or patch changes, a major version change usually represents a break in the contract (service interface) and/or a change so radical in the integration it is likely to have sufficient impact on the use of the service to warrant other possible changes.  As you change will most likely demand change from the integration client(s) it becomes necessary to run both services in parrallel for a period to allow a change in usage. As a result you need ICS to support both versions running.

To illustrate the behaviour we can repeat the process of changing the integration that we applied in the minor version number change.  This time taking the version number from 1.10.0000 to 2.0.0000.  With the changes all safely applied, we will see in the integrations view the same integration with version 1.0 inactive, 1.10 active and version 2.0 inactive. Now we should activate version 2.0, rather than 1.10 being deactivated, we will see both active as shown in the next screenshot:

So the question begs, how does the invocation differentiate between the two instances?  If we look at the information for each integration we can see the answer, as we can see below:

V1.10 versioning information

V2 versioning information

As you can see the URL is different, with the URL version number having changed from/v01 to /v02. So if you trigger your test without any change we will see the v1.10 integration execute as before.  Now change the test harnesses URL to reference the v02 rather than v01 and then fire the test call. This time you’ll see your new instance execute.

Summary

As you can see we can transition integration changes without creating downtime on the endpoints. We also have a means to parallel run some integration changes.

Designer Content Filters – ICS Definitive Tip #1

Definitive Tips, like our Definitive Guide posts address features could not be incorporated into the book, either simply because the feature is new or simply down to the fact the new feature arrived since the book has been published. Unlike the Definitive Guide entries, the tips will always remain very short. The first tip will address content filters in the designer.

Designer Content Filters

Release 16.4.5 introduced a new filter feature to help improve the location of Integrations, Connections and other designer elements. If you visit the Integrations  parts of the designer for example, you will now see  additional options included in the banner as highlighted in the following screenshot:

Some of the designer views only have one filter option, although most have two. The following tables show the different filters available in each part of the Designer.

Integrations
Connection Filters  Lookup Filters

Agent Filters Package Filters

Adapter Filters

Scheduling Orchestration Integrations – ICS Definitive Guide #2

This is the second article in the Definitive Guide series. This time we take a look at the ability to Schedule an Orchestration integration within ICS has arrived with release 16.4.5 release of ICS,   To demonstrate this, we are going to reuse our FlightSitRep endpoint from the book’s Chapter 3. If you have not got the book yet – it’s a WSDL one way integration that is connected to Mockable.io.

Creating Scheduled Orchestration

From the Integrations screen we need to click on the New Integration button, and select the Orchestration pattern in the pop-up dialogue. The first thing we will see when creating an integration is the Create New Integration dialog, as shown here:

create new orchestration wizard

As you can see the dialog has a new toggle option which defaults to represent the common approach to Orchestration, that being event triggered through an event or object defined by a connection.

Read More

For Each activity for processing batches – ICS Definitive Guide #1

This is the first blog in our ICS Definitive Guide series. In this series we pick one feature that ICS offers and write the most detailed guide on it. This blog introduces the capability of processing batches using the For Each activity in orchestration type integrations.

These blogs follow the same writing style as the book. We sometimes use the same resources as we used in the book, which can be downloaded from the website of Packt. For this blog we are using the same WSDL as we used in chapter 5, Going Social with Twitter and Google, and as in chapter 10, Advanced orchestration with branching and asynchronous flows, but with a little change to one element.

The integration we are going to build simulates a notification send from our Airport Flight Schedules system with a list of persons that are missing / late for their flight. For each flight a list of persons are included in the notification. For each missing person a task is added to a Trello Board, used by the ground personal, to remove the passengers luggage from airplane.

Read More

iPaaS and Cloud Security

Oracle iPaaS SecurityWithin the Oracle iPaaS products you will encounter features that exist to help provide security or mitigate the risks of someone misusing Oracle cloud accidentally (or otherwise). These features look to mitigate security issues such as a Denial of Service (DoS) attack (e.g. flooding a web service with more calls than it can handle preventing genuine service calls for being serviced).

We thought it might be useful to pull some articles together that engage and talk about the challenges that can exist:

Page 1 of 2

Powered by WordPress & Theme by Anders Norén