Integration – what technology when?

Integration is a key element of many of the projects that our customers choose to build on the WEM platform. WEM offers a large variety of integration options, all available in every project and most easily setup, even by the most novice user. This blog highlights some of the integration standards available in WEM and when and where you may use them. For details on the how of using these technologies I invite you to read the blogs on that specific technology or attend our integration training listed on our training schedules.

SOAP

SOAP “Simple Object Access Protocol” is a highly standardized and self-documenting messaging framework, where everything you can do is strictly defined in a WSDL (“Web Service Description/Definition Language”) document. SOAP is a very rigid integration standard, which makes it especially easy to use for novice or non-technical users in WEM, because both systems integrating with each other have a pre-defined way of communicating and WEM can hide the really complex technical stuff. When using SOAP, the data exchanged is always in the XML format, but you’ll never know, as WEM does all the translation from and to XML into the familiar WEM datamodels. WEM can do both, Expose a Service (provide a service to let others connect to your WEM Project) and Consume a Service (to let you connect your Project to other external systems). Exposing and Consuming SOAP Web Services with a minimum of technical skills - that is typical for WEM. Integration between WEM Projects is possible and even fairly easy, using the SOAP standard. You use this option when integrating projects where you don’t want to just expose all data to the other projects, but rather need specific parts of data or specific modifications/preparations based on who is asking.

REST

REST “Representational State Transfer” is not so much a standard for integration but more of a standard architecture for creating web based API’s (“Application Programming Interface”). While that makes REST very flexible, it also means you need more technical know-how to integrate your WEM Project with other systems. The flexibility of REST also means you can choose your own data formats. Most often, JSON is used, but XML, CSV and others are also possible and supported by WEM. WEM can do both, Expose (provide a service to let others connect to your WEM Project) and Consume (to let you connect your Project to other external systems) REST Web Services, but both require some technical skills and knowledge of data structures like JSON. To aid in consuming the REST services exposed by your WEM project, the WEM Modeler generates Documentation with examples in both JavaScript and TypeScript with all the specifics of your exposed services. The REST features in WEM are best used when integrating with an external application that already offers a REST based API.

OData

The OData standard is in many ways the most flexible and easiest integration option for WEM, you can expose all your data from your WEM project and have any other application supporting the OData standard interact with that data, where reading, updating and writing are all possible. WEM can also consume OData exposed services. These capabilities are offered as a standard feature on many (if not all) applications from Microsoft, SalesForce, PowerBI, Oracle and many others. To use the OData integration option you don’t need much technical knowledge, but there are limitations around security and data integrity that you should consider, as well as possible limitations on filtering options or large data transfers. Some of the many interesting uses of the OData standard is integrating your WEM project with Microsoft O365, CRM systems or other administrative applications.

HTTP(s)

HTTP “Hypertext Transfer Protocol” is what most people call a website, plain and simple it’s the World Wide Web Browser way of requesting data and receiving responses. Using WEM you can easily just execute a call to any website and retrieve the information that would normally be shown on the screen of your web browser and process that information in your WEM project. While this is not the nicest way of integrating, especially from a technical perspective, sometimes applications offer a nice human readable web interface but no fancy API, or maybe the API comes at an additional cost. The raw HTTPS (please, use the secure version with the “S” at the end!) is your solution of last resort when integration with an application or website that doesn’t offer integration with SOAP, REST or OData formats.

File Import/Export

Using WEM you can import and export a host of different file formats. Some of these, like PDF, are mostly read-only, others like pictures can be edited by using some of the more advanced Widget features WEM offers and yet others are fully machine readable. WEM supports many different formats like CSV, Excel, XML and JSON using the available import and export nodes; as well as the ability to generate documents as Word (docx) or PDF using the specific Process Nodes. The format you use is mostly dictated by the data you want to import or export and the available formats in the applications you want to integrate with. Sending an invoice to a customer? Show the invoice in the body of the email as HTML and attach a copy in PDF. Want to do statistical analyses and your statistics application doesn’t support OData or other integrations, use CSV (or Excel, XML or JSON depending on what they do support). Want to import a large list from your previous administration built in Excel, use XLSX or CSV. The file integration is most often used when batch processing with human intervention is required, this is not a commonly used integration for application to application interaction.

Cheat sheet

Consuming a JSON webservice

This article explains how to consume a JSON webservice from within your WEM project. At the end of this article we will have built a simple application that uses the Open Movie Database API to search for a movie and show the details. You can already see the final application working at https://omdb.live.wem.io.

This part of this article explores a new feature in WEM that allows you to perform any HTTP web request. The second part will focus on JSON messages and how to handle them in WEM.

Introducing a new flowchart node

HTTP (Hypertext Transfer Protocol) is the protocol that is used to communicate with web servers. Your browser sends an HTTP request to a web server any time you request a new web page, while webservers use HTTP to communicate with each other through webservices.

We recently added a new node type to the flowchart editor that can perform any HTTP web request, and conveniently called it the “web request node“. When you drag this node in your flowchart, you can specify all the aspects that make up an HTTP request message:

• The url (this includes the scheme, which is either http or https).

• The request method (GET and POST are the most common methods, but WEM supports all common methods).

• Request headers – The header fields Accept, Connection and Content-Type are predefined. You can specify any number of additional headers.

• Query fields – You can manually put the queryfields in the url yourself, but I recommend to specify them separately. This way WEM will take care of proper url encoding.

• The request body, if applicable. This can either be plain text (for instance, json or xml) or a document such as an image or a pdf file.

When it comes to security, WEM supports basic authentication and client certificates out of the box. Other authentication schemes are often possible through custom HTTP headers.

The web request node also lets you specify how to handle the web response that you get back from the server. You can map each response header to a data field, and the response body can be stored in either a text field or a file field. The node has an error exit that is followed when the server is not available or when the server responds with a 50x Server Error. You can also add exits to handle specific server responses, such as 404 Not Found or 401 Unauthorized.

Using the Open Movie Database API

Searching the Open Movie Database is very easy. It consists of a simple GET request to https://www.omdbapi.com/with some query fields:

• s – The movie title to search for.

• y – The year of release (_optional_).

• r – The message format of the response (json or xml).

There are more optional query fields, but these are the three that are used in the demo application. You can even try the API in your browser, and inspect the JSON response:

http://www.omdbapi.com?s=Matrix&y=1999&r=json

For our example I created a simple flowchart:

The first node contains a simple form where the user can enter (a part of) the title and year of the movie. I created two data fields named [Search.Title] and [Search.Year] for this form.

The second node is the new web request node. It performs a GET request to the url "http://www.omdbapi.com/". No authentication is needed. The response body is stored in an other data field [Response.Json]. This is a plain text field. The following three query fields are added for this request:

Note that the query field r contains the literal text "json". This instructs this particular webservice to send the response in json format.

The last node in the flowchart is an overlay that shows the JSON message. When I searched for “The Godfather”, I received the following json response:

Importing aJSON message into a data list

The first part of this article described how to use the web request node to send an HTTP web request to a json webservice. The response was a JSON message. This part shows how to import the JSON message into WEM data fields.

WEM has data import and export nodes, a convenient way to import or export excel or CSV files. We recently added the option to import JSON data.

Importing JSON data is different from importing tabular data.

What is a JSON message?

JSON stands for JavasScript Object Notation. It describes data in a text-based and human-readable format. JSON is often used in conjunction with REST APIs and web-based services. It is more powerfull than for example CSV, because it isn’t restricted to tabular data.

The following example is a JSON message that contains information about a person “John Smith”:

The basic construct for JSON messages is the so called object. An object is a list of “key/value” pairs surrounded by curly brackets ({}). The message itself is an object. In the example above, the line "firstName": "John" is a key/value pair. The key in this case is firstName, and the value is "John".

Key is always surrounded by quotes. The value can be any of the following type:

• A string (e.g. text). For example, the value "John" is a string. Strings are always surrounded by quotes.

• A number. In the example above, the age and height are numbers. Numbers are not surrounded by quotes.

• A boolean (e.g. true or false). In the example, a boolean is used for the key "isMember".

• The value null can be used when the actual value is unknown or missing.

• A value can again be an object. This allows for nesting of objects. In the example above, the address is an object that contains the keys "street", "postcode" and "place".

• A value can also be a list (called an _array_) of values. This list is surrounded by square brackets ([]). In the example above "phoneNumbers" is a list of objects.

An important aspect is that JSON messages can be deeply nested. It isn’t unusual to see a message that contains a list of objects, where each object again contains a list of objects, etc…

Importing a JSON message

In part one of this blog we used the Open Movie Database API. This API contains a method to search for movies, given a title and optionally a year. The following query searches for the Jurassic Park movies: http://www.omdbapi.com/?s=Jurassic+Park&r=json

We will now import this JSON response using the data import node. A new drop down is added to the data import node, that allows you to choose the file format. It this case we choose JSON. Next, we have to specify the data source. This sould be the data field that was used by the web request node to store the response body.

If we click Edit Mappings, an overlay appears that allows you to describe the structure of the JSON message, and map the fields and arrays to data fields in WEM. For CSV, you could specify a flat list of columns. JSON has a nested structure however, so you define this structure by building a tree. This tree consists of objects, arrays, text fields, number fields or “true/false” fields, which corresponds to the different JSON value types described above. The tree that describes the response of the OMDB query looks as follows:

Now, instead of building this tree structure manually, I used the feature “Create mapping from example”. This allows you to copy & paste the JSON result and let WEM infer the JSON structure based on this result. This saves a lot of time, and you can always refine the structure if WEM wasn’t able to infer everything from one example.

The next step is to map the the JSON fields to data fields in WEM. For this, I created the following data model in WEM:

For the next step, we have to map the JSON fields to the corresponding WEM data fields. You do this simply by clicking on the field, and choosing the corresponding data field by clicking on the “Map to” button. After all the relevant fields are mapped, the tree should look like this:

Now that we have imported the data, we can simply build a page that shows the results. In this example, we used a repeater and a panel to list all the movies:

Single Sign-on

This article gives two examples on how to use the authentication node to realize single sign-on functionality. For these examples we have made the assumption that an authentication provider (based on Microsoft Office 365) had been created.

Example 1 – a simple way of using single sign-on

We’ll start with a simple way of using the authentication node to authenticate and authorize users to access your WEM application. In a very basic way. You only need a few steps:

• Open the Home flowchart that is always available

• Now add an authentication node to the flowchart. You are presented with a popup to select the authentication provider you want to use. If you don’t already have a provider, you can read here how to do that. Select the provider and click on Ok.

• You can now change the properties of the authentication node you just added. Check whether the property Actionis set to SignOn. Because this is the action we want the node to execute.

• Finally define in your flowchart what should happen when the user is authenticated. E.g. start a flowchart that takes the user to the actual application. When you draw an arrow between the authentication node and the nedt node (e.g. a subflowchart), the exit that should be used is the authenticated exit: the user is authenticated and can now access the rest of the application.

You now have something like this:

Example 2 – Using attributes from your authentication provider

The previous example was the most basic way to use the authenticated node. You can do a lot more with it and we’ll explore that in this example. We will expand the previous flowchart and add an overlay that displays some information, like your email address, whenever you log in.

Step 1: add fields to store information

We will start with creating a coupld of data fields we need:

• Create a new folder in the Data model, and call it ‘Single sign-on’

• Add three text fields:

  • Email – this will store the email address of the person logging in

  • Display name- this will store the Display name of the person loggin in

  • Response message – the store the message the authentication node returns when logging in

So now we have:

Step 2: create the flowchart

The next step is to create the flowchart that is used to log in the user. Create a new flowchart called sso that we will use for this example.

• Open the sso flowchart. There is only the Start node.

• Add three Assignment nodes and use the to clear the values of the fields we just added.

• Add an Authentication node in the same way as in the previous example. So select your authentication provider and make sure the Action is set to SignOn

• Connect the nodes. You now have a flowchar that looks like:

Step 3: add an overlay to display information

The idea of this example is to show that there is more information that you can get out of the authentication process. To show this, we will add an interaction node to our flowchart and add a few labels to display that information:

• Add the interaction node. Name it ‘Authenticated’ and make it an overlay

• Add three labels to the overlayand give the the following values:

  • “Welcome “+ select the Display name field you created earlier

  • “Your email address: “+ select the Email field you create earlier

  • “Response message from Azure AD: “+ newline + select the Response message field tou created earlier:

And the interaction node looks like this:

Step 4: complete the flowchart

Now that we have created the overlay to display the information, we need to complete the flowchart:

• Connect the Authentication node with the Interaction node, and select the Authenticated exit.

• Create to exits for the interaction node

  • ‘Sign out’, and add this button to the button bar. When the user clicks this butten, the user will log out

  • ‘Continue’, and add this button to the button bar as well. When the user clicks this button, we will continue with the application.

• Add a second Authentication node

  • Select the same authentication provider as the first Athentication node

  • Set the Action value to Logout. When this node is reached, the user will now be logged out

  • Connect the interaction node with the second Authentication node and use the ‘Sign out’ exit to connect the two

  • Add an End node and connect the Authentication node, using the Not authenticated exit.

• Add a Enter subflowchart node and select the flowchart that needs to be started after the user has logged on (you can create a very simple flowchart for this purpose). Connect the interaction node with this Enter subflowchart node and use the ‘Continue’ exit to make the connection

Your flowchart should now look something like this:

Step 5: create navigation points

Since you created a new flowchart for this example, we need to make the flowchart is executed by the application. This can be done by calling this flowchart from e.g. the default Home flowchart, but in this case we’ll add a menu option to sign in:

• Open Navigation in the project tree and click on on Main menu. You now see the various main menu navigation points that exist

• Click on New navigation point to create a new menu option (a navigation point)

  • Name it ‘Sign in’

  • The Action is a Deeplink

  • Select the sso flowchart

You now have a menu option available to sign in, using your companies authentication provider.

Step 6: configure the authentication provider settings

To mke it all work, we need to configure the settings for the authentication provider. This is needed to make sure all the additional information is actually provider by the authentication service. There are two ways of make the information available:

1. Using response handlers

2. Adding/managing attributes

First, we will use the response headers:

• In the Project navigation tree, open Authentication providers and click on your authentication provider. You will no see the settings page of this provider. On the right there is the column Response handlers:

• In the Response Handlers column, change the Map response message to value to point to the Response message field you creater earlier

Next, we add attributes for the Display name and the Email address that we want to show in the overlay window when a user signs in:

• In the toolbar, click on Manage attributes. This shows the form to manage attributes that are used by this authentication provider

• Click on Add to add a new attribute. Fill in the form with the following values:

  • Friendly name: ‘Display name’

  • Name: ‘http://schemas.microsoft.com/identity/claims/displayname’

  • Name format: ‘TXT’

  • Data field: select the Display name field you created earlier. This attribute is now connected to the data field.

• Save this attribute, and Add another:

  • Friendly name: ‘Email’

  • Name: ‘http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name’

  • Name format: ‘TXT’

  • Data field: select the Email field you created earlier. This attribute is now connected to the data field.

• Save this attribute as well

You now have two attributes defined that you actually use in the interaction:

Step 7: preview

You have everything in place and can now preview your example. When you log in, or when you were already logged into your company network, you would get a form with your name, email address and the actual Azure AD response message.

Office365 offers many services that we can integrate with. E.g.:

1. Outlook

2. Calendar

3. OneDrive

We’ve created a template project where you can begin using the Office365 API. In this guide we’ll show you how to set it up.

The OAuth 2.0 authentication protocol

To be able to safely make use of external providers there’s always some authentication procedure involved. The WEM Modeler already supports the two main standard industry protocols, SAML 2.0 and OAuth 2.0.

In this guide we’ll be using the OAuth 2.0 protocol. Below is a little bit of technical protocol information that helps you understand the actual integration. The OAuth 2.0 protocol requires 5 pieces of information to work properly:

1. An Authorization Endpoint, a link where we’ll redirect to when we start the authentication procedure.

2. A Token Endpoint, a link which will give us an access token after we’ve successfully validated ourselves with the ‘Authorization Endpoint’.

3. A Client ID, this will validate our application to the external service provider.

4. A Client Secret, this will also validate our application to the external service provider.

5. Redirect URLs, after successfully logging into the external service we’ll be redirected to these URLs.

After retrieving the access token we’ll be able to use the external service provider’s API.

(An API, application programming interface, is like a toolbox that grants us access to a bunch of useful functions. For example, Outlook’s API allows you to retrieve all your unread mails.)

Step 1 – Copy the example project

Step 2 – Setting up the project (optional)

Let’s say we end up working on our project for a long time. Then it would be nice if we could publish the project right? To be able to do this we first need to link a domain address with our project.

Go to the ‘Project settings’ and configure the hostnames for the staging and live environment. (If the hostname isn’t taken, you can generally use the name of your project and the postfix “.staging.wem.io” or “.live.wem.io” as hostnames.) And with that our hostnames are set up. Don’t forget to save the changes you’ve made!

Step 3 – Getting your redirect URLs

Click on the authentication provider Office 365.

Copy and write all the Redirect URLs down somewhere. The redirect URLs in our case are:

Step 4 – Creating your Microsoft Azure App

Click on Azure Active Directory and after that click on App registrations.

Click on the button New application registration and fill in the form that pops up.

In the text field Name you can enter any name you prefer. In the field Application type you need to select the option Web app / API. In the field Sign-on URL you can enter any URL you prefer, it doesn’t matter which URL you enter.

In the next screen you can see some of the settings of your application. Write down the ‘Application ID’, that will be your client id. After that’s done click on the Settings button.

Step 5 – Entering the Redirect URLs

Afterwards click on the Reply URLs button.

Enter the Redirect URLs you’ve written down before, and press the Save button afterwards.

Step 6 – Creating a client secret

Click on the ‘Keys’ button.

Enter a new password. Write in the Description field any name you prefer, we choose Client Secret. Select in the Expires field the option Never expires.

After that’s done click on the Save button. The actual key will now appear in the Value field.

Write the key down somewhere, that is our client secret.

Step 7 – Selecting the permissions of the app

Click on the required permissions button.

Click on the Add button.

Select the Microsoft Graph API.

Select all the permissions, this way we aren’t restricted in the way we’ll use the app.

After that’s done, do not forget to click the Grant Permissions button.

Step 8 – Get the OAuth2.0 endpoints

Go to the App registrations overview.

Click on the Endpoints button.

Write down the OAuth 2.0 endpoints somewhere.

For our purposes we need to change the endpoints a little bit, since we’ll be using the 2.0 version of the API. In both URLs add after ‘oauth2/’ the text ‘v2.0/’. The URLs will now look like this:

Step 9 – Setting up the authentication provider

Click on the ‘Office 365’ Authentication provider.

Write your Client ID and Client Secret in these fields. You’ve written them down in step 4 and step 6.

Write down the appropriate URLs in the ‘Authorization Endpoint’ field and the ‘Token Endpoint’ field. You’ve written them down in step 8. Under the ‘Scope’ field you need to write down ‘mail.read’, what you write down in this field is not that important as long as we do not leave it empty. And with that your authentication provider is done!

Step 10 – Trying out the template project

The template project can now be used.

Click on the highlighted ‘Preview’ button to try out the template project! It’s recommended to look through all the flowcharts in the project to see how it all works.