Process large amounts of Elasticsearch data using TIBCO ActiveMatrix BusinessWorks 5

Sometimes you want to create a longitudinal study of patterns in your Elasticsearch data and you want to analyze the entire event stream matching your criteria. The scroll API provides a mechanism for asking Elasticsearch for every last entry matching a query and then to get the results back in chunks which sequentially represent the entire set of matching records.

New blog design!

Trying to keep it fresh. My last redesign was from like ages ago. I liked the minimalistic feel but it missed some juice and felt outdated. So I decided to search for a responsive Blogger template and found one.

GQueues Task Template for your Weekly Review Checklist (GTD)

This is my Weekly Review Checklist task template which I run (almost) every week using the powerful "Quick Add Window" capability of my task manager GQueues:

Weekly Review today
- GET CLEAR
-- Collect Loose Papers and Materials :: Gather all accumulated business cards, receipts, and miscellaneous paper-based materials into your in-tray.
-- Get “IN” to zero :: Process completely all outstanding paper materials, meeting notes, voicemails, dictation, PC Desktop, email (Gmail, Pnl, Rbx), Google Keep, GoogleDrive, Rabobank, Yammer, LinkedIN, Facebook, Pocket, Feedly
-- Empty Your Head :: Put in writing and process any uncaptured new projects, action items, waiting-for’s, someday maybe’s, etc.
- GET CURRENT
-- Review Action Lists :: Mark off completed actions. Review for reminders of further action steps to record.
-- Review Previous Calendar Data :: Review past calendar in detail for remaining action items, reference data, etc., and transfer into the active system.
-- Review Upcoming Calendar :: Review upcoming calendar events - long and short term. Capture actions triggered.
-- Review Waiting-For List :: Record appropriate actions for any needed follow-up. Check off received ones. https://www.gqueues.com/main/tag/active/%23waitingfor
-- Review Project (and Larger Outcome) Lists :: Evaluate status of projects, goals and outcomes, one by one, ensuring at least one current action item on each. Check mindmaps "Balancing Work and Life".
-- Review Any Relevant Checklists :: Use as a trigger for any new actions.
-GET CREATIVE
-- Review Someday/Maybe List :: Review for any projects which may now have become active, and transfer to “Projects.” Delete items no longer of interest. Pocket and Inbox-Saved articles.
-- Be Creative & Courageous :: Any new, wonderful, hare-brained, creative, thought-provoking, risk-taking ideas to add into your system??? https://facilethings.com/blog/en/basics-natural-planning

Storing XML documents in ElasticSearch using TIBCO BusinessWorks 5x

When you need a fast and simple solution to store your XML documents, ElasticSearch is there to help. Elasticsearch is a distributed, open source search and analytics engine, designed for horizontal scalability, reliability, and easy management. It combines the speed of search with the power of analytics via a sophisticated, developer-friendly query language covering structured, unstructured, and time-series data. For more info, check out the reference documentation.

Below you'll find a image of a TIBCO BusinessWorks 5 process definition that shows how easy it is to set it up using the "Send HTTP Request" activity:

Some pointers on the process definition:

• When you want to log your XML documents in a time-based manner, you need to make sure you use a certain name pattern for your index like: logs-[date]. Reference: https://www.elastic.co/guide/en/elasticsearch/guide/current/time-based.html
• We're using Kibana to visualize the XML data in a web browser. We needed to configure a dateTime formatted field in Kibana. This is the reason why we added field "post_date" in the JSON payload.
• On my dev machine, I needed to remove unwanted control chars from the XML or ElasticSearch would return an error. In the Java Code activity, we're using this small snippet: out_var_1 = in_var_1.replaceAll("\\p{Cc}", "");

Thanks for reading and stay tuned for more Elastic usage within our systems integration endeavours!

WebDav connectie maken naar TransIP STACK met de Cyberduck client

Na wat uitzoekwerk, is het me gelukt om een connectie te maken naar STACK via het WebDav protocol. Zie onder een screenshot van mijn Cyberduck config scherm (mijn username heb ik even weggemoffeld).

Howto solve SoapUI scaling issues on a high DPI screen of a Windows 10 machine

I was fed up with SoapUI high dpi issues with my Dell XPS15 on Windows 10.

This trick is what I picked up from the SmartBear Community:

First add a registry key

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\SideBySide\PreferExternalManifest (DWORD) to 1

Then add a manifest file next to your exe
SoapUI-5.2.1.exe.manifest

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0" xmlns:asmv3="urn:schemas-microsoft-com:asm.v3">
    <description>eclipse</description>
    <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
        <security>
            <requestedPrivileges>
                <requestedExecutionLevel xmlns:ms_asmv3="urn:schemas-microsoft-com:asm.v3"
                               level="asInvoker"
                               ms_asmv3:uiAccess="false">
                </requestedExecutionLevel>
            </requestedPrivileges>
        </security>
    </trustInfo>
    <asmv3:application>
        <asmv3:windowsSettings xmlns="http://schemas.microsoft.com/SMI/2005/WindowsSettings">
            <ms_windowsSettings:dpiAware xmlns:ms_windowsSettings="http://schemas.microsoft.com/SMI/2005/WindowsSettings">false</ms_windowsSettings:dpiAware>
        </asmv3:windowsSettings>
    </asmv3:application>
</assembly>

Invoking a OAuth2 REST API with the TIBCO BusinessWorks 5.x plug-in for REST and JSON

In this post I will show you some pointers on how to invoke a OAuth2.0 REST service with the TIBCO ActiveMatrix BusinessWorks plug-in for REST and JSON plugged into TIBCOActiveMatrix BusinessWorks™. OAuth2.0 is an authorization framework that enables applications to obtain limited access to user accounts on an HTTP service, such as Facebook, GitHub, Google, Twitter and Trakt.tv. It works by delegating user authentication to the service that hosts the user account, and authorizing third-party applications to access the user account.

For this demo I'm using the Trakt.tv add-to-watchlist API. Trakt.tv is a website that collects lots of interesting information about what TV shows and movies that everyone is watching by scrobbling media players or by manual input from its users. They've exposed their movie watchlist capabilities with their OAuth secured API.

General process definition

The REST and JSON plug-in allows you to invoke RESTful Web Service APIs and translate data between JSON format and XML format (and vice versa) very easily. In this demo, the source of the movie watchlist data comes from my IMDB watchlist which I exported to a CSV file using the export web tool.

The simple BW Process Definition flow:

• Parse Data - Use a Data Format shared resource to be able to parse the data from the file. 
• Map Data - Use this to prepare for JSON rendering. You can also use a XSD generated from a sample JSON tekst using the "Generate XML schema from JSON" tool.
• Render JSON - Render the XML output from Map Data to JSON format.
• Invoke REST API - invoke the Trakt.tv API using the JSON data as the payload.
  

JSON conversion

The XML output from Map Data will be converted to JSON using the the Render JSON activity. Looking at the Input Editor config, the chosen cardinality of movies and shows might look odd but this is the only way to get a proper JSON rendering.

The output of Render JSON will be like in below image. You can see three movies and two TV series (shows).

Invoke REST API using OAuth2.0

Obtaining the Access Token

This part can be a bit tricky. You can for example write your own oauth client program using libraries from Google or some other open source initiative. Or you can follow the manual approach below:

1. First, create a client_id and client_secret by registering a Trakt.tv app. Click here for some helpful links to get you started.
2. Grab the code parameter value which you need to obtain from your Trakt.tv app redirect url querystring. You will use this in step 3.
3. Open SoapUI (by SmartBear) and use the Access Token Retrieval tool. Fill in the obtained code from step 2 in the Client Identification field. The Redirect URI could be the same as what you have used in Step 1. For OAuth 2 Flow, choose Implicit Grant as we don't need web browser interaction. After you have clicked Get Access Token button, a access_token should be presented to you.

Configure the Invoke REST API activity

Make sure to leave Protocol to None. Add the Resourc URI which you can get from the Trakt API page. 

Add two header fields: trakt-api-version and trakt-api-key.

Finalize the input mapping. The OAuth2TokenValue is the Access Token we just obtained.

Final

After running the process, the Invoke REST API activity should output something like this:

Go to your Trakt.tv Watchlist page (http://trakt.tv/users/[username]/watchlist) to see the result:

Happy watching!