Connectors Overview

Introduction

Connectors and Gateways are edge computing software applications that run on premise. Each connector reads data from a local data source and sends that data to the associated ThinkIQ platform instance in the cloud via a Gateway. Communication from the Gateway to the cloud is secure. By default, the data is transmitted over https on port 433 so it is not necessary to open any new ports in your firewalls, in most situations.

Each data source will have a specific connector. Connectors are multi-platform and can be installed on Windows or Linux (including Raspberry Pi) unless the data source lacks support for a particular platform. 

Gateways automatically support data buffering. This allows the data coming from connectors to be stored when Internet connectivity is lost, and then forwarded once connectivity is restored. Buffering also ensures compact data transmission, thus reducing the Internet bandwidth needed. The size and duration of the buffer is configurable.

Connectors that connect to data sources with historical data support data back-filling. Back-filling can be used to import large amounts of existing history and to fill any data gaps that may occur if connectivity to the data source is lost. 

Installation and Network Considerations

By default, communication between the ThinkIQ edge, which runs on premise, and the ThinkIQ platform, which runs in the cloud, is driven through OPC UA. Communication is always authenticated and encrypted. Communication uses a wide range of different OPC UA techniques to exchange information models, live data, historical, and transaction data. For this to work all that is necessary from a security perspective on the edge, is to open one outbound port, and it's the same port that is usually used for secure communication between a browser and a website via HTTPS.

It is never necessary to open any inbound port for the ThinkIQ platform.

HTTPS by default runs over port 443, and in most environments that outbound port will already be open. On most modern firewalls you would open port 443 explicitly for HTTPS. On older firewalls, that might simply be TCP. OPC UA runs over the TCP/IP protocol and can use HTTPS, but is far more performant using the TCP protocol over port 51210. 

URL	Protocol	Port (outbound)
yourinstancename.thinkiq.net	https	443
opcua.yourinstancename.thinkiq.net	tcp	51210
meta.iioto.org	https	443

 

For security purposes, we recommend gateways and connectors not be installed on the physical process control network because our protocols are remotable. Rather, we recommend the connector be installed on a network that has access to the Internet and has firewall and other routing protection between it and the process control network.

The basic architecture of ThinkIQ edge software is that there is a service that communicates from on-premise to the ThinkIQ platform in the cloud. This is commonly referred to as the northbound service. This is a software component, called a gateway, that is responsible for all communication between that computer and the ThinkIQ platform. There can be many gateways connected to the ThinkIQ platform. Each gateway can have one or more connectors that communicate to local data sources on the on-premise network. This is referred to as the southbound service. There are a wide range of southbound protocols as described later in this section.

The gateway and connecting infrastructure is multi-platform which means it can run on all current versions and many older versions of Windows. It can also run on almost all distributions and flavors of Linux.

Store and Forward

If communication is lost over the public Internet causing communication to be lost between the gateway and the cloud platform, the system will automatically perform store and forward functionality. When connectivity is resumed the system will automatically backfill and send all buffered data to the ThinkIQ platform.

Hardware Considerations

The hardware required to run the ThinkIQ gateway and connector is somewhat dependent on the number of tags that a system will track and send to the cloud. The minimum configuration recommended is:

• 2 CPU
• 4 GB memory

This configuration should support a system collecting 250,000 tags. This configuration assumes the host computer is dedicated to running the ThinkIQ gateway and connectors, and that other resource intensive services are not running on the same machine.

Controlling What Tag Data is sent

The following mechanisms provide control over which Tag data are sent to the ThinkIQ platform:

• Meta-data about the Tags and/or the Information Model. This includes information such as tag names, descriptions, groupings, range, and engineering units.  By default, all this data is sent to and synchronized with the ThinkIQ platform. This is typically a very small data set and is maintained on an on-change basis consuming negligible bandwidth. The meta data can be restricted through two mechanisms:
  • "blacklist".  Metadata for Tags in this list are not sent. Typically used to hide the existence of sensitive data.
  • "whitelist".  Used to restrict metadata to only a specific set of tags. Not usually used because this greatly complicates the task of building the solution.
• By default, "live data" is not sent to the ThinkIQ platform until the subscription is turned on. This is done on a per-tag basis, typically in these situations:
  • a Tag is mapped to an Attribute in the Model.
  • a Tag is explicitly set to be historized in the ThinkIQ platform.

 Minimizing Outbound Bandwidth

In order to minimize outbound bandwidth, the following approach is used by default: 

• Data is collected on-change.
• Duplicate data is removed.
• Many data changes (mutiple Tags and mutliple samples per Tag) are buffered together and then sent periodically. This has the following impacts:
  • Data fidelity (accuracy and granularity) is preserved. The cost is increased latency.
  • The overhead of chatty realtime protocols is avoided.
  • Information is compressed.

Perceptions about Data Quantities and Cost

It is a common misconception that sensor data is enormous and expensive to transmit. In reality, the sensor data costs are dwarfed by the actual cost of repeated retrieval, data science, and processing algorithms. When cameras are part of the solution, the images and videos generated consume massively more resources than sensor data. To help put this into perspective, here are some actual numbers from an actual implementation:

• Tags in the Historian: 18,583
• Tags on subscription: 1,285
• Years of data stored to date: 8
• Physical size of History on disk: 39Gb (less than 5% of the space used and 0.4% of the available space)

To illustrate the transmission cost: Following a break the Historian was upgraded and moved to a new server. This resulted in the system doing a backfill of nearly seven days of data. Transmission of that back-fill data took 39 seconds, even though the plant does not have a particularly large uplink.

 

Process Historians

Process historians are applications that connect to a wide range of real-time data sources and store the data over long periods of time. They make great initial data sources for any ThinkIQ installation because the ThinkIQ platform can immediately be primed with existing history.  

Specific Historians

Wonderware/AVEVA Historian

A popular tag-based historian, Wonderware Historian, previously known as Wonderware IndustrialSQL Server, is now AVEVA Historian. The AVEVA Historian connector supports most versions of the AVEVA historian. The connector reads the entire tag namespace and loads it into the ThinkIQ Model as Tag Definitions (see whitelist note below). This tag definition load includes all tag properties including look-ups such as Engineering Units and Message Pairs. It does not include the namespace hierarchy.

No historical data will be collected until a tag is flagged as "Historized" in the Model in the ThinkIQ Platform. Flagging can be done either explicitly in the Tag List in the Model Explorer or implicitly whenever a Tag is mapped to an attribute in the Model.

The AVEVA Historian connector can be installed on the same server running the historian (in which case it will login to the historian using Microsoft SQL Server integrated security) or it can run remotely (in which case it connects using Microsoft SQL Server client libraries and database credentials that the user will need to provide).

Back Fill: Supported

Store and Forward: Supported

Multi-OS: yes

OSIsoft PI System

The OSI Historian connector supports most versions of the OSIsoft product.  The connector reads the entire tag namespace and loads it into the ThinkIQ Model as Tag Definitions (see whitelist note below). This tag definition load includes all tag properties including look-ups such as Engineering Units and Message Pairs. It does not include the AF templating system.

No historical data will be collected until a tag is flagged as "Historized" in the Model in the ThinkIQ Platform. Flagging can be done either explicitly in the Tag List in the Model Explorer or implicitly whenever a Tag is mapped to an attribute in the Model.

This connector can be installed on the same server running the historian or it can run remotely (in which case it connects using the Pi SDK and OPC HDA which is an option when installing the SDK that must be selected).

Back Fill: Supported

Store and Forward: Supported

Multi-OS: No

Historians Supporting OPC HDA

Many other historians such as Honeywell PhD and Emerson support the OPC HDA (Historical Data Access) standard. The level of compliance support for the standard varies greatly from vendor to vendor and specific version. Please contact ThinkIQ support for assistance with your particular installation.

Live Data Sources 

Many data sources in industrial environments provide streams of real-time data values. These sources include Programmable Logic Controllers (PLCs), Programmable Automation Controllers (PACs), Distributed Control Systems (DCSs), Smart Instruments, Variable Frequency Drives (VFDs), Sampling Equipment to mention a few.

It is common practice with ThinkIQ to initially connect to a Process Historian and then, later, to bypass the Historian accessing the Live Data sources directly. The advantage of connecting to historians is to provide metadata about tags (such as descriptions, limits and Engineering Units) that may not be available from the Live Data sources and, of course, historians contain history. The disadvantage of historians is that many introduce serious data quality problems (usually due to poor configuration, weak scan rates, lossy compression algorithms and tag bloat due to weak calculation engines). 

No live data will be collected until a tag or variable is flagged as "Historized" in the Model in the ThinkIQ Platform. Note that in this case, "Historized" does not indicate historical data but rather live data. Flagging can be done either explicitly in the Tag List in the Model Editor or implicitly whenever a Tag is mapped to an attribute in the Model.

Specific Live Data Protocols

OPC DA (Data Access)  

Support for OPC DA (also known as OPC Classic) is pervasive in the industrial automation industry. OPC DA is a Windows-specific protocol originally driven by Microsoft using COM and DCOM. This ThinkIQ connector is an OPC DA client which will read and load the Tag List automatically if the source supports OPC DA name space browsing. If the source does not support browsing, the Tag List can be configured in a local text file.

OPC UA (Universal Access - Live Data)  

OPC Unified Architecture (UA) is a standard and protocol driven by the OPC Foundation. Released in 2008, it is a platform independent service-oriented architecture that integrates all the functionality of the individual OPC Classic specifications into one extensible framework. This ThinkIQ connector is an OPC UA client which will read and load the Variable List automatically if the source supports OPC UA Discovery. If the source does not support discovery ,the Tag List can be configured in a local text file.

If the data source supports both OPC DA and OPC UA it is almost always better to use OPC UA.

Follow the common installation instructions to install OPC Live Data connectors on Windows, and review the specific information about OPC UA Live Data sources here. 

OPC Bridges 

If your live data source does not directly support OPC DA or OPC UA, it is very likely that the connection will be made through one of the many OPC Bridges and Aggregators available from vendors such as Matrikon and Kepware.     

Linux Foundation FLEDGE Connectivity

ThinkIQ leverages the Linux Foundation to provide Open Source connectors to a broad range of sources. We also partner with the original creators of FogLAMP (now contributed to FLEDGE) , Dianomic, who provide commercial support and tools to manage many connectors at scale. The advantage of FLEDGE is that, in addition to data collection, it provides the following additional capabilities at the edge:

• Aggregation - combine and organize data
• Transform - filter and transform data
• Buffer - protect data
• Analytics - understand data

Currently any part of the ThinkIQ Model can be exposed through the ThinkIQ OPC UA Server on the cloud instance. 

To use FLEDGE to get data from ThinkIQ back to the edge, there is no need to install a ThinkIQ connector; instead one configures a FLEDGE OPC UA South Service to read from ThinkIQ.

At the time of writing (5/2022) FLEDGE has more than 40 plug-ins including COAP, CSV Files, Modbus, Flir Cameras, CANBUS, MQTT, OPC UA , Azure IoT, Google Cloud IoT Core, Kafka and OMF.

Weather Connector

The weather connector collects weather data for a specific location and inserts it directly into the Model. This connector does not follow the same pattern as "normal" connectors, instead it showcases how to use several core capabilities of the platform, including:

• Libraries to package a solution
• Model to tie data together
• Development Platform to create scripts to collect data from a REST service
• Browser Scripts to create a custom user interface. 

For details about how to use the Weather connector see Weather Connector Setup

Connector Toolkit (Ability to develop custom connectors)

If the options above do not meet the user's needs, ThinkIQ provides a Connector Toolkit that enables developers to create their own connectors that plug into the ThinkIQ Edge Gateways. The toolkit makes it simple to create cross-platform connectors written in the .Net language, C#. 

Tag Whitelists

Most ThinkIQ connectors will read the namespace of the data source and send it to the ThinkIQ platform. This makes it easy to see what is in the data source and to map the desired tags into the Model. If this is not desirable, a whitelist on premise can be provided as a text file. The connector will only send data for those tags.