Article: Telemetry data to Google Analytics 2017-08-18

Collect telemetry data to Google Analytics

In a real life project we had a whole bunch of node instances that needed to send back telemetry data to us for usage statistic purposes. In order to make it as a little burden as possible for the clients we decided to send the data as a small AES encrypted json, sent as an UDP package with an interval.

Nodes to Foopipes to Google Analytics

What we now needed was a way to collect the data and let us aggregate it. There are many great products for this, such as Elasticsearch/Kibana. But why not just Google Analytics? That would be enough for now and wouldn't create any new hosting requirements.

An example telemetry event looked like this:

{
	"Time": "2017-08-18T06:53:05.1554106Z",
	"EventType": 2,
	"NodeId": "a7124fcf-f557-4c9c-b294-558ee29f7eef",
	"Count": 56,
}

One of the goals with Foopipes is to set up pipelines without being required to write code. With the upcoming 2.4 release we could use a couple of existing Addins, and only needed to write a small yml configuration. The service was easily deployed with Docker Cloud, and then we're done!

Ok, I admit it, the addins were not previously written. But they are now and available for you in the public Github addin repository: https://github.com/AreteraAB/Foopipes.Addins

Foopipes.yml:

version: 2
addins:
  - url: https://raw.githubusercontent.com/AreteraAB/Foopipes.Addins/master/Aes/aes.csx
  - url: https://raw.githubusercontent.com/AreteraAB/Foopipes.Addins/master/GoogleAnalytics/GoogleAnalytics.csx

services: 
  udp: 
    type: udplistener
    port: 8080
  
  analytics:
    type: googleAnalytics
    trackingId: UA-xxxxxx-yy

pipelines: 
  - 
    when: 
      - udp
    do: 
      - aes.decryptjson
        key: "${key}"     # environment variables, base 64 strings
        iv: "${iv}"
      
      - log

      - ga.event
        service: analytics
        clientId: "#{NodeId}"
        eventCategory: "telemetry"
        eventAction: "#{EventType}"
        eventValue: "#{Count}"
        userIP: "#{metadata:remoteEndPoint}"

Dockerfile:

FROM aretera/foopipes:next
ADD *.yml /project/
EXPOSE 8080/udp