Foopipes - Fetch, transform and push [foo] between services

Poll restful services using nothing but yaml configuration.

Data flows are set up with yaml configuration files or as Javascript modules.


plugins:
  - Elasticsearch

services:
  poller: 
    type: scheduler
    interval: "00:01:00"

  elasticdb:
    type: elasticsearch 
    url: "http://localhost:9200/"

pipelines: 
  -
    when: 
      - poller
    from: 
      - http: "http://myservice/api/content_as_json_or_xml"
    do: 
      - map: "$.entries[*]"    # json path expression
    to: 
      - store: elasticdb
        index: myindex
        dataType: entry
        key: "#{entryId}"      # data binding expression
    then:
      - publish: sync_done

Process the data with C# scripts

You can pretty much do whatever you want with the data with your own or community created C# Roslyn scripts.


addins:
  - url: "https://raw.githubusercontent.com/AreteraAB/Foopipes.Addins/master/Mailgun/mailgun.csx"

pipelines: 
  -
    when: 
      - queue: entry_submitted
    do: 
      - script: |
          Json((context, json, ct) => {
              json.Data["firstname"] = json.Data["fullname"].Split(' ').First();
              return json;
          });
    to:
      - mailgun.send
        to: "#{email}"
        subject: "Thank you, #{firstname}!"
        text: Thank you for your submission!

Process the data with Node.js modules.

Data processing can also be performed with your own or third party Javascript modules running in Node.js.


exports.somefunc = function(data, context) 
{
    data.someValue = data.someOtherValue.toLowerCase();
    return data;
}

Listen and react to the outside world.

You can easily create APIs for your consumers.


services: 
  postapi:
    type: httplistener
    path: mypath

  searchapi:
    type: httplistener
    path: search/{searchstring}
    responseBody: json

pipelines: 
  -
    when: 
      - postapi
    do: 
      - map: "$.body"
    to: 
      - store: elasticdb
        index: myindex
        dataType: entry
        key: "#{entryId}"
  -
    when: 
      - searchapi
    do: 
      - search: elasticdb
        index: myindex
        dataType: entry
        value: "#{route.searchstring}"
      - map: "$.hits[*]"

Set up asynchronous event chains.

Foopipes is built with an event driven, message based architecture with the possibility to use both internal and external message queues.


pipelines: 
  -
    when: 
      - queue: sync_done
    do:
      - http: "http://myserver/api/flushcache"
    then: 
      - publish: cache_flushed

Asynchronous task execution.


pipelines: 
  -
    when: 
      - webhook
    then:
      - publish: source_updated 
  -
    when: 
      - queue: source_updated
    do:
      - command: "bash"
        args: "build.sh"
    then: 
      - publish: build_completed

Use plugins, or build your own in .NET core.

The functionality of Foopipes can be extended with third party plugins written in .NET Core.


plugins:
  - Elasticsearch
  - RavenDB
  - RabbitMQ
  - Me.MyPlugin

Get Started!

Foopipes runs in Docker. Stand alone installations can be arranged for Linux, Mac or Windows.

Install Docker, open a terminal and copy/paste:

Linux/Bash


# Create default config file
docker run aretera/foopipes config >foopipes.yml

# Start and mount current directory 
docker run -v $(pwd):/project aretera/foopipes

Windows


docker run aretera/foopipes config >foopipes.yml
docker run -v %CD%:/project aretera/foopipes

Want more examples?

Check out the examples in the documentation.

#{Foopipes}

Fetch, transform and push [foo] between services.