#{Foopipes}

Fetch, transform and push [foo] between services.

Foopipes is a lightweight pipeline engine for asynchronous pipelining of json and other data formats between services.

Poll restful services using nothing but yaml configuration.

Data flows are set up with yaml configuration files or as Javascript modules.


plugins:
  - Elasticsearch

services:
  poller: 
    type: scheduler
    interval: "00:01:00"

  elasticdb:
    type: elasticsearch 
    url: "http://localhost:9200/"

pipelines: 
  -
    when: 
      - poller
    from: 
      - http: "http://myservice/api/content_as_json_or_xml"
    do: 
      - map: "$.entries[*]"    # json path expression
    to: 
      - store: elasticdb
        index: myindex
        dataType: entry
        key: "#{entryId}"      # data binding expression
    then:
      - publish: sync_done
                

Process the data with C# scripts

You can pretty much do whatever you want with the data with your own or community created C# Roslyn scripts.


addins:
  - url: "https://raw.githubusercontent.com/AreteraAB/Foopipes.Addins/master/Mailgun/mailgun.csx"

pipelines: 
  -
    when: 
      - queue: entry_submitted
    do: 
      - script: |
          Json((context, json, ct) => {
              json.Data["firstname"] = json.Data["fullname"].Split(' ').First();
              return json;
          });
    to:
      - mailgun.send
        to: "#{email}"
        subject: "Thank you, #{firstname}!"
        text: Thank you for your submission!
                    

Process the data with Node.js modules.

Data processing can also be performed with your own or third party Javascript modules running in Node.js.


exports.somefunc = function(data, context) 
{
    data.someValue = data.someOtherValue.toLowerCase();
    return data;
}
                    

Listen and react to the outside world.

You can easily create APIs for your consumers.


services: 
  postapi:
    type: httplistener
    path: mypath

  searchapi:
    type: httplistener
    path: search/{searchstring}
    responseBody: json

pipelines: 
  -
    when: 
      - postapi
    do: 
      - map: "$.body"
    to: 
      - store: elasticdb
        index: myindex
        dataType: entry
        key: "#{entryId}"
  -
    when: 
      - searchapi
    do: 
      - search: elasticdb
        index: myindex
        dataType: entry
        value: "#{route.searchstring}"
      - map: "$.hits[*]" 
                

Set up asynchronous event chains.

Foopipes is built with an event driven, message based architecture with the possibility to use both internal and external message queues.


pipelines: 
  -
    when: 
      - queue: sync_done
    do:
      - http: "http://myserver/api/flushcache"
    then: 
      - publish: cache_flushed
                            

Asynchronous task execution.


pipelines: 
  -
    when: 
      - webhook
    then:
      - publish: source_updated 
  -
    when: 
      - queue: source_updated
    do:
      - command: "bash"
        args: "build.sh"
    then: 
      - publish: build_completed 
                            

Use plugins, or build your own in .NET core.

The functionality of Foopipes can be extended with third party plugins written in .NET Core.


plugins:
  - Elasticsearch
  - RavenDB
  - RabbitMQ
  - Me.MyPlugin
                            

On the shoulder of giants

Get Started!

Docker certified

Foopipes runs in Docker. Stand alone installations can be arranged for Linux, Mac or Windows.

Install Docker, open a terminal and copy/paste:

Linux/Bash


# Create default config file
docker run aretera/foopipes config >foopipes.yml

# Start and mount current directory 
docker run -v $(pwd):/project aretera/foopipes
                            

Windows


docker run aretera/foopipes config >foopipes.yml
docker run -v %CD%:/project aretera/foopipes
                            

Want more examples?

Check out the examples in the documentation.