Foopipes is a lightweight pipeline engine for asynchronous pipelining of json and other data formats between services.
Data flows are set up with yaml configuration files or as Javascript modules.
plugins:
- Elasticsearch
services:
poller:
type: scheduler
interval: "00:01:00"
elasticdb:
type: elasticsearch
url: "http://localhost:9200/"
pipelines:
-
when:
- poller
from:
- http: "http://myservice/api/content_as_json_or_xml"
do:
- map: "$.entries[*]" # json path expression
to:
- store: elasticdb
index: myindex
dataType: entry
key: "#{entryId}" # data binding expression
then:
- publish: sync_done
You can pretty much do whatever you want with the data with your own or community created C# Roslyn scripts.
addins:
- url: "https://raw.githubusercontent.com/AreteraAB/Foopipes.Addins/master/Mailgun/mailgun.csx"
pipelines:
-
when:
- queue: entry_submitted
do:
- script: |
Json((context, json, ct) => {
json.Data["firstname"] = json.Data["fullname"].Split(' ').First();
return json;
});
to:
- mailgun.send
to: "#{email}"
subject: "Thank you, #{firstname}!"
text: Thank you for your submission!
Data processing can also be performed with your own or third party Javascript modules running in Node.js.
exports.somefunc = function(data, context)
{
data.someValue = data.someOtherValue.toLowerCase();
return data;
}
You can easily create APIs for your consumers.
services:
postapi:
type: httplistener
path: mypath
searchapi:
type: httplistener
path: search/{searchstring}
responseBody: json
pipelines:
-
when:
- postapi
do:
- map: "$.body"
to:
- store: elasticdb
index: myindex
dataType: entry
key: "#{entryId}"
-
when:
- searchapi
do:
- search: elasticdb
index: myindex
dataType: entry
value: "#{route.searchstring}"
- map: "$.hits[*]"
Foopipes is built with an event driven, message based architecture with the possibility to use both internal and external message queues.
pipelines:
-
when:
- queue: sync_done
do:
- http: "http://myserver/api/flushcache"
then:
- publish: cache_flushed
pipelines:
-
when:
- webhook
then:
- publish: source_updated
-
when:
- queue: source_updated
do:
- command: "bash"
args: "build.sh"
then:
- publish: build_completed
The functionality of Foopipes can be extended with third party plugins written in .NET Core.
plugins:
- Elasticsearch
- RavenDB
- RabbitMQ
- Me.MyPlugin
Foopipes runs in Docker. Stand alone installations can be arranged for Linux, Mac or Windows.
Install Docker, open a terminal and copy/paste:
Linux/Bash
# Create default config file
docker run aretera/foopipes config >foopipes.yml
# Start and mount current directory
docker run -v $(pwd):/project aretera/foopipes
Windows
docker run aretera/foopipes config >foopipes.yml
docker run -v %CD%:/project aretera/foopipes
Check out the examples in the documentation.