Foopipes 2.2 released! 2017-06-09

Foopipes 2.2 released!

A new minor version of Foopipes has just been pushed to Docker hub with new features and enhancements.

What's new

  • The most noticeable difference is the new Shorthand config format. Some config parsing sugar makes it possible to write denser pipeline config. For example, the following config is valid:
pipelines: 
  - 
    when: 
      - queue: started
      - scheduler
    from:
      - { http: "https://jsonplaceholder.typicode.com/posts", method: get }
    to:
      - { file: "post_#{id}.json", path: "." }
    finally:
      - exit

The old format is still supported.

  • The httplistener service has been greatly enhanced and now works as it was intended from the beginning where the pipeline's results can be returned as http response. It also supports route templates.
services: 
  api:
    type: httplistener
    path: search/{searchstring}
    responseBody: json

pipelines: 
  - 
    when: 
      - { api }
    from:
      - { search: elastic, index: myindex, dataType: myarticle, value: "#{route.searchstring}" }
    do:
      - { select: "$.hits[*]" }
      - { select, Name: "#{Name1}, #{Name2}", Id: "#{ArticleId}" }
  • The http task now supports custom headers.
  • The http task now set the response headers as metadata.httpHeaders. Here is an example of using the ETag header for incremental polling a RSS feed.
services:
  etagstore:
    type: file
    path: .

pipelines:
  -
    when:
      - queue: started
    from:
      - http: "http://csr200.blogspot.com/feeds/posts/default?alt=rss"
        format: xml
        headers:
          If-None-Match: "#{etagstore:etag}"
    do:
      - { set: "etagstore:etag", value: "#{metadata:httpHeaders.ETag}" }
      - { select: "$.rss.channel.item[*]" }
      - { select, id: "#{guid.#text}", title: "#{title}", link: "#{link}", description: "#{description}" }
    to:
      - log 
    finally:
      - exit