The grass is rarely greener, but it's always different

Big Data Integrator (BDI) Integrated Development Environment (IDE)

In the Big Data Europe framework, the Big Data Integrator is an application that can be thought as a "starter kit" to start working and implementing big data pipelines in your process. It is the minimal standalone system so you can create a project with multiple docker containers, upload it & make it run using a nice GUI.


You can think of the Big Data Integrator as a placeholder. It acts as a "skeleton" application where you can plug & play different big data services from the big data europe platform, and add and develop your own.

At it's core it is a simple web application that will render each different service's frontends inside it, so it is easy to navigate between each system providing a sense of continuity in your workflow.

The basic application to start from is constituted of several components:

That service will be the "referee" that imposes the steps defined in the workflow builder. For more information check it's repository.

Systems are organized following a microservices architecture and run together using a docker-compose script, some of them sharing microserviecs common to all architectures, like the identifier, dispatcher, or resource. This is a more visual representation of the basic architecture:


Installation & Usage

How to add new services

    image: bde2020/new-service-frontend:latest
    - csswrapper
    - identifier:backend
    - "80"
    VIRTUAL_PORT: "80"
  "data": [
    ...etc .. ,
      "id": 1,
      "type": "user-interfaces",
      "attributes": {
        "label": "My new Service",
        "base-url": "",
        "append-path": ""

Have fun with it!

#bigdata #docker