Merge pull request #5 from antonia2702/main

Expanded documentation

Author: david-woelfle

.python-version

@@ -0,0 +1 @@
+esg_venv

docs/source/01_concepts.rst

@@ -0,0 +1,92 @@
+Concepts
+========
+
+This section explains how to apply the ESG framework and gives an overview of the basic technical concepts.  For more in-depth information especially regarding the discussion of the approach, please refer to the published `research article <https://doi.org/10.48550/arXiv.2402.15230>`__.
+
+Framework
+---------
+The below figure shows the basic architecture of an EMS that integrates the ESG framework.
+
+.. figure:: graphics/ems_architecture_with_services.png
+   :alt: EMS Architecture
+   :align: center
+
+   EMS Architecture
+
+
+The API of the framework is based on the RESTful principles with authorization being handled through JSON Web Tokens (JWTs). It enables the 
+
+* retrieval of a forecast or optimized schedule and
+* fitting of system specific parameters of a service.
+
+The intended interaction of the EMS with the API, consists of the following calls in the given order:
+
+1. Issue a POST call to the ``/{version}/request`` or to the ``/{version}/fit-parameters/`` endpoint.
+2. Poll the GET endpoint ``/{version}/request/{task_ID}/status/``  or ``/{version}/fit-parameters/{task_ID}/status/`` respectively for the status of the job initiated in step 1. The task_ID is returned in the response of the POST call.
+3. Once the status is ``ready``, issue a GET call to the ``/{version}/request/{task_ID}/result/`` or ``/{version}/fit-parameters/{task_ID}/result/`` endpoint to retrieve the result of the computation.
+
+The format of the input data for calls to the ``/request/`` and ``/fit-parameters/`` endpoints and of the output returned by the corresponding ``/result/`` endpoints has to be specified by the developer. However, the ESG package provides building blocks for data models (see section :ref:`data_model`) to reduce the time spent on defining them.
+
+Service Components
+------------------
+A functional service that implements a forecasting or optimization algorithm consists of three categories of components:
+
+1. **Base**: Contains the components necessary for executing the code of the service.
+2. **Service Framework**: Contains all components generic to all services.
+3. **Service Specific**: Contains all components a service provider must implement to derive a functional service.
+
+.. figure:: graphics/service_components.png
+   :alt: Service Components
+   :align: center
+   
+   Service Components
+
+The worker, which is the second component within the service framework, enables concurrent processing of requests, effectively decoupling the API from interacting with forecasting or optimization code. To enhance performance, the worker and API should be operated in distinct processes connected only through a message broker.
+
+.. figure:: graphics/service_architecture_full.png
+   :alt: Full Service Architecture
+   :align: center
+   
+   Full Service Architecture
+
+The illustration above shows the full architecture of a service. The API is the entry point for the client software and interacts with the worker through the message broker. For each valid POST request, the API publishes a task on the message broker and assigns it a unique ID. A worker then fetches the task and starts computing the result. It regularly publishes status updates on the processing progress to the broker and finally the result of the computation. If the EMS calls the ``/status/`` endpoint, the API fetches the latest update regarding the corresponding task and returns the information to the client. Accordinngly, if the EMS calls the ``/result/`` endpoint, the API fetches the result from the broker and returns it to the client. 
+To ensure the message broker is kept clean, a garbage collector deletes task-related data from the broker that are likely not required anymore. 
+
+Operation Concepts
+------------------
+All components, also referred to as processes, of the service need to be wrapped in individual Docker containers in order to enable parallel execution of multiple instances of the same process and distribution of the processes across different machines. 
+For commercial applications, the orchestrator `Kubernetes <https://kubernetes.io/>`__ is well suited, while academic applications are well off choosing `Docker Swarm <https://docs.docker.com/engine/swarm/>`__ as an orchestrator.
+
+There are two supportive applications that service providers will habe to operate:
+
+* A **Gateway** (also known as an ingress or reverse proxy): This makes the API containers accessible to the EMS and balances the distribution of requests across the API containers. Ideally, it also takes care of encrypting the communication between the client and the service through HTTPS.
+* An **Identity Provider (IdP)**: The IdP issues the JWT tokens to the client software using the OpenID Connect (OIDC) protocol (e.g. the application `Keycloak <https://www.keycloak.org/>`__).
+
+Implementation
+==============
+A reference implementation of the design concept can be found in `open source repository <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics>`__. While this implementation is in Python, the concepts are described sufficiently generic to allow for implementation in another programming language, e.g., Java. 
+
+Inter-process Communication
+---------------------------
+The ESG utilizes an open source python library for distributed task execution. `celery <https://github.com/celery/celery>`__ is here employed for the inter-process communication between API and worker. It is worth noting that celery supports different message brokers, e.g. Redis and RabbitMQ.
+
+Worker
+------
+The worker uses celery to interact with the message broker. ESG Framework already provides a generic worker that invokes the service specific forecasting or optimization code in the ESG package.* 
+
+Garbage Collector
+-----------------
+ The garbage collector integrated in the ESG framework is the one provided by celery. Depending on the choice of the message broker no dedicated process is required (`see celery documentation <https://docs.celeryq.dev/en/stable/userguide/configuration.html#result-expires>`__). 
+
+API
+---
+The framework selected for the implementation of the API is `FastAPI <https://fastapi.tiangolo.com/>`__. FastAPI already provides some functionality out of the box, such as automatic generation of OpenAPI schema documents and SwaggerUI. They can usually be accessed at `localhost:8800 <localhost:8800>`__. 
+
+The API is fully functional, except for the definition of the data models. Service providers only have to define the input data for the ``POST /request/`` and ``POST /fit-parameters/ `` API methods and the corresponding  output data for the ``GET /result/`` API methods. As mentioned above, the ESG package contains building blocks for data models to reduce time spent on defining them (see section :ref:`data_model`). The data format for all data exchange is JSON. 
+
+The example implementation utilizes `PyJWT <https://pyjwt.readthedocs.io/en/stable/>`__ to verify calls to the API endpoints before using celery to publish tasks to the message broker.
+
+Additional Functionality
+------------------------
+* **Generic Client**: The ESG package comes with a generic client that can be used to trigger calls to services from python source code.
+* **Utility functions**: The ESG packages includes useful utility functions, e.g., for parsing pandas DataFrames from JSON data.

docs/source/02_example_service.rst

@@ -0,0 +1,88 @@
+Example service implementation
+==============================
+This section demonstrates a simple but fully functional implementation of a photovoltaic (PV) power gerneration forecast service. For more in-depth information about the individual components, please refer to the :doc:`01_concepts` section or the published `research article <https://doi.org/10.48550/arXiv.2402.15230>`__.
+
+1. Preparation of the forecast or optimization code
+---------------------------------------------------
+To implement a service, the code that is to be the payload of the service needs to be prepared. This code is responsible for the actual forecast or optimization task. In this example, the `pvlib <https://github.com/pvlib/pvlib-python>`__ has been utilized, which computes the PV power production forecast. As mentioned already though, the framework does not impose any restrictions on the code that is wrapped by it, i.e., linear programs, classical statistical models, or fully black-box machine learning approaches are all possible.
+
+First the input data needs to be provided to the library. To keep this example simple, only the geographic position, i.e. latitude and longitude, as well as the geometry of the PV system, i.e. azimuth and inclination, and the peak power are selected to describe the PV system. The second part of the required input to compute PV power prediction consists of meteorological forecast data, especially forecasts of solar irradiance.
+
+However, this example service is intended to produce PV power generation forecasts for systems for which geometry and peak power values may be unknown and need to be estimated from power production measurements. Therefore, the parameter fitting has been implemented with a simple least squares approach. Although it should be noted that this choice has no particular relevance for the present example. Thus, the input data necessary to obtain a forecast is separated into two groups: 
+
+* **arguments**: here latitude and longitude
+* **parameters**: here azimuth, inclination, and peak power
+
+Finally, it should be considered that it may not be a good choice not to demand all input data as client input. In the present example, the service instead fetches the meteorological data automatically from a third-party web service, which would, in practice, make the interaction with the service more convenient and less error-prone for the client.
+
+The actual format of input_data and output_data is implicitly defined in the corresponding data
+models, which are introduced in the following section.
+
+Below is an excerpt of the forecast code implemented in this exemplary service. Only the functions handle_request and fit_parameters are shown here, as they are the only part of the service implementation that actually interacts with the forecasting or optimization algorithm. The functions predict_pv_power, fetch_meteo_data as well as fit_with_least_squares have been omitted from the listing, as the practical implementation details of those are not necessary for a developer wanting to implement their service. However, the code of the omitted functions can be found in the repository of the `ESG framework <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics>`__.  Implementing fit_parameters is optional and can be omitted for services without fittable parameters. 
+
+.. literalinclude:: ../examples/basic_example/source/service/fooc.py
+   :pyobject: handle_request
+
+.. literalinclude:: ../examples/basic_example/source/service/fooc.py
+   :pyobject: fit_parameters
+
+The code shown above can be found in the file `fooc.py <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/docs/examples/basic_example/source/service/fooc.py>`__.
+
+.. _data_model:
+
+2. Definition of the data model
+-------------------------------
+The data model is the second component that is service specific and which must thus be defined by the service developer. The data models define the format of the data the client exchanges with the service. For a service without fittable parameters, i.e. a service with ``/request/`` endpoints only, it is sufficient to define the arguments required for computing the request as well as the result of the computation. The corresponding data models are called ``RequestArguments`` and ``RequestOutput``.
+
+In the case of a service with fittable parameters, it is additionally necessary to define the data format for the input and output data of the ``/fit-parameters/`` endpoints. The data models specifying the input for the fitting process are referred to as ``FitParameterArguments`` and ``Observations``, and the corresponding output is ``FittedParameters``. As the simple PV power generation forecast service used as an example is designed to provide functionality to fit parameters, it is necessary to define all five data models.
+
+.. literalinclude:: ../examples/basic_example/source/service/data_model.py
+   :pyobject: RequestArguments
+
+.. literalinclude:: ../examples/basic_example/source/service/data_model.py
+   :pyobject: RequestOutput
+
+.. literalinclude:: ../examples/basic_example/source/service/data_model.py
+   :pyobject: Observations
+
+.. literalinclude:: ../examples/basic_example/source/service/data_model.py
+   :pyobject: FitParameterArguments
+   
+.. literalinclude:: ../examples/basic_example/source/service/data_model.py
+   :pyobject: FittedParameters
+
+The above code can be found in the `data_model.py <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/docs/examples/basic_example/source/service/data_model.py>`__ file.
+
+The ESG package provides ready-to-use **building blocks for data models**, which can be found in the file `metadata.py <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/source/esg/models/metadata.py>`__. For example, in the code above, ``GeographicPosition`` is imported from ESG. ``GeographicPosition`` is a data model itself, which defines that a geographic position consists of latitude and longitude. The data models also serve a documentational purpose as they define the data structure of the downstream application, i.e. the EMS. Additionally, they define permitted ranges for values, which helps in automatically validating the input provided by clients. 
+
+3. Implementation of the worker component
+-----------------------------------------
+The worker component is responsible for executing the tasks, i.e. computing requests or fitting
+parameters by invoking the forecasting or optimization code, as well as task scheduling. While the ESG framework utilizes the Celery library for implementing the worker, it extends it with functionality to make the implementation of services more convenient, for example by utilizing the data models for de-/serialization of input and output data. Thus, the main objective for implementing a worker is to wire up the data models with the forecasting or optimization code. This is usually a rather simple program, as displayed in the code excerpt below, taken from the PV power generation forecast example service.
+
+.. literalinclude:: ../examples/basic_example/source/service/worker.py
+   :pyobject: request_task
+
+.. literalinclude:: ../examples/basic_example/source/service/worker.py
+   :pyobject: fit_parameters_task
+
+The above code can be found in the `worker.py <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/docs/examples/basic_example/source/service/worker.py>`__ file.
+
+4. Customization of the API component
+-------------------------------------
+The implementation of the API component is available ready-to-use in the ESG framework. However, in order to operate the API it is necessary, similar to the worker, to wire up the API with the other components, in particular with the data model and the worker. Furthermore, some information like name and version number must be provided too. Nevertheless, the necessary code to instantiate an API component is fairly simple as seen in the code excerpt below.
+
+.. literalinclude:: ../examples/basic_example/source/service/api.py
+   :language: python
+   :start-at: api =
+
+The above code can be found in the `api.py <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/docs/examples/basic_example/source/service/api.py>`__ file.
+
+5. Building docker images to derive functional services
+-------------------------------------------------------
+In order for the service developer to derive functional services, they need to build docker images that can be run e.g. on Kubernetes or Docker Swarm. It is necessary to build two distinct images, one for the API (which includes the data model) and one for the worker (which includes the data model and the forecasting or optimization code). The build instructions for both images are implemented as `Dockerfile <https://docs.docker.com/reference/dockerfile/>`__ as seen below.
+
+.. literalinclude:: ../examples/basic_example/source/Dockerfile-API
+.. literalinclude:: ../examples/basic_example/source/Dockerfile-worker
+
+The above code can be found in the `Dockerfile-API <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/docs/examples/basic_example/source/Dockerfile-API>`__ and the `Dockerfile-worker <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/docs/examples/basic_example/source/Dockerfile-worker>`__ files.

docs/source/03_example_client.rst

@@ -0,0 +1,23 @@
+Example client implementation
+=============================
+This section showcases two example client implementations for the basic example service. The first example is a minimal client implemented in a UNIX shell script. The second example is a more sophisticated client implemented in Python using the ESG package. 
+
+Minimal client implementation
+----------------------------------------------
+This minimal client implementation is to be executed on the same machine as the service. The service should be reachable via a localhost port. The client is implemented using a simple UNIX shell script and requires the additional packages **curl** and **jq**. The script below can be used to simulate a client that requests a photovoltaic power generation forecast from the basic example service. Note that this interaction with the ``/fit-parameters/`` endpoint is omitted in this example but would follow the same structure.
+
+.. literalinclude:: ../examples/minimial_client/source/client/main.sh
+    :language: bash
+    :start-at: SERVICE_BASE_URL
+
+Client implementation in python
+-------------------------------
+The service integration in a production EMS will probably require a more sophisticated client implementation and require additional functionality. However, the concept of the client remains the same. As mentioned before, the ESG package contains a `generic client <https://github.com/fzi-forschungszentrum-informatik/energy-service-generics/blob/main/source/esg/clients/service.py>`__ providing additional functionality such as parsing the result into the format the EMS expects and network error handling.
+
+.. literalinclude:: ../examples/client_using_ESG/source/client/main.py
+    :language: python
+    :start-at: import
+
+It is recommended that the client developer specifies the data models themself in order to document the data structure of the downstream application, i.e. the EMS.  This should reduce tedious debugging deep within the downstream application and produce the error right inside the client code. However, it is possible to fetch the data models automatically from the service.
+
+It is to be noted, that the generic client provided by the ESG package can only be used in a python environment. Nevertheless, if the client code needs to be in a different language it should be possible to implement the logic rather quickly.  `Swagger Codegen <https://swagger.io/tools/swagger-codegen/>`__ can help to partially automate the generation of the client program.