Cloud Computing Project: - Indexing Large Geospatial Data with Distributed FaaS Method

This is our implementation of a faas-based system for the Cloud Computing exam. Our work is based on the following papers:

• Klimatic: A Virtual Data Lake for Harvesting and Distribution of Geospatial Data;
• Serverless Workflows for Indexing Large Scientific Data;

The system we designed uses faas, in particular the funcX framework (Documentation). The extraction of the metadata is carried out on files in NetCDF format (Documentation). A DB (PosgreSQL) is used on a MS Azure server to store the links of the files that are later used to extract the data requested by the user.

Architecture

Below is an image of the infrastructure of the system created:

As the image shows, the system is divided into two modules. The module on the left is responsible for calling up parsers via funcX and then executing the code remotely. Endpoints save the tuples they extract to the DB. The other module queries the DB using funcX.

Installation and Execution

• Clone this repository and enter it:

git clone https://github.com/pasqualedetrino/Cloud-Computing---Indexing-Large-Geospatial-Data-with-Distributed-FaaS-Method.git

• Do not forget to enter the endpoint codes and change the parameters for connection to the DB;
• Install all requirements;
• Execute ExtractMetadata.py using python (version 3.7+). This is an example:

python ExtractMetadata.py

The ExtractMetadata.py file does not need input parameters as it performs direct parsers for each type of measurement.

• Execute Query.py using python (version 3.7+). This is an example:

python Query.py --lat_min <float> --lat_max <float> --long_min <float> --long_max <float> --data_min <string> --data_max <string> --misura ["Air Surface Temperature Anomaly", "Precipitation", "Outgoing Longwave Radiation"]

The Query.py file needs the input parameters, in particular the minimum and maximum latitude, the minimum and maximum longitude, the minimum and maximum date and the type of measurement you are looking for.

Evaluation

To run the entire pipeline (extraction and query), we used two machines, the first with the following tecnichal specifications:

• CPU: 2 x CPU Intel(R) Xeon(R) Xeon 16-Core 5218 2.3Ghz 22MB.
• CORE NUMBER: 32
• RAM: 192 GB

The second machine has the following tecnichal specification:

• CPU: 1 x CPU Intel(R) i3-4010U(R) 1.7 Ghz
• CORE NUMBER: 4
• RAM: 12 GB

Contact

The presentation explains in detail the system.

For questions about code, please contact pasqualedetrino or GennaroIannuzzo.