HSRW EOLab Students Wiki

User Tools

Site Tools

Trace: • sensors • openhype
eolab:openhype:start

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

eolab:openhype:start [2022/09/19 11:46] – [3 Dashboard] sina001eolab:openhype:start [2023/01/05 14:38] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +/*
 ~~NOTOC~~ ~~NOTOC~~
-====== OpenHyPE - Förderprojekt ====== +*/ 
- +====== NRW-Förderprojekt: OpenHygrisC Data Processing for Education (OpenHyPE) ======
-===== NRW Groundwater Data – OpenHygrisC Data Processing for Education (OpenHyPE) ===== +
- +
-  * Gefördert durch das [[https://www.umwelt.nrw.de/|Ministerium für Umwelt, Landwirtschaft, Natur- und Verbraucherschutz NRW (MULNV)]] +
-  * Laufzeit: 15.12.2021 - 31.12.2022+
  
 +  * Gefördert durch das [[https://www.mlv.nrw.de/|Ministerium für Landwirtschaft und Verbraucherschutz NRW (MLV)]]
 +  * Laufzeit: 15.12.2021 - 30.06.2023
  
 +/*
 =====Table of Contents===== =====Table of Contents=====
  
Line 39: Line 39:
  
 5. Project Codes  5. Project Codes
 +*/
  
  
Line 97: Line 97:
 ===== 2.2 PostgreSQL/PostGIS ====== ===== 2.2 PostgreSQL/PostGIS ======
  
-PostgreSQL is also known as Postgres is a free and open-source relation database management system and according to the PGadmin website, "Postgres Database is the most advanced open-source database in the world". We can store the time series data as well as geometry data in Postgres. In this project, we have used PGadmin which is the most popular and powerful open-source administration platform for Postgres Database.+PostgreSQL, also known as Postgres, is a free and open-source relational database management system, and according to the official website of pgadmin, "Postgres Database is the most advanced open-source database in the world." In Postgres, we can store both time series and geometry data. In this project, we have used PGadmin, which is the most popular and powerful open-source administration platform for Postgres databases.
  
-**PostGIS:** PostGIS is technically an extension of the PostgreSQL database which helps to add support for geographical objects. PostGIS is open-source and free to use.+**PostGIS:** PostGIS is technically an extension of the PostgreSQL database, which helps add support for geographical objects. PostGIS is open-source and free to use.
  
-The below image shows the PGadmin.+The below image shows the PGadmin tool.
  
 |  {{ :eolab:openhype:pgadmin.jpg?800 |}}  | |  {{ :eolab:openhype:pgadmin.jpg?800 |}}  |
Line 142: Line 142:
    * Programming applications      * Programming applications  
  
-In this project, we have used Python for data engineering, data pre-processing and data analysis. Jupyter Notebook is used in this project to write python codes. The Jupyter Notebook is an open-source web application that data scientists simply can write the code and make it easier to document it. Simply we can combine python codes, text, images, comments and the result of the codes on the same page. The below image shows how code, text and the result of the code can be seen on a single page.+In this project, we have used Python for data engineering, data pre-processing, and data analysis. Jupyter Notebook is used in this project to write Python codes. The Jupyter Notebook is an open-source web application that data scientists can simply write the code for and make it easier to document. Simply, we can combine Python codes, text, images, comments, and the result of the codes on the same page. The below image shows how code, text, and the result of the code can be seen on a single page.
  
 |  {{ :eolab:openhype:combination_code_text.jpg?800  }}  | |  {{ :eolab:openhype:combination_code_text.jpg?800  }}  |
Line 179: Line 179:
  
  
-**openhype environment:** In this project, openhype environment has been created in order to install all the necessary packages which are needed. openhype environment has been created based on the above video.+**openhype environment:** In this project, "openhype" environment has been created in order to install all the necessary packages which are needed. openhype environment has been created based on the above video.
  
  
Line 212: Line 212:
    * ** geopandas:** With the help of this package we are able to work with geospatial data in Python. according to the geopandas website " It     * ** geopandas:** With the help of this package we are able to work with geospatial data in Python. according to the geopandas website " It 
         is extends the data types used by pandas to allow spatial operations on geometric types".          is extends the data types used by pandas to allow spatial operations on geometric types". 
 +        
 <code> <code>
 conda install --channel conda-forge geopandas conda install --channel conda-forge geopandas
 </code> </code>
 +
 +Some packages need to specify the channel to install and that's why in the above code we have specified the channel.
  
  <code>  <code>
Line 243: Line 246:
 </code> </code>
                  
- Since we have downloaded the four CSV files in the previous chapter, now is the time to read our CSV files and start to clean them in order to make them ready to import to our database. + Since we have downloaded the four CSV files in the previous chapter, now is the time to read our CSV files and start to clean them in order to make them ready to import into our database. 
  
  
Line 255: Line 258:
 There below four notebooks should be run separately, in order to import data into the database.  There below four notebooks should be run separately, in order to import data into the database. 
  
-  * import_gemeinde.ipynb +  * import_gemeinde.ipynb:  
-  * import_katalog_stoff.ipynb +In this notebook, we will import the data of all geminde into the database. 
-  * import_messstelle.ipynb +  * import_katalog_stoff.ipynb: 
-  * import_messwert.ipynb+In this notebook, we will import the data of all the catalogue substances into the database. 
 +  * import_messstelle.ipynb:  
 +In this notebook, we will import the data of all stations into the database. 
 +  * import_messwert.ipynb: 
 +In this notebook, we will import the data of all values into the database.
  
  
Line 269: Line 276:
  
  
-In the previous section, we have downloaded the data, cleaned and imported them to the database successfully and now it's time to see the data in the database. as we know, the SQL command is valid in the Postgres database, as a result of that, we will run some basic SQL commands to see the data. +In the previous section, we downloaded the data, cleaned and imported them to the database successfully and now it's time to see the data in the database. as we know, the SQL command is valid in the PostgresSQL database, as a result of that, we will run some basic SQL commands to see the data. 
  
 First, we want to see our tables, with the below code, we are selecting all the columns (***** means all the columns) from our schema which is consist of our table (in this case is **sina**) and with the name of the table. and then because the size of the table is huge and we want only to see the first 100 rows then we just limit it to 100.  First, we want to see our tables, with the below code, we are selecting all the columns (***** means all the columns) from our schema which is consist of our table (in this case is **sina**) and with the name of the table. and then because the size of the table is huge and we want only to see the first 100 rows then we just limit it to 100. 
Line 471: Line 478:
 Plotly offers online graphing, analytics, and statistics tools for their users, as well as scientific graphing libraries for Python, R, MATLAB, Perl, Julia, Arduino, and REST. Plotly offers several open-source and enterprise products such as Dash which have been used for creating simple and interactive dashboards in this project. Plotly offers online graphing, analytics, and statistics tools for their users, as well as scientific graphing libraries for Python, R, MATLAB, Perl, Julia, Arduino, and REST. Plotly offers several open-source and enterprise products such as Dash which have been used for creating simple and interactive dashboards in this project.
  
-** Dash: ** Dash is a framework to build data apps rapidly not only in Python but also in R, Julia, and F#. According to Plotly official website, Dash is downloaded 800,000 times per month which shows that nowadays Dash getting more popular. Dash is a great framework for anyone who uses data with a customised user interface. With only a couple of patterns, Dash  +** Dash: ** Dash is a framework to build data apps rapidly not only in Python but also in R, Julia, and F#. According to Plotly official website, Dash is downloaded 800,000 times per month which shows that nowadays Dash getting more popular. Dash is a great framework for anyone who uses data with a customised user interface. Through a couple of simple patterns, Dash eliminated all of the technologies as well as protocols that are needed to make a full-stack web app with interactive data considerations. Another good feature is that Dash is running on web browsers so it means that no other application needs to run it.
- +
-Through a couple of simple patterns, Dash eliminated all of the technologies as well as protocols that are needed to make a full-stack web app with interactive data considerations. Another good feature is that Dash is running on web browsers so it means that no other application needs to run it.+
  
-If you would like to know more about Dashboard with Plotly Dash, click the link below.+If you would like to know more about Dashboard with Plotly Dash, click the link below. In the below link, you will find full tutorials about how to create a simple dashboard with Plotly Dash.
  
  
Line 531: Line 536:
 \\ \\
  
-All the codes are available in below link. +All the codes are available in the below link. 
  
 [[ https://github.com/sina7272/LANUV_Groundwater |Click here for project codes]] [[ https://github.com/sina7272/LANUV_Groundwater |Click here for project codes]]
eolab/openhype/start.1663580803.txt.gz · Last modified: 2023/01/05 14:38 (external edit)