FuzzyCorr¶
This repository contains the work developed for a Master Thesis on fuzzy map comparison methods to evaluate the performance of hydro-morphodynamic numerical models. Please read the License terms for code usage and re-distribution.
Sediment transport and hydraulic processes can be reproduced with numerical models such as SSIIMM, Hydro_AS_2D, TELEMAC and many more. The accuracy of numerical models is assessed through comparing the simulated and the observed datasets, which constitutes a model validation. With the purpose of analyzing simulated and observed bed elevation change, two methods of comparison can be applied:
Comparison via statistical methods such as RMSE (Root Mean Squared Error) or visual human comparison. However, local measures of similarity (or a similarity) like the RMSE are very sensible to uncertainty of location and amount, thus indicating low agreement even when overall patterns were adequately simulated.
Visual comparison captures global similarity, which is one of the reasons why modelers often use it for model validation. Humans are capable of finding patterns without deliberately trying, and therefore, this type of comparison provides substantial advantages over local similarity measures. Nevertheless, more research has to be done to implement automated validation tools that emulate human thinking. This is necessary because human comparison is not transparent, prone to subjective interpretations, time consuming, and hardly reproducible.
In this context, the concept of fuzzy set theory has capacities to consider similarity of spatial pattern analogous to human thinking. For instance, fuzziness of location introduces a tolerance regarding spatial uncertainty in the results of hydro-morphodynamic models. To this end, fuzzy logic enables an objective validation of such models by overcoming uncertainties in the model structure, parameters and input data.
The algorithms provided with fuzzycorr
address the necessity in evaluating (or validating) model performance through the use of fuzzy map comparison. Future developments aim to go beyond a one-way validation towards a two-way communication between the validation algorithms and the models. The two-way communication represents a feedback loop that will eventually enable an automated calibration of numerical hydro-morphodynamic models.
Usage¶
Basics¶
The following code block exemplifies the usage of fuzzycorr to explore the fuzzy correlation between two (e.g., observed and modeled) maps (in GeoTIFF format):
from flusstools import fuzzycorr as fc
Example (showcase)¶
The best way to learn the usage is by examples. In the directory examples
, the usage of the modules are demonstrated in a case study. Inside the folder salzach_case
, the results from a hydro-morphodynamic numerical simulation ( i.e., simulated bed elevation change, deltaZ) are located in raw_data
. For more details on the hydro-morphodynamic numerical refer to Beckers et al. (2020).
The following showcase scripts live in ROOT/examples/fuzzycorr-showcase/:
prepro_salzach.py
: example of the usage of the classFuzzyPreProcessor
of the moduleprepro.py
, where vector data is interpolated and rasterized.classification_salzach.py
: example of the usage of the classPreProCategorization
of the moduleprepro.py
.fuzzycomparison_salzach.py
: example of the usage of the classFuzzyComparison
of the modulefuzzycomp.py
, which creates a correlation (similarity) measure between simulated and observed datasets.plot_salzach.py
,plot_class_rasters.py
andperformance_salzach
: example of the usage of the moduleplotter.py
.random_map
: example of generating a raster following a uniform random distribution, which uses the moduleprepro.py
.
Structure¶
This package contains the following modules, which were designed in Python 3.6:
prepro.py includes functions for reading, normalizing and rasterizing vector data. These are preprocessing steps for fuzzy map comparison (module fuzzycomp).
fuzzycomp.py provides routines for fuzzy map comparison in continuous valued rasters. Refer to Hagen (2006) for more details.
plotter.py: Visualization routines for output and input rasters.
The package documentation is located in the folder docs.
Pre-processing: prepro.py¶
- Hints:
many class methods could be imported from geotools
already removed: clip_raster, which is a duplicate of raster_mgmt
-
class
flusstools.fuzzycorr.prepro.
CategorizationPreProcessor
(raster)[source]¶ Structured for … (UNCLEAR)
- Parameters
raster – string, path of the raster to be categorized
-
class
flusstools.fuzzycorr.prepro.
FuzzyPreProcessor
(df, attribute, crs, nodatavalue, res=None, ulc=(nan, nan), lrc=(nan, nan))[source]¶ Parent pre-processing structure for the comparison of numeric maps
- Parameters
df – pandas dataframe, can be obtained by reading the textfile as pandas dataframe
attribute – string, name of the attribute to burn in the raster (ex.: deltaZ, Z)
crs – string, coordinate reference system
nodatavalue – float, value to indicate nodata cells
res – float, resolution of the cell (cell size), is the same for x and y
ulc – tuple of floats, upper left corner coordinate, optional
lrc – tuple of floats, lower right corner coordinate, optional
-
array2raster
(array, raster_file, save_ascii=True)[source]¶ Saves a raster using interpolation
- Parameters
raster_file – string, path to save the rasterfile
save_ascii – boolean, true to save also an ascii raster
- Returns
saves the raster with the selected filename
Hint
Function can be moved to geotools/raster_mgmt
-
create_polygon
(shape_polygon, alpha=nan)[source]¶ Creates a polygon surrounding a cloud of shapepoints
- Parameters
shape_polygon – string, path to save the shapefile
alpha – float, excentricity of the alphashape (polygon) to be created
- Returns
saves the polygon (*.shp) with the selected filename
Hint
Function can be moved to geotools/shp_mgmt
-
norm_array
(method='linear')[source]¶ Normalizes the raw data in equally distanced points depending on the selected resolution
- Returns
interpolated and normalized array with selected resolution
Hint
Read more at https://github.com/rosskush/skspatial
-
plain_raster
(shapefile, raster_file, res)[source]¶ Converts a shapefile(.shp) to a GeoTIFF raster without normalizing
-
points_to_grid
()[source]¶ Creates a grid of new points in the target resolution
- Returns
array of size nrow, ncol
- Hints:
Read more at http://chris35wills.github.io/gridding_data/
Fuzzy map comparison core: fuzzycomp.py¶
Description
-
class
flusstools.fuzzycorr.fuzzycomp.
FuzzyComparison
(raster_a, raster_b, neigh=4, halving_distance=2)[source]¶ Performing fuzzy map comparison :param raster_a: string, path of the raster to be compared with rasterB :param raster_b: string, path of the raster to be compared with rasterA :param neigh: integer, neighborhood being considered (number of cells from the central cell), default is 4 :param halving_distance: integer, distance (in cells) to which the membership decays to its half, default is 2
-
fuzzy_numerical
(comparison_name, save_dir, map_of_comparison=True)[source]¶ Compares a pair of raster maps using fuzzy numerical spatial comparison
- Parameters
save_dir – string, directory where to save the results
comparison_name – string, name of the comparison
map_of_comparison – boolean, create map of comparison in the project directory if True
- Returns
Global Fuzzy Similarity and comparison map
-
fuzzy_rmse
(comparison_name, save_dir, map_of_comparison=True)[source]¶ Compares a pair of raster maps using fuzzy root mean square error as spatial comparison
- Parameters
comparison_name – string, name of the comparison
save_dir – string, directory where to save the results of the map comparison
map_of_comparison – boolean, if True it creates map of of local squared errors (in the project directory)
- Returns
global fuzzy RMSE and comparison map
-
get_neighbours
(array, x, y)[source]¶ Captures the neighbours and their memberships :param array: array A or B :param x: int, cell in x :param y: int, cell in y :return: np.array (float) membership of the neighbours (without mask), np.array (float) neighbours’ cells (without mask)
-
-
flusstools.fuzzycorr.fuzzycomp.
f_similarity
(centre_cell, neighbours)[source]¶ Calculates the similarity function for each pair of values (fuzzy numerical method)
- Parameters
centre_cell – float, cell under analysis in map A
neighbours – np.array of floats, neighbours in map B
- Returns
np.array of floats, each similarity between each of two cells
-
flusstools.fuzzycorr.fuzzycomp.
squared_error
(centre_cell, neighbours)[source]¶ Calculates the error measure fuzzy rmse
- Parameters
centre_cell – float, cell under analysis in map A
neighbours – np.array of floats, neighbours in map B
- Returns
np.array of floats, each similarity between each of two cells
Plot routines: plotter.py¶
Description
-
class
flusstools.fuzzycorr.plotter.
RasterDataPlotter
(path)[source]¶ Class of raster for plotting
- Parameters
path – string, path of the raster to be plotted
-
make_hist
(legendx, legendy, fontsize, output_file, figsize, set_ylim=None, set_xlim=None)[source]¶ Creates a histogram of numerical raster
- Parameters
legendx – string, legend of the x axis of he histogram
legendy – string, legend of the y axis of he histogram
fontsize – integer, size of the font
output_file – string, path for the output file
figsize – tuple of integers, size of the width x height of the figure
set_ylim – float, set the maximum limit of the y axis
set_ylim – float, set the maximum limit of the x axis
- Returns
saves the figure of the histogram
-
plot_categorical_raster
(output_file, labels, cmap, box=True)[source]¶ Creates a figure of a categorical raster
- Parameters
output_file – path, file path of the figure
labels – list of strings, labels (i.e., titles)for the categories
cmap – string, colormap to plot the raster
box – boolean, if False it sets off the frame of the picture
- Returns
saves the figure of the raster
-
plot_categorical_w_window
(output_file, labels, cmap, xy, width, height, box=True)[source]¶ Creates a figure of a categorical raster with a zoomed window
- Parameters
output_file – path, file path of the figure
labels – list of strings, labels (i.e., titles)for the categories
cmap – string, colormap to plot the raster
xy – tuple (x,y), origin of the zoomed window, the upper left corner
width – integer, width (number of cells) of the zoomed window
height – integer, height (number of cells) of the zoomed window
- Returns
saves the figure of the raster
-
plot_continuous_raster
(output_file, cmap, vmax=nan, vmin=nan, box=True)[source]¶ Creates a figure of a continuous valued raster
- Parameters
output_file – path, file path of the figure
cmap – string, colormap to plot the raster
vmax – float, optional, value maximum of the scale, this value is used in the normalization of the colormap
vmin – float, optional, value minimum of the scale, this value is used in the normalization of the colormap
box – boolean, if False it sets off the frame of the picture
- Returns
saves the figure of the raster
-
plot_continuous_w_window
(output_file, xy, width, height, bounds, cmap=None, list_colors=None)[source]¶ Create a figure of a raster with a zoomed window :param output_file: path, file path of the figure :param xy: tuple (x,y), origin of the zoomed window, the upper left corner :param width: integer, width (number of cells) of the zoomed window :param height: integer, height (number of cells) of the zoomed window :param bounds: list of float, limits for each color of the colormap :param cmap: string, optional, colormap to plot the raster :param list_colors: list of colors (str), optional, as alternative to using a colormap :returns: saves the figure of the raster