Check data quality
Time reference check
Use the Time reference check tool for identification and correction of time shifts, time drifts and other time-related issues based on testing diurnal symmetry - morning vs. afternoon (in kt/h0 space). The time reference is critical for all subsequent quality tests, such as calculated solar geometry (position of sun). Moreover, the correct time reference is a basic prerequisite for the comparison of data sources from different datasets. After correction of time-related issues, the operator saves all required time corrections and the quality status of the dataset is updated.
The user can manually identify the shifts by diurnal symmetry test - morning vs. afternoon (in kt/h0 space). The second option is to run auto-detection, which automatically suggests the location and value of shifts and drifts in the data. The detected shifts can be manually modified or deleted. The detection runs from a few seconds to a few minutes depending on the volume of data. The automatic recognition identifies these shifts within the recognition uncertainty. Always select the original data if you would like to shift the data manually.
Component | Description |
Times series graph | Line graph of original records, records after shifting, and clear-sky. Lines is possible to switch off by clicking on the legend (without pan or zoom tool) |
Graph tools | Tools: Reset original view, Back to the previous view, Forward to next view, Pan, Zoom, Edit plot, Edit axis, curve and image parameters, Save the figure. Pan tool is able to use as a zoom tool by right click. |
Kt graphs | Graphs show diurnal symmetry (morning vs afternoon) of original and shifted data |
Shift overview graph | Line graph showing detected shifts in the dataset with uncertainty. The data are broken into segments, each segment has a uniform shift and its uncertainty. |
Dataset | Input dataset for Time reference check |
Column | Input column for Time reference check. The detection of time shifts is only based on this column. It must be either GHI or GTI type column. |
Time shifts arrow | Shows/hides the list of existing shifts |
Existing shifts | List of existing shifts |
New shifts | List of new shifts. You can disable time shift by checkbox. After running the auto-detection, the shift is shown with its uncertainty. The color shows the confidence of detected shift: green - high, yellow - medium, red - low. |
Auto-detection | Runs the automatic shift recognition. |
Import | Opens time shifts importer. You can choose another dataset and import time shifts from them. |
New | Creates a new shift. It is possible to create a new shift by selecting the time span in Time series graph (pan and zoom tools must be disabled). |
Remove | Deletes shift |
Reorder | Reorder time shifts from lowest to highest time stamp. |
Time from | Datetime setting the beginning of the span. |
Reset min | Resets datetime field to minimum possible value. |
Time to | Datetime setting the end of the span. |
Reset max | Resets datetime field to maximum possible value. |
Time shifts | Sets time shift value |
Draw kt diagrams only for the active shift | Allows you to display all records or only records from the actual time shift span. |
Redraw graphs | Redraws graphs by actual shifts and settings |
Settings | Opens settings window consisting of marker size setting and marker transparency setting for kt plots. |
Save | Saves changes |
Final save | Saves changes and QC status “TIME_REFERENCE_CHECK” will be assign to the dataset. |
Save as | Saves changes into the copy of dataset and QC status “TIME_REFERENCE_CHECK” will be assign to the dataset. |
Close | Closes window |
Limitations
Limitation | Probability | Severity |
Shift error compared to manual shifting is usually around 2 minutes | high | low |
The smallest possible recognizable shift can be between 0.5 and 1 minute depending on shift postprocessing. | - | - |
This method is suitable only for GHI and GTI measurements, It cannot be performed on DNI. | - | - |
The shift correction is based on selected radiation within the dataset and all columns have the same correction. Shift correction of each column separately is not allowed. | - | - |
This method uses postprocessing which unifies shifts through the dataset. Running the detection after shifting for the second time can give small non-zero shifts because of the applied postprocessing. | medium | low |
There can be no shift computation for a non-complete day (part of a day missing). Also, it can happen for a full day but surrounded by days of missing values. | low | high |
The computed shift for a corrupted day measurement has a larger error. By corrupted it means:
| medium | medium |
The time shift is computed for the whole day. We cannot correct the shift that happened in the middle of the day. | low | high |
The time shift is in some cases not computed (set to zero). It can happen when there are only missing values or invalid values during the day. | medium | low |
Radiation check
As soon as the dataset is successfully "time-reference-checked", the next step is to check the quality of the data. The Analyst specialized module (Automatic quality check of irradiation) brings a set of quality tests used today in Solargis. These include detection of invalid values, nighttime/daytime, artificial static values, breaking physical limits, consistency of irradiance components (only for GHI, DNI and DIF). Each record that failed in these tests is flagged by a pre-defined flag value. The user has some control over this process (i.e., disable/enable tests, changing test properties). After quality tests are performed the user is provided with a preview of a summary quality report (plots and text information). At the end of this process, generated flags are saved with the data and the quality status.
There are four main tabs in this window:
Heatmap flag plot. The heatmap flag plot depicts the occurrence of quality flags from automatic QC in time.
Time series flag plot. The Time series flag plot allows the basic data visualization of the quality control flags in time.
Consistency plots. Consistency plots depict the consistency between measured and calculated values separately for GHI, DNI and DIF.
Quality control summary. The Quality control summary resumes the overview of the statistics after the quality control.
How test groups work:
Multi component tests compare related parameters against each other within a test group.
Single component test requires a parameter within a test group.
We recommend to group related parameters e.g. SPN1 GHI and DNI measurements.
Use flag icons to indicate which quality control results to save for particular parameter.
Flags of a parameter can be saved only within one test group.
Analyst pre-fills test groups by default after import. To change parameter assignment or add / remove test groups use test group editor.
Component | Description |
Irradiance columns | Select columns of irradiance data type you want to quality check. Flags will be calculated into respective flag columns. Assign columns into groups - this is required for performing consistency test - testing calculated irradiance components versus measured. |
Advanced settings | Opens separated dialog with automatic quality check advanced settings. |
run between | Quality check can be performed on a subset of the dataset. If checked, values from both calendar widgets are taken into account. |
date from, date to | Defines a subset of the dataset by selecting two dates. |
Graph | The graphs including the QC flags are displayed. |
Irradiance components | The user can select the irradiances component to be displayed. Plot settings menu is active only for these Graph tabs: Heatmap flag plot, Time series flag plot. |
split heatmap by year | If checked, the heatmap is split by years. This checkbox is active only for the Heatmap flag plot. |
view in time zone | The time zone can be selected. |
Edit test groups | The user can edit, create or delete a test group. |
Run | Will start the process of automatic quality check. The result of the procedure is a flagged instance of every data record (if there are no flag columns, they will be created). Preview dialog (Quality report) is provided to let the user review the results. |
Save flags | This button is only enabled after running the quality check. It permanently saves generated flags into the dataset. QC status “AUTOMATIC_QUALITY_CHECK_IRRADIATION” will be assign to the dataset. |
Edit test groups
Edit test groups to change parameter assignment or add / remove test groups.
Test Groups are prefilled by default based on column order. In case default test groups do not correctly assign parameters, it is recommended to reorder columns via metadata editor and set default test groups. It is also possible to manually modify test groups.
Advanced settings
This dialog allows the user to control the test methodology and to select which automatic QC tests should be performed. Some of the tests have additional settings affecting their outcome.
Component | Description |
Test methodology | Solargis or BSRN methodology can be selected. |
List of tests | The user can select the automatic tests to be performed. |
Acceptable minima | For particular irradiance component the user can override Analyst default values of minimum physical value. |
Consecutive static values limits | User can override the default value of how many consecutive static values of the irradiance component will be considered as artificial (and flagged out from valid values). The limit can be set up separately for DIF values. There is a special treatment for DNI component as consecutive zero values can be valid (i.e., the test is performed for DNI values other than zero). |
Periods to be excluded | If checked, periods from the list will be excluded from GHI-DNI-DIF consistency test. |
list of excluded periods | Sometimes you want to exclude one or more time periods from checking the consistency of irradiance components because you know of the malfunction of one of the sensors (i.e., corrupted rotating shadow band). In that case, you do not want to exclude other irradiance components as they may be correct. |
Add period | Opens input dialog to add a time period to the list of excluded periods. |
Remove period | Remove selected time period from the list. |
Remove all periods | Clear the list of periods. |
Select flag values | The user can select the flag values that should be updated |
Advanced radiation check
This module brings semi-automatic tools for the recognition of several issues in data.
Tracker malfunction
The tracker malfunction issue occurs when the instrument for tracking sun position is not working properly. The duration of this malfunction can be very short (a few minutes), but it can also take several days. For the automatic detection you need to provide a GHI which serves as a reference. The DNI and DIF parameters can be checked. In case you do not have a DIF, set this parameter to “Do not use“.
After clicking on Auto detection, the model recognizes tracker malfunction and creates a mask to save. The mask is highlighted in green color. This module is to be run after Time reference control, basic Solar radiation check and Shading detection are done.
The user can add to the mask by selecting data and clicking "Add to mask". Similarly, the user can remove from the mask, by selecting data and clicking on "Remove from mask". The selected data have pink color. This detection can run from a few seconds up to a few minutes based on the volume of data.
Component | Description |
Bottom plot | Area where data are visualized as time series plot. |
Top plot | A plot of the selected column as a function of day and time. Mask and selected candidates are highlighted in green and pink color, respectively. |
Toolbar - Interactive navigation | Navigation toolbar, which can be used to navigate through the data set. |
GHI | The global horizontal radiation used for the tracker malfunction detection. The GHI is used as a reference to check the tracker malfunction on DIF and DNI. |
DNI | The direct normal radiation to be checked for the tracker malfunction. |
DIF | The diffuse radiation. If there is no measured DIF or you do not wish to use it for detection, then its value can be set to "Do not use". |
Model | Issue detection to run, in this case, "Tracker issue". |
Column in upper chart | Radiation to show in upper chart. |
Columns to show/flag | Checkbox selects the columns which are shown in the bottom plot. The small flags indicate which columns should be flagged. |
Load background data | Opens the dialog for loading data from other datasets currently loaded in the project. |
Selectable flags | Displays value, color, and description of flags for the given dataset as loaded from dataset's flag scheme. User can select via checkbox which flags are selectable by given detection - which flags the detection can change. |
Undo | undo the last action. |
Deselect | Deselect selected data. |
Add to mask | Add selected data to mask. |
Remove from mask | Remove selected data from the mask. |
Overview | Statistics for a selected column. |
Run model | Runs the automatic detection of the tracker malfunction issue. |
Quality report | Open Quality report dialog. |
Save | Save user changes to data. |
Limitations
Limitation | Probability | Severity |
The partial tracker issue (when the instrument "sees" part of the sun) cannot be fully flagged. The flagging needs to be manually completed. | Low | High |
Tracker malfunction identification can sometimes flag data with other issues e.g. shading. | Medium | Low |
Cloudy weather in low sun elevations can be misidentified as tracker malfunction. | Low | Medium |
Meteo check
Automatic quality control of meteorological parameters allows the user to run the automatic tests used today in Solargis to quality check the meteorological parameters. The automatic tests (checks of invalid values, consecutive static values, data below/above physical minima/maxima) assign a predefined flag to each record that failed in the test. The user has some control over this process (i.e. disable/enable tests, change test properties). Once the tests are performed a preview of the Quality report summary can be displayed. At the end of this process, the generated flags are saved with the data and quality status.
There are three main tabs in this window:
Heatmap flag plot. The Heatmap flag plot depicts the occurrence of quality flags from Meteo parameters QC in time.
Time series flag plot. Allows the basic data visualization of the quality control flags in time.
Quality control summary. The Quality control summary resumes the overview of the statistics after the quality control.
Component | Description |
Graph tabs | The user can choose a plot type to be displayed. |
Dataset selection | Selected dataset for the quality check |
Meteo parameters columns | Select columns of meteo parameters data type you want to quality check. Flags will be calculated into respective flag columns. |
Advanced settings | Opens separated dialog with automatic quality check advanced settings. |
run between | Quality check can be performed on a subset of the dataset. If checked, values from both calendar widgets are taken into account. |
date from, date to | Define a subset of the dataset by selecting two dates. |
Graphs | The Time series graphs including the QC flags are displayed. |
Plot settings | The user can select the meteo parameter to be displayed. Plot settings menu is active only for these Graph tabs: Heatmap flag plot, Time series flag plot. |
split heatmap by year | If checked, the heatmap is split by years. This checkbox is active only for Heatmap flag plot. |
view in time zone | The time zone can be selected. |
Run | Will start the process of automatic quality check. The result of the procedure is flagging for every record of data (if there are no flag columns, they will be created). Preview dialog (Quality report) is provided to let users review the results. |
Save flags | This button is only enabled after running the quality check. It permanently saves generated flags into the dataset. |
Advanced settings
This dialog allows users to select Solargis test methodology and to select which automatic QC tests should be performed. Some of the tests have additional settings affecting their outcome.
Component | Description |
|---|---|
Test methodology | Solargis methodology can be selected. |
Model data | Model dataset can be selected and used during automatic quality control |
Meteo parameter | User is able to parametrize automatic quality control for each parameter type. |
List of tests | The user can select the automatic tests to be performed. |
List of test settings | For a particular meteo parameter, users can override Analyst default values. |
Select flag values | The user can select the flag values that should be updated |
Default settings | Reset all parameters to default values. |
Interactive QC
After automatic quality check, the user can continue with a visual inspection of already flagged data values in the Interactive QC. Flagged data values can be explored in a plot, selected and assigned with the new flag value. The entire flag scheme of the dataset can be modified this way. At the end of the process, flag values are saved with the data and the quality status is updated.
This tool enables to perform a manual quality assessment for the selected dataset by manually selecting data by cursor or expression and flagging them.
Component | Description |
Graph | Area where data are visualized as time series plot. |
Toolbar | See toolbar |
Columns to show | Columns of dataset shown in tree form. By checking user can visualize individual columns. Users can control the visibility/flagging of columns by clicking on the flag icon. |
Add data | Opens the dialog for loading data from other datasets currently loaded in the project. |
Selectable flags | Displays color, description and value of flags for the given dataset as loaded from dataset's flag scheme. Users can temporarily change color of flag by clicking on the color box. If checked, the flag will be included in cursor/expression selection. |
Apply flags | Clicking on the flag next to Apply flags opens table widget dialog with flag scheme |
Flag value | Number that user wants to use as a flag. If a flag value doesn't exist, a new record will be created in the flag scheme. |
Set | Apply value from Flag value to the current selection. |
Undo | Undo last flagging |
Advanced | Advanced options. See details below |
Quality report | Opens Quality report dialog. |
Close | Close Manual QA window. |
Save | Save user changes to data, but dataset is not marked as manually checked. Users can save the last position where ended flagging to automatically continue from that place when opening manual control next time. |
Final save | Final save when operator flagged everything and dataset will be marked as manually checked. |
Advanced settings
Component | Description |
Maintenance log | Show hide vertical lines representing maintenance events (if available). The dotted vertical line represents maintenance events without exact time, the dashed line represents exact time. |
Flag style | User is able to set basic parameters - size, opacity and marker style of flag. These settings are automatically previewed on the graph. These settings last only for the current session and after closing dialog is reset. |
Advanced interactive QC
In addition to Interactive QC dialog, this tool enables to visualize data in form of heatmap and sun position plot, which allows detecting specific data problems such as shading.
Component | Description |
Graphs | Area where data are visualized as time series, sun position and heatmap plot |
Toolbar | See toolbar |
Column to show | Column of dataset that is plotted in upper graphs of graph area |
Data type | If the column to show is of the type of GHI or DNI, the user can visualize this data in Kt space |
Columns to show | Columns of dataset shown in tree form. By checking user can visualize individual columns. Users can control visibility/flagging of columns by clicking on the flag icon. |
Add data | Opens the dialog for loading data from other datasets currently loaded in the project. |
Selectable flags | Displays color, description and value of flags for the given dataset as loaded from dataset's flag scheme. Users can temporarily change the color of the flag by clicking on the color box. If checked, the flag will be included in cursor/expression selection. |
Apply flags | Clicking on the flag next to Apply flags opens table widget dialog with flag scheme |
Flag value | Number that user wants to use as a flag. If a flag value doesn't exist, a new record will be created in the flag scheme. |
Set | Apply value from Flag value to the current selection. |
Undo | Undo last flagging |
Advanced | Advanced options. See details here |
Filter | Filter data by min/max date. Useful when working with large, multi-year datasets. |
Quality report | Opens Quality report dialog. |
Close | Close Manual QA window. |
Save | Save user changes to data, but dataset is not marked as manually checked. Users can save the last position where flagging ended to automatically continue from that place when opening manual control next time. |
Final save | Final save, when operator flagged everything and dataset will be marked as manually checked. QC status “MANUAL_QUALITY_CHECK” will be assign to the dataset. |
Post-filtering
After irradiance quality check, it is possible that some scattered valid flags are left in the dataset. These individual flags often don't provide much value, and we want to remove them. Automatic post-filtering provides the user the possibility to remove such flags.
Component | Description |
Heatmap flag plot | Area where flagged data are visualized as series of yearly heatmaps or one big heatmap plot showing the overall data picture, with post-filtered data highlighted. |
Time series flag plot | Visualization of irradiance flagged data as time series and bar plot. One plot per one irrad. component. Post-filtered data are highlighted. |
Quality control tab | Text summary of flags for the given dataset. |
Dataset | Dataset on which to perform post-filtering control. |
Irradiation columns | List of irradiation columns on which to perform post-filtering. |
Central hours of the day | Minimum required size of a continuous group of values. For example, if value is 3, all groups of flags with size 3 or less will be filtered. |
Morning hours | Minimum required size of a continuous group of morning values. For example, if value is 3, all groups of flags with size 3 or less will be filtered. |
Evening hours | Minimum required size of a continuous group of evening values. For example, if value is 3, all groups of flags with size 3 or less will be filtered. |
Minimum required valid flags | Percentage of daytime records of valid flags, that will be filtered out. |
Run between | If checked, run post-filtering control between custom dates. |
Plot settings | Column to plot in graphs. |
Split heatmap by year | If checked, plots are split by years. |
View in time zone | If checked, data are shown in UTC chosen. |
Run | Will start the process of postfiltering. The result of the procedure is flagging for every record of data. Preview dialog (Quality report) is provided to let users review the results. |
Save flags | Save flags and QC status “POST_FILTERING_CHECK” will be assign to the dataset. |
Cancel | Close Postfiltering window. |