Drivers¶
This is the core module of package pyroSAR.
It contains the drivers for the different SAR image formats and offers
functionality for retrieving metadata, unpacking images, downloading ancillary files like DEMs and
Orbit State Vector files as well as archiving scenes in a database.
The ID
class and its subclasses allow easy and standardized access to the metadata of
images from different SAR sensors.
classes
Abstract class for SAR meta data handlers |
|
Handler class for BEAM-DIMAP data |
|
Handler class for ALOS-PALSAR data in CEOS format |
|
Handler class for ERS data in CEOS format |
|
Handler class for ALOS-2/PALSAR-2 data in EORC (Earth Observation Research Center) Path format |
|
Handler class for SAR data in ESA format (Envisat ASAR, ERS-1/2) |
|
Handler class for Sentinel-1 data |
|
Handler class for TerraSAR-X and TanDEM-X data |
|
Utility for storing SAR image metadata in a database |
functions
identify a SAR scene and return the appropriate metadata handler object |
|
wrapper function for returning metadata handlers of all valid scenes in a list, similar to function |
|
Filter a list of pyroSAR objects to those that have not yet been processed and stored in the defined directory. |
|
Load a file in a SAR scene archive into a readable file object. |
|
this function gathers known time formats provided in the different SAR products and converts them to a common standard of the form YYYYMMDDTHHMMSS |
|
drop (delete) a scene database |
- class pyroSAR.drivers.Archive(dbfile, custom_fields=None, postgres=False, user='postgres', password='1234', host='localhost', port=5432, cleanup=True)[source]¶
Bases:
object
Utility for storing SAR image metadata in a database
- Parameters
dbfile (str) – the filename for the SpatiaLite database. This might either point to an existing database or will be created otherwise. If postgres is set to True, this will be the name for the PostgreSQL database.
custom_fields (dict) – a dictionary containing additional non-standard database column names and data types; the names must be attributes of the SAR scenes to be inserted (i.e. id.attr) or keys in their meta attribute (i.e. id.meta[‘attr’])
postgres (bool) – enable postgres driver for the database. Default: False
user (str) – required for postgres driver: username to access the database. Default: ‘postgres’
password (str) – required for postgres driver: password to access the database. Default: ‘1234’
host (str) – required for postgres driver: host where the database is hosted. Default: ‘localhost’
port (int) – required for postgres driver: port number to the database. Default: 5432
cleanup (bool) – check whether all registered scenes exist and remove missing entries?
Examples
Ingest all Sentinel-1 scenes in a directory and its sub-directories into the database:
>>> from pyroSAR import Archive, identify >>> from spatialist.ancillary import finder >>> dbfile = '/.../scenelist.db' >>> archive_s1 = '/.../sentinel1/GRD' >>> scenes_s1 = finder(archive_s1, [r'^S1[AB].*\.zip'], regex=True, recursive=True) >>> with Archive(dbfile) as archive: >>> archive.insert(scenes_s1)
select all Sentinel-1 A/B scenes stored in the database, which
overlap with a test site
were acquired in Ground-Range-Detected (GRD) Interferometric Wide Swath (IW) mode before 2018
contain a VV polarization image
have not been processed to directory outdir before
>>> from pyroSAR import Archive >>> from spatialist import Vector >>> archive = Archive('/.../scenelist.db') >>> site = Vector('/path/to/site.shp') >>> outdir = '/path/to/processed/results' >>> maxdate = '20171231T235959' >>> selection_proc = archive.select(vectorobject=site, processdir=outdir, >>> maxdate=maxdate, sensor=('S1A', 'S1B'), >>> product='GRD', acquisition_mode='IW', vv=1) >>> archive.close()
Alternatively, the with statement can be used. In this case to just check whether one particular scene is already registered in the database:
>>> from pyroSAR import identify, Archive >>> scene = identify('S1A_IW_SLC__1SDV_20150330T170734_20150330T170801_005264_006A6C_DA69.zip') >>> with Archive('/.../scenelist.db') as archive: >>> print(archive.is_registered(scene.scene))
When providing ‘postgres’ as driver, a PostgreSQL database will be created at a given host. Additional arguments are required.
>>> from pyroSAR import Archive, identify >>> from spatialist.ancillary import finder >>> dbfile = 'scenelist_db' >>> archive_s1 = '/.../sentinel1/GRD' >>> scenes_s1 = finder(archive_s1, [r'^S1[AB].*\.zip'], regex=True, recursive=True) >>> with Archive(dbfile, driver='postgres', user='user', password='password', host='host', port=5432) as archive: >>> archive.insert(scenes_s1)
- add_tables(tables)[source]¶
Add tables to the database per
sqlalchemy.schema.Table
Tables provided here will be added to the database.Note
Columns using Geometry must have setting management=True for SQLite, for example:
bbox = Column(Geometry('POLYGON', management=True, srid=4326))
- Parameters
tables (
sqlalchemy.schema.Table
orlist
ofsqlalchemy.schema.Table
) – The table(s) to be added to the database.
- cleanup()[source]¶
Remove all scenes from the database, which are no longer stored in their registered location
- drop_element(scene, with_duplicates=False)[source]¶
Drop a scene from the data table. If duplicates table contains matching entry, it will be moved to the data table.
- export2shp(path, table='data')[source]¶
export the database to a shapefile
- Parameters
path (str) – the path of the shapefile to be written. This will overwrite other files with the same name. If a folder is given in path it is created if not existing. If the file extension is missing ‘.shp’ is added.
table (str) – the table to write to the shapefile; either ‘data’ (default) or ‘duplicates’
- filter_scenelist(scenelist)[source]¶
Filter a list of scenes by file names already registered in the database.
- Parameters
scenelist (
list
ofstr
orpyroSAR.drivers.ID
) – the scenes to be filtered- Returns
the file names of the scenes whose basename is not yet registered in the database
- Return type
- get_colnames(table='data')[source]¶
Return the names of all columns of a table.
- Returns
the column names of the chosen table
- Return type
- get_unique_directories()[source]¶
Get a list of directories containing registered scenes
- Returns
the directory names
- Return type
- import_outdated(dbfile)[source]¶
import an older data base in csv format
- Parameters
dbfile (str) – the file name of the old data base
- move(scenelist, directory, pbar=False)[source]¶
Move a list of files while keeping the database entries up to date. If a scene is registered in the database (in either the data or duplicates table), the scene entry is directly changed to the new location.
- select(vectorobject=None, mindate=None, maxdate=None, processdir=None, recursive=False, polarizations=None, **args)[source]¶
select scenes from the database
- Parameters
vectorobject (
Vector
) – a geometry with which the scenes need to overlapmindate (str or datetime.datetime, optional) – the minimum acquisition date; strings must be in format YYYYmmddTHHMMSS; default: None
maxdate (str or datetime.datetime, optional) – the maximum acquisition date; strings must be in format YYYYmmddTHHMMSS; default: None
processdir (str, optional) – A directory to be scanned for already processed scenes; the selected scenes will be filtered to those that have not yet been processed. Default: None
recursive (bool) – (only if processdir is not None) should also the subdirectories of the processdir be scanned?
polarizations (list) – a list of polarization strings, e.g. [‘HH’, ‘VV’]
**args – any further arguments (columns), which are registered in the database. See
get_colnames()
- Returns
the file names pointing to the selected scenes
- Return type
- class pyroSAR.drivers.BEAM_DIMAP(scene)[source]¶
Bases:
ID
Handler class for BEAM-DIMAP data
- Sensors:
SNAP supported sensors
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.CEOS_ERS(scene)[source]¶
Bases:
ID
Handler class for ERS data in CEOS format
- Sensors:
ERS1
ERS2
- Reference:
ER-IS-EPO-GS-5902-3: Annex C. ERS SAR.SLC/SLC-I. CCT and EXABYTE (ESA 1998)
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.CEOS_PSR(scene)[source]¶
Bases:
ID
Handler class for ALOS-PALSAR data in CEOS format
- Sensors:
PSR1
PSR2
- PALSAR-1:
- References:
- Products / processing levels:
1.0
1.1
1.5
- Acquisition modes:
AB: [SP][HWDPC]
- A: supplemental remarks of the sensor type:
S: Wide observation mode
P: all other modes
- B: observation mode
H: Fine mode
W: ScanSAR mode
D: Direct downlink mode
P: Polarimetry mode
C: Calibration mode
- PALSAR-2:
- Reference:
ALOS-2/PALSAR-2 Level 1.1/1.5/2.1/3.1 CEOS SAR Product Format Description (JAXA 2014).
- Products / processing levels:
1.0
1.1
1.5
- Acquisition modes:
SBS: Spotlight mode
UBS: Ultra-fine mode Single polarization
UBD: Ultra-fine mode Dual polarization
HBS: High-sensitive mode Single polarization
HBD: High-sensitive mode Dual polarization
HBQ: High-sensitive mode Full (Quad.) polarimetry
FBS: Fine mode Single polarization
FBD: Fine mode Dual polarization
FBQ: Fine mode Full (Quad.) polarimetry
WBS: Scan SAR nominal [14MHz] mode Single polarization
WBD: Scan SAR nominal [14MHz] mode Dual polarization
WWS: Scan SAR nominal [28MHz] mode Single polarization
WWD: Scan SAR nominal [28MHz] mode Dual polarization
VBS: Scan SAR wide mode Single polarization
VBD: Scan SAR wide mode Dual polarization
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- property led_filename¶
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.EORC_PSR(scene)[source]¶
Bases:
ID
Handler class for ALOS-2/PALSAR-2 data in EORC (Earth Observation Research Center) Path format
- Sensors:
PALSAR-2
- PALSAR-2:
- Reference:
NDX-150019: ALOS-2/PALSAR-2 EORC Path Product Format Description (JAXA 2016)
- Products / processing levels:
1.5
- Acquisition modes:
FBD: Fine mode Dual polarization
WBD: Scan SAR nominal [14MHz] mode Dual polarization
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- property header_filename¶
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.ESA(scene)[source]¶
Bases:
ID
Handler class for SAR data in ESA format (Envisat ASAR, ERS-1/2)
- Sensors:
ASAR
ERS1
ERS2
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.ID(metadict)[source]¶
Bases:
object
Abstract class for SAR meta data handlers
- bbox(outname=None, driver=None, overwrite=True)[source]¶
get the bounding box of a scene either as a vector object or written to a file
- Parameters
- Returns
the vector object if outname is None, None otherwise
- Return type
Vector or None
- property compression¶
check whether a scene is compressed into an tarfile or zipfile or not at all
- Returns
either ‘zip’, ‘tar’ or None
- Return type
str or None
- examine(include_folders=False)[source]¶
check whether any items in the SAR scene structure (i.e. files/folders) match the regular expression pattern defined by the class. On success the item is registered in the object as attribute file.
- Parameters
include_folders (bool) – also match folder (or just files)?
- Raises
- export2dict()[source]¶
Return the uuid and the metadata that is defined in self.locals as a dictionary
- export2sqlite(dbfile)[source]¶
Export relevant metadata to an SQLite database
- Parameters
dbfile (str) – the database file
- findfiles(pattern, include_folders=False)[source]¶
find files in the scene archive, which match a pattern.
- Parameters
- Returns
the matched file names
- Return type
See also
- gdalinfo()[source]¶
read metadata directly from the GDAL SAR image drivers
- Returns
the metadata attributes
- Return type
- geometry(outname=None, driver=None, overwrite=True)[source]¶
get the footprint geometry of a scene either as a vector object or written to a file
- Parameters
- Returns
the vector object if outname is None, None otherwise
- Return type
Vector or None
- abstract getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- getFileObj(filename)[source]¶
Load a file into a readable file object.
- Parameters
filename (str) – the name of a file in the scene archive, easiest to get with method
findfiles()
- Returns
a file pointer object
- Return type
- getGammaImages(directory=None)[source]¶
list all files processed by GAMMA
- Parameters
directory (str or None) – the directory to be scanned; if left empty the object attribute gammadir is scanned
- Returns
the file names of the images processed by GAMMA
- Return type
- Raises
- getHGT()[source]¶
get the names of all SRTM HGT tiles overlapping with the SAR scene
- Returns
names of the SRTM HGT tiles
- Return type
- is_processed(outdir, recursive=False)[source]¶
check whether a scene has already been processed and stored in the defined output directory (and subdirectories if scanned recursively)
- outname_base(extensions=None)[source]¶
parse a string containing basic information about the scene in standardized format. Currently this id contains the sensor (4 digits), acquisition mode (4 digits), orbit (1 digit) and acquisition start time (15 digits)., e.g. S1A__IW___A_20150523T122350
- static parse_date(x)[source]¶
this function gathers known time formats provided in the different SAR products and converts them to a common standard of the form YYYYMMDDTHHMMSS.
- abstract quicklook(outname, format='kmz')[source]¶
export a quick look image of the scene
- Parameters
Examples
>>> from pyroSAR import identify >>> scene = identify('S1A_IW_GRDH_1SDV_20180101T170648_20180101T170713_019964_021FFD_DA78.zip') >>> scene.quicklook('S1A__IW___A_20180101T170648.kmz')
- abstract scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.SAFE(scene)[source]¶
Bases:
ID
Handler class for Sentinel-1 data
- Sensors:
S1A
S1B
- References:
S1-RS-MDA-52-7443 Sentinel-1 IPF Auxiliary Product Specification
MPC-0243 Masking “No-value” Pixels on GRD Products generated by the Sentinel-1 ESA IPF
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- getOSV(osvdir=None, osvType='POE', returnMatch=False, useLocal=True, timeout=300, url_option=1)[source]¶
download Orbit State Vector files for the scene
- Parameters
osvdir (str) – the directory of OSV files; subdirectories POEORB and RESORB are created automatically; if no directory is defined, the standard SNAP auxdata location is used
osvType (str or list) – the type of orbit file either ‘POE’, ‘RES’ or a list of both; if both are selected, the best matching file will be retrieved. I.e., POE if available and RES otherwise
returnMatch (bool) – return the best matching orbit file?
useLocal (bool) – use locally existing files and do not search for files online if the right file has been found?
timeout (int or tuple or None) – the timeout in seconds for downloading OSV files as provided to
requests.get()
url_option (int) –
the URL to query for scenes
- Returns
the best matching OSV file if returnMatch is True or None otherwise
- Return type
str or None
See also
- quicklook(outname, format='kmz', na_transparent=True)[source]¶
Write a quicklook file for the scene.
- removeGRDBorderNoise(method='pyroSAR')[source]¶
mask out Sentinel-1 image border noise.
- Parameters
method (str) –
the border noise removal method to be applied; one of the following:
’ESA’: the pure implementation as described by ESA
’pyroSAR’: the ESA method plus the custom pyroSAR refinement
See also
- resolution()[source]¶
Compute the mid-swath resolution of the Sentinel-1 product. For GRD products the resolution is expressed in ground range and in slant range otherwise.
- References:
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.TDM(scene)[source]¶
Bases:
TSX
Handler class for TerraSAR-X and TanDEM-X experimental data
- Sensors:
TDM1
- References:
TD-GS-PS-3028 TanDEM-X Experimental Product Description
- Acquisition modes:
HS: High Resolution SpotLight
SL: SpotLight
SM: StripMap
- Polarisation modes:
Single (S): all acquisition modes
Dual (D): High Resolution SpotLight (HS), SpotLight (SL) and StripMap (SM)
Twin (T): StripMap (SM) (experimental)
Quad (Q): StripMap (SM) (experimental)
- Products:
CoSSCs: (bi-static) SAR co-registered single look slant range complex products (CoSSCs)
Examples
Ingest all Tandem-X Bistatic scenes in a directory and its sub-directories into the database:
>>> from pyroSAR import Archive, identify >>> from spatialist.ancillary import finder >>> dbfile = '/.../scenelist.db' >>> archive_tdm = '/.../TDM/' >>> scenes_tdm = finder(archive_tdm, [r'^TDM1.*'], foldermode=2, regex=True, recursive=True) >>> with Archive(dbfile) as archive: >>> archive.insert(scenes_tdm)
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- class pyroSAR.drivers.TSX(scene)[source]¶
Bases:
ID
Handler class for TerraSAR-X and TanDEM-X data
- Sensors:
TSX1
TDX1
- References:
TX-GS-DD-3302 TerraSAR-X Basic Product Specification Document
TX-GS-DD-3303 TerraSAR-X Experimental Product Description
TD-GS-PS-3028 TanDEM-X Experimental Product Description
TerraSAR-X Image Product Guide (Airbus Defence and Space)
- Acquisition modes:
ST: Staring Spotlight
HS: High Resolution SpotLight
HS300: High Resolution SpotLight 300 MHz
SL: SpotLight
SM: StripMap
SC: ScanSAR
WS: Wide ScanSAR
- Polarisation modes:
Single (S): all acquisition modes
Dual (D): High Resolution SpotLight (HS), SpotLight (SL) and StripMap (SM)
Twin (T): StripMap (SM) (experimental)
Quad (Q): StripMap (SM) (experimental)
- Products:
SSC: Single Look Slant Range Complex
MGD: Multi Look Ground Range Detected
GEC: Geocoded Ellipsoid Corrected
EEC: Enhanced Ellipsoid Corrected
- getCorners()[source]¶
derive the corner coordinates from a SAR scene
- Returns
dictionary with keys xmin, xmax, ymin and ymax
- Return type
- scanMetadata()[source]¶
scan SAR scenes for metadata attributes. The returned dictionary is registered as attribute meta by the class upon object initialization. This dictionary furthermore needs to return a set of standardized attribute keys, which are directly registered as object attributes.
- Returns
the derived attributes
- Return type
- pyroSAR.drivers.drop_archive(archive)[source]¶
drop (delete) a scene database
- Parameters
archive (pyroSAR.drivers.Archive) – the database to be deleted
Examples
>>> pguser = os.environ.get('PGUSER') >>> pgpassword = os.environ.get('PGPASSWORD')
>>> db = Archive('test', postgres=True, port=5432, user=pguser, password=pgpassword) >>> drop_archive(db)
- pyroSAR.drivers.filter_processed(scenelist, outdir, recursive=False)[source]¶
Filter a list of pyroSAR objects to those that have not yet been processed and stored in the defined directory. The search for processed scenes is either done in the directory only or recursively into subdirectories. The scenes must have been processed with pyroSAR in order to follow the right naming scheme.
- pyroSAR.drivers.getFileObj(scene, filename)[source]¶
Load a file in a SAR scene archive into a readable file object.
- Parameters
scene (str) – the scene archive. Can be either a directory or a compressed archive of type zip or tar.gz.
filename (str) – the name of a file in the scene archive, easiest to get with method
findfiles()
- Returns
a file object
- Return type
- pyroSAR.drivers.identify(scene)[source]¶
identify a SAR scene and return the appropriate metadata handler object
- Parameters
scene (str) – a file or directory name
- Returns
a pyroSAR metadata handler
- Return type
a subclass object of
ID
Examples
>>> from pyroSAR import identify >>> filename = 'S1A_IW_GRDH_1SDV_20180829T170656_20180829T170721_023464_028DE0_F7BD.zip' >>> scene = identify(filename) >>> print(scene) pyroSAR ID object of type SAFE acquisition_mode: IW cycleNumber: 148 frameNumber: 167392 lines: 16703 orbit: A orbitNumber_abs: 23464 orbitNumber_rel: 117 polarizations: ['VV', 'VH'] product: GRD projection: +proj=longlat +datum=WGS84 +no_defs samples: 26056 sensor: S1A spacing: (10.0, 10.0) start: 20180829T170656 stop: 20180829T170721
- pyroSAR.drivers.identify_many(scenes, pbar=False, sortkey=None)[source]¶
wrapper function for returning metadata handlers of all valid scenes in a list, similar to function
identify()
.- Parameters
- Returns
a list of pyroSAR metadata handlers
- Return type
Examples
>>> from pyroSAR import identify_many >>> files = finder('/path', ['S1*.zip']) >>> ids = identify_many(files, pbar=False, sortkey='start')