vignettes/sos4R-vignette-04-extensions.Rmd
sos4R-vignette-04-extensions.RmdAbstract
The sos4R package provides simple yet powerful access to OGC Sensor Observation Service instances. The package supports both encapsulation and abstraction from the service interface for novice users as well as powerful request building for specialists. sos4R is motivated by the idea to close the gap between the Sensor Web and tools for (geo-)statistical analyses.
This document shows how to do advanced configurations, such as adding data parsing functions or implementing & registering functions for encoding and decoding.
The package is published under GPL 2 license within the geostatistics community of 52°North Initiative for Geospatial Open Source Software.
library("sos4R") mySOS <- SOS(url = "http://sensorweb.demo.52north.org/52n-sos-webapp/service/pox", binding = "POX", dcpFilter = list("POX" = "/pox"))
The flexibility of the specifications that model the markup requests and responses, especially the observation encoding, is too high to handle all possible cases within sos4R. Thus an equally flexible mechanism for users to adopt the steps of encoding and decoding documents to their needs is needed.
The process of data download comprises
This can be seen as a fixed, ordered workflow a user has to follow where each step build upon the input of the previous. To ensure flexibility within these steps of the workflow but also to maximize reusability of existing functionality, a mechanism to exchange the functions that are used in these steps is provided.
Step 1, the building of requests, is the assembly of the request parameters passed to the sos4R functions into an R object. It is documented in section GetObservation. Step 3, the sending receiving of documents to respectively from a service, does not need to be changed directly but the user, who only can configure the binding.
The remainder of this document explains how to configure steps 2 (encoding), 4 (decoding) and 5 (data parsing) of the process.
The functions used in the exchangeable steps are organized in lists. To base your own list of functions on the existing ones, thereby not having to start from scratch, you can combine the default list of functions with your own. Use the following functions:
To add your own function, simply add it as a named argument. You can add as many as you like in the ... parameter. If a function with that identifier already exists in the default list it will be replaced by your function. For further adjustments you can explicitly include and exclude functions by identifier. Please be aware that inclusion is applied first, then exclusion. It is also important that you also have to include that functions you just added manually!
Examples of function list generation with parsing functions:
parsers <- SosParsingFunctions( "ExceptionReport" = function() { return("Got Exception!") }, include = c("GetObservation", "ExceptionReport")) print(names(parsers))
## [1] "GetObservation" "ExceptionReport"parsers <- SosParsingFunctions( "ExceptionReport" = function() { return("Got Exception!") }, include = c("GetCapabilities")) print(names(parsers))
## [1] "GetCapabilities"The following snipped shows how to remove a large part of parsers using exclude and then prints the names of the remaining ones.
parsers <- SosParsingFunctions( exclude = names(SosParsingFunctions())[5:29]) print(names(parsers))
## [1] "GetCapabilities" "ows:ExceptionReport"
## [3] "DescribeSensor" "DescribeSensorResponse"
## [5] "wml2:MeasurementTimeseries" "text/csv"
## [7] "text/xml;subtype=\"om/1.0.0\"" "text/xml"The current list of a connection’s encoders can be accessed with
sosEncoders(mySOS)
A complete list of the existing encoders names:
names(sosEncoders(mySOS))
## [1] "SOAP" "POX" "KVP"Here the idea of organizing the encoding functions becomes clear: One base encoding function is given, which is a generic method that must exist for all elements that need to be encoded.
myPostEncoding <- function(object, sos, verbose) { return(utils::str(object)) } # Connection will not be established because of mising objects mySOS2 = SOS(sosUrl(mySOS), encoders = SosEncodingFunctions("POST" = myPostEncoding))
Encoding functions can be overridden for many specific objects. The signature of the encoding function consists of the object, obj, a SOS object, sos, and the optional verbose parameter.
showMethods("encodeXML")
## Function: encodeXML (package sos4R)
## obj="character", sos="SOS"
## obj="GmlDirectPosition", sos="SOS"
## obj="GmlEnvelope", sos="SOS"
## obj="GmlLineString", sos="SOS"
## obj="GmlPoint", sos="SOS"
## obj="GmlPointProperty", sos="SOS"
## obj="GmlPolygon", sos="SOS"
## obj="GmlTimeInstant", sos="SOS"
## obj="GmlTimeInstantProperty", sos="SOS"
## obj="GmlTimePeriod", sos="SOS"
## obj="GmlTimePosition", sos="SOS"
## obj="OgcBBOX", sos="SOS"
## obj="OgcComparisonOps", sos="SOS"
## obj="OgcContains", sos="SOS"
## obj="OgcIntersects", sos="SOS"
## obj="OgcOverlaps", sos="SOS"
## obj="POSIXt", sos="SOS"
## obj="SosEventTime", sos="SOS"
## obj="SosFeatureOfInterest", sos="SOS"
## obj="TM_After", sos="SOS"
## obj="TM_Before", sos="SOS"
## obj="TM_During", sos="SOS"
## obj="TM_Equals", sos="SOS"
## obj="xml_document", sos="SOS"
## obj="xml_node", sos="SOS"showMethods("encodeKVP")
## Function: encodeKVP (package sos4R)
## obj="GmlTimeInstant", sos="SOS"
## obj="GmlTimePeriod", sos="SOS"
## obj="GmlTimePosition", sos="SOS"
## obj="OgcBinaryTemporalOp", sos="SOS"
## obj="POSIXt", sos="SOS"
## obj="SosEventTime", sos="SOS"A useful example can be overriding the encoding method for time classes (POSIXt) as presented below – see the demo southesk for the application of this code.
setMethod(f = "encodeXML", signature = signature(obj = "POSIXt", sos = "SOS"), def = function(obj, sos, verbose) { if(verbose) cat("Using my own time encoding... ") # time zone hack to fix that the time format option # %z does not work on windows machines: .time <- obj + 11 * 60 * 60 # add 11 hours formatted <- strftime(x = .time, format = sosTimeFormat(sos)) formatted <- paste(formatted, "+11:00", sep = "") # append 11:00 if(verbose) cat("Formatted ", toString(obj), " to ", formatted, "\n") return(formatted) } )
All later calls for encoding any classes with time will then reference this newly defined method. Be aware that this changes the encoding globally, in contrast to converters and parsers which can be changed for every instance of class SOS.
The terms parsing and decoding are used as synonyms for the process of processing an XML document to create an R object. XML documents are made out of hierarchical elements. That is why the decoding functions are organized in a listed, whose names are the XML elements’ names it parses.
The current list of a connection’s parsers can be accessed with the following function.
sosParsers(mySOS)
A complete list of the elements with existing encoders is shown below. These are not only names of XML elements, but also MIME types. Here the idea of organizing the encoding functions becomes clear: For every XML element or document type that must be parsed there is a function given in the list.
names(sosParsers(mySOS))
## [1] "GetCapabilities" "ows:ExceptionReport"
## [3] "DescribeSensor" "DescribeSensorResponse"
## [5] "GetObservation" "GetObservationById"
## [7] "gda:GetDataAvailabilityResponse" "GetFeatureOfInterestResponse"
## [9] "GetObservationByIdResponse" "GetObservationResponse"
## [11] "om:Measurement" "om:member"
## [13] "om:Observation" "om:ObservationCollection"
## [15] "om:result" "swe:DataArray"
## [17] "swe:elementType" "swe:encoding"
## [19] "swe:values" "swe:Position"
## [21] "swe:location" "swe:Vector"
## [23] "swe:coordinate" "om:GeometryObservation"
## [25] "om:CategoryObservation" "om:CountObservation"
## [27] "om:TruthObservation" "om:TemporalObservation"
## [29] "om:ComplexObservation" "wml2:MeasurementTimeseries"
## [31] "text/csv" "text/xml;subtype=\"om/1.0.0\""
## [33] "text/xml"Parser selection can also be based on the mimeType of the returned document. Please be aware that this also can be a problem if you want to exchange a parse by operation name, which is done after switching the function based on the mime type. In other words, the exchange by operation name only works if the response type is as expected.
If you want to replace only selected parsers use the include parameter as described above. You can also base your own parsing functions on a variety of existing parsing functions. For example you can replace the base function for om:ObservationCollection, element name ObservationCollection, but still use the parsing function for om:Observation within your own function if you include it in the parser list. The existing parsing functions are all named in the pattern parse<ElementName>(). Please be aware that some parsers require a parameter sos of class SOS upon which they might rely for example for formatting information, and some also have verbose (type logical). In case of “unused arguments” errors, please try different signatures.
# Create own parsing function: myER <- function(xml, sos, verbose) { return("EXCEPTION!!!11") } myParsers <- SosParsingFunctions("ows:ExceptionReport" = myER) exceptionParserSOS <- SOS(url = "http://sensorweb.demo.52north.org/52n-sos-webapp/service/pox", parsers = myParsers, binding = "POX", useDCPs = FALSE) # Triggers exception: erroneousResponse <- getObservation(exceptionParserSOS, #verbose = TRUE, offering = sosOfferings(exceptionParserSOS)[[1]], observedProperty = list("Bazinga!")) print(erroneousResponse)
## [1] "EXCEPTION!!!11"To disable all parsing, you can use the function SosDisabledParsers(). This effectively just “passes through”" all received data because the list returned by the function only contains the top-most parsing functions for SOS operations and exception reports.
## [1] "GetCapabilities" "DescribeSensor" "GetObservation"
## [4] "GetObservationById" "ows:ExceptionReport"This is also the recommended way to start if you want to set-up your own parsers (given you have responses in XML) and an alternative to debugging if you want to inspect responses directly.
The next example shows how the response (in this case the request is intentionally incorrect and triggers an exception) is passed through as an object of class xml_document:
disabledParserSOS <- SOS(sosUrl(mySOS), parsers = SosDisabledParsers(), binding = sosBinding(mySOS), dcpFilter = mySOS@dcpFilter) unparsed <- getObservation(disabledParserSOS, offering = sosOfferings(disabledParserSOS)[[1]], observedProperty = list("Bazinga!")) class(unparsed)
## [1] "xml_document" "xml_node"# (Using XML functions here for accesing the root of a # document and the name of an element.) unparsed
## {xml_document}
## <ExceptionReport version="1.0.0" schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsAll.xsd" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
## [1] <ows:Exception exceptionCode="InvalidParameterValue" locator="observedPro ...A list of named functions to be used by the parsing methods to convert data values to the correct R type, which are mostly based on the unit of measurement code.
The conversion functions always take two parameters: x is the object to be converted, sos is the service where the request was received from.
The available functions are basically wrappers for coercion functions, for example as.double(). The only method exploiting the second argument is the one for conversion of time stamps which uses the time format saved with the object of class SOS in a call to strptime.
value <- 2.0 value.string <- sosConvertString(x = value, sos = mySOS) print(class(value.string))
## [1] "character"value <- "2.0" value.double <- sosConvertDouble(x = value, sos = mySOS) print(class(value.double))
## [1] "numeric"value <- "1" value.logical <- sosConvertLogical(x = value, sos = mySOS) print(class(value.logical))
## [1] "logical"value <- "2010-01-01T12:00:00.000" value.time <- sosConvertTime(x = value, sos = mySOS) print(class(value.time))
## [1] "POSIXct" "POSIXt"The full list of currently supported units can be seen below. It mostly contains common numerical units which are converted to type double.
## [1] "fallBack"
## [2] "urn:ogc:data:time:iso8601"
## [3] "urn:ogc:property:time:iso8601"
## [4] "urn:ogc:phenomenon:time:iso8601"
## [5] "http://www.opengis.net/def/property/OGC/0/SamplingTime"
## [6] "http://www.opengis.net/def/property/OGC/0/PhenomenonTime"
## [7] "urn:ogc:def:parameter:x-istsos:1.0:time:iso8601"
## [8] "sos:time"
## [9] "m"
## [10] "m2"
## [11] "m3"
## [12] "m^3/s"
## [13] "s"
## [14] "ms"
## [15] "us"
## [16] "g"
## [17] "rad"
## [18] "K"
## [19] "C"
## [20] "cd"
## [21] "%"
## [22] "ppth"
## [23] "ppm"
## [24] "ppb"
## [25] "pptr"
## [26] "mol"
## [27] "sr"
## [28] "Hz"
## [29] "N"
## [30] "Pa"
## [31] "J"
## [32] "W"
## [33] "A"
## [34] "V"
## [35] "F"
## [36] "Ohm"
## [37] "S"
## [38] "Wb"
## [39] "Cel"
## [40] "T"
## [41] "H"
## [42] "lm"
## [43] "lx"
## [44] "Bq"
## [45] "Gy"
## [46] "Sv"
## [47] "gon"
## [48] "deg"
## [49] "'"
## [50] "''"
## [51] "l"
## [52] "L"
## [53] "ar"
## [54] "t"
## [55] "bar"
## [56] "u"
## [57] "eV"
## [58] "AU"
## [59] "pc"
## [60] "degF"
## [61] "hPa"
## [62] "mm"
## [63] "nm"
## [64] "cm"
## [65] "km"
## [66] "m/s"
## [67] "m2/s"
## [68] "m3/s"
## [69] "kg"
## [70] "mg"
## [71] "uom"
## [72] "urn:ogc:data:feature"
## [73] "http://www.opengis.net/def/property/OGC/0/FeatureOfInterest"
## [74] "ug/m3"
## [75] "http://www.opengis.net/def/uom/ISO-8601/0/Gregorian"
## [76] "degC"
## [77] "°C"
## [78] "http://www.opengis.net/def/property/OGC/0/PhenomenonTime"
## [79] "factor"
## [80] "numeric"
## [81] "integer"
## [82] "character"
## [83] "logical"
## [84] "POSIXct"The current list of a SOS connection’s converters can be accessed with
sosDataFieldConverters(mySOS)
The following connection shows a typical workflow of connecting to a new SOS for the first time, what the errors for missing converters look like, and how to add them to the SOS connection.
testsos <- SOS("http://sensorweb.demo.52north.org/52n-sos-webapp/sos/pox", binding = "POX", dcpFilter = list("POX" = "/pox")) testsosoffering <- sosOfferings(testsos)[["http___www.52north.org_test_offering_1"]] testsosobsprop <- sosObservedProperties(testsosoffering)[1] getObservation(sos = testsos, offering = testsosoffering, observedProperty = testsosobsprop)
## Object of class OmObservationCollection with 1 members.Looking at the raw response data gives us a hint at a suitable type.
getObservation(sos = testsos, offering = testsosoffering, observedProperty = testsosobsprop, inspect = TRUE)
## [.sosRequest_1.0.0] REQUEST:
## {xml_document}
## <GetObservation service="SOS" version="1.0.0" xmlns="http://www.opengis.net/sos/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:sos="http://www.opengis.net/sos/1.0" xmlns:om="http://www.opengis.net/om/1.0" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml">
## [1] <sos:offering>http___www.52north.org_test_offering_1</sos:offering>
## [2] <sos:observedProperty>http://www.52north.org/test/observableProperty/1</s ...
## [3] <sos:responseFormat>text/xml;subtype="om/1.0.0"</sos:responseFormat>
## [.getObservation_1.0.0] RESPONSE DOC:
## {xml_document}
## <ObservationCollection id="oc_1594227625460" schemaLocation="http://www.opengis.net/om/1.0 http://schemas.opengis.net/om/1.0.0/om.xsd http://www.opengis.net/sampling/1.0 http://schemas.opengis.net/sampling/1.0.0/sampling.xsd http://www.opengis.net/sos/1.0 http://schemas.opengis.net/sos/1.0.0/sosAll.xsd http://www.opengis.net/gml http://schemas.opengis.net/gml/3.1.1/base/gml.xsd http://www.opengis.net/swe/1.0.1 http://schemas.opengis.net/sweCommon/1.0.1/swe.xsd" xmlns:om="http://www.opengis.net/om/1.0" xmlns:gml="http://www.opengis.net/gml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:sa="http://www.opengis.net/sampling/1.0" xmlns:swe="http://www.opengis.net/swe/1.0.1">
## [1] <gml:boundedBy>\n <gml:Envelope srsName="urn:ogc:def:crs:EPSG::4326">\n ...
## [2] <om:member>\n <om:Observation gml:id="o_1594227625461">\n <om:samplin ...## Object of class OmObservationCollection with 1 members.Converters may be matched by XML properties swe:Quantity[@definition] or swe:uom[@code].
testconverters <- SosDataFieldConvertingFunctions( # one of the following would suffice "test_unit_1" = sosConvertDouble, "http://www.52north.org/test/observableProperty/1" = sosConvertDouble ) testsos <- SOS("http://sensorweb.demo.52north.org/52n-sos-webapp/sos/pox", binding = "POX", dcpFilter = list("POX" = "/pox"), dataFieldConverters = testconverters) testsosoffering <- sosOfferings(testsos)[["http___www.52north.org_test_offering_1"]] data <- getObservation(sos = testsos, offering = testsosoffering, observedProperty = testsosobsprop)
Then retrieve the data with the correct type.
## phenomenonTime
## o_1594227626270.1 2012-11-19 13:00:00
## o_1594227626270.2 2012-11-19 13:01:00
## o_1594227626270.3 2012-11-19 13:02:00
## o_1594227626270.4 2012-11-19 13:03:00
## o_1594227626270.5 2012-11-19 13:04:00
## o_1594227626270.6 2012-11-19 13:05:00
## http://www.52north.org/test/observableProperty/1
## o_1594227626270.1 1.0
## o_1594227626270.2 1.1
## o_1594227626270.3 1.2
## o_1594227626270.4 1.3
## o_1594227626270.5 1.4
## o_1594227626270.6 1.5The metadata of the observed property is also accessible.
attributes(sosResult(data)[[1]])
## $class
## [1] "POSIXct" "POSIXt"
##
## $tzone
## [1] "UTC"
##
## $original_value1
## [1] "2012-11-19T13:00:00.000Z"
##
## $original_value2
## [1] "2012-11-19T13:01:00.000Z"
##
## $original_value3
## [1] "2012-11-19T13:02:00.000Z"
##
## $original_value4
## [1] "2012-11-19T13:03:00.000Z"
##
## $original_value5
## [1] "2012-11-19T13:04:00.000Z"
##
## $original_value6
## [1] "2012-11-19T13:05:00.000Z"
##
## $original_value7
## [1] "2012-11-19T13:06:00.000Z"
##
## $original_value8
## [1] "2012-11-19T13:07:00.000Z"
##
## $original_value9
## [1] "2012-11-19T13:08:00.000Z"
##
## $original_value10
## [1] "2012-11-19T13:09:00.000Z"
##
## $name
## [1] "phenomenonTime"
##
## $definition
## [1] "http://www.opengis.net/def/property/OGC/0/PhenomenonTime"
##
## $`unit of measurement`
## [1] "http://www.opengis.net/def/uom/ISO-8601/0/Gregorian"
##
## $`R class of values`
## [1] "POSIXct"Warnings may also include messages if no converter is available for a unit of measurement, for example:
In .valParser(values = obj[[sweValuesName]], fields = .fields, ... :
No converter for the unit of measurement S/m with the definition http://mmisw.org/ont/cf/parameter/conductivity ! Trying a default, but you can add one when creating a SOS using SosDataFieldConvertingFunctions().
In .valParser(values = obj[[sweValuesName]], fields = .fields, ... :
No converter found! Skipping field Conductivity
No converter found! Skipping field http://mmisw.org/ont/cf/parameter/conductivity
No converter found! Skipping field S/mThis shows warnings about unknown units of measurement and a swe:Quantity element (which describes a numeric field) without a given unit of measurement (which it should have as a numeric field). The next example creates conversion functions for these fields and repeats the operation.
myConverters <- SosDataFieldConvertingFunctions( "S/m" = sosConvertDouble, "http://mmisw.org/ont/cf/parameter/sea_water_salinity" = sosConvertDouble)
When working with sos4R, two kinds of errors must be handled: service exceptions and errors within the package. The former can occur when a request is invalid or a service encounters internal exceptions. The latter can mean a bug or illegal settings within the package.
To understand both types of erroneous states, this sections explains the contents of the exception reports returned by the service and the functionalities to investigate the inner workings of the package.
The service exceptions returned by a SOS are described in OGC Web Services Common (Whiteside, 2007) clause 8. The classes to handle the returned exceptions in sos4R are OwsExceptionReport, which contains a list of exception reports, and OwsException, which contains slots for the parameters exception text(s), exception code, and locator. These are defined as follows and can be implementation specific.
The standard exception codes and meanings are accessible by calling the following function.
library("knitr") knitr::kable(OwsExceptionsData())
| exceptionCode | meaningOfCode | locator | httpStatusCode | httpMessage |
|---|---|---|---|---|
| OperationNotSupported | Request is for an operation that is not supported by this server | Name of operation not supported | 501 | Not Implemented |
| MissingParameterValue | Operation request does not include a parameter value, and this server did not declare a default parameter value for that parameter | Name of missing parameter | 400 | Bad request |
| InvalidParameterValue | Operation request contains an invalid parameter value | Name of parameter with invalid value | 400 | Bad request |
| VersionNegotiationFailed | List of versions in ‘AcceptVersions’ parameter value in GetCapabilities operation request did not include any version supported by this server | None, omit ‘locator’ parameter | 400 | Bad request |
| InvalidUpdateSequence | Value of (optional) updateSequence parameter in GetCapabilities operation request is greater than current value of service metadata updateSequence number | None, omit ‘locator’ parameter | 400 | Bad request |
| OptionNotSupported | Request is for an option that is not supported by this server | Identifier of option not supported | 501 | Not implemented |
| NoApplicableCode | No other exceptionCode specified by this service and server applies to this exception | None, omit ‘locator’ parameter | 3xx, 4xx, 5xx | Internal Server Error |
response <- try(getObservationById(sos = mySOS, observationId = ""))
## Warning in .handleExceptionReport(sos, response): Object of class OwsExceptionReport; version: 1.0.0; lang: NA;
## 1 exception(s) (code @ locator : text):
## MissingParameterValue @ ObservationId :
## The value for the parameter 'ObservationId' is missing in the request!The exception is also stored in the response object.
response## Object of class OwsExceptionReport; version: 1.0.0; lang: NA;
## 1 exception(s) (code @ locator : text):
## MissingParameterValue @ ObservationId :
## The value for the parameter 'ObservationId' is missing in the request!
##