HTTPMOD

HTTPMOD
Zweck / Funktion
Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices
Allgemein
Typ	Gerätemodul
Details
Dokumentation	EN / DE
Support (Forum)	Sonstiges
Modulname	98_HTTPMOD.pm
Ersteller	StefanStrobel (Forum / Wiki)
Wichtig: sofern vorhanden, gilt im Zweifel immer die (englische) Beschreibung in der commandref!

This page documents the new version of HTTPMOD which has some advanced features compared to the initial version. HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes.

From the HTTP Response it extracts Readings named in attributes using Regexes also defined by attributes.

In an advanced configuration the module can also send information to devices. To do this a generic set option can be configured using attributes.

Availability

The module has been checked in on 14th of December 2014

Prerequisites

This module uses the non blocking HTTP function HttpUtils_NonblockingGet provided by FHEM's HttpUtils in a new version published in December 2013. If not already installed in your environment, please update FHEM or install it manually using appropriate commands from your environment.

Define

define <name> HTTPMOD <URL> <Interval>

The module connects to the given URL every Interval seconds, sends optional headers and data and then parses the response.

Example:

define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60

Set-Commands

can be defined using attributes, see advanced configuration

Get-Commands

None implemented so far

simple Attributes

do_not_notify

readingFnAttributes

requestHeader.*

Define an additional HTTP Header to set in the HTTP request

requestData

POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module

reading[0-9]Name

the name of a reading to extract with the corresponding readingRegex

reading[0-9]Regex

defines the regex to be used for extracting the reading. The value to extract should be in a sub expression e.g. ([\d\.]+) in the above example

reading[0-9]Expr

defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.

simple Configuration of HTTP Devices

If your device expects special HTTP-headers then specify them as attr requestHeader1 to attr requestHeaderX. If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as attr requestData. To get the readings, specify pairs of attr readingXName and attr readingXRegex to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax attr readingsNameX and attr readingsRegexX is still supported but the new one with attr readingXName and attr readingXRegex should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)

Example for a PoolManager 5:

The PoolManager Web GUI can be queried with HTTP POST Requests like this one:

POST /cgi-bin/webgui.fcgi HTTP/1.1
Host: 192.168.70.90
Accept: */*
Content-Type: application/json;charset=UTF-8
Content-Length: 60

{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}

The resulting HTTP Response would look like this:

HTTP/1.1 200 OK
Content-type: application/json; charset=UTF-8
Expires: 0
Cache-Control: no-cache
Date: Sun, 12 Jan 2014 12:23:11 GMT
Server: lighttpd/1.4.26
Content-Length: 179

{
	"data":	{
		"34.4001.value":	"7.00",
		"34.4008.value":	"0.52",
		"34.4033.value":	"24.8"
	},
	"status":	{
		"code":	0
	},
	"event":	{
		"type":	1,
		"data":	"48.30000.0"
	}
}

To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds.

Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.

Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as attr PM requestHeader1 and attr PM requestHeader2

Then the names of the readings to be extracted would be set with attributes

Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().

Example:

define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60
attr PM reading01Name PH
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"

attr PM reading02Name CL
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"

attr PM reading03Name3TEMP
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"

attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}
attr PM requestHeader1 Content-Type: application/json
attr PM requestHeader2 Accept: */*
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}

If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute readingXExpr which can use the raw value from the variable $val

Example:

attr PM reading03Expr $val * 10

Example for AmbientMonitor

AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.

This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor

In this example an HTTP GET is sufficent, so no requestData is needed. The device provides temperature and humidity readings in an HTTP response that looks like:

HTTP/1.0 200 OK 
Content-Type: text/html 

myCB({'temperature':22.00,'humidity':46.00})

the definition could be:

define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300
attr AmbientMonitor requestHeader Content-Type: application/json
attr AmbientMonitor reading1Name Temperatur
attr AmbientMonitor reading1Regex temperature':([\d\.]+)
attr AmbientMonitor reading2Name Feuchtigkeit
attr AmbientMonitor reading2Regex humidity':([\d\.]+)
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}

Some help with Regular Expressions

If HTTPMOD seems not to work and the FHEM Logfile contains a message like

HTTPMOD: Response didn't match Reading ...

then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful.

For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like [\d\.]+ to match the number itself. The expression matches the number characters (\d) or a . if one of these characters occurs at least once.

To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between (). A ([\d\.]+) alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for. if there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above humidity':([\d\.]+) is looking for the number that immediately follows the text humidity': without any blanks in between.

Additional notes

If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite BurpSuite to track requests and responses

Future extensions might include attributes to define set commands in a generic way. If a device allows setting of values via HTTP then for each set a name and a corresponding URL with headers and data might be a possible way forward.

Advanced configuration to define a set and send data to a device

When a set option is defined by attributes, the module will use the value given to the set command and translate it into an HTTP-Request that sends the value to the device.

Extension to the above example for a PoolManager 5:

attr PM set01Name HeizungSoll
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid
attr PM set01Hint 6,10,20,30
attr PM set01Min 6
attr PM set01Max 30
attr PM setHeader1 Content-Type: application/json
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}

This example defines a set option with the name HeizungSoll. By issuing set PM HeizungSoll 10 in FHEM, the value 10 will be sent in the defined HTTP Post to URL http://MyPoolManager/cgi-bin/webgui.fcgi in the Post Data as

{"set" :{"34.3118.value" :"10" }}

The optional attributes set01Min and set01Max define input validations that will be checked in the set function. The optional attribute set01Hint will define a selection list for the Fhemweb GUI.

Advanced configuration to create a valid session id that might be necessary in set options

when sending data to an HTTP-Device in a set, HTTPMOD will replace any $sid in the URL, Headers and Post data with the internal $hash->{sid}. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes:

sid[0-9]*URL

sid[0-9]*IDRegex

sid[0-9]*Data.*

sid[0-9]*Header.*

sid[0-9]*IgnoreRedirects

Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into $hash->{sid}. HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of $hash->{sid}.

Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.

To determine when this login procedure is necessary, HTTPMOD will first try to do a set without doing the login procedure. If the Attribute ReAuthRegex is defined, it will then compare the HTTP Response to the set request with the regular expression from ReAuthRegex. If it matches, then a login is performed. The ReAuthRegex is meant to match the error page a device returns if authentication or reauthentication is required e.g. because a session timeout has expired.

If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a sidURL, sidData.* or sidHeader.* Attribue (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.

Example for a multi step login procedure:

attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid
attr PM sidHeader1 Content-Type: application/json
attr PM sid1IDRegex wui.init\('([^']+)'
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}

Advanced attributes

ReAuthRegex

regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.

sid[0-9]*URL

different URLs or one common URL to be used for each step of an optional login procedure.

sid[0-9]*IDRegex

different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response

sid[0-9]*Data.*

data part for each step to be sent as POST data to the corresponding URL

sid[0-9]*Header.*

HTTP Headers to be sent to the URL for the corresponding step

sid[0-9]*IgnoreRedirects

Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.

set[0-9]+Name

Name of a set option

set[0-9]*URL

URL to be requested for the set option

set[0-9]*Data

Data to be sent to the device as POST data when the set is executed

set[0-9]*Header

HTTP Headers to be sent to the device when the set is executed

set[0-9]+Min

Minimum value for input validation.

set[0-9]+Max

Maximum value for input validation.

set[0-9]+Expr

Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.

set[0-9]+Map

Map that defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the visible values or select a value with a slider.

set[0-9]+Hint

Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.

set[0-9]*ReAuthRegex

Regex that will detect when a session has expired an a new login needs to be performed.

queueDelay

HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.

queueMax

Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue

minSendDelay

Defines the minimum time between two HTTP Requests.

Links

• Thread in Fhem Forum that discusses the first version of this module
• Introduction to regular expressions
• BurpSuite: Tool (local proxy) to help analyze http traffic