Logging Multiple Sensors

[PeterW]

5 hours ago, JSHarris said:

 

Would such a data set be of use?

 

Moved from here 

 

 

Yes..! I'm currently planning where to put 10 DB18S20 sensors around the slab (including one under the insulation) and the fabric of the building. 

 

Current list is :

- Outside Air (northerly wall)

- Under slab

- Kitchen Slab

- Utility Slab

- Lounge Slab

- Cellar

 

When finished, they will be added to :

- UVC

- ASHP Feed

- ASHP Return

- MVHR Inlet

- MVHR Supply

- Internal Air 

 

question is ... how to link them all easily as I'm an rPi fan at heart but something tells me Arduino would be easier ..?

 

or is there an off the shelf logger at a reasonable price that can do this sort of sensor..?

[SteamyTea]

22 minutes ago, PeterW said:

question is ... how to link them all easily as I'm an rPi fan at heart but something tells me Arduino would be easier ..?

Start a new Coughing Corner thread on this as I am currently doing the same. It is easy enough for us to swap code, improve it and test it out on each others RPis.  That is one of the beauties of the RPi, it is cheap.

 

Saves messing up this thread with my feeble attempts.

 

Edited February 11, 2017 by SteamyTea

[SteamyTea]

I am about to go to work, but shall get back to you on this and explain what I am doing.

[PeterW]

New thread created ...

[Jeremy Harris]

The RPi makes using one-wire sensors pretty easy, I think, as the port and code for them already exists AFAIK.  Each DS18B20 has a unique serial number, which it responds to when interrogated, so you can daisy-chain them up on the same cable.  The term one-wire is a bit of a misnomer, as they really need either two or three wires depending on the mode you choose to use them in.  I use them in three wire mode, because I don't bother using the one-wire code.  Speeds things up slightly too, as they take 750ms anyway to take a temperature reading on command and spit out 12 bit data.

 

ST has more experience than me of using one-wire devices in one-wire mode, so is the better person to answer RPi related one-wire stuff I think.  My data logger stuff all uses PICs.

 

Edited to add:

 

Just found this pdf tutorial from the inimitable LadyAda.  She's good at explaining stuff, IMHO, and has given some sample code (Python) for the RPi and DS18B20 sensors.  The tutorial download is here: https://cdn-learn.adafruit.com/downloads/pdf/adafruits-raspberry-pi-lesson-11-ds18b20-temperature-sensing.pdf

Edited February 11, 2017 by JSHarris
Edited to add extra info

[reddal]

Cool project!

 

By co-incidence I was just doing something where this kind of device would have been really useful.

 

I have been creating a 'curing chamber' for drying chorizo. Basically you need a chamber with fairly precise control and monitoring of temperature and humidity that the chorizo cures in for several weeks. I've done this using a fridge with a heater, humidifier and dehumidifier inside it. The power for all those is attached to a clever little box that switches the fridge on if the temp is too high, the heater on if the temp is too low, etc etc. That way it keeps the temp and humidity in the box in a fairly tight range throughout.

 

However the control box is a commercial product - and it doesn't do any logging or anything to allow monitoring - so I needed a seperate way to be able to check its working and graph changes etc.

 

By chance I happened to have an old environmental monitoring device left over from a data center. This has sensors for temp and humidity and does data logging and alerting. However it also connects to a LAN and broadcasts the current values as SNMP (a network protocol that allows devices to make monitoring info available). The nice thing about the temp and humidity being available on SNMP is there is loads of nice and free software tools you can run on your PC that know how to do logging, graphing etc etc for SNMP sources. (the one I'm using is called PRTG). It works really well.

 

However - that relies on me having the env monitoring device lying around unused - it would be too expensive to buy for this purpose. I can't see a sensibly priced alternative to achieve this?

 

You guys should build you RPi device that takes an arbitary number of DS18B20 inputs (and the equiv for humidity etc) - does basic logging - and makes the data available over the network. You could sell that for £100 each would be my guess :). Loads of people would find such a device useful - but most aren't going to have the skills to build one themselves (myself included).

[Jeremy Harris]

I have a few loggers, all home built, a couple log to µSD cards, one logs to a standard SD card and the one in front of me right now, as I'm trying to get the code to work, stores data to a USB stick. 

 

I've never bothered with networking them, as it's easy enough to just pull the card and stick the data in a spreadsheet (the data files are all comma separated value, .csv, that any spreadsheet programme should open OK).

 

Easy enough to get even the £4 Raspberry Pi Zero to chuck the same data on to the local network, as I've found a cheap way to get them to work with a bare board wifi module and connect to any wifi network.  I may get around to trying this one day, but right now have to get back to seeing why my "write data to a USB stick"  code isn't behaving as it should!

[SteamyTea]

Right

For a simple, and basic temperature logger the DS18B20 is pretty good.  You have to set the RPi up to accept the 1Wire protocol, which is done though the raspi-config screen.  The General Purpose In Out (GPIO) defaults to GPIO4 (this can be changed with a bit of trickery).

One thing that the RPi lacks is a real time clock (RTC).  I bought some cheap ones a few years back but they never worked reliably, so have ordered some new ones (still waiting for them).  Hopefully this will sort the problem.  If the RPi is on a network, it just gets the time from a Network Time Protocol server (NTP).  This can cause a few problems on a reboot as it has not always updated.

The code in Python is simple, I can post some up, but there are many websites that have examples.

Just recently I have tried to put a bit of error correction in, but have hit a snag.  If there is an error, it stops logging.  So I shall go back to post processing the data (if you now what you are looking for, error are easy to spot).  Or I can learn now to allow it to recheck say 10 times (2 seconds), then carry on.

A few months back I found a bit of free FTP space and played about uploading data to that, it worked well (until I hit my upload limit).  I used a bit of simple shell (sh) script.  I have also looked into using email, and splitting large data files into daily files.  This is a bit 'work in progress' as there are some quirks between different versions of Raspbian OSs and getting a reliable connection (more of that later).  But it does give one the opportunity to upload from a remote location via a cheap old mobile phone.  As data compression is easy to do from a sh script, any old 3G connection is perfectly adequate.

I have only scratched the surface of shell scripts, but they seem a very powerful tool to use.  Many of the commands can be used within Python, but as I am not a programmer and have little interest in programming, I like to use simple scripts that I can understand (and write/modify), and then just get them to do their thing when needed.  My CurrentCost data logger seems very reliable these days, unlike the copy I got from the internet.  It takes a few minutes to set up initially, but then just chugs along writing to a comma separated file (.csv).  Once you have data in that format, you can do just about anything with it.

Other project I am currently working on are using DHT22 temperature and relative humidity sensors.  These are a bit harder to set up as they need a separate GPIO pin for each sensor, and they do not connect easily to the RPi.  To get around this I just download some C (a programming language) from AdaFruit, follow the instructions and eventually get it working.  word of caution though, don't by the much cheaper DHT11s, they have limited temperature range and are not that accurate, pay the extra for the DHT22s (think I pay less than £2.50 for them, will need to check).

I have also made a bit of Python script that senses when a switch is open or closed and puts a time stamp against it.  This is so I can sense, via cheap 240V relay (still waiting for it to turn up) wired in parallel to a load, when that load is on or off.  This is for a heat meter for a solar thermal system, but could also be used monitor just about any load, as long as wired correctly and safely.  This is cheaper than buying in a flow meter, but I have an idea on how to make on of those.

At the moment I only log data, I don't use it to control anything (not really interested in that side of things too much).  I have got LEDs flashing for a laugh as an alarm, easy step from there to getting something fully controlled.  I bought a PiFace, which is a board that fits onto the GPIO pins and has 240V/5A relays on it.  But never got around to using it.  Something for a rainy day.

 

Now problems I have had. Well one major one that I cannot fathom out.  With the old Raspbian OS Wheezy, setting up a static IP address for both the local area network (LAN) and the wireless network (WLAN) was easy, just edit one file.

Since I bought a Raspberry Pi Zero (a cut down RPi, but just as good as my Model A), I decided to use the later OS, Jessie.  There is two versions of this, one with a graphic user interface (GUI), and one without.  As the RPi Zero (RPiZ) is really used with a screen and keyboard, I decided to use Jessie Light (the non GUI version).  This has made it impossible, for me, to set up a reliable wireless connection.  It works fine on the USB to LAN adapter, but changes the IP address every now and again, which means I loose connectivity.

To get around this, just this week I have set up one of my old Model As with the GUI version of Jessie, connected via the main screen to my wireless network, and it seems to be working fine with the static IP I put in.  I have searched though the files that seems to be associated with the networking (think there is 4 of them), but can see no mention of the static IP address I typed in.  But for the last few days it has been working reliably.  A mystery to me.

 

There are a few projects I would like to try out, one it to make a reliable, but cheap, wireless energy monitor.  This can be either via a clamp on meter (plenty accurate enough for most things) or a sensor to read the flashing LED on the meter.  If CurrentCost where still in business, I would just buy their kit, but they seem to have gone (annoying that they sold a million plus units to the energy companies, who then gave them away to customers, who never used them, there must be half a million of these boxed and unused).

There are cheap transmitters/receivers for the RPi, just a matter of fitting a sensor to them, finding out a way to transmit the data and then read it (bit beyond me to be honest).  But all the 'bits' are cheap, so can't be that hard.  Maybe even 'reverse engineer' a CurrentCost one.

I also have an analogue to digital board (ADC) that I have only used once, there is a fair bit of scope for sensing there, and they are cheap.  I have a £100 LabJack, which is a great bit of kit for playing with, and works very well with the RPi.  But it is really only for lab use, not a practical, or cheap thing to use for everyday monitoring.

 

Right it is late, I need some sleep. 

 

Edited February 12, 2017 by SteamyTea
Unedited for confusion, smoke and mirrors.

[Jeremy Harris]

ST, not sure if this helps or not, but I have chopped down a version of the "full fat" Jessie and loaded it to a Pi Zero, with a home brew (read cheap and chopped about) USB wifi module soldered to the Pi Zero.  To cut down Jessie I just took out Libre Office, Wolfram all the games etc, and got the image down to not a lot bigger than Jessie Lite.  I found that with Jessie Lite I couldn't get it to run with a fixed IP or get the wifi to work very reliably; with the cut-down "full fat" version the PIXEL GUI and wifi config menu is still there, so once the thing has connected via DHCP you can just VNC in and make the change to a static IP.  I've not enough experience with the RPi to say why this is - I only bought my first one just before Christmas, and then only to use as a backup device for a website, nothing more. 

 

I need to sit down and do some serious work with the Pi Zero, as for virtually everything I want to do it's better.  BTW, I'm now using a cheap GPS module as a RTC.  The tiny UBLOX 6 modules work fine indoors (one is sat a foot away from me on my desk reliably squirting out the time).  They also only need three wires, 0V, +5V and a UART Rx port that is set to 9600 baud.  UBLOX have a utility that allows you to connect the GPS module to a FTDI USB to serial port converter (powering the GPS from it) and set what data you want transmitted.  I have mine set to only transmit the $GPZDA string once a second, which is just ASCII UTC time and easy to parse into whatever you want.  There is also a battery-backed RTC on the module, so if it loses the signal it still carries on spitting out pretty accurate time data whilst powered up, and will re-sync itself as soon as it acquires just a single SV.

 

[SteamyTea]

6 minutes ago, JSHarris said:

ST, not sure if this helps or not, but I have chopped down a version of the "full fat" Jessie and loaded it to a Pi Zero

What I did in the end.

 

sudo apt-get purge libreoffice wolfram-engine sonic-pi scratch minecraft-pi chromium-browser

sudo apt-get autoremove –y

Sudo apt-get clean -y

 

But when I updated and ungraded it just loaded LibreOffice again.  I need to look at it again as I may have missed something.

6 minutes ago, JSHarris said:

BTW, I'm now using a cheap GPS module as a RTC.

That seems like a good work around.  I may try that out.  RTCs seem expensive for what they are and do take up the first 6 GPIO pins. 

Edited February 12, 2017 by SteamyTea

[Jeremy Harris]

You can either go for an £6 UBLOX GPS from China  , like this: http://www.ebay.co.uk/itm/Ublox-NEO-6M-GPS-Module-Aircraft-Flight-Controller-For-Arduino-MWC-IMU-APM2-UK-/401215045349?hash=item5d6a47bee5:g:CcUAAOSwB09YEire  , which is the same as the module I'm using, I think, or go for a slightly more expensive all-in-one module from the UK for just under a tenner, like this one: http://www.ebay.co.uk/itm/Ublox-NEO-6M-GPS-Module-Aircraft-Flight-Controller-For-Arduino-MWC-IMU-APM2-New-/281957215228?hash=item41a5f5abfc:g:5tkAAOSwYlJW3VYf

 

Both use the same UBLOX  U Center utility to set them up, available to download here:https://www.u-blox.com/en/product/u-center-windows

 

You also need a USB to serial converter, like an FTDI board, to connect the GPS module to a PC to use that set up utility.  You can actually set the GPS up from the RPi, if you connect both the Rx and Tx UART connections, set the RPi to 9600 baud and then just send the commands directly to the GPS module to set it up.  It's a bit of a faff, and nowhere near as useful as the U Center software, as that gives graphical feedback on a whole load of stuff, including how many SVs are being received, what the signal strength looks like etc, which give confidence that it will work OK where you have it positioned.

[TerryE]

I use ESP8266 devices for my sensors.  And I program them in a language called Lua using the NodeMCU firmware.   However, in the interests of full disclosure, I am a little biased since I am one of the core developers of the NodeMCU firmware.  I use these to communicate to a central full RPi3 for logging and home automation.  The ESP modules do the data collection and relay control, etc., but the RPi does all of the complex stuff. 

 

If you don't want to ESP programming at all and essentially use it as a fairly dumb Wifi connected chip, then an alternative that my son-in-law uses and recommends is ESPeasy.  This project by default recommends Nodo Domotica as the Home Automation (HA) system, but I would suggest that you stick to either Pimatic or OpenHAB.

 

I run my RPi3s with the SD cards pretty much RO and attach a small USB-powered HDD as the main storage device. You can buy these for £50 up or you can buy SSD ones such as this for £60.  I run a standard Rasbian Server installed through NOOBS.  It is basically a bog standard Debian build so there is little point in doing anything else.  The only frill that you might consider is a Li/LiPo battery hat if you want a battery backup (UPS).

 

As far was the ESP kit is concerned, have a look at the WeMos range. These use a standard form factor and edge pin-out so that you can stack them.  You can get a whole range of hats for relay and sensors, and if you want to have multiple DS18B20s, then use the prototyping hat to put the headers for the DS18B20s, resistor and cap on.  You can make up a complete battery-backed 6 temperature sensor with WiFi for about £20 if you are willing for delivery from China and buying the stuff from AliExpress.

 

BTW the Wemos ESP boards which cost about £3 already have Arduino-style USB serial + power onboard + FlashRAM which is used for both firmware load and as a pseudo HDD -- as well as Wifi connectivity of course.

 

PS.  I've just ordered a USB 120Gb SSD from AliExpress: £39.74 free shipping.

PPS.  I've decided to SikaFlex my RPi3 box that I use for my house server to the wall of the cupboard under the stairs.  That way there is no bloody chance of Jan covering it crap so that it over-heats this time!  

[PeterW]

5 hours ago, TerryE said:

I've decided to SikaFlex my RPi3 box that I use for my house server to the wall of the cupboard under the stairs.  That way there is no bloody chance of Jan covering it crap so that it over-heats this time!  

 

I saw some tiny 30mm fans in Maplin of all places - may put one on my rPi to keep it cool. 

[TerryE]

@PeterW, it's an ARM chip so this shouldn't really be necessary if you are using it as a server, since the overall CPU utilisation will be well under 100% and the on-chip Graphics Processors (GPUs) will be idle.  If you are concerned, then you can buy a passive component like this.  I really don't think that you will need an active heat-sink unless you are making heavy use of the OpenGL libraries.  (These are a standard API libraries used to access the functions provided in the hardware GPUs in a portable fashion; most 2D and 3D applications and OS functions will use these standard interfaces rather than get into proprietary Broadcom hardware internals).

 

[Steptoe]

On 12/02/2017 at 09:22, SteamyTea said:

What I did in the end.

 

sudo apt-get purge libreoffice wolfram-engine sonic-pi scratch minecraft-pi chromium-browser

sudo apt-get autoremove –y

Sudo apt-get clean -y

 

But when I updated and ungraded it just loaded LibreOffice again.  I need to look at it again as I may have missed something.

That seems like a good work around.  I may try that out.  RTCs seem expensive for what they are and do take up the first 6 GPIO pins. 

You may have to remove before you purge,

I'm not massively into exactly how Raspbian works, but some *nix flavours require it this way,

Also , apt-get has mostly been changed to just apt now, but again, I'm not sure if this applies to Raspbian either,

Although I do have some issues with some code I've transferred from Wheezy to Jessie , I'm not convinced Jessie isn't without its problems tbh. I'm reverting that one back to Jessie as the rest of my Pi's are running fine on Wheezy, but the one on Jessie is giving problems.

 

[SteamyTea]

7 hours ago, Steptoe said:

but the one on Jessie is giving problems.

Been my experience when it comes to networking.

 

I shall try the remove then purge and see if it works.

[TerryE]

9 hours ago, Steptoe said:

You may have to remove before you purge

 

? apt-get purge is just short hand for a apt-get remove followed by apt-get clean.

 

I'll just be doing this again in a few days, so if you want, I will update my notes, but IIRC, I just edit my NOOBS config.txt to set the HDMI screen to text mode only -- or you can buy a HDMI to VGA convertor for about £5 on eBay (but watch out, some come from China) and use an old monitor instead.  You can also do this through rasp-confg.  As well localisation, enable SSH.  As far as using the RPi, I don't enable GPIOs or anything since I do this with my ESPs.  I just configure it as a bog standard LAMP server plus SSH, then just do an autoclean / autopurge.

 

Just search for raspberry pi raspbian lamp server in YouTube if you want an online video on how to do this.  There are euvalent vidoes for using RPis for embedded use.

 

 

[SteamyTea]

15 minutes ago, TerryE said:

apt-get purge is just short hand for a apt-get remove followed by apt-get clean.

Yes, I had a quick Google and seems that using 'purge' and 'autoremove' should do it.

 

NOOBS = New Inexperienced Person

HDMI = High Definition Multimedia Interface

VGA = Video Graphics Array

ESP = Event Stream Processing ?

LAMP = Linux Operating System, Apache HTTP Server, MySQL and PHP language.

SSH = Secure Shell

[TerryE]

@SteamyTea Nick, I assumed that you know how to use Google and have a base level of experience here on configuring the device that you want to use.    However, I will expand on these for other readers and rather than go into great detail, I've just linked to the relevant Wikipedia article.

• HDMI and VGA. These are two hardware display interfaces.  You plug old-style monitors (and some TVs) into VGA; and modern TVS into HDMI.  The RPi has an HDMI interface on board so you need a little converter like this if you want to use VGA.
• Raspberry Pi
• ESP.  Sorry I assumed that you'd read / not ignored / not forgotten my earlier post on this page where I explained all this.
• NOOBS, the RaspberryPi.org recommended and supported (and easiest) way to setup a Pi.  If you haven't heard of this then no wonder you are struggling to setup you Pi.  RTFM
• Server vs Workstation.  In simple terms you have a display / mouse / keyboard and graphics interface on a workstation.  You typically access a server using one of more encrypted SSH links, either for backup, console sessions (think of DOS box), and file browsing.  This doesn't mean that you don't use a graphics interface to develop / maintain the server; you just use your PC/Laptops and open as many windows / SSH connections onto server as you need.  The main advantage of doing it this way is that you don't need to install the OS desktop components or desktop applications on your server.  Less footprint both in term of HDD and memory; less security vulnerabilities.
• SSH
• LAMP.  The standard user interface for servers these days is either a web application or your mobile phone using RESTful API.  To do this you need to configure your Pi as a webserver.  The LAMP stack is the de facto free open-source stack.  Installing it on the Pi is a check-box on the configuration menu.   Nearly all automation application assume that you have this installed.

Google, YouTube (and your brain) are your best friends here.

[Jeremy Harris]

Thanks for that, Terry.  Having acquired my first RPi (a RPi3) in December, I have to say I found the learning curve, in terms of setting up the LAMP stack, installing Wordpress and then setting up automatic backups from our web site was remarkably straightforward, my hat goes off to the RPi developers for making this stuff work so seamlessly.  When I tried to do the same in Windows 7 last year it was a bit of a nightmare by comparison.

 

Getting back to logging, I have now set up a logger to record the slab centre surface temperature, so together with the data I already log from the house built-in sensors I should be able to post a reasonable data set in a couple of weeks.  I think it needs a fair time to be sure we get enough outside temperature variation to be able to test a model over as wide a range of conditions as possible.

 

As an aside, the portable logger that I've just moved indoors has been sat in our water treatment plant "shed", a timber building at the rear of the house, outside the house thermal envelope, that houses two 300 litre accumulators, the ozone generator and contact tank and the main filtration system tank.  I fitted a frost protection heater in there, with a stat set to come on at 4 deg C, and wanted to see how it dealt with the sub-zero temperatures we had during a few nights over the monitoring period.  The data's interesting, as the coldest the inside of this shed got was 4.3 deg C, and then only on one night.  It sat at around 8 to 9 deg C pretty much all the time.  The other interesting thing is that when the filtration system backwashes, the shed temperature increases very rapidly to 9 deg C - so rapidly that there are big spikes every 4 days when the data is plotted.  That's just from the accumulators being completely emptied and refilled with warmer borehole water, that must still be coming in at around 9 deg C, even in mid-winter.  It seems that I probably didn't need to fit the tubular heater and frost stat in there at all, as the incoming "warm" water will be more than enough to keep the shed warm enough to prevent freezing.  It also means that my calcs for the DHW capacity need revising, to allow for having a warmer cold supply than I had thought.

Edited February 14, 2017 by JSHarris