Wanted: Broadband speed monitoring / logging program

[SteamyTea]

9 minutes ago, ProDave said:

But Sorry to say @SteamyTea your script is still returning "1" for every test so not logging the fault.

That is odd, shall see if I can get it running here, it should show when I take the phone away.

May take me a while to set up a spare RPi.

[epsilonGreedy]

@prodave do you know how far away the broadband street cabinet is? I ask because your experience sounds like my previous property which was about a mile from the cabinet and delivered 2mb on a good day or zero when playing up.

 

It would be useful to know how fast your broadband is when working ok because a web page or streaming radio station does not stress even a poorly 2mb link.

 

This test page gives a useful 10 minute polling log of link speed.

 

http://startrinity.com/InternetQuality/ContinuousBandwidthTester.aspx#cst_windows

[Temp]

Can you get into the modem status pages? Things like signal to noise ratios. How long you have been connected for etc

[epsilonGreedy]

1 minute ago, Temp said:

Can you get into the modem status pages? Things like signal to noise ratios. How long you have been connected for etc

 

If it is a BT hub just follow the instructions on the slide out password panel.

[ProDave]

We are about 3 miles as the crow flies to the exchange but the copper path is longer.  Typical download speeds 3.6Mbps  The router is currently synced at4167kbps.  I can't see anything in the router to give me signal to noise ratios etc.  the router is showing it has been connected for just over a day at the moment.  Even while we are suffering an "outage" is has not disconnected or tried to re sync, which makes me feel the problem is further down the network.

[epsilonGreedy]

1 hour ago, ProDave said:

We are about 3 miles as the crow flies to the exchange but the copper path is longer.  Typical download speeds 3.6Mbps.

 

 

+3 miles!

 

Just go for GSM and save yourself years of grief, I wish I had.

 

There are some technical antenna options sold for canal boats and yachts.

 

Edit: Are you sure there isn't an intermediate broadband street cabinet closer to your house?

Edited May 21, 2020 by epsilonGreedy

[Temp]

I think you are probably close to or over the recommended max distance. In some countries they install loop repeaters to improve performance at long distance but for some reason I don't think they do that in the UK.

 

 

[Temp]

Some people say particular modems are better on long distance lines but tuning them up is a hobby all of its own...

https://forums.thinkbroadband.com/dslrouter/4544026-best-adsl-modem-for-long-line.html?page=7&fpart=all

 

 

[ProDave]

Our old house, 100 metres down the road used to consistently get 2.5Mbps

 

When we first moved into this house we got barely 1Mbps so reported a fault and an engineer spent several hours here. I was impressed with the test kit he had that looked for reflections in the line and he had every junction pit up in the road (our line goes through 4 junctions just to reach the top of the road !!!!)  and re made a lot of connections and got us up to 3.5Mbps reliably.  It is possible of course that those connections have gone bad again, which would not surprise me as last time there were so called waterproof junctions sitting in a pool of water at the bottom of the pit and when opened, water came out.  The state of the network I am surprised a voice telephone line works let alone the bodge that is ADSL.

 

The replacement hub has arrived and I will be fitting that shortly. If that does not fix it I will be insisting on an engineers visit.

[ProDave]

I have fitted the new hub and it is currently working so we shall see if that has fixed the problem, or it's just one of it's good spells.

 

@SteamyTea your script did return a load of 0's for the time the router was off and until I had entered the password and connected the laptop to the new router.  So it is working.   What can we read into that, that your script still connects even when everything else is struggling?

[SteamyTea]

Can you make a copy and put in your router IP address.

Run the two together and see what comes out.

Will need to change the .csv file name.

Edited May 21, 2020 by SteamyTea

[andyscotland]

48 minutes ago, ProDave said:

I have fitted the new hub and it is currently working so we shall see if that has fixed the problem, or it's just one of it's good spells.

 

@SteamyTea your script did return a load of 0's for the time the router was off and until I had entered the password and connected the laptop to the new router.  So it is working.   What can we read into that, that your script still connects even when everything else is struggling?

 

How regular are the time/date stamps in your CSV file for the periods it was giving 1 but not working?

 

My hunch is the issue is this bit `wget -q --tries=10 --timeout=20 -O - http://google.com > /dev/null`

 

By default wget will gradually increase the delay between retries, I think starting at 1s increasing 1s at a time up to 10s. So the 10 retry waits themselves could take 55 seconds, and depending on the type of failure there could be up to another 200 seconds on the combined timeout of each request. I also vaguely recall wget can behave unexpectedly if a domain name resolves to more than one IP, which Google possibly does, as it may retry each IP address in turn (e.g. 10 retries * number of IPs).

 

So it could be that it's failing for a fair period but the connection recovers in time for one of the retries to succeed and so it writes a 1.

 

I'd set it to --tries=1 and probably drop the timeout much lower (benchmark what a "normal" response time from Google is and say triple it to give a bit of a margin - for most connections I'd think 2-3 seconds should be perfectly adequate).

[SteamyTea]

30 minutes ago, andyscotland said:

By default wget will gradually increase the delay between retries, I think starting at 1s increasing 1s at a time up to 10s. So the 10 retry waits themselves could take 55 seconds, and depending on the type of failure there could be up to another 200 seconds on the combined timeout of each request. I also vaguely recall wget can behave unexpectedly if a domain name resolves to more than one IP, which Google possibly does, as it may retry each IP address in turn (e.g. 10 retries * number of IPs).

I did not know about that.

 

Feel free to modify and improve my code.

It would be quite good to see if it is possible to make something decent, but extremely lightweight.

I did think of putting the info you get when you ping an address into a text file, but never got around to it.

I just dumped the landline.

[Temp]

3 hours ago, ProDave said:

Our old house, 100 metres down the road used to consistently get 2.5Mbps

 

When we first moved into this house we got barely 1Mbps so reported a fault and an engineer spent several hours here. I was impressed with the test kit he had that looked for reflections in the line and he had every junction pit up in the road (our line goes through 4 junctions just to reach the top of the road !!!!)  and re made a lot of connections and got us up to 3.5Mbps reliably.  

 

They have some scope to trade off speed and reliability but check the speed Vs distance graph down this page...

 

https://www.increasebroadbandspeed.co.uk/2012/graph-ADSL-speed-versus-distance

 

5km is going to be problematic for them to guarantee any particular speed.

 

 

 

[ProDave]

That chart would suggest the exchange is closer than I thought at more like 4Km for the speed we are getting.

 

The new router has not dropped out yet, so who knows it could have fixed the problem?  Time will tell.

[epsilonGreedy]

50 minutes ago, ProDave said:

That chart would suggest the exchange is closer than I thought at more like 4Km for the speed we are getting.

 

 

Another variable is the cable quality. There is standard thin copper, fat (good) copper and old legacy aluminium cabling.

 

My old problematic broadband property was cursed by aluminium telephone cables, a visiting engineer told me this stuff was put in back in the 1970's to save money.

 

BTW: Your references to "the exchange" is dated thinking. These days Open Reach lay on fibre to neighbourhood roadside broadband cabinets and the ADSL signal quality fall off is measured as the distance from the cabinet.

[ProDave]

19 minutes ago, epsilonGreedy said:

 

Another variable is the cable quality. There is standard thin copper, fat (good) copper and old legacy aluminium cabling.

 

My old problematic broadband property was cursed by aluminium telephone cables, a visiting engineer told me this stuff was put in back in the 1970's to save money.

 

BTW: Your references to "the exchange" is dated thinking. These days Open Reach lay on fibre to neighbourhood roadside broadband cabinets and the ADSL signal quality fall off is measured as the distance from the cabinet.

On my first call to BT over this, the man got my hopes up by saying FTTC was available and an upgrade would be the best way to fix the fault. But when he tried to book it, it turned out FTTC was available at the exchange but not at "my cabinet"

 

Just to show how clapped out the copper network is, when the line was installed we took  the last spare pair in the cable down our street, and it then took them a week to find a working pair from the junction pit at the top of the road to the exchange.

[dpmiller]

Surely roadside fibre cabinets only apply to FTTC, plain old ADSL *is* copper pair back to the exchange...

 

In our case the exchange would be about 8km, but the Cabinet isn't much better as it's at the far end of the nearby village, so still about 4km of copper.

[ProDave]

Yes I am not being clear.  At the moment we only have copper all the way back to the exchange.  I have seen the "super fast fibre" vans at the pit at the top of the road pulling something from a drum so I assume they are pulling fibre throughout the network, and one day, when they can be bothered, they might install a cabinet and some hardware at the top of the road and offer us FTTC still with the last few hundred metres through the old copper network.

[SteamyTea]

Did a bit of poking around the interweb and found this script.  It is better.

It gives a 1 or a 0 depending on if it is connected or not, and the number of seconds it is connected, as well as a timestamp.

 

#!/bin/bash

TIMESTAMP=`date +%s`
while [ 1 ]
  do
    nc -z -w 5 8.8.8.8 53  >/dev/null 2>&1
online=$?
    TIME=`date +%s`
    if [ $online -eq 0 ]; then
      echo "`date +%Y-%m-%d_%H:%M:%S_%Z` 1 $(($TIME-$TIMESTAMP))" | tee -a connection.csv
    else
      echo "`date +%Y-%m-%d_%H:%M:%S_%Z` 0 $(($TIME-$TIMESTAMP))" | tee -a connection.csv
    fi
    TIMESTAMP=$TIME
    sleep 60
  done;
 

[andyscotland]

@ProDave you could also take a look at https://samknows.com/solutions/consumers - it's a free monitoring box, if you fit the statistical sample of providers/connections/locations etc they're looking for and it captures pretty detailed and robust metrics that you get access to and can use to evidence a fault as well as feeding into their national monitoring program. We got one for my mother in law's last house a couple of years ago and it was very helpful in finally convincing BT there was an intermittent fault on the line.

 

If they want you as a volunteer they post you a box and you plug it in and that's basically that.

[SteamyTea]

I am a bit wary of these broadband monitoring services.

It is a great way to collect data on people.

[andyscotland]

I'd generally be cautious too, but the SamKnows one is part of a programme working with Ofcom/ISPs and only collects data on your upstream connection - their business model is they're funded by providers and regulators to research and report on connection quality. They're not e.g. selling data for advertising (which many of the commercial broadband monitoring sites do).

 

The box is basically equivalent to a raspberry pi or similar running on your network and regularly running the same kinds of tests your bash scripts are but more sophisticated and with better visualisation of the results. And sending the connection stats back to them for overall analysis of how the different providers compare. It's not directly monitoring your own connections, where you're browsing, etc - it watches to see if there's activity on the network so that it can schedule it's tests when you're not using it (to avoid the test affecting your download speed/streaming or vice versa) but that's it.

 

I haven't looked at the current version of the box, but the one we got a couple of years ago was internally just a dumb ethernet hub with the analytics hardware on one of the spare connections so it really couldn't see anything other than "is the network in use at the moment". In other words it sat alongside your connection, not as a "man in the middle". I think the code it runs was open source too, though my memory is hazy there.

[SteamyTea]

1 hour ago, andyscotland said:

upstream connection

So accessed websites and passwords then, to a government department.

[andyscotland]

1 minute ago, SteamyTea said:

So accessed websites and passwords then, to a government department.

 

Nope, not remotely correct as I said in the second part of the post. The upstream connection it's measuring is the one from your house to the wider internet, nothing to do with what you're doing on the connection yourself.

 

It sits beside the other devices on your network, same as any other device in the house, not in between them and the outside world. It does not / cannot see what websites you're accessing.

 

Likewise it does not see passwords - even if you are sending them over an unencrypted connection which to be honest you shouldn't be anyway. If you send a password over https (as you always should) then nothing bar the browser on your computer and the remote server of the site you're accessing can see it.

 

All that box does is make its own connections to the SamKnows servers to measure the download speed/upload speed/latency/jitter etc.

 

It does not attempt to measure the performance of the connections from your own pc, although it does look at whether there is any traffic (equivalent to "is there a dial tone or is the line engaged") to identify good times to run its tests.

 

It is functionally identical to your bash script, except it doesn't try to connect to Google and sends the results to SamKnows instead of a CSV file.

 

And finally it doesn't send any of the data to a government department direct, it sends it to SamKnows. They provide various reports based on aggregate stats e.g. " BT lines in this area are 95% reliable/have an average speed of XMb/have good download speed but too much jitter to be usable for video calls".