Tracking online prices - web scraping

[Moonshine]

Following on from the thread on timber frame price increases, i have made a simple python code to scrape prices from consumer websites to track how outlet prices may go up or down.

 

My plan is to run the code automatically daily to scrape prices from websites and save them to an excel file for select products, which can be tracked over time. A sample of the data input CSV file is below;

 

Item	Store	Description	Website
1	Wickes	18mm 2440 x 1220 OSB	https://www.wickes.co.uk/Wickes-General-Purpose-Oriented-Standard-Board-3-OSB-3---18-x-1220-x-2440mm/p/110517
2	Wickes	Blue Circle General Purpose Cement - 25kg	https://www.wickes.co.uk/Blue-Circle-General-Purpose-Cement---25kg/p/224661
3	Wickes	Studwork CLS Timber - 38 X 89 X 2400mm	https://www.wickes.co.uk/Wickes-Studwork-CLS-Timber---38-x-89-x-2400mm/p/107713
4	B&Q	18mm 2440 x 1220 OSB	https://www.diy.com/departments/osb-3-board-l-2-44m-w-1-22m-t-18mm/1696229_BQ.prd

 

A sample of the data output is below, which will be amended on a daily basis;

 

	Wickes	Wickes	Wickes	B&Q
	18mm 2440 x 1220 OSB	Blue Circle General Purpose Cement - 25kg	Studwork CLS Timber - 38 X 89 X 2400mm	18mm 2440 x 1220 OSB
07/07/2021 10:16	£45.00	£4.15	£10.45	£39.00
07/07/2021 10:18	£45.00	£4.15	£10.45	£39.00
07/07/2021 10:22	£45.00	£4.15	£10.45	£39.00

 

The question is what sort of products / stores I should be including to get a good spread of different goods, such as the consumer price inflation basket of goods and services, but for construction.

 

Any suggestions?

[Olf]

Multifinish, concrete blocks, insulation boards please

We may find the prices possibly dropping: few weeks ago I had to buy OSB urgently (collapsing foundation trenches), rented a van, sprinted to B&Q and paid £29 a board. 

3 days later doing order for roof boards I was hit with £39 price - 1/3 hike! But now it stays. I suspect some materials got 'future increases' included on the wave of panic buying, with all the desperate ones served and shortage of labour, stock will have to be shifted so the price adjust. Though OSB doesn't have best before date, otherwise there would be soon pallets given away, like with plaster...

[WWilts]

Amazing technology. Great work.

Building Merchants: Travis Perkins, Jewsons, Bradfords, Buildbase, a couple of independent Building Merchants, a couple of online merchants, a couple of specialists (roofing goods, drainage goods, flooring, tiles, sanitary ware, UFH, other specialist heating/ventilation)

Goods: Bricks, standard blocks, rebar, pea gravel, MOT type 1, roofing supplies, aircrete blocks, cement (ready mixed / bagged), cast stone, lintels, windows, tiles

[Moonshine]

Just to cavate it, any builders merchants will the publicly available prices, not on account. I want to get a snap shot of a range products rather than going into too much detail. Though it may be interesting to see the same product from a couple of suppliers to see if the changes are consistant.

 

if you can add the URL's of products to the list below, it would make my life easier.

1. https://www.wickes.co.uk/Wickes-General-Purpose-Oriented-Standard-Board-3-OSB-3---18-x-1220-x-2440mm/p/110517
2. https://www.wickes.co.uk/Blue-Circle-General-Purpose-Cement---25kg/p/224661
3. https://www.wickes.co.uk/Wickes-Studwork-CLS-Timber---38-x-89-x-2400mm/p/107713
4. https://www.diy.com/departments/osb-3-board-l-2-44m-w-1-22m-t-18mm/1696229_BQ.prd

[nod]

Timber prices will have to fall once stocks start to mount up

A 70 mil C stud has just gone up in price to 2.88

Less than half the price of equivalent timber 

[Moonshine]

6 minutes ago, nod said:

A 70 mil C stud has just gone up in price to 2.88

Less than half the price of equivalent timber 

 

I am wondering given the increased costs if people may start to looking at metal framing as a cost effective solution for internal walls (with other benefits)

[Carrerahill]

2 hours ago, Moonshine said:

Following on from the thread on timber frame price increases, i have made a simple python code to scrape prices from consumer websites to track how outlet prices may go up or down.

 

My plan is to run the code automatically daily to scrape prices from websites and save them to an excel file for select products, which can be tracked over time. A sample of the data input CSV file is below;

 

Item	Store	Description	Website
1	Wickes	18mm 2440 x 1220 OSB	https://www.wickes.co.uk/Wickes-General-Purpose-Oriented-Standard-Board-3-OSB-3---18-x-1220-x-2440mm/p/110517
2	Wickes	Blue Circle General Purpose Cement - 25kg	https://www.wickes.co.uk/Blue-Circle-General-Purpose-Cement---25kg/p/224661
3	Wickes	Studwork CLS Timber - 38 X 89 X 2400mm	https://www.wickes.co.uk/Wickes-Studwork-CLS-Timber---38-x-89-x-2400mm/p/107713
4	B&Q	18mm 2440 x 1220 OSB	https://www.diy.com/departments/osb-3-board-l-2-44m-w-1-22m-t-18mm/1696229_BQ.prd

 

A sample of the data output is below, which will be amended on a daily basis;

 

	Wickes	Wickes	Wickes	B&Q
	18mm 2440 x 1220 OSB	Blue Circle General Purpose Cement - 25kg	Studwork CLS Timber - 38 X 89 X 2400mm	18mm 2440 x 1220 OSB
07/07/2021 10:16	£45.00	£4.15	£10.45	£39.00
07/07/2021 10:18	£45.00	£4.15	£10.45	£39.00
07/07/2021 10:22	£45.00	£4.15	£10.45	£39.00

 

The question is what sort of products / stores I should be including to get a good spread of different goods, such as the consumer price inflation basket of goods and services, but for construction.

 

Any suggestions?

It would be pretty good if you could export the results and use them on a QS spreadhseet, when I priced my builds I just used an excel and manually entered the data, things were more stable then, if that could update with an input from this that would be epic. 

 

I am not suggesting that you spend the time and effort doing this personally but could be an option for someone who is maybe tracking a build. 

[Carrerahill]

2 minutes ago, Moonshine said:

 

I am wondering given the increased costs if people may start to looking at metal framing as a cost effective solution for internal walls (with other benefits)

This sort of stuff is already happening, we are providing M&E design services on a restaurant build, they have just converted it to a traditional brick build because they could not get the cladding and and the portals and things for reasonable money. 

[Moonshine]

16 minutes ago, Carrerahill said:

It would be pretty good if you could export the results and use them on a QS spreadhseet

 

The exported file is a CSV file (Comma separated variable) so easily opened in Excel

[nod]

2 hours ago, Moonshine said:

 

I am wondering given the increased costs if people may start to looking at metal framing as a cost effective solution for internal walls (with other benefits)

I’ve had two enquires both medium sized housing sites 

We can also cut there labour costs by 50% 

Normally MF is slightly more expensive to supply 

[Barryscotland]

On 07/07/2021 at 13:05, Moonshine said:

 

I am wondering given the increased costs if people may start to looking at metal framing as a cost effective solution for internal walls (with other benefits)

The price of metal has gone up almost as much as timber. A252 3.6x2 mesh when bought for founds was £27 in April, bought  it for subfloor in July £38, was a sheet short when laid out this week £68. 

[SteamyTea]

On 07/07/2021 at 10:37, Moonshine said:

i have made a simple python code to scrape prices from consumer websites to track how outlet prices may go up or down.

Good idea, I used to use WebWhacker to do the same, then they started charging money for it.

Any chance you want to share the code then others can join it.

Data is King.

 

For the non technically minded, a similar thing can be done with Excel, using the Web Query to import some data.

Edited July 8, 2021 by SteamyTea

[Moonshine]

11 hours ago, SteamyTea said:

Any chance you want to share the code then others can join it.

 

The basic code is on another thread, but its not like its that complex.

 

Interestingly Wickes have just put their cement up to £4.60, where at B&Q is still at £4.12

[Pocster]

Can you please add tracking for walk on glazing ?

[Mr Punter]

4 hours ago, pocster said:

Can you please add tracking for walk on glazing ?

 

No point as someone in Bristol is distorting the market by hoarding redundant stock.

[PeterW]

1 hour ago, Mr Punter said:

 

No point as someone in Bristol is distorting the market by hoarding redundant stock.

Depends - if it new old stock it will have a premium value to a collector who wants to recreate or restore a leaking old dungeon to “as built” then I suppose there is a market force … 

[Pocster]

2 hours ago, PeterW said:

Depends - if it new old stock it will have a premium value to a collector who wants to recreate or restore a leaking old dungeon to “as built” then I suppose there is a market force … 

It’s unique stock . Once in a lifetime never to be repeated . I’m struggling to not up the price by 20% like all other building materials. It won’t stay at this discount price for long . When it’s gone it’s gone !

[Pocster]

3 hours ago, Mr Punter said:

 

No point as someone in Bristol is distorting the market by hoarding redundant stock.

Excuse me . I’m simply seeing it as an investment opportunity. It was this or Tesla shares . I felt Elon didn’t deserve my money .

[Onoff]

1 hour ago, pocster said:

It’s unique stock . Once in a lifetime never to be repeated . I’m struggling to not up the price by 20% like all other building materials. It won’t stay at this discount price for long . When it’s gone it’s gone !

 

Probably a glass ceiling price wise for such a niche product...

[SteamyTea]

13 hours ago, pocster said:

I felt Elon didn’t deserve my money .

But the French did, so you got a Zoe.