WHAT ATTRACTED YOUR ATTENTION TODAY?
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Wendy, I will keep the FaceBook link in mind. Thanks.
I know I'm in my own little world, but it's OK... they know me here. ![Smile :)](./images/smilies/icon_e_smile.gif)
![Smile :)](./images/smilies/icon_e_smile.gif)
- Stanley
- Global Moderator
- Posts: 90859
- Joined: 23 Jan 2012, 12:01
- Location: Barnoldswick. Nearer to Heaven than Gloria.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Peter, it's asking me for my password frequently as well. I don't think it's personal!
Stanley Challenger Graham
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
- PanBiker
- Site Administrator
- Posts: 16561
- Joined: 23 Jan 2012, 13:07
- Location: Barnoldswick - In the West Riding of Yorkshire, always was, always will be.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
No changes to the password requirements. At the moment anything can happen I suppose a bit like "Stingray"
![Extra Wink :extrawink:](./images/smilies/board/winkextra.gif)
I have given the site 4 hours of my life today and I am at a point near to getting my ducks in a row. I will sprinkle the Firewall with some more IP ranges to block later. I'm going to have a break now and wash the car!
We seem to be in a trough at the moment.
Ian
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Ian,
It is very much appreciated all the effort you are putting in to sort this, I will start a round of applause for you![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
It is very much appreciated all the effort you are putting in to sort this, I will start a round of applause for you
Gloria
Now an Honorary Chief Engineer who'd be dangerous with a brain!!!
http://www.briercliffesociety.co.uk
http://www.lfhhs.org.uk
Now an Honorary Chief Engineer who'd be dangerous with a brain!!!
http://www.briercliffesociety.co.uk
http://www.lfhhs.org.uk
- PanBiker
- Site Administrator
- Posts: 16561
- Joined: 23 Jan 2012, 13:07
- Location: Barnoldswick - In the West Riding of Yorkshire, always was, always will be.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
This is what we are up against. The graphs should be a relatively straight line somewhere up the scale when operating normally.
I have isolated and added to the firewall 16,903,955 IP addresses that are causing problems, that is roughly half of what is in the logs. All these belong to Amazon Web Servers that have been attacked by spammers who are using the Amazon addresses
I have isolated and added to the firewall 16,903,955 IP addresses that are causing problems, that is roughly half of what is in the logs. All these belong to Amazon Web Servers that have been attacked by spammers who are using the Amazon addresses
You do not have the required permissions to view the files attached to this post.
Ian
- Stanley
- Global Moderator
- Posts: 90859
- Joined: 23 Jan 2012, 12:01
- Location: Barnoldswick. Nearer to Heaven than Gloria.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
It sounds like a massive task Ian. Thanks for the effort.
Stanley Challenger Graham
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
There is activity going on 'under the bonnet', Ian is researching another service that Ionos can provide to block the IP addresses. Hopefully he'll be able to provide an update on progress later on.
Kev
Stylish Fashion Icon.
![🍹](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f379.svg)
Stylish Fashion Icon.
- PanBiker
- Site Administrator
- Posts: 16561
- Joined: 23 Jan 2012, 13:07
- Location: Barnoldswick - In the West Riding of Yorkshire, always was, always will be.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Kev is very good at ferreting the options out. Trying to find out what we can get for free as a start. There are subscription or paid options available also but no pricing yet. Spam blocking is probably very good business, probably replaced to some degree anti-virus threats. Apparently a Cloud Expert will be in contact soon. ![Extra Wink :extrawink:](./images/smilies/board/winkextra.gif)
![Extra Wink :extrawink:](./images/smilies/board/winkextra.gif)
Ian
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
I've upgraded the mySQL database that's 'behind' the site. The monitoring is currently showing a decrease in the CPU load average, not sure if we're out of the woods yet. I will continue to monitor.
Kev
Stylish Fashion Icon.
![🍹](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f379.svg)
Stylish Fashion Icon.
- Stanley
- Global Moderator
- Posts: 90859
- Joined: 23 Jan 2012, 12:01
- Location: Barnoldswick. Nearer to Heaven than Gloria.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Only a very slight delay in connecting so far this morning. It is accompanied by the bouncing ball in the Oneguy tab marker.
Such a luxury to be able to function. We lose sight of the really important things of life and the last week has definitely reinforced how important Oneguy is to me!
Let me know if you need any subsidy for anti-spam activities.
Interesting comment there about the relative value of anti-spam compared to anti-virus....
Such a luxury to be able to function. We lose sight of the really important things of life and the last week has definitely reinforced how important Oneguy is to me!
Let me know if you need any subsidy for anti-spam activities.
Interesting comment there about the relative value of anti-spam compared to anti-virus....
Stanley Challenger Graham
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
I emailed abuse@amazonaws.com yesterday with a list of their server IP addresses that are spamming the site. I have been told they take these things seriously.
Kev
Stylish Fashion Icon.
![🍹](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f379.svg)
Stylish Fashion Icon.
- PanBiker
- Site Administrator
- Posts: 16561
- Joined: 23 Jan 2012, 13:07
- Location: Barnoldswick - In the West Riding of Yorkshire, always was, always will be.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Site is operational this morning. I have checked the traces and the CPU is running at around 80% which is still too high and well above normal. We are no doubt still being hit with spam packets but I have given up trying to chase these. It's like flogging a dead horse, time consuming and very frustrating because as soon as you identify one lot that are doing damage they just steal another batch of different IP's to do the same. So you have to start again.
We have to move to an automated defence system which is what we are working on at the moment. Ball is in the IONOS court at the moment and I am waiting for feedback.
The option is not user configurable so we cant install it ourselves. Effectively it will be a separate database of known rogue IP addresses which is maintained by IONOS. The site traffic will be redirected, filtered, and packets compared with the database before being processed. Any packets from known bad IP's will be automatically blocked.
There is a free version and an enhanced version with more control, bells and whistles. I have asked for details of both. Free version will definitely be a benefit for the site and may well give us a bit of breathing space.
We have to move to an automated defence system which is what we are working on at the moment. Ball is in the IONOS court at the moment and I am waiting for feedback.
The option is not user configurable so we cant install it ourselves. Effectively it will be a separate database of known rogue IP addresses which is maintained by IONOS. The site traffic will be redirected, filtered, and packets compared with the database before being processed. Any packets from known bad IP's will be automatically blocked.
There is a free version and an enhanced version with more control, bells and whistles. I have asked for details of both. Free version will definitely be a benefit for the site and may well give us a bit of breathing space.
Ian
- Stanley
- Global Moderator
- Posts: 90859
- Joined: 23 Jan 2012, 12:01
- Location: Barnoldswick. Nearer to Heaven than Gloria.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
"Site is operational this morning."
Depends what you mean by operational.....
I'm still getting routine 502 errors and delays.
I've just been listening to Marta Lane Fox on 'The Week in Westminster' and I have a big problem with her and many others like her. They talk so fast that if you miss one word and the context goes it becomes gibberish. I had no problem with the other guests or the presenter so don't blame my hearing or the speakers!
Depends what you mean by operational.....
![Big Grin :biggrin2:](./images/smilies/board/biggrin.gif)
I've just been listening to Marta Lane Fox on 'The Week in Westminster' and I have a big problem with her and many others like her. They talk so fast that if you miss one word and the context goes it becomes gibberish. I had no problem with the other guests or the presenter so don't blame my hearing or the speakers!
Stanley Challenger Graham
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
Stanley's View
scg1936 at talktalk.net
"Beware of certitude" (Jimmy Reid)
The floggings will continue until morale improves!
- PanBiker
- Site Administrator
- Posts: 16561
- Joined: 23 Jan 2012, 13:07
- Location: Barnoldswick - In the West Riding of Yorkshire, always was, always will be.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
By operational I meant it was working at the time I posted as it is now. If you look at the CPU trace I put up earlier you can see why the site is up and down. The graph is basically a square wave but the highs which flood the CPU are much longer duration than the lows which allows the site to function. As I said before, under normal usage and not under attack the CPU usually runs at 25% - 30%, anything above about 85% will reflect as the site slowing down when posting etc and eventually the 502 errors when it is fully overwhelmed.
To put into context what we are trying to deal with. The current 32 bit addressing used by the IPv4 network protocol creates 4,294,967,296 individual and unique IP addresses. Large IT corporations such as Microsoft, Amazon, Google etc own large numbers of IP addresses (hundreds of thousanad or even millions) from this pool which they either sell on or lease to smaller companies and users which is how they make money. It's a trickle down business with each owner making a few bob on each address. IONOS will have a pool which they vend out to their customers like us. We pay for use of the IP addresses that run the site.
The spammers relay on these large pools and steal huge chunks of them, not usually in continuous blocks and then use them effectively by allocating them to spam bots, thousands of them in fact. They set them off on their merry way with whatever payload or action they have been given. Once triggered the operation is fully automatic and only needs updating when the IP pool has been rumbled and added to thousand of firewalls and other anti spam defence systems. When this happens they simply refresh their pool with different stolen address. The other day I banned over 16 million addresses in our firewall that seem to belong to Amazon Web Servers but have been stolen by the spammers. DOS attacks (Denial of Service) are effectively the same as kidnapping as they can be used to hold target companies, institutions, businesses to ransom. Invariably the spammers are working from VPN's and using Proxy servers which effectively mask who and where they are.
The world has actually run out of 32 bit IP addresses and the entire system is now running on the legacy technology that supports it. Since it's inception all the security vulnerabilities have been found and are relied upon by the spammers.
To kill two birds with one stone a new protocol using 128bit encoding has been developed this is the next step in the evolution of the global networks. This is IPv6 and instead of the 4 x 8 bits used in 32bit addressing, IPv6 uses 6 x 128 bit addressing. If you multiply this out you arrive at 340, Trillion, Trillion, Trillion unique addresses. Each address in it's binary form will be 768 digits long! I'm glad I wont be around to have to deal with any problems with that network protocol.
To put into context what we are trying to deal with. The current 32 bit addressing used by the IPv4 network protocol creates 4,294,967,296 individual and unique IP addresses. Large IT corporations such as Microsoft, Amazon, Google etc own large numbers of IP addresses (hundreds of thousanad or even millions) from this pool which they either sell on or lease to smaller companies and users which is how they make money. It's a trickle down business with each owner making a few bob on each address. IONOS will have a pool which they vend out to their customers like us. We pay for use of the IP addresses that run the site.
The spammers relay on these large pools and steal huge chunks of them, not usually in continuous blocks and then use them effectively by allocating them to spam bots, thousands of them in fact. They set them off on their merry way with whatever payload or action they have been given. Once triggered the operation is fully automatic and only needs updating when the IP pool has been rumbled and added to thousand of firewalls and other anti spam defence systems. When this happens they simply refresh their pool with different stolen address. The other day I banned over 16 million addresses in our firewall that seem to belong to Amazon Web Servers but have been stolen by the spammers. DOS attacks (Denial of Service) are effectively the same as kidnapping as they can be used to hold target companies, institutions, businesses to ransom. Invariably the spammers are working from VPN's and using Proxy servers which effectively mask who and where they are.
The world has actually run out of 32 bit IP addresses and the entire system is now running on the legacy technology that supports it. Since it's inception all the security vulnerabilities have been found and are relied upon by the spammers.
To kill two birds with one stone a new protocol using 128bit encoding has been developed this is the next step in the evolution of the global networks. This is IPv6 and instead of the 4 x 8 bits used in 32bit addressing, IPv6 uses 6 x 128 bit addressing. If you multiply this out you arrive at 340, Trillion, Trillion, Trillion unique addresses. Each address in it's binary form will be 768 digits long! I'm glad I wont be around to have to deal with any problems with that network protocol.
Ian
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
I have managed to apply the firewall changes, the lists of AWS IP addresses that Ian added are now actively blocked. I have seen a big drop in CPU usage and will continue to monitor this afternoon for anything unusual.
A bit later, CPU is around 22% which is promising. I can see regular users logged into the site, good to see Mick Brett logged in as a lot of the blocked IPs are US based
The CPU trace for the last hour or so.
I'm still monitoring![Big Grin :biggrin2:](./images/smilies/board/biggrin.gif)
A bit later, CPU is around 22% which is promising. I can see regular users logged into the site, good to see Mick Brett logged in as a lot of the blocked IPs are US based
![Good :good:](./images/smilies/board/good.gif)
I'm still monitoring
![Big Grin :biggrin2:](./images/smilies/board/biggrin.gif)
You do not have the required permissions to view the files attached to this post.
Kev
Stylish Fashion Icon.
![🍹](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f379.svg)
Stylish Fashion Icon.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
I'd best get back to my day job now ![Big Grin :biggrin2:](./images/smilies/board/biggrin.gif)
![Big Grin :biggrin2:](./images/smilies/board/biggrin.gif)
Kev
Stylish Fashion Icon.
![🍹](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f379.svg)
Stylish Fashion Icon.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Last hour CPU
![Good :good:](./images/smilies/board/good.gif)
Tomorrow morning will be the real test ![Good :good:](./images/smilies/board/good.gif)
You do not have the required permissions to view the files attached to this post.
Kev
Stylish Fashion Icon.
![🍹](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f379.svg)
Stylish Fashion Icon.
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Excellent ![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
![👏](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f44f.svg)
Gloria
Now an Honorary Chief Engineer who'd be dangerous with a brain!!!
http://www.briercliffesociety.co.uk
http://www.lfhhs.org.uk
Now an Honorary Chief Engineer who'd be dangerous with a brain!!!
http://www.briercliffesociety.co.uk
http://www.lfhhs.org.uk
- PostmanPete
- Regular User
- Posts: 245
- Joined: 24 Jan 2012, 09:22
- Location: Barnoldswick
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Thanks Ian and Kev for all the work you have done over the last few days. Hopefully you have got the site back to normal now and can get back to your normal routine
It looks to be working fine at my end.
![Good :good:](./images/smilies/board/good.gif)
"Always carry a large flagon of whisky in case of snakebite and furthermore always carry a small snake."
W.C. Fields (1880-1946)
W.C. Fields (1880-1946)
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
That's good to hear Pete, there was one hiccup when I accidentally blocked Wendy
All sorted now though. This did involve removing a couple of the newly applied firewall rules but, hopefully, as the range is UK based, it may not impact things.
CPU usage is still good and there's nowt weird in the logs.
![Big Grin :biggrin2:](./images/smilies/board/biggrin.gif)
All sorted now though. This did involve removing a couple of the newly applied firewall rules but, hopefully, as the range is UK based, it may not impact things.
CPU usage is still good and there's nowt weird in the logs.
Kev
Stylish Fashion Icon.
![🍹](//cdn.jsdelivr.net/gh/twitter/twemoji@latest/assets/svg/1f379.svg)
Stylish Fashion Icon.
- Wendyf
- Global Moderator
- Posts: 9496
- Joined: 23 Jan 2012, 12:26
- Location: Lower Burnt Hill, looking out over Barlick
Re: WHAT ATTRACTED YOUR ATTENTION TODAY?
Perhaps I was causing the trouble all along.
I'll get my coat.....
![Sad :sad:](./images/smilies/sad.gif)