Recent purging of site content?

[Itsallaguess]

‎
Clariman,

Can I please ask you to check with Stooz if there's been an unannounced purging of legacy site content recently?

If there has been, can you please clarify where that purging has occurred, how large it was, and why it's been necessary?

Cheers,

Itsallaguess

[gryffron]

AIUI
CAN is purged regularly of old content. Because the large number of posts there makes the database too large.
Although I don't know for sure, it is quite likely this may have occurred during the recent outage, in order to shrink the files a bit.
This does of course mean you will lose old Recs/Post count.

The investment boards are not purged. Although a handful of posts may have been lost through the current "troubles".

Gryffron (moderator)

[chas49]

This process was announced (and debated) here:

viewtopic.php?t=41390

Have you noticed missing content that would not have been caught by this?

(chas49)

[Itsallaguess]

This process was announced (and debated) here:

https://www.lemonfool.co.uk/viewtopic.php?t=41390

But the initial rule-set on that announcement thread discussed the purging of Current Affairs & News posts that are more than 12 months old.

Given that the current oldest post on that board is dated December 10th 2024, then I thought it was worth asking what the current state of play is regarding the most recent purging process, and if it's still only that board that's being culled, and if so, why there's been an unannounced change in position regarding the previously stated purging period...

Cheers,

Itsallaguess

[csearle]

Given that the current oldest post on that board is dated December 10th 2024, then I thought it was worth asking what the current state of play is regarding the most recent purging process, and if it's still only that board that's being culled, and if so, why there's been an unannounced change in position regarding the previously stated purging period...

You might get a quicker answer if you email stooz. If you do, please be sure to let us know the gist of his response. C.

[Itsallaguess]

You might get a quicker answer if you email stooz.

Thanks Chris, but I'd prefer a public answer if possible, so we all know what the situation is given that things seem to have shifted from the last public announcement regarding the purging of posts.

Cheers,

Itsallaguess

[stooz]

Hi the above assumptions are correct.
During the outage we have been trying to migrate to another host who expect a smaller database. As such we increased the purge t 150 days on current news room. This was based on the fact current means that, any posts are deemed non current after a matter of weeks. Let alone the current months of capture.

We are still upgrading and may face further purges to aid transfer.

LF.

[csearle]

During the outage we have been trying to migrate to another host who expect a smaller database.

To me this seems very odd. Ok so we waffle on and post a mixture of scathingly brilliant financial insight and utter tosh, but we don't host images or videos so the entire sum of the textual content of the database MUST be a drop in the proverbial ocean. (I suspect I could probably store the entire 9-year-long Lemon Fool content on my phone.) C.

[Itsallaguess]

During the outage we have been trying to migrate to another host who expect a smaller database.

To me this seems very odd.

OK so we waffle on and post a mixture of scathingly brilliant financial insight and utter tosh, but we don't host images or videos so the entire sum of the textual content of the database MUST be a drop in the proverbial ocean. (I suspect I could probably store the entire 9-year-long Lemon Fool content on my phone.)

Agreed Chris - and regarding the above, it's obviously worth mentioning the additional implication that eventually other boards may require legacy posts purging as well, if there's been a current need to get down to a small number of months worth of Current Affairs & News posts for this proposed move.

This is a clearly a worrying development and I would hope that any future changes in purging policy might be discussed with a little more openness and clarity.

Of course we're all indebted to Stooz's efforts here, but it really doesn't take long to keep people informed in a much better way regarding these issues, and important changes to site policy like this should be clearly advertised in a much better and more timely manner.

Cheers,

Itsallaguess

[88V8]

During the outage we have been trying to migrate to another host who expect a smaller database.

To me this seems very odd....

Perhaps one could just purge all the posts mentioning Brexit.

V8

[mc2fool]

During the outage we have been trying to migrate to another host who expect a smaller database.

To me this seems very odd.

OK so we waffle on and post a mixture of scathingly brilliant financial insight and utter tosh, but we don't host images or videos so the entire sum of the textual content of the database MUST be a drop in the proverbial ocean. (I suspect I could probably store the entire 9-year-long Lemon Fool content on my phone.)

Agreed Chris - and regarding the above, it's obviously worth mentioning the additional implication that eventually other boards may require legacy posts purging as well, if there's been a current need to get down to a small number of months worth of Current Affairs & News posts for this proposed move.

This is a clearly a worrying development and I would hope that any future changes in purging policy might be discussed with a little more openness and clarity.

Of course we're all indebted to Stooz's efforts here, but it really doesn't take long to keep people informed in a much better way regarding these issues, and important changes to site policy like this should be clearly advertised in a much better and more timely manner.

Cheers,

Itsallaguess

The site currently has 628,040 posts, all of which are just text, and even if we allow a generous 10Kb per post that's only 6Gb. The word index (for searches) will typically add 10% to that and even if we add a pile for DB overheads it's surely going to be less than 10Gb, most likely lots less as most posts are much smaller than that (FYI, this one is under 3Kb).

I rather suspect that what "another host who expect a smaller database" refers to is the disk space limit of a particular hosting package and wanting a bit more space will require the next package up, at extra £cost.

Of course, as you imply, whatever the reason for the limit, if the site is having to be squeezed now to fit under it that implies little to no space for growth, and so to allow for new posts going forward there'll have to be additional -- and ongoing -- purges.

Thing is, by all accounts phpBB itself is pretty lightweight and doesn't take a lot of resources, of any kind, and, let's be honest, TLF is not exactly a big forum site in terms of users and usage, and I suspect that -- provided the bots problem was sorted out -- it'd work fine on yer average ~£100pa hosting package; going to heavier duty (more expensive) packages just giving more horsepower to the bots.

While I appreciate and understand the desire to move away from the current hosting provider (I did so some years ago), it's not going to help if the bots problem isn't dealt with....

[Itsallaguess]

While I appreciate and understand the desire to move away from the current hosting provider (I did so some years ago), it's not going to help if the bots problem isn't dealt with....

100% this...

Cheers,

Itsallaguess

[stooz]

The database in total is only 2gb, but many hosters only allow 1gb. But they allow 650 databases!



Sadly i can't split the tables out of the box.
That's another job...

Anyone that takes larger doesn't offer maintenance of the servers. You are expected to manage it yourselves.
Any recommendations welcome!!

As for not informing you, sadly the site was offline and we were taking emergency actions.
We would normally give warnings where we can.

[mc2fool]

The database in total is only 2gb, but many hosters only allow 1gb. But they allow 650 databases!

Sadly i can't split the tables out of the box.
That's another job...

Anyone that takes larger doesn't offer maintenance of the servers. You are expected to manage it yourselves.
Any recommendations welcome!!

Dunno what providers you're looking at but I've already given you one. viewtopic.php?p=689649#p689649

https://hostpresto.com/cpanel-hosting/, CPanel "Mega" will give you more than you need and is only £84pa (inc VAT). And, yes, they manage the servers. You can even try them out for a mere £1 (you'll get severe resource limitations but at least you can check it out from the inside).

[moorfield]

To me this seems very odd....

Perhaps one could just purge all the posts mentioning Brexit.

V8

You could add the Coronavirus Discussions to that list. Surely they can all be removed. Covid is "done" now, isn't it ?

And maybe all of idpickerings "news" posts. As well meaning as they are, they do clutter the place up.

[XFool]

Perhaps one could just purge all the posts mentioning Brexit.

I can see how that would be eagerly welcomed by some.

You could add the Coronavirus Discussions to that list. Surely they can all be removed. Covid is "done" now, isn't it ?

Nope.

Anymore than influenza is "done", now that 1918 is long ago.

[moorfield]

Anymore than influenza is "done", now that 1918 is long ago.

So we should have an Influenza Discussions board too then, right ?

So we should have an Alzheimers Discussions board too then, right ?


And so on.

My point being If there's of talk of smaller databases and storage capability, there's an awful lot of irrelevant fat that can be cut from what was once just a Shares, Investment and Personal Finance Discussion Forum. Scope creep!

[XFool]

Anymore than influenza is "done", now that 1918 is long ago.

So we should have an Influenza Discussions board too then, right ?

Well... in Health & Wellbeing we already have: viewtopic.php?t=41366

So we should have an Alzheimers Discussions board too then, right ?

Have to say there are already quite a few related posts scattered around the site.

My point being If there's of talk of smaller databases and storage capability, there's an awful lot of irrelevant fat that can be cut from what was once just a Shares, Investment and Personal Finance Discussion Forum. Scope creep!

The creep of the scope or the scope of the creeps?