Archiving TMF boards

[mc2fool]

I've just noticed something about the WayBack Machine (https://archive.org/) that may give us a way of archiving most or all of entire TMF boards.

If you try to go to a TMF post on the WayBack Machine that it doesn't have it gives you the option to "Help make the Wayback Machine more complete! Save this url in the Wayback Machine". Ok, so you click to do so and it does that. Fine.

However, if you then navigate from that post, by e.g. using Prev Thread | Prev | Next | Next Thread, it seems to go into auto-save mode, saving in the WayBack Machine every post you navigate to. Now, if that's true (and I've only made a cursory test), it means we can get large amounts of the TMF boards archived into the WayBack Machine by simply getting a few volunteers for each board to get into that mode on the WayBack Machine and blindly repeatedly clicking Next or Next Thread while watching the tele.

One of the challenges is to figure out how that'd work with TMF's different ways of viewing posts (threaded, unthreaded, whole thread) and different ways of addressing posts (text urls, message no urls, etc) and the ways of viewing the list of posts in a board (threaded, unthreaded, collapsed). Clearly saving whole threads would be a lot less clicking (of Prev Thread | Next Thread) than each individual post (Prev | Next) for the volunteers.

So, what seems to be needed is to work out what works and is practical, and then post instructions telling folks what to do and soliciting volunteers and, in the end, put up a sticky post in each forum that says "For the archived TMF board, start at <url-on-wayback-machine>".

I know it's a bit cheeky to come up with an idea and then ask others to do the work for it, but unfortunately I'm quite short of free time at the moment. So, it'd be great if somebody would be willing to check out the first part, "work out what works and is practical", so we can, hopefully, see if there's a way of getting the old boards archived on the WayBack Machine....

[seekingbalance]

Great post! Now hopefully people can see how useful recs are!

Have a rec

SB

[seekingbalance]

I'll look into this when I get home, but I wonder whether it is possible to request an archive of the entire site. Well, the forums at least, as the rest is pretty useless.

[Gengulphus]

I'd be cautious about this idea. Many sites will have policies against misuse that essentially amount to saying "we'll take action against attempts to use up lots of our resources on things that aren't really in accordance with our aims". I haven't investigated the WayBack Machine's policies or aims, but large-scale 'blind' archiving (as opposed to archiving of things people have decided they want to be able to look at again) might fall foul of something like that...

Gengulphus

[seekingbalance]

Yep, I agree with that point. Looking at the site it does not however appear to say that (though as a US abased site it does have a lot of legalese). As the site has a stated aim to archive any and all "historically important" information, and as it already has an extensive but not exhaustive archive of the Fool, I think it is reasonable to assume they mean it when they say that if you can't find a page you want just add it, so we can certainly do that.

As I said I would, above, I have hadn't a look at the site, especially the help and FAQ sections. It isn't not obvious whether you can ask for them to do a specific crawl, so I have emailed them to ask.

Don't ask, don't get!

We'll see what they come back with. If they say it is possible, maybe we should wait until final archive day from the Fool and get it done then.

SB

[Gengulphus]

Sorry, SB, when I said "this idea", I was actually referring to the idea in mc2fool's OP of human volunteers doing the crawling - I'd not quite understood that what you'd said in reply was about the WayBack Machine doing it. That sounds an excellent suggestion to me if they'll do it, and it's certainly worth asking - the worst they can do is say no!

Gengulphus

[88V8]

Surely, the TMF fora are a significant social history snapshot, and uniquely worth saving.

I would be willing to help with this.

V8

[nigelpm]

I would imagine given how "nice" the TMF structure/web address was there must be a way of extracting every post through a script into a csv file.

Then it might be possible to create an archive board on Lemon - and extract into there?

TMF have given their ok on use of the content.

Must be do-able - although I wouldn't know but I know folk who have done this in kind of exercise in the past.

[uspaul666]

Doable but they hinted it was large and "raw"
Guess at about 400 boards, 5000 posts each, each 10 lines of 80 characters. That's 1.6 GB. I think that's a conservative estimate. It's probably is some kind of weird format that is only readable by some really old software. It might take several days for a regular PC to chew it's way through all that data. If it's going to be processed in the cloud then there may be considerable bandwidths costs connected with uploading, downloading and transferring it all plus it might be necessary to repeat the whole procedure several times until the conversion logic is perfected.
Doable but not trivial. I'd find it a fun and worthwhile project but it'd be hard work.

[wickham]

I wonder if it would be possible to take over the domain name and hosting service TMF UK uses and give a guarantee to TMF that the forum will always be Read Only, so then Lemon Fool would carry on paying for the domain and hosting and just put an link to the archive on this forum. As long as no new members or posts are added, the software would probably carry on working for a while if hackers couldn't get in, but crashes might be a problem.

I administer a forum which has done just that, but both the old and new forums are owned by the same club.

[melonfool]

TMF Tarantula has said that TMF will help with this:

"Would it be possible to export Board content to another system? Or licence it for 3rd party use?

Yes, either of those may be possible, but there is a very large amount of data and it's in a relatively 'raw' format, so it may be difficult to import it gracefully into another system. However, we could potentially pursue this option if a viable solution can be found."

http://boards.fool.co.uk/hi-everyone-ra ... 58233.aspx

Mel

[nigelpm]

Sounds like a nice challenge for a student or someone as a uni project - they'd lap it up.

1.6Gb - peanuts by today's standards.

[88V8]

I just scraped this from TMF... perhaps he/she - RandomAmbler will post it over here. They've joined but not yet posted.

If it's of interest to anyone else I have knocked up a tool to download all of an author's posts on a particular board. The idea is that I can archive my posts before they are lost forever (whether they're worth keeping is another matter!) and it's a lot easier to do this with a tool than by hand.

At the moment I've only used this on Chrome and I know that it doesn't work on IE (yet!).

Anyway the tool is here: https://damiancannon.github.io/MotleyFoolDownloader/

Hopefully the instructions are clear enough. If anyone has any questions, feedback or problems then feel free to let me know.

NB when used for the first time Chrome will complain that you're trying to download too many files at once and will, eventually, show a dialog asking if you want to allow this - click yes to complete the downloading.

see #4541
http://boards.fool.co.uk/closure-of-the ... e#13458561

V8

[modellingman]

TMF Tarantula has said that TMF will help with this:

"Would it be possible to export Board content to another system? Or licence it for 3rd party use?

Yes, either of those may be possible, but there is a very large amount of data and it's in a relatively 'raw' format, so it may be difficult to import it gracefully into another system. However, we could potentially pursue this option if a viable solution can be found."

http://boards.fool.co.uk/hi-everyone-ra ... 58233.aspx

Mel

I'm glad this is starting to get noticed.

I've just written about what I think the implications of this might be here: viewtopic.php?f=21&t=197#p2164

[RandomAmbler]

I just scraped this from TMF... perhaps he/she - RandomAmbler will post it over here. They've joined but not yet posted.

If it's of interest to anyone else I have knocked up a tool to download all of an author's posts on a particular board. The idea is that I can archive my posts before they are lost forever (whether they're worth keeping is another matter!) and it's a lot easier to do this with a tool than by hand.

At the moment I've only used this on Chrome and I know that it doesn't work on IE (yet!).

Anyway the tool is here: https://damiancannon.github.io/MotleyFoolDownloader/

Hopefully the instructions are clear enough. If anyone has any questions, feedback or problems then feel free to let me know.

NB when used for the first time Chrome will complain that you're trying to download too many files at once and will, eventually, show a dialog asking if you want to allow this - click yes to complete the downloading.

see #4541
http://boards.fool.co.uk/closure-of-the ... e#13458561

V8

Hi V8,

Yes indeed I'm here and available for questions. The download tool appears to be working at the moment but this is in just a single environment (Chrome on Windows); I'm quite sure that there will be some issues on other environments as I only hacked this together on Sunday!

That said while I've limited the tool to download just the posts from a single user on a single board at the moment it would be feasible to extract all of the posts from any particular board - although we wouldn't want everyone doing it! Probably best to see if a data dump of the entire forum can be obtained first though.

Feel free to send me any questions/issues and I'll do my best to help.

Damian

[seekingbalance]

Update - I emailed the Wayback machine people asking if they have a process for a manual crawl for the entire boards section of the site. A week later, no reply from them!

I will try again tomorrow.

SB

[mc2fool]

Update - I emailed the Wayback machine people asking if they have a process for a manual crawl for the entire boards section of the site. A week later, no reply from them!

I will try again tomorrow.

SB

They do have forums and one of those may be of utility (sorry, once again I'm short of time to research it myself )

https://archive.org/iathreads/forums.php

[TawnyOwl]

I cannot see any reason why saving the old TMF boards is worthwhile. Basically nobody reads old posts, it's a waste of time and effort.

Nowadays there are so many sites, so much stuff, out there that nobody can ever keep up with it. I used to post on many boards on many sites but have basically given them all up because it's like posting into a black hole. I'm still here simply because I have an interest in investment and might find or contribute something useful.

Tawny

[mc2fool]

I cannot see any reason why saving the old TMF boards is worthwhile. Basically nobody reads old posts, it's a waste of time and effort.

That may be true for some boards but for others, e.g. the Pensions - Practical Problems board, folks regularly refer -- and link -- back to previous detailed explanations of complex matters and their discussions, so I must disagree with the assertion that "nobody" reads old posts and it's a waste of time and effort.

[poundcoin]

Unlike most others , there were only a couple of threads I initiated on TMF that are worth me saving .

I viewed them in whole thread mode , highlighted ,copied and pasted into an email and sent it to myself . As long as my yahoo account doesn't get hijacked should be there for ever .

Seems to view in the exact same format as TMF page and the external URLs all work .

I realise that this method would be tedious for those wanting to save hundreds of posts .