More Glitches in the ATS Matrix

[Maxmars]

I feel like discussing scraping the ATS database is like metaphorically "waiting for the other shoe to drop."

At some point, the answer seems to point to being technically able to rescue the posts and research of the deteriorating ATS platform, but not without breaking some notional contract using the preposterous Madison Avenue/Hollywood-esque verbiage "... everything is ours in perpetuity in this universe and all others."  (I find such legal contrivances laughably shameful.)

I must point out that the member/creator does not "transfer" their rights to TAN by by posting... they only "grant" specific rights... creators; by default they retain their own rights to use their created material as they wish.  This isn't Hollywood or Madison Avenue - as much as the "owners" wished it were.

Ultimately, this quandary is left to be resolved when the eventual "owner" surfaces.

[FlyingClayDisk]

....but not without breaking some notional contract using the preposterous Madison Avenue/Hollywood-esque verbiage "... everything is ours in perpetuity in this universe and all others."  (I find such legal contrivances laughably shameful.)

...

LOL!  Not to mention 'unenforceable'!

I got a kick out of your description.  I call those types of clauses and language the "God save us all" clauses.

In the Engineering contracts I work with on a daily basis, I refer to them as the... "God save us all...and everything else we forgot or screwed up...is all YOUR responsibility!!"...clause.

[ArMaP]

It sounds like a pretty efficient process.  I think I'd probably be inclined to export the data into an application where I could easily cull out the extraneous data.  For example, Post and Thread ID aren't really relevant to anyone other than ATS.

By keeping the post and thread ID I can know what post is being answered when there's only a link, as that link has both. Also, the thread ID points to the threads list, where I have the thread's name and in which forum it was posted.

Is your program automated, or do you have to extract the data on a manual thread by thread basis?  In other words, do you have to know the name/title of the the thread before you can capture the data, or have you developed some kind of a wildcard to automate this process?  I would think it would be a pretty tedious slog through the mud without some automation like that.

It's automated. I choose a forum from a list and the program reads all threads from the threads list, then goes through the list and reads all posts from each thread. For each post it saves only the HTML that creates the post area, where the content is.

In summary, I guess my posture would be; things being what they are over there on ATS, I seriously doubt there is anyone with enough legal standing to care one way or the other what happens to the data.  They don't seem to care about anything else, and I think we all agree that the content data is the most valuable part of what remains of ATS.

One thing for which I don't have an answer and that could change things is what happened to TAN.

If TAN still exists then all references to TAN apply.
If TAN exists but wasn't sold along with the site then the original owners are the co-owners of the content, not the new owners.
If TAN doesn't exist anymore, what happens? Are the individual posters the only owners of the content they created?

Well, you have one of my highest compliments if you do it like that.  You speak excellent English.

Thanks.

I suppose I do it like that because I learned English by watching subtitled movies and TV shows, so I don't have the basic knowledge other people that learn a language by "traditional" methods have.
 

When I first started learning Spanish I used to do it that way also, but it became too difficult after a while primarily because of the differences in sentence structure / word order.  I think native tongue English speakers are at a disadvantage learning other languages because of exactly this.

Other languages, like Japanese, have a completely different structure that does not follow either the Romance or Germanic languages' structures, which makes it hard to be learned by people that speak either Romance or Germanic languages.

Because of that I don't have much hope of learning Japanese just by watching Japanese animation, but I already learned some words and a four or five characters. Now I only need to learn the other  300. 
 

I really ran into trouble with my skills while working down in Peru.  There, Portuguese and other dialects of Spanish creep into the language.  We were working with a team out of our Madrid office, and I had a gal tell me one time that she was born and raised in Mexico and now lives in Spain.  That transition was no big deal to her, but when she attempted to communicate in Peru..."people look at me like I have two heads; they don't understand a word I'm saying"  I was sure glad to learn I wasn't the only one!!  It's like a whole other language.

Yes, the South American versions of Spanish have many other influences. In Argentina, for example, they have a relatively strong Italian influence.

[ArMaP]

I think it is still available on the WayBack Machine...

1401 pages captured from March 2011 - September 2013:

Japan declares nuclear emergency after quake - Part I

After going through the first 300 pages, 10 pages are already missing. 

[Encia22]

After going through the first 300 pages, 10 pages are already missing. 

Sorry to hear that, ArMap. 

I wonder why the W.B Machine missed them. Can you deduce if they are consecutive pages or just random?

One thing that may have created blanks could have to do with the way ATS paginates threads. I remember a certain condition or glitch when there is only one new post on a new page, and the page number appears in the page navigation menu, but the page itself then appears blank. After a second reply is posted, then the page loads properly.

Was that happening back then, too? Could something like that have caused the WBM spider to skip pages?

 

[Notran]

I’m confused. If it’s a right wing echo chamber, and anyone that disagrees gets banned — how is it that you still have an account there?

I didn't see everyone who disagrees gets banned.
The more vocals are banned or shadowbanned and the less vocals and the others who have different opinions are subjected to self censorship.
Take a look at the threads on ATS. Mostly it's about Trump and Trumpism with no opposition from the democrats.
If you Google to find out what the site is, it says a tight wing conspiracy site. And they are correct.

---

Maybe.

I don't. If, as suspected, there was a change on the email servers that, without an admin, cannot be applied to ATS, it will keep on trying to use the wrong server configuration and no emails will be sent.

But mods have the power to ban members, so not letting new members on purpose could backfire, as those that the hypothetical hidden admin wanted to keep on posting could be banned.
For a hidden admin with an agenda it would be better to allow new members and to post ban the members with the opposing point of view. That way ATS would become a controlled echo chamber instead of a possible echo chamber.

Or... To not allow anyone else in platform making it more of an echo chamber than it is and then start banning everyone including the mods.
Wipeout all members and mods while the site still exists.

[DBCowboy]

I didn't see everyone who disagrees gets banned.
The more vocals are banned or shadowbanned and the less vocals and the others who have different opinions are subjected to self censorship.
Take a look at the threads on ATS. Mostly it's about Trump and Trumpism with no opposition from the democrats.
If you Google to find out what the site is, it says a tight wing conspiracy site. And they are correct.

---

Or... To not allow anyone else in platform making it more of an echo chamber than it is and then start banning everyone including the mods.
Wipeout all members and mods while the site still exists.

ANY site that doesn't censor conservative voices is considered a "right-wing" site.

 

[ArMaP]

Sorry to hear that, ArMap. 

I wonder why the W.B Machine missed them. Can you deduce if they are consecutive pages or just random?

Some consecutive, some apparently random.
These are the missing pages up to page 300: 112, 195, 202, 203, 209, 217, 220, 277, 290 and 291.

One thing that may have created blanks could have to do with the way ATS paginates threads. I remember a certain condition or glitch when there is only one new post on a new page, and the page number appears in the page navigation menu, but the page itself then appears blank. After a second reply is posted, then the page loads properly.

Was that happening back then, too? Could something like that have caused the WBM spider to skip pages?

Yes, it has always happened, but I doubt that messed up WBM's spider, as it could save those pages when the page count increased.

I think the problem is that the Wayback Machine's copies of ATS pages were not done as a complete copy of the whole site but requested by some means, that's why we have some ATS pages with many copies and other pages with no copies at all.

[Encia22]

Some consecutive, some apparently random.
These are the missing pages up to page 300: 112, 195, 202, 203, 209, 217, 220, 277, 290 and 291.

Yes, it has always happened, but I doubt that messed up WBM's spider, as it could save those pages when the page count increased.

I think the problem is that the Wayback Machine's copies of ATS pages were not done as a complete copy of the whole site but requested by some means, that's why we have some ATS pages with many copies and other pages with no copies at all.

Ok, understood and thanks for letting us know which pages are missing. I don't know what criteria the WBM uses, so it's anyone's guess why some pages are missing, especially the random ones.