|
|
The Sex Blog Of Record
Friday, July 19th, 2013 -- by Bacchus
I was among the first to discover back on May 15 that Tumblr was using an exclusionary robots.txt file to hide the contents of blogs flagged “Adult” from all search engines, the Internet Archive crawler, and any other internet service that respects robots.txt files. A few days later when I was poking at that unpleasant fact, I also discovered that Tumblr was excluding these blogs from its internal tag-search function:
Worse yet? Tumblr blogs flagged “adult” aren’t searchable even with Tumblr’s own internal search. You can test this yourself. Log into your Tumblr dashboard, go to your settings, and make sure you haven’t checked the “Browse tag pages in Safe Mode (Hide content from NSFW blogs)” setting:
Unlike the one that doesn’t actually “allow search engines to index your blog”, this checkbox appears to actually work in the narrow sense that if it is not checked, you can search for blogs flagged “NSWF” within the Tumblr tag search interface. But this checkbox lies by omission. You’ve got the option to search tag pages of NSFW blogs (or not) but opting to search them does not let you search blogs that have the deeper-level-of-perdition “adult” flag.
I even proved it with a careful set of screenshots:
My test for this is to search for a recent post at Wicked Knickers, which I used as my “adult” flagged example in the Thou Shalt Not Search Adult Tumblr Blogs post:
The post we will be looking for in the Tumblr dashboard tag search has a time stamp of 9:30pm yesterday, May 18, and is tagged “ziegfeld” which makes it a nice handy and recent thing to search for.
So, what happens in the Tumblr tag search interface? If you’re logged in, this is what you see when you search for tumblr posts with the “ziegfeld” tag. The posts returned are listed in date order (most recent first) and dates are visible as tooltips on the live page, so I’ve added them in the margin with red arrows and white text. You’ll see that the Wicked Knickers post is not returned by the Tumblr search:
Thus you can imagine my surprise when Twitter started blowing up yesterday with outraged Tumblr-users who had only just discovered that their adult blogs had gone missing (2-3 months ago) from the Tumblr tag search interface. Of course it was all over Tumblr as well. I didn’t pay it any mind; I was busy yesterday and figured it was just one of those moments when public consciousness crystallizes about a long-established injustice.
It wasn’t until this morning, when I finally had time to get caught up, that I discovered what had caused the moment of crystallization. At first I thought it was the Daily Dot article that appeared yesterday, which I only skimmed at the time due to it being such ancient news to me: New NSFW content restrictions enrage Tumblr users. But then this morning Violet Blue published a similar (but much better, and I’d say that even if she hadn’t quoted my May reportage in detail) article at ZDNet: Adult Tumblr blogs now removed from every form of search possible. Violet’s too good a reporter to jump on a bandwagon just because it’s starting to speed up, so I took a second look at both articles to see what, if anything, had actually happened recently and wasn’t old news. Finally I twigged to it. Although the Tumblr internal search (which has always been a tags-only search) hasn’t revealed content from blogs flagged “adult” for months, the “new thing” is that Tumblr’s cryptic little internal checklist has finally (yesterday? Nobody seems to know just when) been updated to reveal that fact. Here’s the box as it existed on May 15, with my red arrow:
And here’s the box as it is today, which Violet linked to and the Daily Dot printed:
I’ve outlined what’s actually new in the graphic. Once again, it isn’t new policy at Tumblr; these are the rules since (near as I can tell) some time in February. What’s new is that Tumblr is now admitting what the rules have been for some time.
Now we come to the happy fun-time “evasions and denials” section of this post!
Remember, first, that Tumblr’s internal search has always been a tags-and-titles-only affair. So, look again at that line in the first red box of mine: “Posts appear in tag pages and search pages for logged-out users.” They are playing games with multiple pairs of yes-no variables here, for maximum confusion. The one checked box is for unflagged blogs; the unstated obverse is that “posts on flagged Adult and NSFW blogs do not appear for logged-out users”. Fair enough so far. What does the statement “do not appear for logged-out users” imply? Well, it implies that the posts do appear for logged-in users, which would make this a fairly benign (if still nanny-ish) attempt to make sure everybody who sees porn has opted-in to see it.
But that benign implication is false. Remember the other new (red boxed) line in the graphic? It says “Blog indexed by Tumblr search” and shows an unchecked box under “Adult”. So, in the first red box they say “you can’t see adult adult or NSFW blogs when logged out”, implying that logged in users can see them. But then in the second red box they carve the adult ones away, because they admit that Tumblr search (which is tag search) won’t index these blogs at all, meaning that logged-in users won’t see them either. NOBODY WILL SEE THEM except your logged-in followers, and that appears to mean legacy followers only, because how would anybody not already a follower of an adult-flagged blog ever discover it now? If your blog is flagged adult on Tumblr, you’re blogging inside a sealed black box, and you have been for months.
So much for evasion — now for the outright false denials. The reporters for the Daily Dot sought comment from Tumblr, and they got this from Tumblr Head of Communications Katherine Barna. I’m including my commentary inline [in italicized brackets like this]:
Tumblr’s longstanding policy regarding NSFW content has not changed [true, if a few months is “longstanding”] and emphasizes the importance of free expression. [Bullshit!] As addressed in these policies, we are constantly taking measures to ensure our users can avoid this content [true so far] unless they’d like to see it. [A lie — if they’d like to see it, they still can’t find it because Tumblr doesn’t index it.] You can read about some of these features here: http://www.tumblr.com/docs/en/nsfw
Adult and NSFW content will be visible to anyone who has opted-in via their Settings page. [“Visible” only if you know the link already, but not searchable for anyone. In other words, more bullshit. Unless the “will be” future phrasing means Tumblr plans to change this? Do not die holding your breath.]
Different app environments have different requirements that we do our best to adhere to. [Presumably true, but freighted with the false implication that this is why Tumblr restricts adult content in its apps, given that they restrict it elsewhere likewise] Users can also find all content with Tumblr search in their mobile web browser. [Flat lie. Tumblr’s own policy chart newly shows that Tumblr search does not index the adult-flagged blogs.]
So, there you have it, folks! Months after making a policy change to exclude adult-flagged blogs from its internal search, Tumblr updates its docs to disclose the change. Internet goes wild, so their spokeswoman sends email that contradicts the new policy document and falsely denies the change.
Similar Sex Blogging:
Tuesday, June 11th, 2013 -- by Bacchus
Recently I became aware of an ironic lapse: the most succinct statement of Bacchus’s First Rule that exists on the internet is to be found in a two-year-old tweet on somebody else’s server. Doh!
By embedding the tweet here, I am trying to live by my own Rule. A side benefit is that I’m putting it somewhere that’s easier to find and link to. It’s true that the embedded tweet still depends on access to Twitter’s servers for styling information, but the text will still be here if Twitter’s servers go away. (By “here” I mean “on my server for as long as I have one” and subsequently in whatever archives of ErosBlog may persist.)
I should note that sometimes these days I phrase the rule a little bit differently than I did at first. The “at your own domain” phrasing comes from an era when search engine optimization (SEO) loomed larger in my thinking. Why create content that will generate search traffic to another domain instead of to your own? But focusing on the domain name can be somewhat misleading.
One reason is that in these modern times, search engines can’t be trusted to reliably send traffic to web locations that aren’t in their corporate-partner data-silo complexes, especially if the content is disfavored, like dirty porn or instructions for downloading stuff. SEO has become a mug’s game for the most part; it’s necessary but not sufficient to guarantee your web presence. You can’t disregard it entirely but it’s a will-o-wisp that will mire you in the swamp if you make it your guiding star.
But that’s actually a side issue. The core of the problem with putting your creative output on free blog hosting services and what we’ve come to call “social networking sites” never was maintaining visibility; often, the social networking sites will send you more traffic than you ever could hope to get on your own. No, at the heart of the problem is control. And for that, you want your own server more than you want your own domain, because the server is more important to your own control and (if you want it) long-lived web presence.
Your “ownership” of a domain name is anyway a somewhat fragile thing; a domain can be fairly easily taken away by litigation or state action. The two things you can actually control are your files and the server that hosts them. What gives your web presence its best hope of permanence is you being the owner of your own server or, more commonly, being a lessee of server space from some commodity hosting provider who can be instantly replaced when (if) they stop respecting your prerogatives. If you keep your files in order (fresh backups!) you can get new server space in under four hours; hosting businesses are highly competitive and eager for your money. So now I tend to state the rule this way:
Bacchus’s First Rule Of The Internet: “Anything worth doing on the internet is worth doing on your own server that you control.”
2022 update: Never build your house on someone else’s land.
Similar Sex Blogging:
Saturday, June 1st, 2013 -- by Bacchus
Lots of people (including me) started bitching after I discovered that Tumblr was locking away all the adult blogs behind a robots.txt designed to make them impossible to search or archive. Dr. Faustus has actually been doing something constructive about it, by exporting the best images from his Tumblr to individual self-hosted blog posts and this very fine thumbnail gallery. He explains his motives thusly:
In May 2013 Tumblr was acquired by Yahoo, a major corporation with at best a checkered record for dealing with its acquisitions.  While Yahoo executives insist that they’re not going to screw up Tumblr, their past record combined with their obvious interest in having the site generate more corporate advertising revenue off the site makes me profoundly pessimistic about its future, especially with respect to adult content. And my pessimism is shared by some of the most astute and experienced observers in the adult Internet world. I fear the coming of a day when my tumblr is simply deleted for its violation of some vague “community standard,” which deletion will take place without warning or possibility of appeal. The content that I will have worked so hard to curate will be lost. I am determined to have it not be lost, and that is why I am keeping it here on my own site that I control.
Even if this ugly day never comes, there is still a major problem with Tumblr, which is that its content is simply not searchable. Tumblr blocks access to the crawlers of search engines and the Internet archive, probably to make itself more appealing to the corporate suits who are now its overlords. To me, that is not the Internet. The Internet means openness. It means having content that people who want to find can find. And it means backups for the historical record.
I will not stand to see the content I have curated hidden – tucked away as if it were something shameful – any more than I can stand to see it lost. I intend to save at least some of it, and that is the purpose of both my “Tumblr favorites” blogging and of this thumbnail index gallery.
Similar Sex Blogging:
Sunday, May 19th, 2013 -- by Bacchus
The full implications of Tumblr making adult-flagged porn Tumblr blogs non-searchable, and hiding their content from the search engines, are only just starting to sink in for people.
For instance: if you have an adult tumblr, now you can’t even search your own blog to find an old post.
I’m getting panicky emails from people with huge adult Tumblrs, thousands of posts. Apparently internal Tumblr search has never worked well (you can search for one tag, or for blog names, but not for post content and there are no multi-keyword searches) and it’s impractical to scroll back very far in your own Tumblr dashboard. So they were in the habit of typing [keyword] [their own tmblr url] into Google, and hey presto! There would be the post they were looking for.
Now their blogs have a non-consensual robots.txt file that excludes Google, and all of those search results are gone from Google.
Worse yet? Tumblr blogs flagged “adult” aren’t searchable even with Tumblr’s own internal search. You can test this yourself. Log into your Tumblr dashboard, go to your settings, and make sure you haven’t checked the “Browse tag pages in Safe Mode (Hide content from NSFW blogs)” setting:
Unlike the one that doesn’t actually “allow search engines to index your blog”, this checkbox appears to actually work in the narrow sense that if it is not checked, you can search for blogs flagged “NSWF” within the Tumblr tag search interface. But this checkbox lies by omission. You’ve got the option to search tag pages of NSFW blogs (or not) but opting to search them does not let you search blogs that have the deeper-level-of-perdition “adult” flag.
My test for this is to search for a recent post at Wicked Knickers, which I used as my “adult” flagged example in the Thou Shalt Not Search Adult Tumblr Blogs post:
The post we will be looking for in the Tumblr dashboard tag search has a time stamp of 9:30pm yesterday, May 18, and is tagged “ziegfeld” which makes it a nice handy and recent thing to search for.
We already know that Google no longer has access to the posts on an adult-flagged Tumblr like this:
So, what happens in the Tumblr tag search interface? If you’re logged in, this is what you see when you search for tumblr posts with the “ziegfeld” tag. The posts returned are listed in date order (most recent first) and dates are visible as tooltips on the live page, so I’ve added them in the margin with red arrows and white text. You’ll see that the Wicked Knickers post is not returned by the Tumblr search:
Interestingly, that logged-in Tumblr dashboard search result is displayed at a URL ( http://www.tumblr.com/tagged/ziegfeld ) that returns something very different (but still no sign of our Wicked Knickers post) if you navigate to it as a not-logged-in person:
Try it yourself if you’ve got an adult-flagged Tumblr blog. Log in and try to search for your own posts in the search box on your own dashboard. You will, sadly, fail.
So, what is to be done? How can you search your own Tumblr blog?
The answer is, quite simply, you cannot — not while it’s on Tumblr’s server behind their robots.txt that you do not have the power to alter or remove.
But, all is not lost. Be ye not in despair. If you could only back up your adult Tumblr blog — make a complete copy of it, on your local hard drive — you could search it there with any file searching tool. Or, if you have a web server of your own, you could upload that copy (mirror it) onto your own web space, where it would once again be indexed and searched by Google.
That’s all I got. It’s the only way. It’s also a very good idea, because eventually The Pornocalypse Comes For Us All, and because Anything Worth Doing On The Internet Is Worth Doing At Your Own Domain That You Control.
Your next logical question is “But how do I do that? How do I back up a Tumblr blog?”
It’s not a simple question. The answer isn’t simple either. But, it can be done. So, that’s my next post.
Similar Sex Blogging:
Wednesday, May 15th, 2013 -- by Bacchus
If you’ve got an adult blog on Tumblr, there’s a good chance Tumblr uses robots.txt to exclude the search engines from indexing it. Did you know that?
Two weeks ago in The Pornocalypse Comes For Us All, I wrote:
Who is next? My guess would be Tumblr. Tumblr is, of all the big platforms, perhaps the most porn friendly; there’s lots of porn on there and the Terms of Service do not prohibit it… But Tumblr is, famously, a popular platform in search of a revenue-generating business model. And we’ve learned that the suits have no loyalty to the porn users who made their platform popular. So, my bold prediction is that as Tumblr casts about for a business model, one of their steps will be to “clean this place up”…
And now, guess what? I’ve discovered that Tumblr uses robots.txt to bar all search engine access to blogs flagged as adult. If you’ve got an adult Tumblr, go look at your own settings. Do you see that first checkbox, the one that says “allow search engines to index your blog”?
That checkbox is a lie. It’s nicely checked, it’s not greyed out, but if your blog is flagged “adult” it’s a lie. Do you see the “Learn more about what this means” link under “Your blog was flagged NSFW” selector? It leads to this page, where Tumblr requests users to appropriately self-flag their blogs:
Please respect the choices of people in our community and flag your blog as NSFW or Adult from your blog Settings page.
- NSFW blogs contain occasional nudity or mature/adult-oriented content.
- Adult blogs contain substantial nudity or mature/adult-oriented content.
If you’re not sure if you should flag your blog you can leave it unflagged, but keep in mind that we might flag it later if we see a lot of mature/adult-oriented content.
To answer the question “What happens to blogs that are flagged NSFW or Adult?” Tumblr offers this handy chart. The key piece of information is the white space indicated by my red superimposed arrow:
That’s right — where the “Blog indexed by Google” row intersects the “Adult Blogs” column, we find a ringing silence.
Would you have noticed? None of the adult Tumblr bloggers I know ever did. I knew from my porn researching that adult Tumblrs tended to be poorly represented in Google search results, but I chalked it up to the sheer scale of Tumblr and Google’s growing bias against returning porn search results. Nope, I found out the truth in one stark moment of astonishment, summed up by this image:
Let’s click the “See wickedknickers.tumblr.com robots.txt page” link:
From me: Aghast. Fucking. Gulp.
In robot, that means, roughly “All robots: stay out!” No search spiders allowed. No Internet Archive crawler. The Wicked Knickers tumblr is there, but you have to know about it, or you have to be linked to it. You won’t find it in Google, you won’t find it in any other search engine that honors robots.txt, and when Tumblr decides to stop hosting it, you won’t find the pages in the Wayback Machine — it will be gone for good, lost to humanity unless somebody with the technical chops and outlaw sensibilities of Archive Team finds a way to archive it anyway, robots.txt be damned.
Wicked Knickers is just an example, one that has some meaning to me because it’s one of the first Tumblr blogs I ever noticed, and I’ve been linking to it since 2010. That’s almost 6,000 vintage erotica posts since January 2009, and none of those pages are in Google or the Wayback Machine. It was only when I twigged to that anomaly that I finally understood what Tumblr is doing to adult blogs.
In all the years that I’ve been preaching Bacchus’s First Rule (“Anything worth doing on the internet is worth doing on your own domain that you control”), I’ll confess that I never considered the power of robots.txt, or what it means to be putting stuff on an internet site where somebody else controls what robots.txt says. Not only do they control your visibility to search engines, they control whether history will remember what you said. That strikes me as a high price to pay for a “free” blogging platform.
It’s worth noting that there’s still rather a lot we don’t know about the Tumblr robots.txt blockade on adult Tumblr sites. Unanswered questions include:
- Does Tumblr have any flexibility on this? Would their support, if asked, remove or modify the robots.txt barrier in specific cases?
- When did Tumblr start using robots.txt to block Google from adult blogs? Has it always been like this, or is it a recent innovation?
- Why does Tumblr display the misleading checkbox that falsely implies that search engines can see flagged adult blogs?
- What is the actual reason for excluding adult Tumblrs from search engine and (especially) archive crawls?
In an unusual move for me, I actually reached out to press@tumblr.com, told Tumblr I was going to write this post, and asked them for answers to those questions. That was on May 11th. No response so far. If they ever do answer, I’ll be sure to update this post.
Similar Sex Blogging:
Tuesday, October 18th, 2011 -- by Bacchus
Thomas Roche likes to dive into those bullshit sex stories that always get the old media all moist in their collective sex-negative panties, just so he can mock them comprehensively from the perspective of an web citizen who understands about search engines and how you can, y’know, check that shit before you write about it. I admire this, because it takes work as well as wit, and as a lazy half-wit, usually I can’t be bothered; once my advanced spidey sense tells me “this is bullshit”, I typically want to forget I ever saw that stupid headline as rapidly as possible, and thus I get to spend the next three days cursing as people retweet it at me. But Roche makes sex-story-stupidity-debunking funny, and I suspect serves a salutatory educational function while he does it. What’s more, in the process he typically does a better job reporting the actual stories in question than anybody previously has done. This may indeed be what 21st-century journalism looks like, folks!
His latest takedown is of the “ZOMG, sex while skydiving!” story, the breathless headlines about which have been infesting my RSS and twitter feeds of late and causing, thereby, extreme weariness in the finger I’ve been using to click on something else, ANYTHING else, “NO PLEASE FEDERAL NETWORK, I would NOT like to know more!” I’m not even going to link that shit, because Roche did it already and to link it, I’d have to look at it, and that would make my brain hurt more than it already does.
I’m posting this for an entirely different reason, namely, that I just about lost my cookies laughing at this bit in Roche’s story, where he mocks people who write for the web but who don’t seem to possess even the most rudimentary web search skills:
Incidentally, before we get started: multiple news sources (and Gawker) have said that the “original video” of the flying fuck between porn stars Alex Torres (aka Voodoo) and Hope Howell (possibly aka Maryjanesexy and Paige Thomas) was quickly “taken down from YouTube,” and “only fakes” exist. I have to tell you, if I got my porn from YouTube, I would think babies were made by Florida blondes in bikinis telling me about the Transitive Verb of the Day. Yes, the forces of decency did conspire to keep that video out of Gawker’s hairy palms. But I am one l33t h4ck1ng muth4fukk4. I did have to pull out some serious Neuromancer-era cyberpunk mojo to find the video that the FAA doesn’t want you to see, but after changing into my black PVC catsuit and Converse Hi-Tops, putting on night-vision goggles, drinking a six-pack of Jolt and eating a box of Hot Pockets while listening to Skinny Puppy at top volume, I “hacked the mainframe” and scored that tasty morsel of thoughtcrime contraband! I liberated that dirty video even though it was kept in a secure location guarded by Pentagon infobots…on Alex Torres’ website. Information wants to be free!
Muah ha ha ha ha, I can’t stop laughing.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
|
|
