It all started with a simple innocent tweet. As Vivek Wadhwa was finishing up his excellent post on outsourcing he tweeted about it and I replied with a tweet of my own:
Note the URL I tweeted to him was shortened using the xrl.in service.
Now, Vivek is a TweetDeck user, and TweetDeck uses bit.ly to shorten links, even links that don't need shortening, like mine. So what do you think happened when he retweeted me and responded?
That's right, my already shortened link was reshortened with bit.ly.
Now what do you think you see when you click that link? The site I was trying to link to?
No, you see a warning page that implies that my original link leads to a malware, phishing, spam, or forgery site.
So my link to a reputable site is being called a bad site by bit.ly because I used a competing URL shortening service in my original tweet.
Note that there is an email link to report mistakes on bit.ly's warning page. What do you think happens when you click it and report a mistake? Do they check the link and remove the flag if the site is ok?
No, they don't. They told me to make a new bit.ly link and give it out to people, as if that would undo the damage that was done, change the links in other people's tweets, and prevent others from retweeting the bad reputation damaging bit.ly link that I never made in the first place.
Then they apologized and told me some day their service will be better, still not removing the reputation damaging flag from the link.
Can something like this affect you and your website? You bet it can.
If you or anyone else ever tweets a link to your site using a competing URL shortening service, and that link gets retweeted with any twitter client that shortens all URLs with bit.ly (whether they need it or not) the resulting link to your site will be flagged as a bad site.
What happens when the average person clicks that link and sees that warning page? Well, the average person believes it and won't visit your site. And if your name is on the original tweet, it will also be on the retweet, giving you the reputation of passing out links to bad sites.
Will a smart web savvy user believe the bit.ly warning page? Maybe, maybe not. But the average person probably will. As a developer I have seen the average person believe a lot of things that were not true, placing their complete trust in things like that bit.ly page and other false positives, and believing the worst, even spreading the word about it as if it were the truth, telling others that a website or application was harmful. They think that companies don't do things like this without ironclad proof, so they believe every word of it.
What alarms me the most is the attitude of bit.ly with regards to this problem and how they refuse to remove the flag from innocent links. They don't care if your reputation is damaged.
And it is in the best interest of bit.ly not to fix the problem, since it makes more people use their service, worried that if they use another service and get retweeted, they could end up with a reputation damaging link to their site. So this whole problem serves to make more people use bit.ly, out of fear, rather than convenience or because they have a better service.
I am not a bit.ly user because it is not convenient for me. I have a browser plugin that uses xrl.in, that with a single click of a button, copies the shortened link to whatever page I am on to my clipboard, ready to paste anywhere I want.
Other links in the stuff I tweet use ff.im, because they are cross-posts from friendfeed. I was pretty sure if I post something on friendfeed, cross-posted to twitter and it gets retweeted, reshortened with bit.ly, it will result in one of those warning pages, too.
But I was wrong, it doesn't. In fact, there are a few other shorteners that don't get warning pages either, probably because they are so popular that they would get noticed pretty quick if bit.ly decided to pick on them.
Bit.ly does white list the following shorteners:
- ff.im
- youtu.be
- goo.gl
There is no reason why bit.ly can't white list the rest of the popular shorteners if they can white list those. But those others are competing services and they don't feel like being nice guys about it. They would rather ruin the reputations of innocent people like you and me.
So what can you do to ensure you won't become a victim of bit.ly's "bad site" interstitial page?
There are only 2 things you can do:
1. Always use bit.ly to shorten your links and make sure everyone else in the world does too. (highly impractical, because you can't control what other people do.)
2. Let bit.ly know this is unacceptable. Tell them to play fair and white list the other shorteners. The choice of url shortener you use should be yours and not theirs, and you should not be punished with a reputation damaging warning page because you use a competing service. (easy to do)
- Send them an email: support@bit.ly
- Tweet them: @bitly
- Spread the word by tweeting this post
- Share this post on social networking sites
- Blog about it
- Tell your friends
Don't stop making noise about this till bit.ly stops damaging the reputations of innocent people.
UPDATE May 8, 2010: Bit.ly has changed their interstitial page, slightly. Instead of the top banner saying "WARNING - visiting this website may harm your computer" it now says "Stop - there might be a problem with the requested link"
But it goes on to say
- Some URL-shorteners re-use their links, so bit.ly can't guarantee the validity of this link.
- Some URL-shorteners allow their links to be edited, so bit.ly can't tell where this link will lead you.
- Spam and malware is very often propagated by exploiting these loopholes, neither of which bit.ly allows for.
The link you requested may contain inappropriate content, or even spam or malicious code that could be downloaded to your computer without your consent, or may be a forgery or imitation of another website, designed to trick users into sharing personal or financial information.
I still consider this unacceptable. If bit.ly can detect a shortened link at the time someone clicks, they can detect it at the time someone submits it for shortening, and just not allow it. That page is completely unnecessary and can still be damaging to someone's reputation.
Come on, bit.ly, just do the right thing! How hard is it to just not reshorten?
40 comments:
We have the same problem - all of a sudden our link that was "working"
for oupule years is being marked as a problem. WTF?!?! And we actually
shortened the link with bit.ly to start with!
why not create a whitelist of known and trusted shortening services? then if a link is submitted that is hosted by a provider on the whitelist you can simply spit back the url that was submitted instead of a shortened bit.ly url this way existing technologies can still 'shorten at will' but if they're shortening a url on the provider list, it just gets returned instead of the bit.ly shortened version of it. what i would do anyway i think.
They say they are using 3 different services to check links: VeriSign’s iDefense, Websense Threatseeker Cloud, and Sophos. I don't think an additional check with WOT would be necessary.
The flagging of all competing shorteners I think is their own thing though. I seriously doubt that those 3 services are flagging all shorteners by default, or are incapable of checking the target site of them.
On the page where they announced the whole new security thing, in the comments was a similar complaint as mine, dated December 2, 2009. So they have done nothing to remove flags from false positives in all this time, despite having received at least one other complaint before now. From the nature of that complaint, I am guessing that back then they didn't even have a way to report false positives. http://blog.bit.ly/post/263859706/spam-and-malware-protection#comment-24601400
rediculous, they have good reasons for doing it the way they're doing it. the people you should be talking to are the twitter clients that re-shorten urls. the loopholes that are exploitable without this interstitial in place are worse than your damaged 'reputation' (I've never heard of you)
Hey App. Great post!
I find it amusing that the people who developed bit.ly apparently never implemented any review process and "unflag" functionality.
With reshortens being such a bad thing, it is wild that they do it to begin with. Your suggestion above is such common sense, how in the world did they miss it?
Hell, if I were a developer for bit.ly I probably would have implemented an automatic "Web of Trust" check on reshortens to make sure this kind of thing didn't happen. But, maybe that's just me.
Have you ever tried to reshorten a URL with is.gd? They don't stick an interstitial page in front of the face of the one that goes to the resulting link.
They do something even better...they don't reshorten URL's. They don't create a loophole that can be exploited.
Bit.ly could just as effectively protect people by following the same example. This would force the developers of twitter clients to comply or their apps will break.
If it was only my reputation that was at stake here, I wouldn't have made this post. It's the reputations of everyone that uses a shortener other than bit.ly.
You still don't get it.
301 redirects aren't allowed. It has nothing to do with 'shorteners' specifically. Bit.ly looks up the url and if it gives a 301 they warn you about it. Simple as that.
Since you could post a link that didn't have a 301 one day and did the next, they do this live. They notice that other url shorteners are popular and whitelist some of them that comply with their policy. They haven't picked up your random obscure choice of shortener, so instead of being an asshole to their support people maybe you should just _ask_ for it to be whitelisted... that is, as long as it wont introduce regular users to the possibility of malware sites (say, by allowing editing of their shortened links, etc).
Ok, let's say I never use another shortening service for the rest of my life. Let's say I even never use Twitter again. How do you propose I stop other people from using Twitter, tweeting links to my site, and using a shortener?
I can't control other people. They are going to use twitter whether I want them to or not. And they are going to tweet links to my site. And they are going to use shorteners.
That is out of my control, but it still has me with the same problem. And I am not the only one with this problem. If you have a website, it's your problem too, even if you never used Twitter in your entire life and never plan on using it. Other people do and will, and you can't stop them from tweeting links to your site, and you can't stop them from shortening your URLs, and you can't choose which URL shortener they will use, and you can't stop people from retweeting them. It's out of your control.
Here's a tip: Read and think before you comment, so you won't look like a clueless ass. I presented both the problem and a solution that would work quite well, allowing bit.ly to continue to protect users, without the need of a reputation damaging interstitial page.
I have to agree with Bit.Ly on this one, you can deliberately make Tweetdeck not "shrink" URL's by unchecking it, which is what I do whenever I post an Ow.ly link in Tweetdeck so that it doesn't re-encrypt the ow.ly link as bit.ly. You can also change your default URL shrinking service in Tweetdeck. Double URL shrinking is a way for users to hide malicious links and prevent the URL compression service from determining if it might be a harmful site. A URL shortening service can only follow so many referrals to determine if it is a bad link. Use Hootsuite or another client that doesn't auto shrink URL's or turn off the Tweetdeck autoshrink if you use shrinking from another source.
Actually, that's not true. They are not detecting 301's and they do allow them (I just tested it and didn't get a warning page).
They are black listing known url shorteners. In fact, I'd bet anything if I were to set up my own URL shortener today and use it and then reshorten the links with bit.ly, they won't get flagged because my brand new shortener is so obscure. The more popular a shortener is, the more likely it will be blacklisted, not the other way around.
Asking them to whitelist anything doesn't work. I have tried. My attempts have been documented here on this page. the best you get when attempting that is a line from them that effectively says "so sorry, maybe someday our service won't suck".
And xrl.in isn't obscure. If it were, it wouldn't be blacklisted by bit.ly. In fact, this is how obscure it isn't... it's used in a button for a firefox extension that gets 5,852 weekly downloads: https://addons.mozilla.org/en-US/firefox/addon/2377
And I started out very polite, but there is only so long I can remain patient and polite with the kind of treatment and responses I have received from bit.ly.
Were they guided by some altruistic motive to protect their users (as Todd mentions) or did the lawyers pressure the policy into effect to protect their liability? Why would bit.ly be held liable for a malicious website? And, why haven't the others implemented a similar interstitial? Google does so because they are several magnitudes in size larger and have such deep ties to almost every user's online experience. Does Bit.ly fall into the same category, or exert a similar degree of influence?
I see Bit.ly's intent, but I agree, the implementation is incomplete, and their approach to customer service and support is laughable. I much prefer the idea of preventing multiple redirects being shortened.
Still, atleast they're not picking and choosing the sites they label as malware. Bit.ly thinks it's own site is malware - http://bit.ly/apU51c - created through ow.ly.
I never knew about this as I normally do use bit.ly. But it's ridiculous that they will use these tactics to subversively drive everyone to use their service.
Also, the idea that people are making a living from shortening URLs is frikkin hilarious. Of course they will invent shit to keep them occupied.
Isn't the core problem actually that twitter retweets can be modified? People expect retweets to be an exact verbatim copy but they can easily be spoofed - in this case it gets spoofed automatically by some crappy software. "mom, Tweetdeck is putting words in my mouth".
Changing settings in TweetDeck is fine for just your URL's but exactly how do I force everyone in the world to do that before they retweet me or anyone else that has tweeted a link to my site with another service? How is changing your settings before you tweet an ow.ly link going to stop someone else that hasn't changed their settings from double shortening your ow.ly link by retweeting you with TweetDeck set to use bit.ly? And exactly how is changing your settings going to stop Average Joe from tweeting a link to your site and then Average John from retweeting it with bit.ly? You can't control what other people use, which is why I said option 1 was impractical.
In case you didn't notice, I did say that I am not a bit.ly user and not a TweetDeck user. I have no settings to change, yet my links are being flagged by bit.ly as a bad site. See how impractical your suggestion is? You tell me what I can on my end to prevent everyone in the world from accidentally reshortening my links with bit.ly when they retweet me...if it makes any sense I'll do it.
If double shrinking URL's is so bad, then why does bit.ly allow it in the first place? They shouldn't allow it at all, like the is.gd service doesn't. That would seem to make a lot more sense to me than allowing it and then flagging it later as a bad site without any proof that it does lead to a bad site.
The problem is that bit.ly is doing it all half-fast & backwards, and innocent people are having their reputations hurt.
The bigger problem is there is no remedy for site owners that are being slapped this way. Bit.ly will NOT unflag good links...not even to their own site. There is just something wrong with that.
bit.ly's so called malware checking system is not bulletproof.
http://www.stopthehacker.com/2010/02/19/analyzing-url-shorteners/
What is there to consider? You make the change and the developers of third party tools will have to make changes too, or their apps will break. They are never going to change unless you make the first move and force them to do it. I am sure you know that.
I appreciate the fact you want to protect users on the one side of your interstitial page, but you really need to consider protecting the innocent sites on the other side of that page too.
You and I both know the average user will be scared off by your warning, never even bothering to read the rest of the page beyond the bad description of what may lie ahead.
The link visitor should never see that. The one attempting to reshorten a link should see an error and that explanation of why reshortening is bad. If you can detect a shortened URL at the time the visitor clicks, you can detect it when it is submitted and reject it, eliminating the need for that page.
You have it all backwards.
And I can't think of any reason why you wouldn't seriously consider changing this, unless you are afraid that you might lose some business from those that want to reshorten everything in order to take advantage of your click tracking features.
But I wonder how they would feel if they knew the truth about what they are tweeting. That most of the clicks being tracked from reshortened URLs never reach the final destination, due to a scary warning page. It's a meaningless inflated click count that doesn't reflect the truth.
There is a quadruple problem here:
1. TweetDeck is shortening URL's that don't need shortening.
2. Bit.ly is reshortening urls and then flagging them as bad sites.
3. You can't control the software others use and how they use it.
4. Bit.ly will not remove the flag from good sites after being informed of the mistakes, despite posting an email address to report the mistakes.
That 2nd problem is what is making the first problem possible, which is one of the things I have a problem with.
If bit.ly fixes problem 2, TweetDeck will be forced to fix problem 1, and no other app developer would be able to cause a repeat of problem 1 in the future.
Problem 3 can't be fixed and no suggestion you can make will change that. (you don't seem to understand this very well)
Problem 4 is the worst of them all, and the whole reason why I wrote my post. If they were willing to do something about the mistakes, the bad site flag would have been removed from my link and I would have been happy, and I would never have written my post.
Anti-virus companies that block sites have a means to report mistakes, and when mistakes are reported they investigate, and if they are wrong for blocking a site, they remove the flag.
Never will an antivirus company ever send you an email response that says "Maybe some day we won't suck so much and block good sites. In the mean time you are screwed because we don't remove flags. Sorry we are damaging your reputation that we don't really care about. Thanks for the feedback."
When bit.ly took it upon themselves to "protect the public from bad sites" with an interstitial page labeling websites as malware, phishing, spam, and forgeries, they voluntarily placed themselves into the same category as the anti-virus companies.
Unless they have an effective and fair means of dealing with their false positives when they occur and are reported, they had no business doing what they did, the way they did it.
Bit.ly's failure and refusal to correct known false positives is libelous. And there is nothing you can say that will change that fact.
While bit.ly clearly needs to make further changes, I can't help but wonder why TweetDeck has to resend each link for shortening.
I think TweetDeck needs to share responsibility and make changes, too. Other Twitter clients do not re-shorten links. This is great information. Thank you!
Anita Nelson @ModelSupplies
so...inform people that they should unclick the "automatically shorten link" box in tweetdeck. It doesn't shorten links automatically unless you have that box ticked off.
Hmm... How hard could it be for bit.ly (or Tweetdeck as a matter of fact) to simply set a minimum number of characters in an URL that triggers shortening? Short URLs would get passed through, which would include any URLs short to begin with and the various other shortening services. This just doesn't sound like a rocket science.
I love bit.ly for the extra features it comes with, but its job is to police its own links, not the entire social network.
You might want to be a little nicer to the people at bit.ly. It's amazing how much someone saying Thanks! can help the other person's day/job.
I started out nice and they blew me off. How long am I supposed to be nice and continue being blown off by them?
Read the email conversation I posted.
I am not about to thank them for that.
TweetDeck can do it because bit.ly allows it. If bit.ly doesn't allow it, then not only will TweetDeck not be able to do it, neither will any other twitter client.
Bit.ly can force developers of any twitter client to not reshorten by making that change on their end.
Or force twitter clients to stop doing that by making it impossible for any twitter client to reshorten URLs with bit.ly's service. That would be much more effective.
Todd from bit.ly here.
From day one, we've prized security, transparency, reliability, and openness at bit.ly. Along the way, we've made a number of product decisions based on those tenets. Among those are link permanence (link destinations don't change once created), the avoidance of anything that interferes with user experience (we've never framed, nor will we), and a dedicated focus on spam and malware detection, so that our users can click on bit.ly links with a high degree of confidence.
We take our responsibility as internet citizens seriously, and you'll see this exhibited even in the small details of the ways in which we manage flagged links (you'll notice we never actually disable a redirect, and at most simply insert an interstitial which retains the end destination link).
In the course of analyzing content for spam, malware, and phishing attacks, we rely on a number of systems, both internal and external. Over the course of the past year, a number of spammers have attempted to use various levels of indirection through redirectors (some of which are reconfigurable), in order to obfuscate and cloak their efforts. In fact, the bulk of shortens to bit.ly coming through other URL shorteners have tended to be attempts to spam the system. While our crawlers do of course follow links through redirections, the inclusion of modifiable redirects in the stream, and our analysis of the preponderance of spam attempts via these vectors have made it necessary and appropriate in some cases to block the URL shorteners.
Just to reiterate, the only goal is and always has been to protect the end user clicking on bit.ly links, regardless of the link source. Given that multiple layer wrapping of URL redirectors tends to be an edge case based on inappropriate API usage, confused users, or in the preponderance of cases, attempts to spam, we think this has been a fair approach. As such, you'll note that we did in fact update our interstitial warning pages with language better reflecting the reasoning behind the status. We're happy to see a healthy, vibrant, shortening ecosystem, and have no intention whatsoever to put a damper on other sites in the space.
Some have suggested we simply not shorten URLs already pointing to 3rd party short URLs. While this is a potential possibility, our API responses and the innumerable clients and scripts that use these methods aren't currently designed with this state in mind. Consequently, any changes would have to be carefully considered.
As with any product, bit.ly is a work in progress, and we're always interested in finding ways to best serve our users, while maintaining the integrity and openness of the product.
Nice blog post. Well written, but as I tried to explain to you - our policy is not out to harm anyone. In fact it is there to protect our community. There are too many times that a bit.ly url can be used as a cloak to hide a rogue shortened link.
While I tried to explain that our interstitial page was a bit vague, that page is currently under review. Again, our apologies for the inconvenience, but bit.ly is trying to protect the overall community from rogue short links. We understand the frustration, and hopefully after review, you will understand that we are not out to undermine you or your efforts in promotion.
As I stated last night, this issue is brought to the attention of the entire team at Bit.ly. We do care. Our interstitial page is being looked at, and hopefully will be updated very soon. We do appreciate the feedback.
Your idea above has been fwd on to the engineers and the people that make these types of decisions. Thanks for the feedback.
thanks for alerting us app -- i agree this is just not acceptable behavior. a url shortening service has only a few things they have to get right -- one is not labeling websites malicious when they aren't!
hurting someone's reputation is not ok -- especially after you've been informed about the mistake.
Jesus you're a self-victimizing whiner.
Twitter *may* be promoting Bit.ly with this policy. But they are unquestionably protecting their users (as well as their own asses) from very well-known methods of propagating malicious links.
Here's a tip: stop using URL shorteners. What's that? You can't tweet the full URL? Then stop Tweeting, it's obviously a stupid, non-Web-friendly technology.
Come on bit.ly - you don't think we're that naive do you? As April points out all you have to do is check the url length and not re-shorten those links.
You do know that we are going to call this for what is: an obvious strategy to "encourage" the masses to only use your shortener reminiscent of what Microsoft did to make sure PC users would only use their software. (Yes, we've been around THAT long.)
The Internet is far more powerful at calling companies on their misdeeds than BBs were because even non-geeks can use Twitter and read and comment in blogs. Until someone manages to censor the Internet (and I have no doubt they're working on that) perhaps acting more ethically would be well-advised.
That interstitial page is not vague at all. It clearly states the target site is believed to be a forgery, spam, malware, or phishing. All very damaging words to a site when you have no proof of such activities.
I ask again, why give an email address to report mistakes if you are unwilling to check the links and unflag them if they turn out to lead to safe sites? I gave you enough information to be able to decide if the link lead to a safe site or some rogue site, so why are you unwilling to remove flags from safe sites?
This post would never have been written if you were more cooperative and removed flags from safe sites that were mistakenly marked as bad sites. Instead you gave lame advice that was very unhelpful and apologized for your unwillingness to undo the harm you are doing to innocent people.
Change your ways and stop hurting people and I'll modify my post with an update to reflect the changes.
Reputation is a thing of great value that can take many years to build and only one small careless act to destroy. And in this case the careless act isn't my own, so the reputation damage isn't my fault.
It's yours.
You can change the service on Tweetdeck for those who want to be able to retweet stuff. You have more than one choice in the settings section.
I agree that the service should easy be able to whitelist all other major URL compression services (if they did goo.gl already) but when you retweet links in Tweetdeck though, it doesn't auto shrink the URL's. It only does when you type a link and hit spacebar manually after the link if you have "Auto Shorten" URL's checked. I retweet other peoples ow.ly & Ping.FM links all the time, and they never convert to Bit.ly links. Tweetdeck will only convert the link to Bit.Ly if you manually type or cut/paste a URL and hit spacebar, it doesn't do it on the "send" button.
Fair enough, I see your Point, Bit.Ly also does the same thing with Hootsuite Ow.Ly links: http://bit.ly/cENfOZ for example (which is an encoded ow.ly link) so after testing it is more rampant than just with xrl.in recompressed links. Ow.Ly, Is.gd, xrl.in and TinyURL don't behave the same way and can encrypt any of the other URL compression without causing any type of error or malicious flag. I didn't realize how wide spread this was until I did some more testing and specifically experimented with a variety of other URL services.
I would agree that this would be interpreted as Bit.Ly sabotaging people who use other URL Compression services. I wish Tweetdeck would allow Ow.ly compression honestly, I care more about the click tracking of compressed URL's which is why Ow.ly, Su.Pr and Bit.Ly are my fav. Su.Pr doesn't pass 302 redirects to your blog all the time, and Ow.Ly is a little slow on Hootsuite to collect tracking stats. and BIt.Ly had the really convenient add a + at end of URL to show full analytics of the URL which was really handy.
Wow... I come to this post from Dennis's site, Gail from growmap mentioned this post, gotta say, bit.ly, sort it out, who are you to be deciding what sites are bad or not?
I am a bit.ly user but this is not on.
Do you Bit.ly not like this post and the negative publicity is causes you?
Nor did the owner of this site, when you caused this problem for them!
thanx for inform us,well many times i also feel tht.its a nice informative post.
I too, had this error, but for a slightly different reason. I was trying to share a Dropbox public link to a program I've written through bit.ly, to avail of its custom url shortening, and tracking.
It shortened it fine, but when I tried testing it, it gave the same ridiculous "interstitial page" that the link had already been shortened, which, obviously, looks too threating for anyone to open! I thought it was something to do with the format of dropbox links, but then another dropbox link to a pdf or html file worked. It was just giving problems with exe! Not even .msi, which could be just as damaging if they really cared about security, but just .exe.
So in spite of boasting of 3 scanners for their links, they put a blanket ban of not trusting any exe links. And instead of informing the person shortening the url, they inform the person who's downloading the file! The security is bullshit, coz it doesn't block .msi, and I'm pretty sure it'll allow .scr as well, which is also executable. And worst of all, the page doesn't mention that its just a statuatory warning because the linked file is executable, it instead talks crap about how the link has already been shortened and what not.
Thanks bitly, for nearly making me look like an ass spreading viruses, before I realized and switched over to goo.gl
And kudos to app, for not letting this pass silently. I hope others will join and boycott bitly till they learn that they can't just randomly do this to people!
dumbass, they can't just "investigate and unflag" because they don't know if xrl.in (never heard of it, site is terrible) lets you change the link of a shortened url or not. If they unflag and then you change the xrl.in to point to a malware site... then... well, then what? get over yourself
Post a Comment