Wednesday, January 03, 2007

Vulnerability Pimps

Marcus Ranum has written a very interesting article about code review, secure coding, Fortify, and vulnerability pimps. The meat of his article is about code review, and there are some real lessons to be learned there. You should take his comments to heart, and implement the review processes he recommends. I know I'm going to look into Fortify now.

There are also some interesting minor insights into Marcus' history. Love him or hate him, you should always pay attention to what Marcus has to say. He graciously added an RSS feed to his site at my request, so please use it.

That said, what I can't let go is "vulnerability pimps". I know, story of his life. He tries to tell people things, and they can only pay attention to his politics. Sorry about that, Marcus.

So, yeah, vulnerability pimps. That's awesome. I'm sure he means for it to be pejorative, but for the folks he is describing, I can't see them taking too much offense. I can see the rise of the purple hat hackers even now.

It's the first time I've heard the term, though maybe he didn't coin it. Google says that Rodney Thayer (at least) used it in 2005. I see Marcus using it in February. Of course, Google doesn't know everything, so I'm happy to take corrections. I can't help but think of this as a Ranumism, though.

As for my politics, I could be accused of encouraging, facilitation, and participating in vulnerability research. Though, not with as much skill as most other vulnerability pimps.

I'll keep my counterpoint brief. Marcus throws out the "many eyes" catchphrase, specifically calling it a failure in the face of his findings. If one does not like independent vulnerability research taking place, then where do you think the checks that Fortify performs come from? If the developers and companies aren't going to look, who else will? If you expect the few eyes to be able to see, where are those eyes going to train?

To be fair to Marcus, he just did the same thing himself. In fact, if I wanted to be extremely ungenerous, I could put him in the same category as the kid who just got a new fuzzer and went looking for problems. But he doesn't deserve that.

The difference for him, as he points out, is that he thinks there's no benefit to touting his findings, (presumably) not even after the patch is out. He reports that everyone was cool, and they are going to get the fix out Real Soon Now. So he can get the problem fixed without the fanfare.

I invite Marcus to finish the experiment, and give us a update later about the following:
  • Let us know if you will be taking credit for the finds.
  • Explain how pimping Fortify by searching for vulns in other people's software is different than eEye doing it to pimp Blink.
  • Tell us how long it takes the programmers to release the patch
  • Tell us whether the programmers properly acknowledge that this update fixes a security problem, and that people should update right away
  • Tell us if you spend the extra time to check that the patch correctly fixes the problem you identified


Anonymous said...

(Marcus J. Ranum )

Thanks for your comments, Ryan!! I'm not sure I care who coined the phrase "vulnerability pimps" but I like it - it has the right amount of nuanced contempt, which is what I feel for a lot of these "vulnerability researchers" and media whores. Oh, crap, now I left myself open for you to accuse me of being a media whore, too. :(

By the way, a "grep" through my mail archive shows a personal Email from my to Chris Klaus at ISS dated 12/17/2002 in which I ask him to please stop his company from "vulnerability pimping"

The gist of your counterargument is that "the good guys get their knowledge from the bad guys, therefore they are at least somewhat complicit." Never mind the fact that that's a morally repugnant position (try telling a district attorney that they should be grateful to all the child molesters and murderers and see how they like it!) it's silly logic. Of course, the good guys learn about how to defend their systems from the bad guys!! But that doesn't excuse the entire process of what the bad guys, or marginally bad guys, are doing. It's perfectly legitimate for a real "security researcher" to sit in a lab and hypothesize a new kind of problem that might occur, verify that it might exist, and then warn people about it. Where the line is crossed is where the problem is hypothesized, tested, and published - with full a priori awareness that the result will be a lot of grief for innocent victims.

Perhaps the best example I've ever seen of "doing this right" would be the NSA's handling of improving IBM's Lucipher cipher (later the DES) to strengthen it against differential cryptanalysis without revealing anything about DC, or even how their changes made the cipher stronger. Those nameless guys at NSA were real "security researchers" in my view.

Likening me to a kid who just got a new fuzzer is, as you say, unfair. I used the tool against my own code and made sure that any of the vulnerabilities I disclosed were in ancient code that will result in no forseeable harm for anyone in the community. In fact, the only harm I can see accruing to anyone is to myself -- oh, dear, there I've gone and popped the balloon of the myth of Marcus' code being perfect. You say there never was such a myth? :) You're right. No harm done, then.

I will take zero credit for anything found. I never have. It's OK to talk about this now, so... Back in 1990 I hypothesized FTP bounce attacks, while I was coding the FTP proxy for the DEC SEAL. I wasn't looking for a vuln; I just noticed that FTP sucked and realized it sucked in a particular way. I wrote a test routine and verified that what was later known as "FTP bouncing" worked. Later that night I realized that since FTPd was in the "privileged port" range it could be used to "bounce" a carefully constructed file to rshd. I tested that, too, and it worked. Basically, I had a key that could unlock the entire Internet, such as it was, at that time. Or at least, everyone who wasn't running a DEC SEAL. ;) What did I do? Well, I sent out a few Emails and if you look at the source code for ruserok() you'll notice there is a check to block FTPd. I'm not trying to brag - I tell this story only because you seem to be questioning whether my actions match my words. They do.

How was "pimping" fortify different from Eeye pimping blink? Because I am releasing absolutely no information in any way, shape, or form, that will result in anyone in the public being placed at risk. I never have done that. How many times has Eeye?

With respect to the rest of your questions: will I verify the fixes? Not my problem. In fact, since I'm not telling anyone what I found, it doesn't matter, does it? Do I expect a response or a "thank you" from programmers? Not really. I don't keep score that way. You know what? I'd rather have someone think well of me and maybe say "hi" at a conference than be in another damn press release. You've known me for a while - I've been on CNN, I've been in press releases, and I've had my 15 seconds of fame. But I didn't do it at the expense of the innocent.

Ryan Russell said...

First off, let me apologize to Marcus for comparing him to eEye. I did that on purpose, knowing of course that they are the epitome of how he does NOT like to see disclosure handled. I did not, however, know how personally he was going to take that, and he let me know in a private email. I think human nature is to look for irony and hypocracy, and I went there in spades for the situation, thinking it was an extreme enough leap that the sarcasm of it would be apparent. Marcus didn't think so.

I like Marcus. He's a truly smart guy. But I did the equivalent of arguing politics and religion at a dinner party. So I'm doing my best to say sorry in public for the comparison he took offense to. My only defense is that I didn't mean it like that, and of course from my point of view, it's not an insulting thing to say.

So let me try this again, without doing the equivalent of taking this straight to Godwin's law territory.

First off Marcus, I'm actually happy to think that you may have coined the term after all. As I said, I think that's only appropriate.

To the meat of my point. It appears that you have taken your tool, and audited some Internet infrastructure code. You appear to have found some exploitable bugs, and have notified the vendor. You have set a couple of things in motion:

-You have "informed" people that they can run Fortify against the list of network services you provide, and find some exploitable holes. That's more than enough for today's vulnerability researcher to get to an exploit.

-You have started the process of the vendor releasing a patch. This patch will likely be in source form, given the list of programs you provided. Not that it is needed, but that should make it darn easy to figure out where the hole was. More than enough for a vulnerability researcher to write an exploit.

-You will likely receive credit for the vulnerabilities in vendor advisories and bug databases.

As near as I can tell, what you are skipping is the "gloating" stage.

Also, not that I suspect that this is actually in your heart, but one could make room for a financial motive. This makes Fortify look pretty good, and rightfully so. It may generate some sales, too. I assume being on the TAB is a paid position, though I don't actually know that.

And please keep in mind that I don't see anything wrong with the things I just "accused" you of. For me, that's a valid, ethical set of motivations and outcomes.

And doing it without gloating simply proves that you have some class.

Let me address a couple of your specific points:

'The gist of your counterargument is that "the good guys get their knowledge from the bad guys, therefore they are at least somewhat complicit."'

I think that takes my position to an unnecessary extreme, if I may talk about degrees. I'm not one of those people who think that the bad guys are omnipotent, and that the good guys can't figure anything else out for themselves. Rather, I was trying to indicate that the volume, practice, and motivation of the "bad guys" has significantly advanced the state of the art in what is exploitable. For example, I think that certain kinds of heap bugs and free bugs might have been assumed to be unexploitable if a bunch of "bad guys" weren't beating on them. Or at least, the priority for tools to check for those kinds of bugs might have been much lower.

I'd like to point out that "bad guys" here is my idea of Marcus' standards. I hope I didn't put the wrong words in your mouth. I don't see vulnerability researchers as bad guys, myself. I call the bad guys the ones that are actually using the exploits to break into machines, build botnets, and so on. I see a huge distinction between the two groups.

Marcus also says:
'Perhaps the best example I've ever seen of "doing this right" would be the NSA's handling of improving IBM's Lucipher cipher (later the DES) to strengthen it against differential cryptanalysis without revealing anything about DC, or even how their changes made the cipher stronger. Those nameless guys at NSA were real "security researchers" in my view.'

I also agree that this is a good example. The Wikipedia background seems adequate, for those who aren't familiar:

Here's how that story reads to me:
-The NSA bought themselves an extra 17 years to exploit that weakness in other ciphers
-Other cryptographers couldn't design against DC, because they didn't know about it
-Ultimately it was discovered independently anyway

Had I been a cryptographer rather than a kindergärtner in 1974, I rather think I would have preferred to know about DC. That way, I would know why the mysterious changes were in DES, I could have a better tool for evaluating other ciphers, and I would be a better cipher designer.

The alternative is the equivalent of trying to keep integer overflows a secret so that you can't write checks for them in Fortify.

I realize, of course, that Marcus and I will not come to any kind of agreement on what are ultimately religious issues. But I do believe in trying to have a civil debate where we can spell out our reasoning. That way, people can understand why we have chosen the side of the issue that we have, and maybe some individual bits of information will fall out that can help inform other opinions.

Anonymous said...

(Marcus J. Ranum )

The reason that this debate has been going on so long is because it's not a cut-and-dried problem. Like with most interesting moral problems there are degrees of culpability - which is why the justice system (which is a system of law, not ethics!) contains notions such as "accessory" and "second degree" crimes, etc. I'm sure that, from your perspective, I sound a bit weird talking about something as insignificant as software vulnerabilities as if they are a moral problem - but that's exactly what I say they are. As you know, I've been saying that for a very long time - even before it was fashionable to do so. Last night I got a big laugh re-reading the "vulnerability pimps" Email I sent Klaus in 2002 because I was making nearly exactly the same arguments then as I do now. And, then, as now, I got push-back about the degree of culpability of the vulnerability pimps.

It shouldn't take a moral philosopher or psychologist to figure out that the vulnerability pimps need to feel that what they are doing is moral and socially valuable - otherwise they'd have to confront the reality that they often make decisions and take actions that place thousands - even hundreds of thousands - of innocent computer users at a higher degree of risk than they would otherwise be. In one of my private exchanges with you, Ryan, discussing the "I could crash the Internet" attack, you commented (paraphrasing) "BFD - it'd be back up in a week." Completely ignoring the fact that that would be a week of hell for a lot of system administrators, businessmen, ISP network managers - a week that could cost companies and individuals huge amounts of money, jobs, and emotional well-being. It is exactly that kind of facile attitude that has brought computer security to the moral level of a mafia "fire protection" scheme.

Whenever I find myself debating this issue, I offer over and over again the thought that:
You're either making the problem better, or you're making it worse.
Basically, the notion is that every player in this game, at every level, needs to assess constantly whether their actions are a positive contribution, or not. It's that simple.

And, yes, there are grey areas. Of course there are grey areas. I remember, back in the mid 1990's, when the vulnerability pimps first started getting into gear, it was hard to argue against their ideology. Why? Because the premise (at that time) was "by revealing these bugs we make software better." Now it's easy to see what an utterly empty premise that is. 10+ years later, what do we have to show for it? Is software better? My exploration of this topic says that it is not. Is the Internet a safer place for innocent unsophisticated users? Don't make me laugh. But the premise of the vulnerability pimps remains that they are making things better - essentially, by making things worse first. I think that 10+ years into that experiment, it's time to start calling that the obvious self-serving bullsh*t that it is.

Now, what you're doing with regards to my experiment, is claiming that simply because I've entered into the same ballpark as Eeye that I'm equivalent. Yes, I pointed out that I discovered some problems in some major open source code, but that's a whole different end of the spectrum from what those guys are doing. I can prove that simply:
You assert that by mentioning that there are security flaws in 2 of 5 major OS packages has made it materially easier for "bad guys" to do their jobs: go find them and tell me what they were, then.
You'll find that the amount of information I disclosed was virtually useless. You'd have to start with a code review of 400,000 lines of code. The only place where your starting point would be improved by my information is that I named some targets - targets that are so obvious that all I can say is "Duh?"

Basically, you're attacking my position because I'm making it uncomfortable for you to remain smug in your moral grey area. I would be uncomfortable with that, too, if I were you.

When we're talking about guilt, there's almost always a matter of degree attached. Let's consider an example:
1) "Please lock your car doors; this town has a high crime rate."
2) "It has come to my attention that a car in this neighborhood is unlocked. That is dangerous. If your car is in this neighborhood, please check it immediately and lock it."
3) "It has come to my attention that a car on this city block is unlocked and has a new GPS unit on its dashboard. If it is your car, please check it immediately and lock it."
4) "The blue Chevy parked in front of 115 W 5th St is unlocked and has a nice new GPS unit on the dashboard. If you are its owner, please lock it."
5) "If you are the owner of the blue Chevy that was parked in front of 115 W 5th St, please contact me and I'll return your car to you when you prove to me that it's yours."
6) "If you are the owner of the blue Chevy that was parked in front of 115 W 5th St, post a 1 page ad in the city paper saying 'thank you for showing me how stupid I am, Marcus!' and I'll return your car."

I don't like reasoning by analogy but I've constructed those examples based on some clear parallels to how the vulnerability pimping game is being played. I think it's a reasonable illustration - there are matters of degree.

What you've done is basically claimed that by doing #1 above, I should be mentioned in the same breath as the sleaze that make their living based on doing #4, #5, and #6. I'm sure you can see why anyone might take umbrage at that.

Anyhow - apology accepted, as Darth Vader would say...

Anonymous said...

(Marcus J. Ranum )

One final observation: when I was in 7th grade I used to torment my philosophy/religion teacher by adopting a position that "every action is selfish." In fact, it is possible to assert that there is no such thing as altruism - simply by turning it back on your opponent by asserting, "you just don't want to think of yourself as the kind of person who would/wouldn't X" At that point it's impossible to talk about objectve morality because the parry simply becomes an assertion of subjectivity.

Q: If you saw a blind old man about to step in front of a bus, and you could stop him without risking injury to yourself, surely one could say it's the moral thing to do it.
A: It's just selfishness. You're so selfish you don't want to see the disgusting sight of an old man getting hit by a bus.

There are sound arguments rooted in evolutionary biology that support the notions of "greatest good for greater number" and "I am my brother's keeper." I won't make them here. ;) But I hope I've made it clear that I feel a lot of players in the "internet security game" are acting a lot more selfishly than they would have us believe. As an industry we have been tolerating - even encouraging - these people who are "part of the problem" far too long.

Ryan Russell said...

So, Marcus and I have had a lot more of this conversation in private. For example, the crash the Internet thing. Marcus claims to be sitting on a wipe-out-the Internet, can't be fixed until IPv6 thing. And he probably is. And yes, my reaction is meh, BFD. I got over the Internet being wiped out in 2001/2002, when it started happening on a regular basis.

Marcus challenges me to use Fortify to find his bugs. Well, I have to assume I can't. I'm a pretty poor vulnerability researcher. But I'm not the guy to worry about. The guys to worry about there are the Marc Maiffrets, Dave Aitels, and HD Moores of the world. There's blood in the water, people like that can smell it. That's more than enough of a hint.

Me, I have to wait until the programmer in question puts out the 5 line diff file security patch. That's my speed. And I have full faith that Marcus just set in motion the process that will cause that diff to fall out, and the wannabees like me will have enough information to try making an exploit.

So, does Marcus get a small percentage of the blame that an eEye would get? The difference is that Marcus doesn't plan to put out an advisory. He doesn't plan to drop disassembly snippets or register dumps. And no greetz, I assume.

If you don't want to contribute any to there being more vulnerabilities in the world, then don't look.

To cut to the chase a bit, Marcus thinks it's very appropriate to help the world by finding the bugs and reporting them to the vendor, and then never speaking of it again. Then, he hopes the vendor will slipstream them into large releases where people will not be able to tell the small security fix apart from all the other code changes.

Me, I think that ship sailed 5 or 10 years ago. I've watched way too many smart people rip apart huge codebases or binaries to believe that the obscurity approach is practical. You can find posts from the eEye guys, for example, where they found fixes in XP SP2 that didn't have any corresponding Microsoft bulletins to go with them. XP SP2 is pretty large.

I'm hearing from the vulnerability pimps that, yes, code security is improving. They are reporting that it's much harder to find a remote hole in the current operating systems.

I'm going to have to take the word of the vulnerability pimps that things are getting better. They are the ones in the best position to know. They're the bad guys we're trying to defend against, right?

Anonymous said...


Considering the increased attach surface compared to 10 years ago, the increased amount of vulnerabilities reported isnt what offends me so much.

Ten years ago, I honestly could not name a single company based on the vulnerability pimp business model. Nowadays they seem to be everywhere. This is what offends me.

BTW This is the first mention of "purple hat" hackers I've seen and dang if it didnt get me giggling here at my desk.

Ryan Russell said...


Glad you got a chuckle, you HAVE to see Alan's picture though.

As to being offended by the actual vuln pimps... I think one has to decide whether they are in favor or against finding and publishing vulns to decide if they are going to have a problem with making a business out of it.

Anonymous said...


Great pic!

regarding your differentiation, were they true professionals or researchers, they wouldnt be pimps now would they?

niroa said...
This comment has been removed by a blog administrator.