It is however surprising how the media covers this presentation. IOS images, modified on the binary level to allow unprivileged access, are neither new nor extremely hard to do. When I presented on BlackHat Federal 2003 in Washington DC and mentioned backdoored IOS images in the presentation (Slide 12), nobody in the quite Cisco-aware audience showed any indication of surprise. Some time before that presentation, a random hacker had sent me an email with a few bash scripts which would correct any checksum in an IOS image that was patched, just for my entertainment, and mentioned that the scripts took him about half an hour to complete. What topo excels in is the level of platform independence and automatism his rootkit tools achieve. After all, he works in a team of experts on that topic.

The early backdoors in IOS images are a natural consequence of the software being fairly expensive and priced by features. In the same way that people receive a Windows XP Home version with their new PC, but want to run Windows XP Professional or even Vista, people receive an IOS IPonly image but want to support SSH logins instead of Telnet(1) on their routers. Accordingly, people would trade IOS images on the Internet instead of spending money on legitimate copies, especially since they also have to spend the same money on memory extensions for their routers to support the image. With images traded openly, of course attackers modify the image to allow login with a special password. Traded IOS images are an invite to them, as they essentially say: "Whoever downloads this either uses it in the lab or will install it on critical routing infrastructure in their corporation." There is just no other use for an IOS image.

The fact that IOS images are nothing but one single executable without inner security boundaries makes the modification just more powerful. When backdooring a Juniper router in the same way, the attacker has to modify a number of binaries, as Juniper's operating system is based on FreeBSD and therefore has real processes implementing different functionality (login, routing, etc.). It also has a kernel. Now, on IOS, everything is in the same process / address space / memory privilege zone, so a single patch can reach integrity checking code as easily as it can reach login validation code. Oh the other hand, what's the challenge of backdooring Juniper routers? Any adjusted FreeBSD rootkit will do.

What is left is getting the privileged access to the router in the first place. Best practice security in network design, architecture and configuration should prevent that, but there are people who don't follow those and there are vulnerabilities. As noted in other occasions, security functionality seems to be an excellent place to find bugs in. Even SSH, audited to death on other systems, appears to still be a very interesting target on IOS. In this respect, I can understand some of the service provider folks being unhappy about the presentation, as the availability of an IOS rootkit will most certainly rather fuel the interest in writing code execution exploits against IOS, rather than damp it. This is probably one of the reasons the rootkid maker code is not going to be released. Additionally, the heavily increased protections on common operating systems platforms (e.g. Windows Vista), will draw more attention to 1980s style, C written, complex monolithic systems running on critical Internet infrastructure.

All this isn't exactly news. I could have named this entry "I told you so", but that wouldn't really cut it anyway. What rather happened was the ongoing concern of a few people about this whole subject, several years before it was actually presented. When working on the first (and luckily completely replaced) version of the GDB debugging agent for Zynamics BinNavi that allowed runtime debugging of Cisco IOS in a decent reverse engineering tool, I realized that the GDB debugging capabilities would also finally allow the development of an independent forensics tool for Cisco IOS. I discussed this with Gadi Evron, one of the few people that foresaw the issue as well, during the Chaos Communication Congress in 2006.

This was the beginning of the CIR project, which stands for Cisco Information Retrieval or Cisco Incident Response, whatever you like more. CIR is a framework for analyzing IOS memory core dumps, the only solid and complete evidence that can be collected remotely from a router. While the GDB link provides a better (read: less easy to circumvent) interface, network operators do not appreciate driving to every router they manage and using it remotely would be pretty slow. CIR uses the IOS image as a blueprint for the memory map, as the core dumps do not contain any memory mapping information, and also compares static elements from the IOS image to the actual running code in the core dump. A paper and presentation on the technical background of CIR can be found on the Recurity Labs website. One of the simplest features implemented is the comparison of the .text segment between the image and the core dump.

When topo announced the IOS rootkit talk, I was excited that he was willing to provide a backdoored IOS image to me for testing CIR. This is responsible disclosure at its finest hour, as the patched image means little to Cisco Systems but very much to us for testing CIR. After all, detecting your own modifications to an IOS image is kind of silly. From the beginning on, CIR was meant to be provided as a free service to anyone who suspected that their router might have been attacked, backdoored or suffered from a successful or unsuccessful Denial of Service attack using vulnerabilities in IOS. Accordingly, processing was made available online when we barely had the alpha version working and was announced at BlackHat Federal in Washington DC. Since then, we made significant progress on the functional front, but most work went into code quality, dealing with the broken-beyond-recognition ELF files produced by Cisco's build process and making the framework ready to support bigger iron platforms.

Finally, but still before his talk at EUSecWest, topo provided the backdoored image. We fed it into CIR and successfully detected the .text segment modification. However, it only told us that the image was patched and where the patch begun. A quick change of the respective plugin then produced a detailed report that pointed to all modifications in the .text segment. The resulting report can be viewed on CIR Online for those interested in the details. Getting the exact virtual addresses out of the report makes reading the code differences in IDA a piece of cake: you jump to the respective address in the core file and press C. Out comes topo's backdoor, which is a beauty to read in PPC assembly. Felicidades topo!

So, while still in BETA phase, CIR Online seems to actually work. Of course, it also makes the shortcomings of the core dump method obvious: changing the core dumping function in Cisco IOS together with any other backdoor inserted would ruin the collection of evidence and therefore give nothing to process to CIR. This, as mentioned above, could be overcome by physically connecting to the serial console of the router and use the GDB link to obtain the evidence, which again could be prevented by patching the GDB stub in IOS. You end up with the classical race between attacker and forensics person. However, it is a race worth racing if you realize that the bad guys already left the starting line and the dust cloud at the horizon is them speeding away.

Now that some people actually talk about IOS rootkits, interesting tidbits show up. One person asked me if we have tested CIR with the Russian IOS rootkit that was for sale a few years ago. No, we didn't, but good to know that these exist. Additionally, Cisco finally published an updated and much more detailed response, giving instructions on how to detect and/or prevent rootkits with the on-board tools IOS provides. However, some of the instructions are rather humorous, such as transferring images to the router using a secure protocol when your image does not support SSH. On the other hand, they also published a list of MD5 sums of all 12.x images, which is shortly going to be a test CIR performs.

So finally, we seem to have overcome the discussion on if IOS rootkits work or not and approach the age where it's an accepted fact. We also know that, so far, binary and TCL backdoors are detected by CIR. That means we can concentrate on the crash-cause analysis and exploit attempt identification features again. Stay tuned.

Our Italian friends were exceptionally nice and forthcoming, making sure we had everything and were well entertained all the time. Many thanks go to the organizers of our daily and evening events.

One thing that struck me as strange was the security trade show itself. The exhibitors came almost exclusively from the usual suspect section of security software and appliance vendors and distributors. Many displayed embedded boxes of various sizes with little or no LCD displays that somehow made something secure. As far as we could tell, none of them sent any technical personal to the event and the attendees didn't seem to mind at all.

We talked to one particular vendor's booth personal since we happen to use one of their products and happened to stumble across some 0day vulnerabilities in it. The person did not know what a vulnerability is and, once we started to explain that their embedded product runs on Linux, insisted that we must be wrong, since it only supports Windows and Apple. Oh well. While I'm totally aware of the fact that a trade show booth is not the recommended vulnerability reporting channel, I did actually expect the company representative to know a certain minimum about their product.

Afterwards, it crossed my mind that at every trade show, may it be cars, construction equipment, tools, boats and even food, the exhibitors get out of their way to show the inner workings of their product, like engines, safety mechanisms and any other aspect that highlights the quality and uniqueness. At the security product show, nobody seemed to consider opening their magic appliances to even show the PCB and the hardware within; leave alone explained the inner workings in any considerable detail. And even then, people seemed to like the stuff, as far as we could tell. Very interesting.

The first lecture focused on security engineering and the different ways professional security verification services are performed. We also went into discussions on how software audits are done, including source code review as well as binary code audits.

It is interesting to talk to people with solid education on the subject but less practical experience in the field. What I found most astonishing was that they had little to no preference for a specific programming language. This caused some discussions about which programming languages are better fit for security verification. I had to argue about my point that more modern languages should be used for quite a while. What I think finally drove the point home was the effort needed for testing, especially unit testing and code verification. If your language and especially your runtime environment (for example .NET) does not allow you to play fancy tricks with pointers or address indices outside of the array bounds, neither testing nor automated code verification needs to cover those topics and deal with the problems inherent to such actions. If nothing else convinces you to stop writing your programs in C, it should be the fact that the less flexibility on the lower levels of machine interpretation you have, the less things can go wrong and be turned against you with an exploit.

Of course higher level languages have plenty of their own issues. But given the fact that the fault density actually directly relates to the number of lines of source code, a language in which you write less code allows for a smaller chance to introduce faults. This is why I cringe every time I hear or read someone stating that the prototype is written in Python and works well but the release version is going to be rewritten in C. Please, don't.

The event was more smooth-running than any other CCC Congress we have been to. Herding such a large number of hackers, coordinating and staying within the schedule of three tracks isn't easy. But this year, everything went perfectly well from our point of view. The community responded well to most talks we have attended and a number of follow-up activities arose, like the newly deployed barcode hackers wiki, hosted at cyphertext.de.

This year was also the first time that we could contribute the results of our research activities to the conference. We presented and released PortBunny, a specialized TCP port scanner for professional use. The motivation behind developing a new port scanner was the requirement in the professional security services world to be predictable.

Port scanning a large network can take a significant amount of time. When using the commonly available tools of the trade, the time to scan a network varies a lot, depending on how well the network is protected. But from a professional services point of view, you want to minimize the time tools work on a subject and hereby maximize the time you can work on it yourself. Additionally, being able to predict the time a TCP port scan takes is a huge improvement for estimating the overall time required for the engagement.

PortBunny is designed to run in the Linux Kernel space on dedicated machines. It uses different algorithms than most other scanners to separate the bandwidth saturation detection from the detection of filtered (aka firewalled) ports. The slides of the presentation can be found on our publications page.

PortBunny is released under the GPLv2 and can be downloaded from recurity-labs.com. Any feedback is much appreciated and should be sent to portbunny@recurity-labs.com.

Since Recurity Labs is a small company, we have a small FAX machine. It is one of those that work with thermo paper on a roll, a so-called thermo transfer ribbon. And since Recurity Labs has some significant dealings with layers lately, and lawyers love FAX, our FAX machine complained that the ribbon was used up. So we got a new one.

Being a good German and always caring about waste separation, I wondered about what to do with the old ribbon when I took it out. The regular lists of things to separate don't contain thermo transfer ribbons, so I inspected the foil to see if it would potentially qualify as plastic. To my mild surprise, I found negative copies of all the papers that were faxed to us during the time of use. The quality of what's left on the ribbon after the thermo transfer is impressive, very high definition, almost like film.

Now I wonder how many secretaries, legal departments, accounting departments and everyone else with the need for a separate desktop FAX machine (confidentiality, you know) discard used ribbons, either environmentally correctly or not, without a second thought about the content. We decided to just keep them around in a safe place and call them backups.

This is just one of these security related stories that are so funny, they must be shared. While working on site at a customer, the person maintaining the inline Snort IPS mentioned that he had to disable a rule due to complains from people. Apparently, the rule was preventing the CNN.com website to work correctly. For eye candy, here is the rule:

(msg:"WEB-CLIENT PCRE character class double free overflow attempt";
flow:to_client,established; content:"RegExp"; nocase; content:"[[";
content:"]"; distance:1;
pcre:"/(\w+)\s*=\s*('|")[^\2]*\[\[[^\2]*\][^\2]*\2\s*\;.*new\s+RegExp\s*\(\s*\1|new\s+RegExp\(('|")[^\3]*\[\[[^\3]*\]/smi";
reference:bugtraq,25002; reference:cve,2007-3944;
reference:url,docs.info.apple.com/article.html?artnum=306174;
classtype:attempted-user; sid:12286; rev:1;)

Since time and resources are precious, the rule got disabled and everyone was happy. I have to admit that I didn't really think anything of it, except that the rule might be somehow too broadly matching and therefore would eat legitimate JavaScript code. That is, until I surfed Fefe's blog a few days ago using my Nokia E70 phone and clicked on a link to CNN.com. After my phone tried to load the page for a while, the browser crashed. I remembered the episode with the Snort rule and also suddenly realized that my phone's browser is actually using the Apple WebKit, as does Safari.

I read up on the specific vulnerability ( http://www.securityevaluators.com/iphone/bh07.pdf) and went back to the CNN.com website, only to find that the included file main.js contains:

var htmlRegEx = new RegExp('[\w*|\W*]*<[[\w*|\W*]*|/[\w*|\W*]]>[ \w*|\W*]*');

This looks suspiciously like what is described on Charles Miller's slides mentioned above, namely: "A valid (though odd) pattern that looked like a POSIX character class but used an invalid character after [ (for example [[,abc,]]) caused pcre_compile() to give the error "Failed: internal error: code overflow" or in some cases to crash with a glibc free() error."

Although I didn't verify to all ends that this is what is crashing by phone's browser, I assume it is.

Dear Nokia, if I had wanted a vulnerable, DRM infested future malware platform in my pocket, I had bought an f***ing iPhone in the first place!

Thanks to K.S. for pointing out the Snort rule issue to me!

Update: People report that the iPhone works fine (now?) with CNN.com and I tested the iPod Touch myself (thanks iUte). So where is the Nokia update and security advisory?

It was a great time, hanging out with people like Luis Miras (RE guru, inventor of my favorite non-black t-shirt), Mike Reavey (MSRC), Nick Farr (Hacker Foundation) and Emmanuel Goldstein, who even joined in on the 2 hour traffic jam session back to the airport. Thanks very much to the organizers, especially Luiz, who is a great host.

We did, regardless of the expected nightmares, Fear and Laughing in Oracle.

Since Oracle is shipped as closed software and releases will be provided as binary/executable program only, we analyzed the Linux ELF binary executable files, because a windows version of Oracle 11g seems to be not released yet.

This is, what we messed around with:

setuid setgid ELF 32-bit LSB executable,
Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses
shared libs), not stripped

OK, uses shared libs, right:

-rwsr-s--x 1 oracle dba 145M Aug 31 16:42 oracle

An almost 150 Megabyte sized executable program, using shared libraries. Actually it has 17 shared library dependencies. The other shared libraries provided by Oracle, which are actually linked dynamically by other executables shipped with Oracle 11g, were statically linked into the oracle executable at compile time. We are talking libraries of 30 Megabytes and more linked in as well as sitting next to the binary, just in case.

The first approach of the analysis was to narrow down the relevant cryptographic algorithm and its implementation. Therefore, different techniques were used to find relevant methods and instructions within the executable. Most cryptographic algorithms like ciphers and checksum calculations expose some kind of "signature" or individual tokens like S-Boxes, transformation tables or constant values. Thus it might be easy to detect automatically within the binary, using tools like the FindCrypt IDA plugin or other scripts we developed for our own purposes.

At least 57 places with crypto were found by FindCrypt: DES, MD4, MD5, SHA1, just to name a few. We found at least two independently implemented AES cipher constants, all algorithms were double and triple implemented.

Another obstacle is the fact that the Intel Compiler, which was used to compile the Oracle executable, uses an optimization which led in having no cross references (XREFs) to code or data in several segments. Thus we could not see wherefrom for example an S-Box is accessed in the code. So we used the IDA API to implement a tool which automatically finds these PC-relative offset calculations and adds XREFs to the IDB. One can only assume that Oracle uses the Intel compiler because no other compiler would produce efficient enough code to run this behemoth of a binary in acceptable speed.

We also combined the static analysis by disassembling the Oracle executable with a runtime analysis using ltrace and the precious GNU debugger GDB. Have you ever tried to attach more than two dozens processes with GDB and set a few hundred breakpoints in batch mode? It's real fun.

Anyway, what were we looking for? We're looking for some kind of "create password hash" function, which generates and stores a 30 Byte ASCII-hex encoded sequence in table sys.user in a field called "spare4".

After we ran into different hashing and cipher functions (Oracle actually does Kerberos, AES for TLS, etc) we found the appropriate hashing function.

There is a function called ztv2ghashs, which takes the following arguments: a password, the password length, a salt value, the salt length, a hashing algorithm identifier and a structure which holds the calculated hash value as result.

The "identifier" is a simple integer and is compared against constant values to decide which hashing algorithm to use:

• 0xf00d means: Use MD4
• 0xdead means: Use SHA1
• 0xbeaf [sic!] means: Use MD5

text.hot:0E5A9038    call $+5
text.hot:0E5A903D    pop  ebx
text.hot:0E5A903E    add  ebx, 0EEA4C7h
text.hot:0E5A9044    cmp  eax, 0F00Dh
text.hot:0E5A9049    jz   short loc_E5A9072 ; MD4_Init
text.hot:0E5A904B    cmp  eax, 0BEAFh  ; <- veggies ?!
text.hot:0E5A9050    jnz  short loc_E5A9062 ; SHA1_Init
text.hot:0E5A9052    push edx
text.hot:0E5A9053    call ztchmd5i  ; MD5_Init

So we found out that the final hash is calculated in the following way:

hash_init( HASH_CTX, algorithm );
hash_update( HASH_CTX, password, password_len );
hash_update( HASH_CTX, salt, salt_len );
hash_final( HASH_CTX, result_buf );

Let's try this at home, kids!

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> create user FOOBAR identified by SHAlala;

User created.

SQL> select name,spare4 from sys.user$ where name='FOOBAR';

NAME

------------------------------

SPARE4

-------------------------------------------------------------------------------

FOOBAR

S:2BFCFDF5895014EE9BB2B9BA067B01E0389BB5711B7B5F82B7235E9E182C

SQL> Disconnected from Oracle Database
11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

$ echo -ne "SHAlala\x1B\x7B\x5F\x82\xB7\x23\x5E\x9E\x18\x2C" | sha1sum
2bfcfdf5895014ee9bb2b9ba067b01e0389bb571  -

We would like to thank Alex and Pete for this interesting challenge and all the helpful information regarding Oracle. Fun Inside. There is a lot of funny SQL, Kerberos, etc and definitively worth digging deeper.

And we would like to welcome Oracle Corp. in the year 2007, the century of highly advanced, mixed-case passwords. :) It should be noted that Oracle, in fine tradition, makes the same mistake Microsoft did a decade ago when they put the insecure LANMAN hash next to the brand new NTLM one. The table sys.user$ still holds the case insensitive DES encrypted password version next to the new one.

by THS

Accordingly, Recurity Labs now offers the following research position:

Bring-your-own-topic Researcher

You work at Recurity Labs Berlin on your own topic, receiving regular (shitty) student pay for doing what you want. The rights to your work and resulting publications stay yours, you just give permission to Recurity Labs to use your results in our own work. You also present your progress on a regular basis internally. Additionally, you may book Recurity Labs people to help you with your project. What work we do for you, is largely your choice.

We realize that there may be interested people out there who don't know what topic they should propose but still fancy the idea in general. We can certainly provide a few suggestions if you tell us what skills and areas of interest you have.

If you are interested, send a short summary of who you are and an abstract about your topic to byot@recurity-labs.com.

Up until yesterday, our small little company was known under a different name. We even held the trademark for this name in Germany. We also knew that there was a corporate behemoth called Sabre with a number of fully owned or whatever subsidiaries. Their business is travel networks and services. We didn't think too much of it.

Apparently, they thought differently. Our small company and my poor little self are currently facing massive legal issues, the details of which we cannot disclose at the moment for obvious reasons.

The one thing we can tell you already, if you didn't already notice it, is a new name for us: Please welcome Recurity Labs.

One of the primary reasons for the leading FUD role of the security community might be the mental process of reviewing a new system or set of systems for attack surfaces. In the beginning, the entire system is seen as a whole. Then, gradually, individual parts of functionality, their intention and implementation are considered in greater detail. Most of the time, a gap between intention and design or intention and implementation is uncovered sooner or later. This gap of course is only present in the small part of the system you are currently looking at.

What often follows is the false application of the same process backwards when considering the impact and importance of the discovery. It goes like this:

• I discovered a buffer overflow in program A, which allows me to execute arbitrary code.
• Program A is written in C.
• My operating system is written in C.
• Therefore, my entire operating system must be vulnerable to buffer overflows.
• This operating system is used all over the Net.
• Therefore, the Net is vulnerable to buffer overflows.
• The world is going to end.

Although none of the observations above is provably wrong, the thought process of a security review is not useful for impact considerations. Many other factors play into the impact of a discovery and deserve a special, case-by-case consideration.

The Web 2.0 has all the potential for the next big wave of FUD in security. First of all, it's not done yet. We are seeing new players on the Web but the general direction of developments is sketchy at best. One of the more solid observations is that the Web 2.0 is a work of composition from known technologies at a higher abstraction level than before. Most components are not reinvented but rearranged and adjusted. This leads to some of the lesser-known components and especially patterns [6] to be considered new, revolutionary developments [4].

The new Web primarily teaches us lessons we should already know. Basics like the fact that perimeter security cannot work in networked environments, since they wouldn't be networked if it did - think mesh-ups. Basics like: defence in depth is one of the few paradigms that actually have a chance to work in the wild and keep complex systems alive. But we knew that before, didn't we?

Another indication for a new FUD wave is usually a massive increase in predictions of the future ("Some times, I get the feeling that old generation of security experts and hackers will never grasp this principles the way the upcomming waves will."[4]) and, if the predictions are not coming along fast enough, they receive help from the prognosticator ("The spider that I wrote is anything by malicious. It just spiders. However, keep in mind that it will take less then 5 minutes to make it equipped with the latest AJAX exploits. Therefore, I am not responsible for your actions. Be responsible. Here is the spider source code"[5]).

It should really be noted that there are plenty of security problems to be solved in existing and emerging environments. A security problem is not less sexy just because it doesn't affect millions of innocent users. In fact, the singular focus on the next world-smashing security issues obscures the view onto underlying issues and especially simple and reliable solutions that are sitting just around the corner, waiting to be discovered by the sensationalist crowd. There is really no need for more FUD, we got plenty of real work to do.

Update pdp was nice enough to point me to the following discussion about this article that I want to share: http://sla.ckers.org/forum/read.php?13,13871
Thanks man.

References

1. Fear, Uncertainty and Doubt
2. http://www.blankenese-seiten.de/kirchenkreis/04-Einrichtungen/Seniorenakad/Vortraege/Brunnsteintxt.htm
3. http://www.chscene.ch/ccc/ds/54/032_brunnstein.html
4. http://www.gnucitizen.org/blog/projections
5. http://www.gnucitizen.org/blog/yahoo-site-explorer-spider
6. http://en.wikipedia.org/wiki/Model-view-controller#Implementations (invented 1979)

Some may have seen the movie "The Devil Wears Prada". In one scene, the protagonist is ordered to obtain a copy of the latest Harry Potter book, which is not available in stores yet. This being a movie, she manages to get it. The script author referred to the hype created by the Harry Potter publishers Bloomsbury Publishing Plc. around every single release of the book.

I always wondered why the script of an upcoming Harry Potter is not obtained beforehand simply by breaking into the publisher's network. My guess was that the people with the required abilities and skills probably have better things to do. But of course, the stakes are higher with the (hopefully) last book in the series.

Today, a post on the Full Disclosure mailing list claims that a copy of the script for the upcoming book was successfully obtained and presents a spoiler with the ending of story, as it will be released in 32 days or so. The post mentions that the way to get it was to send an email with a link to a web page that contained some well-known exploit from milw0rm. The post mentions that it is surprising how many people in the company have the script somewhere on their computer. Game over.

A copy of the new Harry Potter: $34.99.
The global value of the Harry Potter brand according to Forbes.com: $1.000.000.000.
Getting the final marketing move p0wned: priceless.

It doesn't really matter if the Full Disclosure post is a fake or really contains the ending of the next book. If your content is as valuable as this script and your marketing campaign is about the fact that nobody knows about the ending, you should better prepare for someone raining onto your parade.

Now would be a good time to sit back and think about the value of your company's intellectual property assets and if you can be sure that nobody else knows about them. Start with the following, non-exhaustive list of checks:

• Is the information known to exist outside of the company? How do you know?
• Is the information always stored encrypted? If so, how do you know?
• Is the information always destroyed when printed? How do you know?
• Are the backups encrypted? Who got the key? How do you know?

Next time when your laptop is stolen and you fill into the forms an estimated monetary loss of more than $4000, it would be an indication that you did in fact think about the questions above.

Last week, I needed a new passport, and quickly at that. Since this involves German government institutions, the process gets more complicated the quicker you need something to be done. To make a very long story short, I ended up with my two hour old passport pictures at a governmental office and got told that the pictures would not work. What happened?

Germany is now rolling out the e-pass. The political smoke screen with Terrorist decoration aside, it was the only way to sell large amounts of otherwise totally useless and probably unreliable biometric detection software to customers that cannot effectively refuse to buy - the citizen. If you need to travel, you have to buy the new passport, which supports the German industry, well, some of them.

One of the core features of the new passport is a frontal picture, as opposed to the slightly sideways one used before. The frontal pictures can be read and interpreted by a piece of software in the government office to calculate biometric data about your face, so all the fancy Frauenhofer-Anti-Terror cameras at airports, train stations, in taxis, hotels and the cabins in the city sex shop may be able to identify you.

For this to work, photographers were told how to take an exact frontal picture of you. They usually charge more for the biometrics ready picture than for a regular one too. So I went to a photographer's shop and got my biometrics ready pictures taken. Before the picture is scanned into the computer at the government office, a transparent sticker with an outline and some orientation points is placed on top of the photograph to tell the software where to look for the face.

The software at the government office kept complaining that my picture was not showing a head in frontal perspective, although it was painfully obvious that it was. The surprisingly helpful lady re-placed and re-adjusted the transparent sticker several times to make the software understand that in fact, this is a frontal picture of someone who just happens to have a real egg head - but to no avail. Finally, she sent me on my way again to another photographer's shop around the corner that supposedly produced acceptable pictures for the software with a 100% success rate. So I went getting new pictures taken.

The photographer at the second shop was a young and obviously street-smart German citizen of Turkish descends. After the second round of pictures was taken, he loaded them into Photoshop and copied one into a template file he had at hand. The template file was composed of rulers and grid lines for Photoshop, which seemed to mirror the requirements of our governmental biometrics software at the passport office. Once it again became obvious that my egg-head would not fit into the boundaries, he scaled it on the X-Axis to make it fit. Kiss your aspect ratio goodbye.

Now, I don't understand much of the biometrics matching algorithms used by our government's software. But the little that I do know about face recognition involves head geometry information, including the distance between eyes in relation to other distances, like the one between your nose base and mouth. Well, if you scale an image without preserving the aspect ratio, this information is changed.

What came out was a head shot that was happily accepted by the passport office software as a perfectly valid front shot and respectively is rendered into my passport as well as placed on the RFID chip digitally. The only issue is: that's not me, or to be more precise, it is not my head geometry. May be I should be happy about this, since it will make life a bit harder for the government officials should my profession become entirely illegal in the future and cause them to search train stations and public places for me.

The story somehow reminded me of NASA's input validation when they put a satellite in place over Antarctica. The satellite neglected the fact that there was a hole in the ozone layer, since the measurements were so far out of average that they were considered false. The same can probably be said about my head geometry compared to the average German citizen.

When you collect sample data that you want to use in detection mechanisms, it's probably not wise to neglect data just because it is different than you expected. This will lead to the detection only detecting things you did expect, which renders the detection slightly useless. On the other hand, the story underlines the fact that overly drastic security measures, governmental or otherwise, will increase the likelihood of people circumventing them, knowingly or not. If you bother people enough, they will start cheating. If you bother them even more, you loose them.

Update:
Some forum posting people suggested the root issue is actually that the person photographed is ugly. While this is obviously not a false statement per se, it would suggest that the biometrics software exhibits automatic taste - an interesting thought ;)

• The big vendors have come to realize that they can sell their products with lots of vulnerabilities in them as long as they appear reactive to vulnerability reports. Nobody gets sacked for buying IBM, a proverb once vent. The same might be true for Sun, Cisco and Symantec today.
• Microsoft invested so much money in the security of their product line and the industry was always "us versus Microsoft", so they decided to kill the giant by going the other direction and strictly and stubbornly not caring about product security and quality (which, in fact, are very close to each other if not the same).

Congratulations to Sun Microsystems, you successfully moved the Internet over a decade back in time. As of today, we have a new worm spreading, exploiting an authentication vulnerability in telnet of all things! In Solaris (SunOS 5.10 and 5.11), you must know, there is no need to actually posses the password of a telnet user. All you need to do to get a shell with the privileges of the user "adm" is:

SomeLinux$ telnet -l "-fadm" my.poor.sun.isp.net

The same would work for root, but luckily the default installation of Solaris does not allow remote root telnet logins. Not only is this an ages old type of vulnerability, it's reintroduced by Sun into their latest operating system. How on earth can QA miss something like that? In 1995, this type of vulnerability hit a long list of UNIX vendors (see here). Therefore, when hacking around in their telnetd implementation, I would expect that at least someone would check if this new feature they are implementing might be a very bad idea indeed.

But Sun just picks up where Cisco is leading the pack right now. Let's take a look at a few of their recent publications:

cisco-sa-20070228-nam
NAMs communicate with the Catalyst system by using the Simple Network Management Protocol (SNMP). By spoofing the SNMP communication between the Catalyst system and the NAM an attacker may obtain complete control of the Catalyst system."

cisco-sa-20070214-pix
Multiple vulnerabilities are found in Cisco PIX 500 Series Security Appliances and the Cisco ASA 5500 Series Adaptive Security Appliances. They affect the following: Enhanced inspection of Malformed Hypertext Transfer Protocol (HTTP) traffic, Inspection of malformed Session Initiation Protocol (SIP) packets, Inspection of a stream of malformed Transmission Control Protocol (TCP) packets [...]

cisco-sa-20070213-iosips
The Intrusion Prevention System (IPS) feature set of Cisco IOS® contains several vulnerabilities. These include: Fragmented IP packets may be used to evade signature inspection, IPS signatures utilizing the regular expression feature of the ATOMIC.TCP signature engine may cause a router to crash resulting in a denial of service.

cisco-sa-20070124-crafted-tcp
The Cisco IOS Transmission Control Protocol (TCP) listener in certain versions of Cisco IOS software is vulnerable to a remotely-exploitable memory leak that may lead to a denial of service condition.

cisco-sa-20070124-crafted-ip-option
Cisco routers and switches running Cisco IOS® or Cisco IOS XR software may be vulnerable to a remotely exploitable crafted IP option Denial of Service (DoS) attack. Exploitation of the vulnerability may potentially allow for arbitrary code execution. The vulnerability may be exploited after processing an Internet Control Message Protocol (ICMP) packet, Protocol Independent Multicast version 2 (PIMv2) packet, Pragmatic General Multicast (PGM) packet, or URL Rendezvous Directory (URD) packet containing a specific crafted IP option in the packet's IP header.

cisco-sa-20070118-certs
The Cisco Security Monitoring, Analysis and Response System (CS-MARS) and the Cisco Adaptive Security Device Manager (ASDM) do not validate the Secure Sockets Layer (SSL)/Transport Layer Security (TLS) certificates or Secure Shell (SSH) public keys presented by devices they are configured to connect to.

cisco-sa-20070105-csacs
Certain versions of Cisco Secure Access Control Server (ACS) for Windows and the Cisco Secure ACS Solution Engine (here after both referred to as purely Cisco Secure ACS) are affected by multiple vulnerabilities that cause specific Cisco Secure services to crash. Two of the vulnerabilities may permit arbitrary code execution after exploitation of the specified vulnerability.

cisco-sa-20061025-csa
Cisco Security Agent (CSA) for Linux contains a denial of service vulnerability involving port scans. By performing a port scan against a system running a vulnerable version of CSA, it is possible to cause the system to become unresponsive. Cisco Unified CallManager (CUCM) and Cisco Unified Presence Server (CUPS) ship with a vulnerable CSA version.

I'm sorry this list gets so long, but I'm really trying to just focus on the glaringly silly ones. To sum it up, Cisco's security software and appliances crash when being presented with port scans or intentionally malformed packets. Duh! Hello Cisco! These are the devices your customers are paying a lot of money for to protect them against the exact threats they are vulnerable against! And a security analysis and response system that doesn't even validate any SSL certificate or SSH key? What did your QA exactly test under the functionality topic of authentication? Something along the lines of: "I logged in - check."?

At least the picture is consistent. Sun, shipping UNIX since 1982, reintroduces a vulnerability type that was considered extinct for more than a decade. Cisco, shipping IP routers since 1987, notices in 2007 that they still don't know how to correctly parse IPv4 options in a ping packet, even with their latest and greatest IOS XR.

So far, there have been no provable relations between a company's turnover, stock price and market share and their security track record. The only exception is of course Microsoft. I wonder if that's what is really needed to make the other big ones understand the enormous responsibility they have due to the cheer amount of today's daily life functionality depending on their code. After all, when looking at the professional and social life in today's Internet, it is indeed 2007 and not back in the 90s. Turn off all Cisco equipment on the Internet and try to do your daily job - it might get a little bit more difficult than usual.

At first sight, the code he showed did indeed look like a variable initialization was missing:

DWORD RS_MDS_Encode(DWORD k0,DWORD k1)
	{
	int i,j;
	DWORD r;

	for (i=r=0;i<2;i++)
		{
		r ^= (i) ? k0 : k1;			/* merge in 32 more key bits */
		for (j=0;j<4;j++)			/* shift one byte at a time */
			RS_rem(r);				
		}
	return r;
	}

The initialization of the variable r wasn't really obvious and I missed it too. Frankly, my annoyance was significant since we are currently helping a customer to design a security protocol with Twofish being one of the cornerstones, due to the cipher's strength and conservative design. And we went to great lengths to get it right, getting a cryptographer on board for the project and verifying everything over and over again. The prospect of looking at a broken reference implementation wasn't really what I wanted to hear.

After reviewing the code today, it quickly became obvious that there was no bug in the function; it initializes r when entering the loop. But, while looking at the reference implementation, I noticed the following comment at the beginning of the file:

Notes:
		*	Pedagogical version (non-optimized)
		*	Tab size is set to 4 characters in this file

While browsing the source code, I immediately stumbled over a few lines such as:

/* works for big and little endian! */
d[i/8] |= b << (4*((i^1)&7));		

outBuffer[n/8] = (outBuffer[n/8] & ~ bit) | 
                 (ctBit ^ ((((BYTE *) x)[0] & 0x80) >> (n&7)));

And I have to say: with all due respect, this is everything but pedagogical code. What is the point of presenting the pedagogical reference implementation of a cryptographic algorithm in a language such as C, where out of bounds array access isn't really noticed and one can write code such as the examples shown above? To teach students that really important code must look like this? If this is not optimized for actual use, why not present the code in a readable form, computing everything one step at a time and preferably in a programming language that doesn't look like a someone rolled an armadillo over his keyboard?

Schneier and Ferguson correctly state in their book "Practical Cryptography" that complexity kills most security systems and that the only known way of handling complexity is to modularize the problem into chunks that the human mind can handle. This advice should be adjusted to read: chunks that an average human mind can handle. I know that many people are proud of their brilliance, rightfully or not, but that's not the point.

Cryptography is fragile and delicate enough by nature. But it is also a very important building block of many security systems. Therefore, many programmers must implement cryptography in their respective programming languages and some of them may not be able to correctly understand either the C gibberish in the reference implementation or the LaTeX special math character party in the official Twofish paper. May be it's just my world view as a consultant, but we always try to explain demanding content as simple as possible while still being correct, which is in fact the real challenge.

A common answer is: If people don't understand it completely, they should not implement crypto. This seems simple enough, but the number of people in the world who do not fall under this rule is too small compared to the amount of software that needs protection. We need security mechanisms most programmers (and auditors) can handle; otherwise we will keep producing insecure software. Not even the most brilliant person can write all his programs himself without the need to trust another, potentially unknown programmer. And the other programmer will, be definition, be less brilliant.

Schneier and Ferguson keep repeating the following important sentence in their book: "We have enough high performance insecure systems, we don't need another one." They are totally right. But with all due respect, please consider this rule yourself next time.

On the 16th of November, Lance James of Secure Science Corporation posted on the SecurityFocus Binary Analysis mailing list and 5 other mailing lists an extensive analysis of a piece of Trojan horse Malware for Windows. The purpose of the Malware was to collect HTTP Post data when certain keywords, such as "bank", "login" or "mail" were matched in the target URL and drop the collected data at some (probably hacked) web server.

The report, written by an anonymous author at Secure Science Corporation and Michael Ligh, explains how the Trojan works and how it encodes and compresses the data collected before dropping it onto the collection sever. Section 11, detailing the encoding and compression, also contains some example material gathered during lab tests and containing only exemplary login data used by Michael Ligh for the testing of the program.

What surprised me was section 12, which detailed that the drop site was changed by an update to the Trojan on October 18 and listed the un-obfuscated IP address of the new drop server. When I saw the authors reporting updates on the drop site at the time of them writing the report and concluding that the site is still very active, we checked it out. Result: It is still alive!!!

Not only did the authors of the report point everyone and their dogs to a live and kicking drop site for stolen banking, email and other accounts but they also detailed how to decode, decompress and interpret the data into a readable and easy to use format. To verify that this is actually the case, we implemented a decoder solely based on the information in the report and tested it with the daily TAR ball of the drop point site from November 17. The TAR ball contains logs from 1196 infected computers, all in all 371 MB of illegally obtained confidential personal data. Our decoder successfully converted the data into readable text format. According to the PHP scripts running on the drop site, it currently maintained 7410 infected machines.

We will assumed in dubio pro reo that Secure Science Corporation tried to contact the hoster or upstream provider of the drop site and simply failed to have the site taken offline due to the contacted party being unreachable or unresponsive. We asked Secure Science Corporation about that and requested taking down the paper and replacing it with a version not disclosing the IP address of the drop site. The only answer was: "many attempts have been made", no comment regarding the paper.

So at the end of the day, we made our own efforts and informed some contacts about the drop site. It is now offline, replaced by a fresh and probably equally insecure Apache installation. May be this happened due to our requests, may be not.

An entertaining side note: A Google search for "FBI computer crime" produces the email address of the FBI National Computer Crime Squad (NCCS) as nccs@fbi.gov. Unfortunately, the address bounces with a "user unknown" SMTP error. Apparently, computer crime has been eradicated in the USA.

The morale of the story: Is the act of publishing illegally obtained information that can only be used to commit a crime itself a criminal act? I don't know. Is it responsible to do so? Certainly not. Irresponsible disclosure is obviously not limited to hackers.

Now, if not speaking were the augmentation of speaking, what would be the antonym of such golden silence? What could be the worst case of mindless babble? The answer smiled at me today from an exhibition advertisement (as in advertising exhibitionism) printed on transparent film. Reading it from both sides was a revealing experience:

... .

Disclaimer: The current temparatures in Berlin do not allow for any straight thinking. Therefore, we have to revert to humor to prevent our brains to from simply stopping operation altogether. Consider this post as the our mental screen saver.

The whole story would not be worth telling if Citibank had actually cared about my reports when I contacted them. But they didn't care and still don't. (well, they finally did, see below for the Update)

Juergen Schmidt and Daniel Bachfeld from Heise publishing supported my at first unsuccessful attempts to inform Citibank of the problem. Not that Citibank did anything about it. The full article is published at Heise Security UK. A German version can be found here.

Most amusing is the response from Mr. Stahlschmidt of Citibank to Daniel Bachfeld from Heise (I never got any feedback from Citibank, I'm just a customer).

Sehr geehrter Herr Bachfeld,

vielen Dank für Ihre E-Mail. Die TAN-Generierung der Citibank beinhaltet
selbstverständlich eine Zufallskomponente. Dabei handelt es sich um eine
zeitbasierte Komponente, die in den Prozess der Generierung von TANs
einfließt. Da hierbei fortlaufend sich ändernde "Zeitpunkte" einfließen, ist
die Generierung der TANs letzten Endes zufällig. Es ist somit
ausgeschlossen, dass sich eine TAN aus anderen TANs errechnen lässt.

Die Einbindung der Zufallskomponente Zeit erklärt, warum die einzelnen TANs
in aufsteigender Reihenfolge sortiert sind - schließlich nimmt die Zeit als
Zufallsgröße während des Prozesses der Generierung zu. Dabei wird jede TAN
unabhängig von den vorigen ermittelt. Der Zahlenraum - der je nach TAN-Liste
wie auch in Ihrem Beispiel beschrieben sehr unterschiedlich dimensioniert
sein kann - ist durch das unterschiedliche Anwachsen der Zeitkomponente
bedingt. Sprich: Die statistischen Häufungen von gleichen Differenzen
zwischen einzelnen TANs sind zufällig und somit je nach TAN-Liste
verschieden. Etwaige gewonnene Erkenntnisse über Abstände zwischen einzelnen
TANs aus einer TAN-Liste lassen sich somit nicht auf eine andere TAN-Liste
übertragen. Insofern ist das von Citibank angewandte
Zufallskomponenten-Verfahren sicher.

Mit freundlichen Grüßen

Rüdiger Stahlschmidt

Citibank Privatkunden AG & Co. KGaA
Unternehmenskommunikation
Kasernenstraße 10
40213 Düsseldorf
Fon +49 (0)211 8984-959
Fax +49 (0)211 8984-164
mailto:ruediger.stahlschmidt@citigroup.com
http://www.citibank.de

And finally, some links to related material. The differential analysis stuff was done using some scripts by Florian Walther that can be found here.
Bindview (now Symantec) used the same method in 2001 to analyse TCP initial sequence number generation. Even Cisco fixed their generation algorithm afterwards and that's TCP sequence numbers, not replacements for your signature.
I never expected to come across a case where Cisco is the example of a good vendor response and the worldwide largest private bank is the counter example.
The best related link award goes to Citibank itself. On the German online banking FAQ page, they state that HBCI is too complicated and therefore PIN/TAN over SSL is good enough. But may be a HBCI terminal is able to generate random numbers.

Update! Today (24.10.2006), I received the following Email forwarded from Daniel Bachfeld (Heise) containing a message from Citibank. And you know what? They will be using a random number generator from now on!

wir möchten Sie informieren, dass die Citibank für die Generierung ihrer TANs ab
sofort einen Zufallsgenerator verwendet. Dies bedeutet, dass die Nummern in den
von nun an vergebenen TAN-Listen rein zufällig sind.

Mit freundlichen Grüßen                                                            

Rüdiger Stahlschmidt

Being invited by mc.fly to speak at the MRMCD event, I found myself talking to a few nice gentlemen about SQL injections and their use. One of them (his name is intentionally left out) lends me his laptop, a brave thing to do in any case, and pointed his browser to a demo web application for employee work time tracking, which he said is used in his work place. The first thing we noticed was that the standard "I'm here to work now" button was labelled "coming", while there was another button labelled for the exceptional case of "coming with a reason". The software vendor selling this application must be an interesting place to work at.

When we played with a few SQL injections in the application's "Search for employee by name" function on the "absence from work" page, a most interesting error message was returned:

The only contributions we made to the statement were a few characters and a single tick, which of course caused the expression to be invalid. The limiting expression

AND p.nummer >= 'AND p.nummer NOT IN (12,17) ORDER BY Nachname'

The purpose of the expression was obviously to not display the time records of employee number 12 and 17, since p.nummer is the short form of Personalnummer, the German's term for employee ID. It is arguable if such a feature would never be noticed (as in "Why is Mr. L33t Coder never on the time records?"). Or, as Fefe would put it: "Das merken die NIE !1!!"

What should be noticed here is:

• The backdoor will never work.
• The backdoor is hard coded and therefore exists in any installation of this software at any customer.
• If the programmer fails to correctly backdoor his own application, it is clear why the entire software fails to filter user input.

If your software is in a state where random hackers refer to it as a perfect training ground for SQL injection techniques, you should be worried. If the hackers in question identify, with the first injection attempt, your developer's backdoor, which doesn't even work, you should be embarrassed.

And the morale of the story: When you buy software, you don't know what it is going to do. Although the task everything but easy, I think it's high time to fix that.

I'm quite used to computer systems crashing for no apparent reason other than my personal proximity, so I were not really surprised to see a Windows message box proclaiming that Internet Explorer had just died and asking if a report of the accident should be sent to Microsoft. This did spark a little interest, so I stopped and walked over to the kiosk box.

Kiosk systems are by design meant to hide the original operating system user interface from the person using the kiosk. I vividly remember many years ago, the direful task of creating such a kiosk system was given to me. To make a long story short, even with specialized kiosk software, it is almost impossible to setup a system in a way that the kiosk mode will provide real web browsing experience but the user will be unable to influence the underlying operating system. It has been shown countless times that there is almost always a way around it. By now, it has become a hobby for many people to break into kiosk systems in airport lounges during the 5 to 10 minute delay before boarding the plane. Actually, the delay is usually much longer, but the kiosk is just good for 5 minutes of fun. At one occasion, the airline lounge people just covered the kiosk by a hand-written "out of order" sign to prevent further misuse, since the kiosk would show an open cmd.exe shell (with NT Authority/SYSTEM privileges) instead of the airline's home page.

So it was no surprise that the little crash report dialog window did indeed provide the expected way around the user interface limitations put in place by the kiosk setup. In this case, it was by simply clicking on the "Microsoft data collection policy", which opens another instance of Internet Explorer. From here, it is just "File", "Open" and a short navigational challenge getting to c:\winnt\explorer.exe using a not-so-small finger and a touch screen that is wet from the rain.

The reason kiosk systems almost never provide the security their makers think they do is simple: Windows was never built to prevent authenticated users from accessing the user interface. While Vista improves significantly on the distinction between a logged on user and a process running under the user's context, even (some times especially) Vista relies on the fact that the OS and the processes running in it have real time access to user interaction functionality.

A kiosk system, on the other hand, must provide the user interface more in the way of a web server providing its content: if you are in doubt, just don't do it. Therefore, the decision to display anything to the user must be made by the kiosk software and not by any other part of the system. Or, in other words, the security functionality of the kiosk system lies in its ability to decide under which circumstances it should extend the possibility for interaction to the user. Which is of course not the way Windows is built.

I have worked with companies that made ATM systems. Due to their obviously higher interest in the security of their products, they invested a lot more time into this issue and basically ended up with an identical concept. The same applies to most ticket machines I have seen from the inside. While many of these systems are based on Windows as well (and some still on MS-DOS), the display and interaction decisions are centrally embedded in kernel space to prevent any process, may it be part of the system or not, from influencing the security relevant interface between the machine and the user.

Kiosk software following the approach of using a standard system and an arbitrary browser and then trying to hide everything else from the user will never work. That's simply because the software would now have to guard every possible way a process can open, modify or close a window on the system - very unlikely for a user land program. I have seen approaches of programs sending "minimize" window messages to every but their own window and claiming to have solved the problem, until they discovered stay-on-top modular message boxes.

It should (hopefully) be clear by now what larger issue I was referring to in the beginning: When designing and building a system, it must be clear which interfaces provide security and validation. Once these are identified and classified, the components providing said security must be designed. If the interface in question can be used by anything else than the product itself, the security components must be placed at a neuralgic point where they can guard the entire interface, not just the part that obviously belongs to the product. Or would you install a PHP remote command shell on the same web server that runs your online banking application?

The aim of this campaign is to inform the general public in informative, thrilling and entertaining ways about the importance of informatics. Another goal seems to be to draw more young people into the fields of computer science. All of the above are noble goals and should be supported.

Security is a topic, even in this campaign. For example, there was a discussion round in February at the university of Magdeburg. Reliability, on the other hand, is not a term that can be found on the campaign web site. This is interesting, since the two are closely related, although this seems to not be widely accepted yet.

The easiest way to understand why reliability and security are very close together is an imagined computer system that works exactly as it is supposed to be and is therefore extremely reliable. Does this system crash when being presented with long user inputs? Unlikely. Dependable and flexible security models that guarantee confidentiality, integrity and availability need reliable software foundations. Otherwise, they will get circumvented all the time by exploitation of faults in the software.

Back on the Informatikjahr web site, what fascinated me most was the event search functionality. When entering a search term, you get a list of events that cover the topic. When accidentally hitting the single tick key [ ' ], you end up with a web page of Wissenschaft im Dialog stating that this area is currently worked on. The image links are broken and the page descriptions are from a physics event in Dresden 2003. Of course, this looks suspiciously like a SQL injection vulnerability. Just to make sure, one can enter % in the search field to get all the events they have in the database.

Inspired by the flexible SQL interface unintentionally provided, I took a look at the HTTP Server string the site returns and found a SuSE installation of an Apache 1.3.27, mod_perl 1.27, mod_ssl 2.8.12, OpenSSL 0.9.6i as well as PHP 4.4.0. Most things in this list are outdated and some have severe known security issues. And just in case you accidentally deleted the icons directory from your Apache document root, you can get a fresh copy from Informatikjahr.de/icons/.

So we have an initiative that aims at showing the people of this country how important computer science is and hereby implicitly how dependent they are on correctly functioning computer systems. This might inspire people to get into computer science, may be to write software and even publish it, potentially open source, which are desirable outcomes. The campaign is underway and there is still half a year left, so may be we will see events concerning reliable systems, secure development, web application security and privacy as well as maintaining open source systems for reliability and security. But one thing is already obvious: We still have a long way to go, thanks to computer science.

While we are waiting for those vendors to take a bite from the Apple from the tree of knowledge and getting banned from their imagined paradise of unbreakable software, we have some time to think about this perception of the vendor's sole responsibility for security. This commonly accepted point of view is partially based on the fact that the vendor who receives the money for the software product must care about the product's quality, security included. While I fully embrace this concept and still hope that one day, you can return software that just doesn't work as advertised [1], it is an insufficient argument when it comes to security. What about free open source software?

Intuitively, GPL software users also hold the maker of the software responsible for its security, and there is nothing wrong with that. The user extended a certain level of trust to the maker(s) by using their software and this trust gets hurt when the user gets owned. But the argument of responsibility based on a monetary transaction clearly doesn't hold in this case. There have been cases where free software makers simply refused to fix their code or committed that they don't care. Others just mention that the user is an unthankful beast and should run something else. After all, the software was free and there is no warranty.

We could try and change the definition of the responsible party to "owner of the source code". After all, you can only secure something if you have access to its building blocks, right? This case would mean the same thing for commercial software vendors, since they do own the code to their products and are the only ones who can access it. If we extend the definition to mean "write access to the main source tree", it would also neatly describe the maintainers of free software. Case closed, or may be not?

The problem with this approach is that nobody has complete control over the entire source code, and even the few who apparently have it don't. Software is made out of modules and every piece of software uses a wide range of modules: from kernel and system calls to statically and dynamically linked libraries, other software handling the events sent by the software and of course firmware and microcode on hardware devices. It is highly unrealistic to expect anyone or any organization to have complete control over all the components their software depends on. Big software companies are much like a collection of small companies that happen to work together on a single large project.

Today, the lack of central understanding and control leads to responsibility being resolved by the social or business equivalent of a call graph. If, for example, a vendor of a complex server software faces a security vulnerability in the image parsing and handling code of said server, he identifies the maker of the library in question. The library maker gets contacted and asked to fix the bug. The library maker in turn realizes that the issue is in a piece of lower level library and contacts its respective maker and so forth, until hopefully one of the elements in this chain feels responsible and fixes the issue. Or this is how it should work.

In reality, it is often not so easy to identify whose code is actually responsible for the security issue. In the post buffer overflow era of software vulnerabilities, many application specific issues arise from the inter-workings of components. Let's assume, to stay with the image library example above, the image in question has a width and height information of -3. The low level library computes that there will be -3 * -3 = 9 bytes space required and provides this much. The upper level library copies image data until a counter reaches 4294967293, the unsigned interpretation of -3 on a 32 Bit machine. Both parts of the behavior are not correct, but who is responsible for fixing the issue? What if they do not agree? That becomes an important point when the issue is larger than just a signed vs. unsigned integer.

It is by now a commonly accepted fact that a good design and architecture can prevent a lot of issues before they actually happen. After all, the term architecture comes from a profession that must plan on things to be not perfect. If an essential part of a building does hold the weight anymore and crashes, you can sue the vendor and demand a fix but your building is already in rumble and dust. Or, you can just design the whole thing to not depend completely on a single element. Some software vendors have understood that and start to build their products accordingly.

Design becomes even more important than it already is when it comes to computer systems, simply since this is the only way to handle all the complexity we are facing. Accordingly, it needs to be more precisely executed and validated by additional people with a different viewing angle on the subject. Also, changes dictated by reality, Murphy, management or customers need to get back-ported into the design documents. Companies who already work by those principles produce amazingly good, secure and easy to manage products, not only in the software world.

We already identified that there is no such thing as the software maker but rather a more or less designed and planned way of putting together components. In that respect, writing software is some times just linking already existing components by specifying their inter-working in arcane grammars called programming languages. Therefore, the primary work is in selecting the right components for the task, design their relations and how data is handed over from one to the other and make the whole thing work.

But if we take this description, we can see that selecting an operating system and installing software on top of it is actually the same process. In fact, on a high abstraction level, there is no difference between writing code and designing and implementing an infrastructure solution like a company's email system. You have to know your components. You have to rely on third parties to tell you exactly what their components can and cannot fulfill and how they work. You have to take explicit and implicit requirements into account and finally design a solution. And you have to work on imperfect data, since almost all facts you take into the calculation may turn out to be false, just like the assumption about the security of this image parser.

A good design is based on the definition of your goals, so I gave this insight a try. The defined goal was an actively used Windows XP system that rots as little as possible over time. Most people are forced to freshly install their Windows from time to time, just because they installed and deinstalled a lot of software and everything left a bit of waste lying around or installed additional components that are not removed when the software is already gone. And, much like with security issues, most people believe that, if you actively use a Windows system and install new stuff from time to time, a performance degrading mess is inevitable, since you don't have the source code and you don't know what happens under the hood.

In my design, I therefore decided to not just install software all the time but rather to perform a minimal verification. This is done by installing it into a VMware Windows installation. What I'm looking at is:

1. Does the software what I want it to do? If not, there is no point in installing it in my production environment.
2. How stable is the software and how good does it handle the data I trust it with? If I can't use the data afterwards in other software I'm already using, the candidate fails.
3. What security implications does the software have on first sight? Things included here are open ports, shared memory, highly privileged processes or system services as well as kernel drivers.
4. On a very high level, what modifications does the software do when being installed? I don't check every detail, just if it puts something in auto start registry keys or loads a process to display a tray bar icon.
5. How much is left when the software gets uninstalled?
6. Do I need the software just once or twice? If this is the case, I can keep it in VMware for that purpose and get rid of it afterwards.

It's a very quick check. The result are two Windows XP systems that I run for over a year now, heavily used and permanently modified but still almost as well performing as when I installed them for the first time and not a single fatal failure that caused data loss.

I'm totally aware that this example is slightly off-topic and that it is common practice in every good IT operation, but it is useful to illustrate how important design and architecture are and how little difference there is between composing an application by code plus libraries and composing a set of software to work together by setup.exe and configuration menus. After all, seasoned UNIX programmers tend to be very good system administrators on the same platform.
For both large system architectures as well as for my question whether I install a piece of software on my production system or not, one thing holds true: you have to decide on inperfect information but once you decided, it's pretty hard to get rid of the consequences.

Coming back to the responsibility question, I tried to show that really good security, namely defense in depth, can only be achieved by a good design and architecture. While having a number of other merits, a well reviewed design can be held accountable for security, despite all the imperfections of the components involved. The only entity that can finally be held responsible for the security of something is whoever designed it. Having an all-embracing system design, reviewed by experienced subject matter experts from different fields, yields a very good result and provides as much security was we can achieve today.

[1] I could also imagine getting reimbursed a percentage of the software's price for every crash dump I send to the vendor via the "please inform CorpX about this problem that just wasted half a day worth of your work" message box.

For those who hid themselves successfully so far from the hype, the term fuzzing is, well, not defined. Everyone talks about it, but it means different things to different people. In general, it means throwing semi-valid data against a system to automate security testing. Many of today's issues in input parsers can be uncovered by constantly throwing data at them and watching them crash at some point in time. Supposedly, many people find their 0day bugs that way. The result is that more and more fuzzing tools appear, talks are held on conferences and people start to think that they can secure their products if they just sit long enough in the line-of-fire of a fuzzer before being shipped.

I was recently very surprised how much vulnerabilities one (me) could find by throwing a lot of junk against the target and just waiting for it to crash. I have to admit that I was first following another, more conventional approach of reading the code. What's also important is that my case was extremely well suited for fuzzing attacks, which I admittedly didn't see right away.

In theory, this is called boundary value testing. Boundary testing in contrast to fuzzing is a well-defined term. If a given function F is implemented in a program and the function has two parameters x and y, these two have known or unknown boundaries a < x < b and c < y < d. What boundary testing does is testing the function F with values of x and y close to or equal to a, b, c and d. In the form that should be used for security related testing, it will be called "Robust Worst Case Tests", at least that's what Paul C. Jorgensen calls it in "Software Testing, a craftsman's approach". Here, you would test x = a-1, x = a, x = a+1, x = b-1, x = b, x = b+1 and for each of these y = c-1, y = c, y = c+1, y = d-1, y = d, y=d+1. You can easily see that this is subject to a combinatorial explosion and will take a lot of time.

Another interesting aspect of boundary testing is it's limitations. Boundary testing only works well on independent bounded physical quantities. Mind you, we are talking about independent values representing physical quantities such as offset values in a file format (physical = in the physical file, no matter how physical files are by themselves). If the variable you are testing has no clear boundaries in the sense of a < x, it's almost impossible to use boundary testing effectively. Jorgensen has the nice example testing a calendar function and the February and leap year problems. There is basically no sense of the nature of those test values or any semantic meaning in the current context, which also means if there is another value referring to the one you are testing, it will not be taken into account and might (or most likely will) invalidate your test results.

So how about the practice? It's easy to see that implementing a good boundary testing tool is fairly hard. None of the tools I have seen so far implements fuzzing as Robust Worst Case Tests. It is readily visible why, given the observation from above about the combinatorial explosion, since none of our protocols or file formats work with only two values. Additionally, security testers have in almost no case a clear understanding of what the boundaries for a given parameter x are, or, to stay with the notation used above, a and b for a < x < b are unknown. Therefore, they cannot implement boundary testing without reading the entire target code to understand the boundaries. But that would be a different testing approach and wash all the sexiness of a quick-and-dirty hacker tool from fuzzing, making it a test case engine applied after a source code audit.

So what is it these tools do? So far, the tools I have seen use what could be considered experience-derived boundaries for testing. 32 Bit integer values are for example mostly tested against 0x00000000, 0xFFFFFFFF, 0x7FFFFFFF and 0x80000000, which are the boundaries for the entire unsigned 32 Bit integer and it's signed representation. Other test values are derived from vulnerabilities in the past, like web servers crashing with long header fields. Most fuzzers concentrate on automatically testing "what's usually broken".

This leaves the question why fuzzing, given all the theory above, is such a successful attack strategy in practice. The short answer is: It shows nicely how bad software quality is in general that a small subset of a very limited testing method uncovers so many bugs. The longer answer is more complicated and I don't understand all aspects of it so far. Part of the story is that some applications are better suited for fuzzing, namely the ones who operate on independent sets of data with physical values (surprise!). Examples of these include packet parsers for connection-less protocols without any checksums. If the data in the packet has to be taken at face value from the parser since it doesn't have any other chance, fuzzing can be very successful. Think HTTP and ISAKMP.

What's not clear so far: Does the success or failure of fuzzing against a specific product tell you anything about the security or quality of the product in question? Or does it just show the limitations of the method and the actual fuzzer used? What is clear is that fuzzing by no means is an adequate product testing tool for vendors. Even if I get a good answer on the questions above, just remember February 29.

I vividly remember the fascination of coming up with your own authentication mechanisms. 14 years ago, I was doing the same. This was at the times of MS-DOS, where there wasn't any such facility provided by the operating system. While it still didn't make too much sense to implement something that was called in autoexec.bat to check for credentials (remember the F5 key?), it wasn't mission critical or even commercial software either, so I hope it's forgivable. Interestingly enough, one year later, I was presented with an access control software that was used by the German postal service to secure access to their MS-DOS machines. The person showing it to me said: "You can't break into this, it is commercial grade software." while I rebooted the machine and pressed F5. Well, yes, I could.

When designing identity checks and object access control, the guiding principle should be: PAL. It is a German slang acronym and stands for "Problem anderer Leute", which can roughly be translated to "someone else's problem". In most cases, your underlying operating system or database does require authentication anyway and does offer a fine granularity of access control mechanisms based on the credentials used in the authentication process. Use them!

When moving into a new house or flat, nobody in his right mind would go into a hardware store (as in "Home Depot", not as in "Radio Shack") and buy metal, rasp and small drill bits to make himself a lock for the front door. The person would instinctively know that the expense in time and money exceeds buying a commercial door lock. Additionally, the person would normally realise that a homemade lock will not provide the same level of security.

In many applications, the homemade approach is favoured. By implementing it's own authentication and authorization schemes and using a single almighty user account on the operating system and/or database, the vendor makes his own door lock out of cardboard or wood and embeds the original key to the commercial door lock in the mechanism, so the original door lock still functions. Sounds silly? It is.

Imagine a kiosk type application, such as a cash register or device control system, running on a modern version of Microsoft Windows and compare the following two design approaches. First, we look at what's usually done, namely the kiosk application implements it's own authentication and authorization scheme:

• For the application to run, the Windows system must be configured with auto-logon, which means leaving the password of the operating system account in clear text in the registry.
• Since the application runs always with the same system account, you cannot use any of the access control mechanisms on file systems or the registry.
• The application must try to prevent access to any other functionality a logged on Windows user normally has (i.e. explorer, regedit). This usually fails and even if not, requires a lot of code writing and additional hooking to take place.
• Any additional credentials the kiosk application needs, for example a database account, must be stored somewhere. But since the application always runs in the same account context, the only place to prevent access to the information is by the application itself. It might even use a fancy crypto algorithm such as AES to protect the information, using a hidden static key.
• For any special functionality, such as archiving stored transactions, creating new users or modifying existing ones, the application needs new code and must check access permissions of the currently logged in application account to perform this operation.

It should be fairly obvious by now that, by implementing it's own authentication, the application actually disabled all build-in security provided. And, regardless of your opinion on Microsoft, it's fair to say that they probably spent significantly more time on making sure their authentication and authorization is correct than the developer(s) of the kiosk application.

If, on the other hand, the application would be designed to use the Windows user accounts as means of authentication, the picture is a lot different:

• If the user does not possess a valid Windows account, she cannot even logon and therefore not run the application nor access any data.
• Access to files in the file system and information in the registry can be restricted to read-only or read/write according to the privileges the account has. The access checks are done by the operating system, which requires no additional code and probably performs better. Also, there is no chance for the application developer to forget about a permission check.
• The application can be integrated as the shell for the respective user account. The user might still be able to access other native Windows functionality, but the harm caused is very limited, since access to the critical files is not allowed, regardless of the tools the attacker uses. The attacker would need a privilege escalation vulnerability to get there.
• The additional credentials the application requires can be stored on a per-user bases using integrated protected storage. The CryptProtectData API function can be used to store a user account for the database. Again, the one-to-one mapping allows for the database to use it's own integrated permission checking and not allow an unprivileged user to change any tables she's not supposed to.
• To manage user accounts, the existing tools of the Windows operating system can be used. The same holds true for the database. No additional code is required.
• Introducing containers, such as groups, into such a design is easy, because it's already there. Additionally, if the application suddenly needs to do central authentication, no code must be changed since the functionality essentially would not change by dropping an Active Directory into the picture.

The same approach works on almost every modern operating system, although the almighty root account on UNIX type systems makes it a little more complicated to securely store user related credentials. The approach also works with most web servers and databases. Side effects include less code to develop, higher security due to the use of well-audited code and kissing goodbye authentication bypass vulnerabilities.

Interestingly enough, some commercial web application developers have already learned this lesson and are using frameworks that handle authentication and session tracking for them. Next time you evaluate the design of a to-be-build solution or a to-be-purchased product, ask how authentication and authorization are implemented. If you hear stories about strong encryption and their own user management, at least ask why.

The driving force for looking at the effectiveness of their network security was of course money, namely the money needed to operate and maintain it. When this network was built, a great deal was spent on the security of it, much like many people in the industry and all security appliance vendors recommend. But when looking at the cost years later, they decided to grab available data and see what type and how many actual attacks happened since the deployment of all the firewalls, IDS sensors and whatnot. The result: very little. So they get rid of some.

On the other hand, think about the security devices as targets and not protections for a minute.

Take a fairly simple protocol, such as HTTP. A web server only needs to implement the functionality of this one clear text protocol correctly and will be fine. Now, how many major web servers do you know that didn't have a HTTP parsing vulnerability in the past? But the parsing of a HTTP request is implemented in a fairly small share of the web server's code base, because it also has to deal with things like MIME types, virtualization of path information, finding and opening files and other stuff.

Compare this to an network intrusion detection system (NIDS) sensor. It needs to do the same thing, namely implement protocol parsing correctly, but for 100 some times complex protocols (the number is from ISS's Proventia). Also, the protocol inspection is it's job, so the lion share of binary code executed is actually attackable surface. In the past, people used TAPs (RX-only network connection) and stated that this prevents attackers from breaking into the NIDS sensors, because they would be unable to communicate with it. Did anybody ever evaluate a fully deployed NIDS under the assumption of one sensor running non-interactive attacker code? I think a not too complex multi-stage shellcode (the term is a bit misleading here) should be able to exploit the central NIDS management server from the implicitly trusted NIDS sensor and phone home from there. Stuff like this has been done before.

Today, everyone deploys inline network IPS. eEye showed nicely [1, 2] what it means to deploy something with such a large attack surface in the middle of your communication stream: namely, your IPS gets owned first and used as a very convenient jump point into the management network. I also vividly remember when K2 presented at DefCon 9 on polymorphic shellcode [3] and the ISS sensor, which was supposed to just not notice the exploit, crashed in front of the audience while parsing the modified exploit. And please don't think this is about ISS's products, this is a general issue.

On the organizational side of larger IT operations, the same or related teams usually manage the firewall and NIDS rules. In practice, this means the firewall will have a rule saying "block port 99" and the NIDS will have a rule saying, "alert on internal port 99 traffic". So you have two types of NIDS sensors deployed, the ones on the outside producing several thousand alarms per minute and the ones on the inside reporting traffic violations and P2P leech. Is this worth thousands of bucks per month operational cost?

But the only actual reason many organizations have these devices is for forensics, since nobody is watching them in real time anyway. If I were to be called into a case where systems are owned using 0day, I would much prefer a full packet dump covering the last 30 days instead of a selection of may-be-accurate information the IDS has for the same time period. Running a full packet log with a rotation of 30 days is going to cost you very little. A common 100 GB hard drive will be easily able to handle a day worth of traffic and one can reproduce the entire day completely without a semi-intelligent piece of hot needle software pre-filtering the data.

Again, the very old recommendation to keep things simple and stupid holds ground. If your deployed software does not act on input such as network traffic, the same can be as malformed as it wishes to be. I wonder how many IDS sensors choke on HTTP chunked encoding reassembly when the "protected" web server behind it does not even support this transfer method.

Consider throwing away at least parts of your IDS and reinvesting the saved money and staff time into their education. Even a little talented network security person will be able to use additional knowledge more intelligently than any of your alert filtering software solutions. And, you don't have to script them: they are voice controllable.

Although I very much want to, I currently cannot change the average fault ratio in software that is stable since the 70s: 0,5 - 2 faults per 1000 lines of source code. While I can't change that number, I can reduce my risk by reducing the number of code lines executed for what I want. You can't screw up that much in a "Hello World" program.

Therefore, I refrained from using some of the major blog solutions because they contain a lot of code and I don't run PHP. Instead, I selected a lightweight Python solution: PyBlosxom. Setting it up caused the usual system administration issues, but that was expected and fairly quickly dealt with.

The big surprise came, when I configured the categories plugin. Selecting the / category twice as a test produced a URL that ended on three slashes: /cgi-bin/pyblosxom.cgi///. To my astonishment, the blog presented several files from the system's /var/spool/ and /etc/ directories in my web browser. After double and triple checking my various configurations, verifying that I was running the latest version and diffing the distribution package I used against a fresh copy of the source, I had to conclude I was indeed looking at a 0day bug.

This left me with two options: reverting the entire installation, selecting another piece of software and beginning from the scratch or finding and eliminating the bug. The later seemed the more feasible, since the installation already did what I wanted. The debug log mode of the software did only produce two entries per request in the log file, so it was little help. Therefore, I had to read the code. My Python skills aren't very good yet, which made it an extra challenge.

After introducing a number of new debug logging messages and some more reading and testing, the fault became clear: When parsing the requested resource, the code would check if it begins with a / and remove it:

if path_info.startswith("/"):
    path_info = path_info[1:]

After that, it would use it's configured directory root and combine the two using os.path.join:

absolute_path = os.path.join(config["datadir"], path_info)

Checking the Python documentation, it would read: "Joins one or more path components intelligently. If any component is an absolute path, all previous components are thrown away, and joining continues" Therefore, requesting any absolute system path from the blog software will cause the path to be read and interpreted as entries directory and displayed. Ergo: arbitrary file read.

I fixed the software. Submitting a bug into their bug tracking system required a login and the site also mentioned you should rather post to the mailing list. My post to the same still awaits moderator approval, since I didn't subscribe.

The story highlights several general issues we are facing: First, writing in a programming language that is considered modern, robust and object oriented does not prevent bugs and these bugs may very well be security issues. Second, nobody has several days spare time to source code audit the software just because she wants to setup a simple blog. We need solutions to those problems, and this Lablog will talk about some of the ways we try to get there.

Update For anyone running PyBlosxom, version 1.3.2 was released, fixing the bug.