The Recurity Lablog

During a non public security event, I saw a presentation by Olaf Kolkman about the new DNS server named Unbound. When he mentioned that the whole thing is written in C for performance reasons, I returned that we should simply stop developing production software in languages that produce unmanaged code. We got into quite some discussions with the whole audience after that, many people stating that there isn't an alternative and everything else is just to slow. I'm not buying this for many obvious reasons, but that's for another post.

To put our abilities where my mouth is, we ended up performing a short source code audit for the Unbound developers. After all, Unbound is an effort to produce a reliable, validating and DNSSEC ready name server, something we all want to have deployed on a larger scale. Sergio Alvarez, who by the way will be speaking at CanSecWest this year, looked at the code and found it surprisingly not riddled with remote code execution bugs. I was certainly happy about that find, because it meant the next generation DNS server deployments wouldn't look at a future comparable to ISC bind's past.

That impression, however, was largely because Sergio compiled the code with all the ASSERT statements intact. Now, people running heavy duty production DNS servers will most certainly try to make it as fast as possible, instructing the compiler to get rid of “debug” features like ASSERTs. That might not be a good idea. So here is another lesson learned: When building for production, you might want to keep those ASSERTs compiled in, since your server crashing on funny packets is probably better than to share the administrative control of the machine.

Other than that, I hope the Unbound team keeps up the good work, so people have one less excuse to not move to DNSSEC.

Some of you might have heard that I gave a talk on Cisco IOS security at the 25C3 this year. The talk was unique for me in many ways, starting with the fact that it covered content going all the way back to the beginnings of Phenoelit up to material that was developed within Recurity Labs. It could be said that the talk was a nexus of research efforts in different areas of my life.

The second unique aspect was the cheer amount of stuff to cover, which prevented more in-depth reflections on some of the issues. This begins with the question of who would actually take over Cisco routers. The short answer is of course: whoever can. But that needs to be taken apart in more detail. Let's focus on attacks that directly apply to the device in question and ignore for now that the easiest way to take over an entire network infrastructure is to attack the unpatched Sun servers running in the Network Operation Center.

Consider successful exploits a question of development cost. An exploit is in that respect not different than any other software: you find someone you think can actually pull it off, present your requirements and have them develop it for you. In almost all cases, this isn't going to be for free, so they will give you a price tag for the work, which in most cases is a linear function of the cost of a work hour for them. This implies that the better they are, the less hours they will need to develop the exploit for you, which makes it cheaper. This is what made exploits against Windows desktops so cheap that attackers mostly relied on them for gaining access to networks from what the network owners considered inside (an outdated but still widespread way of looking at it).

Our research concerning Cisco IOS security was always based on the assumption that there are entities in this world that have reached a reasonable development cost level for IOS exploits. But the publicly available knowledge on how to write IOS exploits didn't fit the bill, as they required to jump into some memory address that is specific to the IOS image running on the target. Assumed you don't know what image is running on that machine, you could still argue that the exploit writer included a list of all possible image dependent addresses into the exploit would try them out, one at a time. This would cause the router to reboot every time a wrong guess was tried.

In the presentation, I said that there are about 100.000 different IOS images in use. This is a very debatable number, as about 15.000 are supported by Cisco at any given time and good network administrators will only run one or two IOS images in their entire network, often times investing several months to figure out which exact image they want. When we however fire up the Cisco Feature Navigator and ask it for all IOS images that support IP Routing, which should be all of them, we get: “Showing 1-50 of 280715 results” at the bottom of the page. Wow, what a number. On page 5615 of the result listing (thank god they have a direct jump feature!), we see that this covers everything from IOS 12.4 to 11.2. Therefore, the number doesn't take very old networks into account, in which you do see images below 11.2 running occasionally.

It should be clear that trying them all out is not an option, especially considering reboot times of 30 seconds and more per attempt. Your exploit would constantly reboot a router for 2339 hours or 97 days. And that doesn't take into the account that you need to get and disassemble them all, estimated time with IDA: 5848 days or 16 years.

Therefore, there is either no one exploiting IOS devices or they have found a better way to do it. Our Cisco Incident Response tool was developed in the hopes of finding people attacking IOS devices, successful or not. But then again, it's hard to write a detection for the unknown, so we also had to look into finding ways to get code execution stable. The method presented at the 25C3 (and documented here, feel free to post questions in the discussion section) only reduces the number of things you have to know about your target, it doesn't eradicate the problem in general. Now, we only need to know the ROMMON version and there are a lot less ROMMONs than IOS images out there.

For smaller machines, such as 2600s, updating ROMMON did not seem to be supported and the version depends on the shipping date. However, after closer inspection, here comes an errata: Cisco does offer 6 updated ROMMONs for 2600 routers. For larger machines, e.g. 7206s, there are about 36 different versions known. That's a few magnitudes smaller than 280715. But it is also still far from the ultimate truth, as you still need to know and have that ROMMON as well as knowing a few things of the box, most importantly the hardware series. Some people like to include the hardware series in their router's DNS records or name the PTR records of the IP addresses bound to a router's interface after the interface itself, which allows to guess what type of metal it is.

Knowing the hardware platform is actually more important for the first and second stage shellcode than it is for getting stable code execution, as the same ROMMON seems to be applicable to a number of subtypes of routers, while one subtype may have memory wired into different addresses than the other. But being lucky is also a valid option, which is what happened when we selected the memory area for a direct write: I assumed the memory at 0x80000000 on a 2600 is used for global IOS variable pointers, which is incorrect. So, errata #2: I was made aware that this is of course the exception vectors, after the MMU is turned on. Accordingly, this is a very good place to store two instructions.

There is still a lot to do and research when it comes to Cisco IOS and security. But the stable, image independent code execution at least allows us finally to draw better assumptions about the attacks we should be looking for. It shows nicely that, even with CIR, we should not try to detect the exploitation while it happens, but focus on the shellcode functionality and the footprints it leaves. And the IP options vulnerability is a perfect example why critical infrastructure should always dump the core files onto its own FLASH device, as dumping core over FTP doesn't really work to well when your “IP Input” process just got popped.

This year's Chaos Communication Congress, better known as 25C3, was an exceptional event in many ways. It begins with a program committee that attracted so many interesting people over the last years that they had ample material to select from, and they did a very good job of that too. Accordingly, the quality and spectrum of the presentations was significantly above many other conferences and we all need to thank the people that put up the program. And while I'm still not done seeing all the video recordings of all presentations, there have been quite a number of highlights.

The hard working organizers and Engel of the CCC apparently are by now so well trained in running a Congress that it almost appeared as stress-less routine to the casual observer. I've never been to a Congress with less shouting and less chaos in terms of organization, and despite the event's name, I think that's a good sign. They even somehow managed to handle the insane amount of people showing up, which, as DEFCON attendees will surely know, is quite a challenge by itself.

And then of course there were the presentations, above all Alexander Sotirov and Jacob Applebaum with their successful creation of a rogue SSL CA certificate. The work shows how the combination of academia research with the practical experience and dedication of world class security professionals can achieve something that was considered a theoretical attack. It also shows how much of a pipe dream the perceived security of browser based communication over SSL/TLS actually is. If all but one trusted CAs belong to the same publicly traded commercial entity, they don't actually need to fulfill their security promise anymore, because they have a monopoly.

The purpose of a publicly traded corporation is to maximize the profit for the share holders*. And if selling certificate signatures generates enough revenue to get your stocks rated as "buy", you did your job. If you need to revoke a large part of these certificates, because you failed to react to previously published research on vulnerabilities in them (MD5), this is similar to a call back of your product and would therefore hurt your reputation on the stock markets. If you however just ignore the problem as long as you can and then trust that very few people will actually understand the problem so it doesn't impact your sales, you can even offer remedy at no additional cost and look good in the press. From a business point of view, that is a remarkable containment stunt. From a security point of view, it's devastating. Not only does it show that revocation simply does not work, but also that the one entity that must be extremely strict with revocation actually doesn't follow it at all.

Interestingly enough, this proves two points made by Dan Kaminsky. The first is about how much of a defense SSL actually is in the light of vulnerabilities like his DNS issue from summer 2008. Dan said in his presentation at BlackHat that SSL proved to be much less of a defense than we all thought it would be. The second point is actually less obvious: The much debated partial disclosure approach Dan followed had a very interesting positive side to it that nobody saw before. The big difference between Alex's and Jake's big-bang presentation and Dan's long process of informing selected people gradually over time is the learning effect they had on all the other people. I think after that summer, we security professionals will not hear that old argument of a vulnerability not being critical because the attacker would need to control DNS in order to exploit it very often anymore. On the other hand, I don't see anyone reviewing their security perception of the trust model that so-called secure web sites are build on. Everyone is just happy that the issue got "fixed" so quickly. I for one have not realized that aspect of making a big fuzz about something enhancing its long term educational value before, and I certainly thank Dan for teaching that lesson to me.

Speaking of thanks, my fellow Phenoelit members, above all Mumpi, need to be thanked too for the awesome party they put on. That also includes DJ Vela and CMOS for playing at that event and in CMOS' case for flying all the way into Berlin to do so. And last but not least, I would like to thank the audience of the 25C3, which again was one of the smartest I had the privilege to speak in front of. I apologize for the suboptimal delivery of the Cisco IOS presentation to everyone who saw it, if you found a stray "Erm" in what I said, you may keep it.

* You could argue that this is the case with any corporation, but working at Recurity Labs, I can tell you it isn't.

Unfortunately, a number of things recently broke, including the main (and only) harddrive of the machine running phenoelit.net and therefore this blog. We are recovering all the services but, as you might have noticed, switched blog engines so we no longer actually run any (potentially vulnerable) code when you hit this site.

Now we just need to port the old entries over in a consistent and permalink friendly way, which might require a few more days.

Thanks for not unsubscribing :)

Update: Things should be back to normal and the two delayed posts should be posted. Please contact me with any complains if you find something missing or broken. Thanks.

... I feel compelled to point at Derek Soeder's VMware Emulation Flaw x64 Guest Privilege Escalation (1/2) post at full disclosure. It seems that it's always the same names that come up with those beautiful exploits. Delighting.

I recently discovered that there seems to be this class of discoveries that is made independently by many people from the same community in roughly the same timeframe. What's interesting about this class of discoveries is that often they are not published. Everyone seems to assume (correctly) that others have discovered the same. One topic that certainly falls in this category is “return oriented programming”, which everyone I know invented on their own once they actually needed it. Finally, at BlackHat USA this year, academia in the names of Erik Buchanan, Shawn Moyer, Ryan Roemer and Stefan Savage took this concept and turned it into solid research. Conceptually no news, it's highly recommended reading for anyone who used this method before, as approach and execution are brilliant.

The second instance of that class this year is TCP connections that don't use the operating system's sockets. If you ever wrote networking attack tools using raw sockets or drivers, you have come across the idea and probably tried it out. The basic idea is that you don't need to keep any state as the TCP client. You can send as many SYN packets as you want and can reply with the final ACK once the SYN/ACK packet comes back. All the information you need is in the answer from the server. You can take this method one packet further and actually send the first payload packet as well, e.g. an HTTP GET request. Check out Dennis' blog post if you need a picture of this.

TCP server software is in most cases written around the accept() call, which blocks until the OS has received the final handshake packet. Once that happens, the accept() call returns and the server software starts handling the connection. If you had sent the first payload packet as well, the server software even goes into the state of processing a request. Many servers are not built to handle the case where they no longer have any communication partner (i.e. they don't time out quickly), and the rate in which they get new ones usually overwhelms the server as well as the OS. On top of it, TCP, being a reliable protocol, tries to keep the connection alive by resending the answers because it assumes lost packets. The attacker can make this worse by playing with some properties of TCP, e.g. the window size, or add other twists to it, like ignoring RST packets or sending out-of-band packets with the PSH flag. Bottom line is, it really depends on the kernel and the server software what happens after the handshake is completed, but the issue is the same all over the place. Side note: things that don't implement a proper common TCP/IP stack fail more miserably, ask people at Cisco.

After I played with this in 2002, I became aware of many earlier implementations, e.g. this one, that one and many others. I recently remembered all that when I met Jack and Robert at the Sec-T conference in Stockholm (side note: great conference, watch for their 2009 edition). We compared notes on what things reacted how when exposed to stateless TCP clients hitting them hard. While it is relatively easy to protect server implementations by limiting the number of concurrent requests served, and operating systems by limiting kernel buffers and number of connections from a single source, so-called security devices and code are entirely different stories. Back in the days, my own Linux kernel gave me problems as ip_contrack, part of the NAT code, kept killing my TCP/IP stack as it tried to keep track of the connections I created, despite the fact that neither firewall nor NAT were actually enabled. I had to remove it from my own kernel before I could perform the attack against other devices.

Today's networks are full of devices whose vendors think it's a good idea to track established TCP/IP connections. IDS and IPS systems are just the tip of the iceberg. Just about every firewall, NAT device, TCP load balancer, transparent proxies and even some network sniffers do it. That's certainly not smart. You should try to avoid tracking state of others you don't control, especially if you don't actually need to. But the ignorance of basic principles and prior work is unfortunately not that uncommon.

It should however be clear that the risk of falling victim to this attack is inherent to offering a service. When you get a phone and publish the phone number, anyone can call you. It is not possible to prevent strangers from calling you every five minutes in the middle of the night when you want to be reachable to people that you have not authorized before. Ask any call center or ask people who run a web server that got slashdoted. If you want others to be able to contact you, they can overwhelm you with contacts. If you open your shop's doors, people can come in and buy stuff, but you could also have a flash mop filling your space. You could post bouncers at your door, but that's neither friendly nor helpful.

It should also be noted that this attack is not stealthy at all. No matter if you use your regular operating system's sockets or a raw and stateless TCP/IP implementation to establish a connection; you cannot spoof your IP address unless you sit on the transit route of the legitimate owner (in which case you can do a lot more). I think the scenario of the low-bandwidth customer at home taking down important servers using full handshakes is less likely. In most cases, his DSL home router would die first. Almost all other scenarios have the same effect to the victim as if he is attacked by a regular DDoS attack today: It's always possible but there are mitigations. The easiest mitigation for stateless TCP clients is to limit the amount of connections from a single source address and get rid of useless connection tracking devices in between.

And if the attacker is located in Germany, he will end up in jail. The much contested change to §303b of the StGB, Germany's criminal law, says: Interference with a data processing system that is of importance to someone else, by committing a crime described in §303a, by entering or sending data with the intention to cause a disadvantage to the system or [...], will be punished with jail up to 3 years or a monetary fine. The jail time can be up to 5 years if the affected system belongs to a company or the government. Therefore, completing too many TCP handshakes is illegal in Germany.

I'm looking forward to Jack and Robert publishing their full findings, as I'm sure it will provide much entertainment to the security community.

Perception is an interesting thing. Since everyone apparently has their own, it is fairly hard to arrive at a common denominator. In today's world, media is the perception softening instance that decided what people see and what not. Using the media to reach a large amount of people is intentionally shaping their perception. If your goal is to make people do something specific, this is a highly effective approach. That's what happened with DNS. Every computer security blog on the planet posted statements about Kaminsky, Halvar and the Domain Name Resolution Protocol, some even unintentionally. The global perception is: this is extremely important. People talk about it, people patch their servers with a workaround and people think about the Internet's safety. Dan has accomplished his mission.

The Kaminsky DNS attack is definitively regarded as the most important vulnerability this year. This, I find highly interesting , as we have seen two other gigantic security failures already in 2008. Debian's NRNG (non-random number generator) is most certainly one of them. But honestly, raise your hands if you have even noticed SNMPv3. Anyone? I give you the quickest of run-downs:

1. SNMPv3 uses HMACs over secret keys for authentication.
2. The packet can carry a shortened HMAC for [fill in silly performance statement here] reasons.
3. Most implementations implement their HMAC match check as:
   memcmp( myHMACbuffer, packetHMACbuffer, packetHMAClength )
4. If "packetHMAClength" == 1, brute force requires 256 UDP packets.

What can an attacker do with this? SNMPv3 is used to manage routers - the routers that forward all your traffic around the world, including your DNS queries. Managing a router means being able to configure it; a.k.a. super user access. Attackers who can configure a router in your path can redirect everything, without you knowing, not just traffic that relies on name resolution.

We have been working with a customer on a security issue scoring system, to help level perceptions. We started off with CVSS, which deserves its own post some time. Let's just say we didn't stay with it very long. When you compare the three big vulnerabilities this year, here are the CVSS scores according to the National Vulnerability Database at NIST:

• Kaminsky's DNS: 9.4 (AV:N/AC:L/Au:N/C:N/I:C/A:C)
• SNMPv3: 6.8 (AV:N/AC:M/Au:N/C:P/I:P/A:P) (see below)
• Debian NRNG: 7.8  (High) (AV:N/AC:L/Au:N/C:C/I:N/A:N)

Interesting to note. So being able to own the entire infrastructure is less important than breaking the SSL certificates of banks, being less important than poisoning DNS. Is that the case, or just the perception?

What I am looking forward to is the hard factual data we will see in the penetration tests and incidents to come over the course of the next year (assumed there is not another disaster). It will tell us, what systems actually got patched to which extend. I don't expect to find many vulnerable Debian keys, I do expect to find many routers ownable by SNMPv3 and I have no idea about the DNS thing yet.

On a final side note, it was wonderful to sit down with Dan on the day he went public, drink German beer in Seattle and discuss this very topic of vulnerability perception (and by the way, he didn't tell me any details, not even after the beers). Dan tried something unprecedented with the way he handled this, a very brave thing to do. May history be a fair judge, the media won't be it.

UPDATE: The people over at NVD/NIST seem to have noticed that owning routers is potentially more dangerous than they initially thought. Accordingly, SNMPv3 got upgraded to a CVSS v2 score of 10.0 (High) (AV:N/AC:L/Au:N/C:C/I:C/A:C). Well done.

A presentation given by Sebastian 'topo' Muniz from CORE Technologies, given at EUSecWest in London and at PH-Neutral 0x7d8, currently receives some considerable attention from the media. The presentation is titled "Killing the myth of Cisco IOS rootkits: DIK (Da Ios rootKit)". I had the pleasure and privilege to read his accompanying paper and can only hope you had a chance to attend the talk if you happened to be at one of these conferences.

It is however surprising how the media covers this presentation. IOS images, modified on the binary level to allow unprivileged access, are neither new nor extremely hard to do. When I presented on BlackHat Federal 2003 in Washington DC and mentioned backdoored IOS images in the presentation (Slide 12), nobody in the quite Cisco-aware audience showed any indication of surprise. Some time before that presentation, a random hacker had sent me an email with a few bash scripts which would correct any checksum in an IOS image that was patched, just for my entertainment, and mentioned that the scripts took him about half an hour to complete. What topo excels in is the level of platform independence and automatism his rootkit tools achieve. After all, he works in a team of experts on that topic.

The early backdoors in IOS images are a natural consequence of the software being fairly expensive and priced by features. In the same way that people receive a Windows XP Home version with their new PC, but want to run Windows XP Professional or even Vista, people receive an IOS IPonly image but want to support SSH logins instead of Telnet(1) on their routers. Accordingly, people would trade IOS images on the Internet instead of spending money on legitimate copies, especially since they also have to spend the same money on memory extensions for their routers to support the image. With images traded openly, of course attackers modify the image to allow login with a special password. Traded IOS images are an invite to them, as they essentially say: "Whoever downloads this either uses it in the lab or will install it on critical routing infrastructure in their corporation." There is just no other use for an IOS image.

The fact that IOS images are nothing but one single executable without inner security boundaries makes the modification just more powerful. When backdooring a Juniper router in the same way, the attacker has to modify a number of binaries, as Juniper's operating system is based on FreeBSD and therefore has real processes implementing different functionality (login, routing, etc.). It also has a kernel. Now, on IOS, everything is in the same process / address space / memory privilege zone, so a single patch can reach integrity checking code as easily as it can reach login validation code. Oh the other hand, what's the challenge of backdooring Juniper routers? Any adjusted FreeBSD rootkit will do.

What is left is getting the privileged access to the router in the first place. Best practice security in network design, architecture and configuration should prevent that, but there are people who don't follow those and there are vulnerabilities. As noted in other occasions, security functionality seems to be an excellent place to find bugs in. Even SSH, audited to death on other systems, appears to still be a very interesting target on IOS. In this respect, I can understand some of the service provider folks being unhappy about the presentation, as the availability of an IOS rootkit will most certainly rather fuel the interest in writing code execution exploits against IOS, rather than damp it. This is probably one of the reasons the rootkid maker code is not going to be released. Additionally, the heavily increased protections on common operating systems platforms (e.g. Windows Vista), will draw more attention to 1980s style, C written, complex monolithic systems running on critical Internet infrastructure.

All this isn't exactly news. I could have named this entry "I told you so", but that wouldn't really cut it anyway. What rather happened was the ongoing concern of a few people about this whole subject, several years before it was actually presented. When working on the first (and luckily completely replaced) version of the GDB debugging agent for Zynamics BinNavi that allowed runtime debugging of Cisco IOS in a decent reverse engineering tool, I realized that the GDB debugging capabilities would also finally allow the development of an independent forensics tool for Cisco IOS. I discussed this with Gadi Evron, one of the few people that foresaw the issue as well, during the Chaos Communication Congress in 2006.

This was the beginning of the CIR project, which stands for Cisco Information Retrieval or Cisco Incident Response, whatever you like more. CIR is a framework for analyzing IOS memory core dumps, the only solid and complete evidence that can be collected remotely from a router. While the GDB link provides a better (read: less easy to circumvent) interface, network operators do not appreciate driving to every router they manage and using it remotely would be pretty slow. CIR uses the IOS image as a blueprint for the memory map, as the core dumps do not contain any memory mapping information, and also compares static elements from the IOS image to the actual running code in the core dump. A paper and presentation on the technical background of CIR can be found on the Recurity Labs website. One of the simplest features implemented is the comparison of the .text segment between the image and the core dump.

When topo announced the IOS rootkit talk, I was excited that he was willing to provide a backdoored IOS image to me for testing CIR. This is responsible disclosure at its finest hour, as the patched image means little to Cisco Systems but very much to us for testing CIR. After all, detecting your own modifications to an IOS image is kind of silly. From the beginning on, CIR was meant to be provided as a free service to anyone who suspected that their router might have been attacked, backdoored or suffered from a successful or unsuccessful Denial of Service attack using vulnerabilities in IOS. Accordingly, processing was made available online when we barely had the alpha version working and was announced at BlackHat Federal in Washington DC. Since then, we made significant progress on the functional front, but most work went into code quality, dealing with the broken-beyond-recognition ELF files produced by Cisco's build process and making the framework ready to support bigger iron platforms.

Finally, but still before his talk at EUSecWest, topo provided the backdoored image. We fed it into CIR and successfully detected the .text segment modification. However, it only told us that the image was patched and where the patch begun. A quick change of the respective plugin then produced a detailed report that pointed to all modifications in the .text segment. The resulting report can be viewed on CIR Online for those interested in the details. Getting the exact virtual addresses out of the report makes reading the code differences in IDA a piece of cake: you jump to the respective address in the core file and press C. Out comes topo's backdoor, which is a beauty to read in PPC assembly. Felicidades topo!

So, while still in BETA phase, CIR Online seems to actually work. Of course, it also makes the shortcomings of the core dump method obvious: changing the core dumping function in Cisco IOS together with any other backdoor inserted would ruin the collection of evidence and therefore give nothing to process to CIR. This, as mentioned above, could be overcome by physically connecting to the serial console of the router and use the GDB link to obtain the evidence, which again could be prevented by patching the GDB stub in IOS. You end up with the classical race between attacker and forensics person. However, it is a race worth racing if you realize that the bad guys already left the starting line and the dust cloud at the horizon is them speeding away.

Now that some people actually talk about IOS rootkits, interesting tidbits show up. One person asked me if we have tested CIR with the Russian IOS rootkit that was for sale a few years ago. No, we didn't, but good to know that these exist. Additionally, Cisco finally published an updated and much more detailed response, giving instructions on how to detect and/or prevent rootkits with the on-board tools IOS provides. However, some of the instructions are rather humorous, such as transferring images to the router using a secure protocol when your image does not support SSH. On the other hand, they also published a list of MD5 sums of all 12.x images, which is shortly going to be a test CIR performs.

So finally, we seem to have overcome the discussion on if IOS rootkits work or not and approach the age where it's an accepted fact. We also know that, so far, binary and TCL backdoors are detected by CIR. That means we can concentrate on the crash-cause analysis and exploit attempt identification features again. Stay tuned.

Found in an ISO / IEEE 11073 Personal Health Data Work Group meeting presentation: "Managers ignore information they do not understand.", referring to one side of the communication interface the group works on. While arcane terminology and acronyms appear awkward at first, they do prevent embarrassing misunderstandings.

Dirk Breiden and FX of Recurity Labs went to an IT security trade show in Milano, Italy, following an invitation of fellow hackers Stefano Zanero, Igor Falcomata, Raoul Chiesa and other members of Sikurezza.org. We gave a talk on the current state of independent research into the security of RIM's BlackBerry solution.

Our Italian friends were exceptionally nice and forthcoming, making sure we had everything and were well entertained all the time. Many thanks go to the organizers of our daily and evening events.

One thing that struck me as strange was the security trade show itself. The exhibitors came almost exclusively from the usual suspect section of security software and appliance vendors and distributors. Many displayed embedded boxes of various sizes with little or no LCD displays that somehow made something secure. As far as we could tell, none of them sent any technical personal to the event and the attendees didn't seem to mind at all.

We talked to one particular vendor's booth personal since we happen to use one of their products and happened to stumble across some 0day vulnerabilities in it. The person did not know what a vulnerability is and, once we started to explain that their embedded product runs on Linux, insisted that we must be wrong, since it only supports Windows and Apple. Oh well. While I'm totally aware of the fact that a trade show booth is not the recommended vulnerability reporting channel, I did actually expect the company representative to know a certain minimum about their product.

Afterwards, it crossed my mind that at every trade show, may it be cars, construction equipment, tools, boats and even food, the exhibitors get out of their way to show the inner workings of their product, like engines, safety mechanisms and any other aspect that highlights the quality and uniqueness. At the security product show, nobody seemed to consider opening their magic appliances to even show the PCB and the hardware within; leave alone explained the inner workings in any considerable detail. And even then, people seemed to like the stuff, as far as we could tell. Very interesting.