Leo Breiman's paper on Random Forests

I was reading the Apache Mahout wiki and got to the Breiman Example page and see that the link to Leo Breiman's paper (Random Forests) is broken. Here is the correct link to the paper. You can download the PDF as well.

 

The subject of the document is document classification.

-- Jack Krupansky

Moving on from Hadoop to Mahout

I've finished reading the Apache Hadoop tuturial (from Yahoo). I didn't do any of the exercises, but at least I have more than a passing familiarity with what Hadoop is all about and how it is well-positioned to cope with Big Data.

 

Now, I'm moving on to reading up on Apache Mahout. Mahout's goal is  to build scalable machine learning libraries for recommendation mining, clustering, classification, frequent itemset mining, and similar purposes. There is actually a book on Mahout (Mahout in Action), but for now I'll focus on "mining" the Mahout wiki, which seems to have a lot of useful info which is likely sufficient for my immediate needs.

 

Mahout is implemented on top of Hadoop.

 

In the back of my head I'm thinking about entity extraction or named-entity extraction or named-entity recognition or NER as it is called. In theory, Mahout greatly facilitates NER.

-- Jack Krupansky

Looking at Hadoop and Mahout

Since I finished my most recent contract work assignment on Friday I'm going to spend some time reading up on Hadoop and Mahout.

 

Hadoop:

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.

Mahout:

The Apache Mahout machine learning library's goal is to build scalable machine learning libraries.

 

Currently Mahout supports mainly four use cases: Recommendation mining takes users' behavior and from that tries to find items users might like. Clustering takes e.g. text documents and groups them into groups of topically related documents. Classification learns from exisiting categorized documents what documents of a specific category look like and is able to assign unlabelled documents to the (hopefully) correct category. Frequent itemset mining takes a set of item groups (terms in a query session, shopping cart content) and identifies, which individual items usually appear together.

In short, Big Data and cluster-based computing.

 

Today I'm reading through the Hadoop Tutorial.

-- Jack Krupansky

Helloo

Helloo

The big problem with storage

As I continue to ponder the question of how to make real progress with software agent technology and a knowledge web, the big problem I keep coming back to is what I will call "The Big Problem with Storage", namely, how to achieve a degree of persistence in the digital networking domain comparable in robustness and reliability to storage in the physical world, and then to go a leap beyond that to achieve truly robust and reliable digital storage. Ultimately this includes communications reliability as well, but we can tolerate a little connectivity flakiness, but storage flakiness is not so tolerable since it generally cannot be recovered. What is needed is a fully redundant and diversified network storage scheme that is 100% robust and reliable so that people can have complete confidence that information and media stored on a digital network is even safer than the best storage in the real world.

 

I have written a proposal for a Vision for a Distributed Virtual Personal Data Storage, but it certainly doesn't appear as if even my limited proposal will happen any time soon.

 

I would add to this requirement that we are in desperate need of connectivity options that are far more reliable than the best offered today. Wired connectivity is probably the most reliable connectivity we have, but has diversity problems. Wireless has greater potential for diversity, but has coverage issues. The sad fact is that if you truly want "always-on" connectivity to your data you need to maintain a local copy on your local computer/network.

 

-- Jack Krupansky

Human Surrogate Travel

Surrogate travel is the concept of using a remotely-controlled robot to simulate travel and sensual experiences in a remote location. The user can move the robot around and listen and see what is around the robot. But this may be a significant logistical challenge given today's robotic and communications technology. So, why not use an actual human in place of the robot? The human robot would have one of more video cameras and microphones to provide sensual experiences to the user as well as a headset and microphone for communications with the user, so the user could audibly direct the human robot to move in a semi-mechanical or intelligent manner and the human robot could give the human user feedback as well.

 

This intermediate form of surrogate travel would be much more technologically feasible at the present time and in some cases maybe even more economical as well as more flexible. It might also be more socially acceptable than a free-roving robot.

-- Jack Krupansky

COBOL strikes again

I was watching "24" online and in one of the final scenes they are showing a video feed from a drone and there was some text scrolling on both sides of the video screen. I replayed the scene and looking carefully and recognized the text... it was COBOL source code! Really! "DATA DIVISION", "FILE SECTION", "WORKING-STORAGE", "PIC", "UNIT-QTY", etc. What a hoot.

I have been enjoying this season of "24" since it is supposedly set in NYC, but other than a very few aerial and stock street shots, the actual action scenes are not NYC at all. Even the UN shots are faked. Oh yeah... I forgot... "24" is put out by FOX. But COBOL? Who would have thought of that for an aerial drone aircraft?

-- Jack Krupansky