Innovation - What does it really mean to innovate software and how do we do it?

I read a great book last week that really peaked my interest in examining how we think about and  perceive innovation, and how we might actually bring innovation to software products.

The book is called "Hooked - How to Build Habit-Forming Products" by Nir Eyal:

One of the most intriguing parts of the book for me is Eyal's discussion of innovation and what innovation really looks like in the world of software development.  The author references a book that heavily influenced his own view of innovation, "Something Really New: Three Simple Steps to Creating Truly Innovative Products" by Denis J. Hauptly:

The author deconstructs the innovation process into three steps:

1.  Understand the reason people use a product or service
2.  Lay out the steps the customer must take to get the job done
3.  Remove steps until you reach the simplest possible process

Eyal  surmises that "any technology or product that significantly reduces the steps to complete a task will enjoy high adoption rates by the people it assists."

He provides a great example of this innovation process, describing the dramatic rise in blogging in the 1990s:  

"Before this era, amateur writers had to purchase their own domain, fiddle with DNS settings, find a web host, and set up a content management system to present their writing.  Suddenly, new companies such as Blogger eliminated most of these steps by allowing users to simply register an account and start posting."

This blog, The Feeder Whispere, itself is an example of this innovation. It would not exist if it were not for Blogger!

After reading this section of the book, my thoughts immediately turned to IBM Design Thinking, and it's advocacy for the utilization of "As-Is Scenario Modeling".  In scenario modeling, designers examine the steps it currently takes to complete a certain task or objective and captures what users are

doing, thinking and feeling during each step:

Opportunities for innovation then practically jump out of the model when the "doing and thinking" items are examined.  The "feeling" section helps with prioritization, as the more identify steps with  frustration or anger, the sooner we should work to eliminate those steps!

 

I think the best part of approaching innovation this way is that it is practical and logical, which fits in nicely with IBM's culture.  Software developers, and engineers are typically logical to a fault, and have trouble "dreaming" up innovative ideas.  In fact, I would advocate that "dreaming" of innovations is more often than not, a fruitless activity.  We should instead spend our time examining existing processes at both a micro and macro level, and look for opportunities to remove steps!

One example of an opportunity for innovation in TM1 can be seen in the process of writing rules and feeders.  At a very high level, the process a TM1 user goes through for rule writing could be described as:

The entire TM1 community would rejoice if we were to simply remove one step from this process:

Imagine a TM1 database without the need to write feeder statements!  I think if we were to do this, at our next TM1 conference we would have an hour long session that simply announces this innovation in the first minute, which would immediately be followed by tears of joy, cries of laughter, and uncontrollable dancing among attendees for the remaining 59 minutes of the session.

An opportunity to innovate TM1 can be seen when examining the existing process for dimension editing.  The current process can be described in 10 simple steps:

All of these are important, but quite a few are redundant, in particular the opening of different editors and the saving their contents.  What would it take to remove or reduce these steps?  A clear opportunity presents itself, in this example, the opportunity to combine these actions in a single editor and reduce the steps required from 10 to 6:

 

You may be thinking that this is not awe-inspiring, awesome, exciting, or even impressive.  I'd say you're correct, its none of those things.  It is instead, quite simply, innovative.

Obfuscate - How to scramble the data in your TM1 database, prior to sharing.

Ob-foo-what?

Obfuscation, according to Google:

I often come across customers who are mortified at the thought of sharing their TM1 databases with us.  Of course we have confidentiality agreements in place, but for some reason they provide very little comfort to customers.  The thing that often makes me chuckle is that customers make their TM1 databases so ridiculously complicated that without a cryptographic key no one can make sense of anything in them.  But that's the subject best covered in another post.

So what can we do to entice customers to share their databases with us for development and testing purposes?  What if we could scramble the data values in each cube to render them obscure, unclear, or unintelligible?  What if we could obfuscate their data?

I bet you're thinking, "Yes!  That would be great!"  Well as it turns out, we can do exactly that, and with minimal effort.

TM1 includes a little known Turbo Integrator function called DebugUtility().  This function is officially reserved for the big, bad, TM1 Server developers; the guys who actually write the code that drives the TM1 database itself.

This function works just like any other Turbo Integrator function.  It requires 6 parameters, and can be called in the Prolog, Metadata, Data or Epilog tabs of a TI process.

DebugUtility( nCode, nVar1, nVar2, sVar1, sVar2, sVar3 );

nCode - a number that identifies which debug utility function should be called.  For the obfuscation process, the magic number is 114

nVar1 - 0

nVar2 - 0

sVar1 - the name of the cube you wish to obfuscate

sVar2 - the directory you wish the scrambled cube to be written to

sVar3 - unused in this case, so must be blank ( ' ' )

Example: 

DebugUtility ( 114 , 0 , 0 , 'myCube' , 'C:\data\obfuscate' , '' ) ;

You might be thinking, "Wow, this is great!" And while that is true, we have to do a little more work to apply this to an entire set of cubes, or an entire database.  I'll walk you through writing your own Turbo Integrator process to obfuscate an entire TM1 database.

The first step is to ensure "Display Control Objects" is turned on in Server Explorer:

Then create a new TI process:

Select a dimension subset as your data source type:

 

The select the All subset of the }Cubes dimension:

 The process's datasource preview pane should now reveal a list of the first ten cubes in your database:

Next, move on to the variables tab of the TI process and change the name of the system generated variable to: vCubeName:



 Now move to the Advanced / Parameters tab and create a new parameter, pDestinationFolder:

Finally, move to the Advanced / Data tab and add the following code:

firstChar = SUBST( vCubeName, 1, 1) ;

  IF ( firstChar @<> '}' );

     DebugUtility( 114, 0, 0, vCubeName, pDestinationFolder, '') ;

  ENDIF;


The variable and the if statement in the code above simply serves to skip control cubes when obfuscating data.  The result of the this process will be a set of new cubes created in the directory you specified:

You then have a number of options for sharing your database structure without sharing your exact data!  The easiest thing to do is to create a fully copy of your database directory, then copy these new cubes over their counterparts in they copied data directory.  You should be left with a fully functional TM1 database with obfuscated data.

Speed, Violence, and Conflict - Critical Components of Movies and MOLAP (TM1)

Last week I was fortunate to participate in "Street Talks", a conference put on by IBM business partner Market Street Solutions. The conference leveraged a TED Talk style agenda, with presenters asked to give 25 minutes "talks" in one of the 5 areas of Business Analytics:

Big Data, Business Intelligence, Performance Management, Predictive Analytics, and Risk Analytics.  

Each category featured a speaker from a university professor, an IBM expert, and either a customer or Market Street employee.  The conference was kicked off with a great presentation by keynote speaker, Tom Davenport, who discussed trends in business analytics.

I served as the IBM expert on performance management, and attempted to draw parallels between the speed, violence, and conflict found in modern day movies, and the same three components found in TM1 database environments.

Speed

 Violence

Conflict 

Three critical components of modern day, blockbuster films.  These three components work together every day to entertain audiences in movie theatres all across our planet.  Somewhat shockingly, something similar can be said of TM1:  speed, violence and conflict work together to capture our attention, and entertain and empower TM1 audiences all across the globe.

When I hear the word “speed”, in the context of movies, my initial thoughts turn a movie of the same name, starring Keanu Reeves and Sandra Bullock.   

For those of you that were living under a rock in the mid-1990s and haven’t seen this movie, let me provide you with a brief plot summary. “A terrorist's elevator bombing plan backfires, so he rigs a bomb to a LA city bus. The stipulation is: once armed, the bus must stay above 50 mph to keep from exploding. If a police officer tries to unload any passengers off the bus, the terrorist will detonate it. “ (IMDB plot summary)

On its opening weekend in June of 1994, “Speed” was the number one movie in theaters worldwide, pulling in over $14,000,000 at the box office.  Since then, the movie has generated over $350,000,000 in worldwide box office revenue (boxofficemojo.com)

 “Speed” also received acclaim and accolades outside of the box office, being nominated for 3 Academy awards, and actually winning two Oscars!  

 And 3 years after Speed’s opening weekend, the movie achieved the pinnacle of Hollywood’s success.

A poorly made sequel, without its primary star (Keanu Reeves), that grossly exceeds its budget, and dramatically under-performs at the box office:  Speed 2 : Cruise Control .  Because boats can explode too!

You likely noticed plenty of violence in the “Speed” movies…apparent in the explosions depicted on the movie posters.  As you’re undoubtedly aware, violence in movies, television and video games has been a hot topic in our society in recent years.

Since 1950, violence in films has more than doubled, the study authors concluded. Perhaps more surprising is that gun violence in PG-13 films has tripled since 1985, even exceeding the amount found in R-rated films in more recent years.

http://pediatrics.aappublications.org/content/early/2013/11/06/peds.2013-1600.abstract

There’s a scene in Hot Shots Part Deux, a spoof type comedy film, in which Charlie Sheen’s character is mowing down  “enemies” with a heavy machine gun while a counter keeps track of his carnage.

The “counter” is keeping track of the “body count” and informs the audience when he’s passed the number of kills in other films like Rambo, Robocop, and Total Recall, until they reach their goal of over 300 kills, and claim to be the “bloodiest film ever”.

Sadly, this warning, delivered back in 1993, had little effect on Hollywood, as movies continued to deliver ever increasing violence:

In 2006, the movie 300, was released.

It should have been called 600, to match the movie’s death toll. (www.moviebodycounts.com)

Yet even the movie 300 couldn’t out-do the violence in Lord of the Rings, The Return of the King.

This film included a whopping 836 deaths, though most of those deaths were probably Orcs, which I'm not certain should count.

In the immortal words of Russel Crowe’s character, Maximus , from the movie Gladiator, 

“Are you not entertained?”   

Clearly we are.  The Lord of the Rings raked in over 1.1 BILLION dollars at the box office.  This works out to be about 1.3 MILLION dollars per death!!

 

Thankfully, for the intellectually elite (and by that I mean the women of our society) there a third component to  blockbuster movies. One which allows us to salvage a little bit of our societal dignity - conflict.   

One could argue that speed and violence are forms of conflict, and I would agree that they are.  However, in my reference to conflict I’m specifically targeting the resolution of conflict, whether its internal, like the thoughts a character may be struggling with, or external, like making sure you’re on the winning side of a war.  This conflict, and its resolution is also often referred to as drama.

In Saving Mr. Banks, Walt Disney (played byTom Hanks) embarks on what would become a 20-year quest to obtain the movie rights to "Mary Poppins” in order to fulfill a promise he made to his daughters. The author, P.L. Travers (played by Emma Thompson), proves to be an uncompromising curmudgeon who has no intention of letting her beloved characters become mangled in the Hollywood machine. (Google plot summary)

Just looking at this image, and Ms. Travers’ body language, we can tell she is mired in conflict.  Spoiler alert: Mr. Disney discovers the root of Ms. Travers’ personal struggles and reveals that like her, his childhood was difficult. Mr. Disney shared that his own father was an active advocate of corporal punishment who never allowed his sons an allowance or any type of plaything.  He also stressed to Travers that he believes his films are a form of art with great healing value. He urges her to not let deeply rooted past disappointments dictate the present.  Powerful, emotional, conflict, and resolution.

Patrick Lencioni describes exactly this type of conflict in his book Death by Meeting. 

In it, he asks, “Why does a 30 minute meeting seem like it lasts 3 hours and a 3 hour movie seem like it lasts 30 minutes?”

The key, Lencioni says, is Conflict.

“Directors and screenwriters learned long ago that movies need conflict to hold the interests of their audiences. Viewers need to believe that there are high stakes on the line, and they need to feel the tension that the characters feel. What is more, they realized if they didn't nurture that conflict - or drama - in the first 10 minutes of a movie, audiences would lose interest and disengage."

"Leaders of meetings need to do the same by putting the right issues - often the most controversial ones - on the table at the beginning of their meetings. By demanding that their people wrestle with those issues until resolution has been achieved, they can create genuine, compelling drama, and prevent their audiences from checking out.”

It was my intent to capture your attention in the opening minutes of this presentation, but now I must reel you back in. This is a software blog after all.

So here’s a fair warning for you…

These same three components equally create and provide just as much excitement when examined in the context of the world’s leading multi-dimensional database, TM1, as they do in the movie theater.  

Riddle me this:

  

To solve the first part of this riddle, I’ll turn to our good friends Maverick and Goose

“I feel the need, the need for speed!”

You need speed?  TM1 has it.  TM1 is an in-memory database that performs calculations in “real-time”.  By real-time, we really mean “on demand”, or “when requested”.  A common OLAP practice for disk based systems is to pre-calculate and store values for aggregated and rule calculated values.  This can take hours, or even days in some cases.  TM1 is much more efficient and only calculates those values that are requested by users, at the point in time they are requested.

It is simply undeniable that RAM based systems achieve retrieval and calculation speeds that disk based systems cannot. The access time to data in RAM is on the order of 50 nanoseconds (5 x 10 **-8 seconds). The access time to data on disk is on the order of 5 milliseconds (5 x 10 **-3 seconds). The difference is 10**5 or a factor of 100 thousand increase in speed.

In TM1 lingo, we translate speed to “Real-time answers” and have the answer to the first part of the riddle:

TM1 is the undisputed leader in performance management software,  providing real - time answers, amidst ________  ________, to _______  ________.

While we haven’t yet achieved Warp Speed (which exceeds the speed of light), in terms of data retrieval from RAM, we are approaching it. 

Now some of you may have experiences with TM1 and, much like Scotty on the USS Enterprise or Chewbacca on the Millennium Falcon, you may need to troubleshoot a poor performing hyper-drive.   

Shameless plug: Market Street Solutions has the knowledge and expertise help you do exactly that.  A properly designed and developed TM1 database will have your crew running the fastest ship in the galaxy.

Back to the Riddler’s puzzle…

TM1 is the undisputed leader in the Performance Management software, providing real-time answers, amidst ________  ________, to ________  ________.

What could part 2 of the puzzle be?  Violence?

Just like a scene from one of one of today’s most popular movies, the TM1 ecosystem is an extremely violent place.  Three commonly used words to describe TM1 operations provide an indication of this:

KILL - user sessions
EXECUTE - processes
DESTROY - cubes, subsets and views

Users request views, which trigger calculations. Results are cached, then users modify data, and the caches are wiped out. Aggregated calculations are served, then annihilated, feeders are fired, metadata changed, dependencies established, destroyed, and re-established.  Users place locks and holds, they spread data from consolidations, leverage data entry shortcuts, create subsets, destroy subsets, create views and destroy views, reference dynamic subsets which get validated and cached, only to be destroyed seconds later.  Rules are modified and saved, feeders are written, rewritten, saved and processed.  Turbo Integrator processes are executed, potentially performing and any number of the above actions.  Process can load terabytes of data, and even call external applications and programs.  Other processes may be killed during their execution.  Threads are spawned and merged, re-spawned and re-merged.  It is amazing that a request is able live a successful life among all of this violence.  Somehow, almost miraculously, they are.

So clearly, one word to describe the violence in a TM1 ecosystem is not enough.  We’ll need two : ) Extreme Violence

TM1 is the undisputed leader in the Performance Management software, providing real-time answers, amidst extreme violence, to ________  ________.

Much like the trend in movies is towards more and more violence, TM1 is becoming more and more capable of handling violence.  10+ years ago, TM1 was completely single threaded, meaning only 1 user request could be handled at a time.  Features like multi-threaded cube loads, parallel interaction, & MTQ have dramatically increased the “violence” within the TM1 ecosystem. 

Additional features like smart cache and non-blocking metadata updates promise this trend will continue, although unlike those of Hollywood movie producers, in TM1’s case, our consciences can remain clear, even with the label of “bloodiest in-memory database ever”.

Let’s complete our riddle!

We’ve discussed speed and violence, the answer to the third part of the riddle should be clear: conflict

Conflict, or in TM1 terms: “Complex questions”

TM1 is the undisputed leader in the Performance Management software, providing real-time answers, amidst extreme violence, to complex questions.

TM1 has no shortage of conflict and drama.  Drama likely abounds within your own TM1 database. TM1 applications are typically created to solve a particular problem, whether its budget and planning, general reporting, or a custom application of some type.  Whatever the questions are that you need answers to, your TM1 application is intended to provide answers.

How TM1 presents these answers, and resolves your internal “conflict” has been a drama of its own in recent years.

When a small company named Applix owned and developed TM1, the software included numerous interface options for developers and end users.  The primary mechanism for interacting with TM1 was “Server Explorer”, which could be exposed within Microsoft Excel as an “add-in” called “Perspectives”, or as a stand-alone application called “Architect”.  The only real difference between these applications was (and remains) the fact that could interface with Excel.  Consultants, developers, report writers, end users…everyone…everyone associated with the tool used the same product with the same interfaces, same gestures and same general experience.  There were and are a handful of additional software components that provided unique experiences, including Turbo Integrator, the Rules Editor, & TM1Top and of course Excel itself, the primary delivery tool for user interfaces.

As time moved along, market demands indicated the need for a web based interface, and TM1Web was born, introducing a similar, yet slightly different paradigm for interacting with TM1.  Not too long afterwards a series of mergers and acquisitions dominated the TM1 landscape, as Cognos purchased Applix and almost immediately afterwards, Cognos was bought by IBM.  

Applix’s software was tightly integrated with Excel, while Cognos eschewed the virtues of an ecosystem free from Excel.   Legacy “Cognos Planning” customers expected TM1 to behave similarly to their past experiences, which led to the creation of Performance Modeler. Cognos BI customers were now looking to leverage TM1’s speed, and expected seamless integration.   In the worst name choice ever for software, Cognos Planning’s Contributor morphed into “The TM1 Applications Web”, not to be confused with the general term “TM1 applications”, or TM1’s “application folders”.   I refer to it now as “The Artist Formerly Known as Contributor”.  

The merger and acquisition trend in the business analytics space has continued, leading to more software products and customers for IBM.  The result is a software system that now includes more than 20 different applications, user interfaces, and interaction experiences utilizing outdated methods for communicating with the TM1 server, and seamless integration expected with numerous others.

Surely you’re on the edge of your seat wondering, “How will IBM resolve this conflict?” 

The answer and the resolution to this internal drama and conflict:

Let’s first examine Cafe.  In its current form, we’re able to connect café to TM1 databases, create flex views and formula views that mimic a lot of the functionality that exists in TM1 Perspectives. 

Excel tends to be used for 3 types of often overlapping activities, Analysis, Reporting and application design.

The first type of activity is Analysis – which is is essentially dimensional slice and dice during which data discovery is more important than formatting. It’s usually focused on one OLAP crosstab

The second type of activity is Reporting and focuses on representing and communicating 

information and requires sophisticated formats and multiple tables and charts. Data is often linked.

Application style workbooks guide users through their consumption and write-back with navigation and a certain amount of business logic. Applications require all of the reporting capabilities plus automation and UI widgets.

Everything you currently do in a TM1 Perspectives workbook today, you can expect to be able to do in CAFÉ in the future. Ongoing work includes ensuring that each TM1 excel function (DBR, DBRW, ELPAR, etc) fully works in the CAFÉ environment.  You’ll be able to create engagement user interfaces in familiar environment (Excel), that directly communicate with the TM1 Server.

Prism is TM1’s new Web Platform, providing a single destination for modeling, authoring, contribution, reporting, and analysis

Prism is part of the Watson Analytics family.  Prism is not a new cube viewer – it’s a workspace for taking action. In this screenshot you can see some familiar and some new objects.  A tree structure on the left for object navigation, a redesigned cube viewer in the middle, and a heat-map visualization on the right.

Why will customers love Prism?

• We have a big picture vision of single Web Platform for all users, no matter their role

• Brand New, Fresh, and Highly Functional Face of TM1

• Modern User Experience for Authoring, Analysis, Visualizations, and Data Entry

• Workspace Layout (“Books”) with multiple Widgets

• MDX foundation permitting use of Hierarchies as Virtual Dimensions, Asymmetrical Nesting, Hierarchical Sorting, On the fly Row & Column Calculations

• New Content Store permitting improved sharing and collaboration

• Synchronization with CAFÉ – create and view objects in Prism or CAFÉ, consume them either place

• API Access so Prism Views and Books can be embedded in custom web apps

• Prism first release is targeted for later this year, in keeping with IBM’s strategic imperatives, Cloud, Data & Collaboration

• The first release will be cloud only.

The SPEED of an in-memory database, he VIOLENCE of user & server activity,  and CONFLICT – questions answered, drama resolved, in PRISM & CAFE.

Thanks the these three components,  I’m certain you’re going to continue to love TM1 as much as you love going to the movies.