Tagged: Operations

The Impact of Groupthink in Decision Systems


NASA 0


“Together with his identical twin brother, Scott, he has laid the groundwork for the future of space exploration as the subjects of an unprecedented NASA study on how space affects the human body, which is featured in Scott’s New York Times best-selling memoir, Endurance: A Year in Space, A Lifetime of Discovery.”

“Currently, Mark is on the Commercial Crew Safety Board at Space X […] and is the co-founder of World View, a full-service commercial space launch provider.”

Endeavour to Succeed. College of DuPage, Department of Physics. February 14 2019.


NASA 6


I managed to attend Captain Mark Kelly’s talk in Chicago just the day before I was leaving for Barcelona’s Mobile World Congress. M. Kelly’s presence and insightful remarks commanded both admiration and utmost respect.

Among many other fascinating topics, he discussed NASA’s None of US is as Dumb as All of Us as a reminder of the negative impact of ‘groupthink‘ in the context of faulty decision making. Most specifically, he referred to dramatic mistakes leading to the Space Shuttle Columbia disaster, which disintegrated upon re-entry in 2003.


“Large-scale engineered systems are more than just a collection of technological artifacts. They are a reflection of the structure, management, procedures, and culture of the engineering organization that created them.”

“They are also, usually, a reflection of the society in which they were created. The causes of accidents are frequently, if not always, rooted in the organization—its culture, management, and structure.”

“Blame for accidents is often placed on equipment failure or operator error without recognizing the social, organizational, and managerial factors that made such errors and defects inevitable.”

Nancy G. Leveson, MIT. Technical and Managerial Factors in the NASA Challenger and Columbia Losses: Looking Forward to the Future. Controversies in Science and Technology Volume 2, Mary Ann Liebert Press, 2008.


NASA 1Groupthink is part of the taxonomy of well-known cognitive biases and takes hold when divergent thinking and disagreement are discouraged (and even repressed) as part of group dynamics.

Hindsight is 20/20 and, statistically speaking, ‘black swan’ events are characterized by seemingly random surprise factors. Groupthink can obfuscate the early detection of predictors such as leading outliers and anomalies, which left unattended can overwhelm a given system over time… and be the source of cascading effects and critical failure.

Groupthink’s negative impact compromises any best intentions such as organizational cohesiveness in the spirit of consensus, agility, productivity, timely project progress and de-escalation management.

Often times, there might be neither adequate situational and risk awareness nor a basis for sense making drawing from the comparative analysis that comes with diligent scenario planning.


Individuals and organizational cultures with a succesful track record can also experience complacency. Over-confidence fosters the sort of behaviors and decisioning that served the group well in the past.

Though, when in the mix of a changing environment defined by new parameters under the radar, only operating within the perimeter of a given set of core competences and comfort zones, makes those specific behaviors blindsight and betray the team’s mission and purpose.

Many plans do not survive first contact (or a subsequent phase for that matter) as their implementation creates ‘ripple effects’ of various shapes and propagating speeds. Some of that can be experienced as ‘sudden risk exposure.’ Once passed the ‘point-of-no-return,’ if that challenge is met with neither contingency planning nor the ability to timely course correct, pivot or even deploy a basic safety-net offsetting the impact, the project fails to ‘cross the chasm’ and is headed for what’s technically known as the ‘valley of death.’


This was one of the key issues discussed by Clyton M. Christiansen when I took his Harvard class on the ‘Innovator’s Dilemma,’ and is also a key point behind Risto Siilasmaa’s ‘Paranoid Optimism’ as well Paul Romer’s ‘Conditional Optimism,’ all of which advocate for scenario planning and sensing optimization to be able to calibrate or re-assess the path forward.


“Michael Shermer stated in the September 2002 issue of Scientific American, ‘smart people believe weird things because they are skilled at defending beliefs they arrived at for nonsmart reasons.”

Groupthink can also manifest itself by means of ‘eco chamber’ effects’ as misguided consensus amplifies what becomes a “self-serving” bias. That is, in effect, a closed feedback loop process that magnifies logical fallacies. These can come across as reasonable enough postulates, though if based on rushed judgement and selective focus they can also suffer from ‘confirmation bias.’ This is the case when new evidence is only used to back-up the existing belief system rather than share new light.


In the context of Decision Support Systems and Cognitive Analytics, the above reasoning deficits become root causes of errors impacting operations. That can involve both (a) Human-Human and (b) Human-Machine interactions, as well as impacting programming work resulting in (c) biased algorithms and automation pitfalls when left unsupervised.


NASA 3


REFERENCES

Carisa Callini. Human Systems Engineering. NASA, August 7 2017. https://www.nasa.gov/content/human-systems-engineering

Carisa Callini. Spaceflight Human Factors. NASA, December 19 2018. https://www.nasa.gov/content/spaceflight-human-factors

Clayton M. Christensen. The Innovator’s Dilemma. Harvard Business Review Press, 1997.

COD Welecomes Astronaut Mark Kelly. Daily Herald, February 13 2019. https://www.dailyherald.com/submitted/20190201/cod-welcomes-astronaut-mark-kelly-feb-17

Geoffrey Moore. Crossing the Chasm. Haper Collins, 1991.

MIT Experts Reflect on Shuttle Tragedy. MIT News, February 3 2003. http://news.mit.edu/2003/shuttle2

Tim Peake. The Astronaut Selection Test Book. Century. London, 2018.

Scott Kelly. Endurance: A Year in Space, a Lifetime of Discovery. Knopf. New York, 2017.

Scott Kelly. Infinite Wonder. Knopf. New York, 2018.

Steve Young. Astronaut: ‘None of Us is as Dumb as All of Us.’ USA Today – Argus Leader, May 13, 2014. https://www.argusleader.com/story/news/2014/05/13/astronaut-none-us-dumb-us/9068537/

Will Knight.  Biased Algorithms are Everywhere, and No One Seems to Care. MIT Technology Review, July 12 2017. https://www.technologyreview.com/s/608248/biased-algorithms-are-everywhere-and-no-one-seems-to-care/

 

Lean NFV Ops: Demo Script.


“This interactive demonstration shows the positive impact of agile service launch subject to Reliability, Availability, Serviceability (RAS) scenarios. It features an application centered system involving sophisticated Virtual Network Functions (VNF) and integrates Operations Support System (OSS), NFV’s Management and Orchestration (MANO) as well as Software Defined Networking (SDN) under a modular and scalable approach.”



“In addition to Alcatel-Lucent’s portfolio, which is represented by Motive Dynamic Operations (MDO), CloudBand Management Platform (CBMS) and Cloud Node, Nuage Networks, Virtual Evolved Packet Core (vEPC), Virtual IP Multimedia Subsystem (vIMS) our conversation illustrates Ecosystem examples involving third party partners, findings from Bell Labs Research and presents opportunities for following up with hands-on activities at the Cloud Innovation Center (CIC).”


00:00 – Hi, my name is Jose. We are going to discuss operations in the context of NFV, Network Functions Virtualization. We will do that for the purpose of delivering service agility because launching new applications in the marketplace should be as easy as getting them deployed with just one click.

00:30 – This is a real environment, this is not a proof of concept. These are products that are either available today or in production in 2015. Namely Motive Dynamic Operations (MDO), the OSS, Nuage Networks’ SDN (Software Defined Networking) framework, the CloudBand platform, which manages the lifecycle of the VNFs (Virtual Network Functions) as well as orchestrating the underlying cloud infrastructure. Last but not least, we will also discuss findings from Bell Labs’ research. To complete the environment that we are operating with today, you will see a fully virtualized RAN (Radio Access Network) as well as the mobile core with the vEPC (virtual Evolved Packet Core) and vIMS (virtual IP Multimedia Subsystem), all working together to deliver this VoLTE (Voice over Long Term Evolution) live video session.

01:20 – We are going to follow two basic principles in this demonstration. Principle number one: these are very sophisticated systems and we are bringing them together, therefore, there is no denial that we need to abstract out complexity to deliver simplicity, that way we can manage operations. Principle number two: no matter what we do in the background operationally speaking, the user experience, the video in this case, should continue to play completely unscratched. At the end of this demonstration we will review these two principles to check how we did.


01:50 – Deploying any application should be as easy as… and here is the virtualization catalog that we use in our labs at the Cloud Innovation Center, it should be as easy as selecting what I need and launching the application to the NFV Operations Center. The heavy lifting is actually performed by CloudBand, the MANO (Management and Orchestration) platform. It understands the application requirements, the lifecycle, and will make sure that things talk to the right components to spin up virtual machines and onboard the service.

02:20 – Moreover, now we need for traffic to flow through this new application, this new service. I am now talking to Nuage Network’s SDN (Software Defined Networking) framework to get that going in a split second. So, I am now working on SFC (Service Function Chaining). And there you are.

02:45 – Now, let’s continue to test more things in the marketplace in real time. I am now delivering yet another application: a content filtering service. Maybe I should also deploy a WebRTC (Web Real Time Communications) server. And here it is. By the way, all the virtual machines in green color are carrying load this minute, the virtual machines shown in blue are on standby. These other are mated pairs for reliability so that we can work in HA, this is a High Availability environment. Moreover, virtual machines laid horizontally are services and products from third party partners also onboarded on the CloudBand platform.

03:25 – As you see, we need to do some more service chaining, and we are now working again with Nuage Networks’s SDN. I am going to do the chaining for this one application. Note that this is fully programmable, everything is fully automated.


03:40 – Let’s discuss what happens when a network operator becomes victim of success. That would be a situation where this video service becomes very popular because it works well. There is [unplanned] pent up demand with more subscribers using the service. Therefore traffic grows. Let’s simulate that kind of situation. These are load generators which I am going to work with to conduct a stress test. As you can see, traffic is ramping up already. The question now is, will we have enough capacity available to meet new demand? Things are not looking that good… but as we detect this trend thanks to Bell Labs analytics, the platform starts spinning up new virtual machines and onboarding necessary services so that we can get some relief. [As a result] now we are working with new subscribers without a glitch.

04:40 – The opposite is also true. Let’s say that there is no longer that much demand for this one service. There aren’t so many subscribers. Traffic is no longer flowing through our system at the same scale. Let’s simulate that. Traffic is going down this minute. The very same way we were scaling and creating more capacity before, we are now going to take down all of those added systems so that we can make the underlying resources for the next batch of successful applications to utilize. As you see, the ones in red are continuously being monitored so that we can clean up and, once again, gracefully terminate those services.


05:20 – We can do all of these things because we are working in a data center environment. These are CloudBand’s Cloud Nodes. This is COTS (Commercial Off The Shelf) infrastructure, these are not dedicated servers. Therefore, we can continue to spin up new virtual machines and onboard applications. We can continue to reuse these resources [compute, memory, storage, networking] at very high utilization levels over and over.


05:50 – If you are successful, in addition to experiencing demand and coping with capacity… at some point you will be facing updates, upgrades… maintenance events. Let’s simulate that too. This is a RAS (Reliability, Availability, Serviceability) test. We could start by opening a maintenance window, the more applications we have, the harder it is to find those at the right time without disrupting the video experience, the user experience. We could trigger a network failure instead, some issue that impacts QoS (Quality of Service) or, perhaps, a cloud failure that could involve a corrupted virtual machine. Let’s cause that last one.

06:30 – The machine that has been compromised has been flagged [in red]. The load has already been placed on the mated pair. There was [service continuity] no disruption of any kind as far as the user experience is concerned. Be have been able to do that thanks to smart placement combined with a distributed architecture. The data center that you see on the left, DC number one, is based at a central location where we have consolidated assets for the purpose of delivering cost efficiencies. [On the right] data center number two is at a distributed location closer to the network’s edge for performance sake instead.


07:10 – Everything that we have been discussing up to this point is available from Alcatel-Lucent’s portfolio in 2015. In the next few minutes, I will share with you research findings from Bell Labs projects. These relate to analytics for smart load placement and autonomics, that is machine learning for NFV.

07:30 – You were able to notice that as I moved the load to the other data center, the service was not disrupted but I lost HA (High Availability) [by operating in a simplex environment instead]. Now I need to look for the best placement for the new mated pair that will become my new backup should something happen to the virtual machine that’s carrying the load right now. The question is: where should I do that?

07:55 – Bell Labs’ recommendations engine is checking cloud requirements and conditions, it couples that with equivalent network requirements and conditions, it understands what any given application needs in the lifecycle. It reads the contract because it does not make sense for me to deploy something in a more expensive environment, which would defeat my business case and cloud economics. By the same token, I cannot deploy the load in an inferior environment, which would not meet the SLA (Service Level Agreement). Additional policies: these could be engineering events or any other kind of rules. This could be weather conditions because I wouldn’t like to move the load to a data center that is going to be compromised by terrible weather for that matter.

08:45 – If I like this recommendation which prompts me to move the load from “cloud one” to the “Barcelona data center” I could just click “accept” and move forward. What if there was a better option? I am going to ask the recommendations engine to present another option. In this other case it says that I should be moving the load to a different data center closer to my next destination, so that the service is provided closer to my location.


09:10 – In any case, at any given point of time, I should be able to do RCA (Root Cause Analysis). For that purpose we get to display fine grained, correlated analytics. We built a dynamic dashboard that we can always check to asses the current situation and do troubleshooting accordingly. The various metrics come, and are fed, by the different solutions that you see represented in the smaller screens on each side of the NFV Ops Center. If this was a false alarm I would then click on “stand down” and nothing would executed. The reality is that false alarms can happen. If I need to buy more time to get more data, to do further analysis, I would then click on “standby” instead.


10:10 – There is research on autonomics as I was sharing before. This means that the recommendations engine, time after time, learns from these behaviors and it becomes more predictive and, eventually, it gives you even better custom recommendations further optimizing system performance as well as any other kind of efficiencies.

10:30 – I am going to accept the recommendation that works best for me, which is the first one. In the background, what you would see are the very same things that we saw early on: virtual machines being spun up, applications being onboarded, networks being created… with all of that happening literally in just minutes. This is very different from PMO (Present Mode of Operations) where it takes filling out forms, scheduling meetings, talking to a lot of people. Then it takes maybe hours, if not days, perhaps, weeks before we get anything done. Here things are programmable, fully automated, and things happen in real time as you can see by means of this demonstration.


11:10 – We have also brought to you a single pane of glass to abstract out complexity. When drilling down, it pays to go to the UI (User Interfaces) of the specific solutions. This [single pane of glass] is not an Alcatel-Lucent product, this is just illustrating a requirement from many of our customers who are asking for the APIs (Application Programming Interfaces) from this various solutions to build their own dashboards and their own screens.

11:30 – Well, this completes the demonstration. As I was saying early on: a 100% real, this is no PoC (Proof of Concept), all of the products with the exception of Bell Labs research. which we just discussed, are currently available or in production in 2015, this year. Thank you.