Category: QbD

QXbD, Quality Experiences by Design – Research Notes, Part 1.2


The preceding post on QXbD research notes – Part 1.1 shared a retrospective with insights from my early college years, which were influenced by Bruno Munari‘s “projected methodology” and the Bauhaus‘ design principles.

Research Notes Part 1.2 (this post) takes me back to BarcelonaTech’s school of engineering in the early 90s, which I joined to study Human Factors Engineering while pursuing my last year of Industrial Design at Escola Massana, an art & design school.



Those days, Donal Norman’s “The Psychology of Everyday Things” and Henry Petroski’s “The Evolution of Useful Things. How Everyday Artifacts Came to be as They Are” became must-read books for anyone interested in thoughtful design principles in a new light. Norman was an Apple Fellow and became VP of the Advanced Technology Group in the mid 1990s. He popularized the term of User Experience.


“Everything that touches that touches upon your experience”.
“Today, that term [User Experience] is terribly misused”.

Petroski was an engineer whose best known work focuses on failure analysis. He stated that the best Industrial Design involves “seeing into the future of a product” and that Human Factors Engineering is concerned with “how anything will behave at the hands of its intended and not intended users.” Here is a summary of some of his design principles:

  • Tools make tools.
  • Artifacts multiply and diversify in an evolutionary way.
  • There always is room for improvement.
  • Good can be better than best.
  • Efficacy can be subjective, want overpowers need.
  • Form follows failure: inventors should be technology’s severest critics.
  • Focus on different faults means different solutions to the same problem.
  • Engineering is invention institutionalized.
  • Sometimes it is about a new job, sometimes about a better or faster job.

“We are all in this together. Artists and engineers are part of the same effort”.

“Though the best designs deal successfully with the future, that does not mean they are futuristic […] There is an apparent reluctance to accept designs too radically different from what they claim to supersede […] if things are redesign too dramatically and the function that they perform can be less obvious”.



Loewy summarized the phenomenom by using the acronym MAYA, standing for most advanced yet acceptable. Dreyfuss emphasized the importance of a survival form, thus making the unusual acceptable to many people who would otherwise reject it [Industrial Designers] have learned to strive for a delicate balance between innovation in order to create interest, and reassuringly identifiable elements”.


Donald Norman pointed to design issues leading to human error and making users unfortunately blame themselves in the process. He claimed that the “paradox of technology” takes effect when added functionality comes with unwanted complexity, which denies the sought-after benefits. These are some of the design principles:

  • Design should be user-centric and consistent.
  • Identify the true root cause of a problem.
  • Well designed products teach the user how to use them.
  • Make things visible, give clear clues, enough information and feedback.
  • Get mapping and system state right, simplify task structure.
  • Design for error, exploit the powers of constraint.
  • Make possible to reverse actions, and make it harder to do what cannot be reversed.

Following up on the topic of technology’s paradoxes, it is worth reviewing Geoffrey A. More’s “Crossing the Chasm“, which was published in 1991. He explored the rationale behind the failure of emerging technologies, which fail to take hold.


Illusion and Disillusion: Cracks in the Bell Curve.
Technology Adoption Lifecycle. Crossing The Chasm, 1991.

There can be a deep chasm between enthusiasts and early adopters and the broader user groups shaping the mass market. Avoiding the Valley of Death starts with an understanding the adoption lifecycle: different user groups come along with different expectations. That prompts the need for the design of specific transitions and adaptations.


Whole Product R&D […] begins not with creative technology but with creative market segmentation. It penetrates not into protons and processes but rather into habits and behaviors […] it implies a new kind of cooperation between organizations traditionally set apart from each other.”


HUMAN-MACHINE-SYSTEM DESIGN PRINCIPLES

BarcelonaTech’s teaching addressed Human-Machine Systems as an interdisciplinary undertaking. Human dynamics entailed the study of individuals and collectives such as teams and organizations. That would encompass the following disciplines and an strengths and limitations

  • Psychology – skills, cognitive appraisal and workload, workstyles…
  • Physiology – form factors, motions, anthropometry, biomechanics…
  • Social Sciences – teamwork, organizational behaviors, culture…

Tools and machines involved hardware and software components. HMS’ holistic approach consistently tackled end-to-end solutions. These were placed in context and in specific physical environments. The sough-after outcomes of “Designing for People” zeroed in on:

  1. The delivery of capable high performance systems as defined by productivity by effectiveness and efficiency metrics, and success rates.
  2. Designing for users’ wellbeing and safety.
  3. Human Error is often a consequence of poor design.
  4. Addressing the broader user base possible, typically set at 95% coverage with adaptations, accounting for diversity rather than designing for just averages.
  5. Extreme case and stress testing, factoring life-long / lifecycle changes as solutions evolve and/or can be deployed in other context and environments.

DESIGN METHODOLOGY

We followed this iterative methodology, starting with due diligence on:

  1. Initial problem statement and goal setting.
  2. Operations assessment: use cases’ current state / present mode.
  3. User Taxonomy and Analysis: jobs, tools, work motion studies (tasks, workflows, success and failure rates) often relying on instrumentation.
  4. Data collection, processing, analysis and insights.
  5. Identification of value based activities, waste and risks.
  6. Critical success factors and possible scenarios at play.
  7. Information, process, hardware and software specifications.
  8. Contextual and environmental considerations.

The next phase focused on Human-Machine-System design, including all relevant subsystems and interactions across them:

  1. Operations review: new target state and mode.
  2. Interaction Matrix* correlating human and design factors.
  3. Prioritization criteria and conflict resolution.
  4. Job and process streamlining, often leading to redesign, or new design.
  5. Goal setting based on metrics optimizing for system wide operability.
  6. Iterative improvement cycling through experiments, prototyping, simulations and testing.

The Ergonomics of Workspaces and Machines. A Design Manual.

The *Interaction Matrix correlated human factors (rows) for a given design option with the following “realization” ones (columns) and the degree to which those relationships were weak, medium or strong (matrix).

  1. Customer acceptance criteria.
  2. Operability levels, including safety.
  3. Conformance with functional requirements.
  4. Reliability and performance levels, as well as maintenance.
  5. Productization feasibility and costs.
  6. Aestetics and affective considerations.

Quality Management

Just a quick reminder about the fact that this article is still discussing topics set all the way back in the early 90s. Those days, Total Quality Management (TQM) and Lean lead the way. Note that ISO 9000 standards had been first released in 1987.


The top three key values were: Customer Intimacy, Operational Excellence and Product Leadership:

customer intimacy: tailoring offerings to match demand […] detailed customer knowledge with operational flexibility […] customizing a product and fulfilling special requests […] engendering tremendous customer loyalty“.

operational excellence: providing customers with reliable products or services at competitive prices and delivered with minimal difficulty or inconvenience“.

product leadership: continuous stream of state-of-the-art products and services. First, they must be creative […] Second, must commercialize their ideas quickly […] business and management processes have to be engineered for speed. Third, product leaders must relentlessly pursue new solutions”.

High operational performance was broken down as follows:

  • Productivity & scalability.
  • Flexibility & adaptability.
  • Mix complexity.

J.M. Juran discussed quality in the context of “Big Q” and “Little Q” where the former addresses a business problem and is all encompassing, while the latter is siloed and focuses on tackling technical issues. Big Q delivers the sort of value that users can appreciate.


Quality Function Deployment (QFD) – House of Quality Template.

Strategic Quality Management was meant to learn from customer experiences and leveraged House of Quality charts to design with.

The first step was to map out a taxonomy of customer attributes (CA) decomposed in primary, secondary and tertiary levels, the latter being the most granular list of customer requirements and expectations… all largely based on surveys and user feedback. This was done for the value chain consisting of end users, consumers, retailers, distributors, regulators, etc. Weightings were set to prioritize attributes based on contextual relevance.

CA items would then be placed on the left rows of the above spreadsheet for the purpose of cross-checking them with technical features to be shown as column headers. That was done by correlating CA and engineering characteristics (EC). The resulting center matrix was used to assess what items were positively and negatively impacted, co-variance, and to what extend. Each cell featured icons and color coding for strong, medium, weak relationships.

The pyramidal roof at the top was filled out afterwards to look into technical synergies and conflicts alone. Basically, becoming aware of how engineering characteristics interact and making decisions on optimizations and conscious trade-offs.


SOME OTHER THOUGHTS…

Attending both Art and Engineering schools was a fascinating experience to say the least. The opportunity to cross-pollinate across disciplines could made anyone feel like being in a reenactment of the Renaissance’s blending of arts and sciences.

Both Industrial Design and Human Factors Engineering optimize for the human experience and, therefore, make their professions be about “Designing for People”. Technology that does not account for human skills, strengths as well as limitations, all in context and in the scenarios and environments will operate under… becomes greatly exposed to failure.

Striving to make designs that fit people’s potential, rather than just expecting users to just fit… does require an interdisciplinary and iterative practice, painstaking attention to detail being critical. At that point, it also became clear that addressing the Big Q also had to do with articulating the business value of design.


BIBLIOGRAPHY

  • D. Norman. The Psychology of Everyday Things. Basic Books, 1988.
  • G.A. Moore. Crossing the Chasm. Harper Business, 1991
  • H. Petroski. The Evolution of Useful Things. Vintage Books, 1992.
  • J. Krafcik and J. Womack. Triumph of the Lean Production System. MIT Sloan Management Review, 1988. Accessed on May 18 2019 http://www.lean.org/downloads/MITSloan.pdf
  • J.R. Houser and D. Clausing. The House of Quality. Harvard Business Review, May 1988. Accessed on May 18, 2019 https://hbr.org/1988/05/the-house-of-qualityT.S. Clark and E.N. Corlett. La Ergonomia de los Lugares de Trabajo y de las Maquinas. Tylor and Francis, 1984.
  • M. Treacy and F. Wiersema. Customer Intimacy and Other Value Disciplines. Harvard Business Review, January – February 1993. Accessed on May 18, 2019 https://hbr.org/1993/01/customer-intimacy-and-other-value-disciplines
  • House of Quality Template. QFD Online. Accessed on May 19, 2019 http://www.qfdonline.com/templates/

Nokia HFE18 Annual Conference (1) #MakeTechHuman


We are gearing up for NOKIA HFE, our annual conference focusing on that it takes to create technologies that deliver unique value by optimizing for early user acceptance and broader adoption.

Human Centered Design (HCD) is the practice addressing the users journeys and technology lifecycles. Human Factors Engineering’s (HFE’s) is the holistic and interdisciplinary science focusing on optimal technological solutions that #MakeTechHuman, which intersects psychological, physiological and sociological factors.



BEING DIGITAL is this year’s theme. As shared in the above brochure, Bell Labs’ Claude Shannon, the father of information theory, first addressed the meaning of “bit” in his landmark paper published in 1948. Just a year earlier, John Karlin, a fellow Bell Labs pioneer, set up the first Human Factors Engineering department in industry. Their combined effort set the stage for today’s digital experiences.

More than seven decades later we get together to explore what “being digital” is now about. This year’s discussion takes place in the context of game changing and pervasive “Digital Transformation” initiatives across industry and public sectors.


CHi RYAN

Chirryl-Lee Ryan is Idean’s Head of Studio in New York, and this year’s keynote speaker.

Chi is an transdisciplinary design practitioner, writer, speaker, coach, and leader, specializing in experience design. Chi believes that design can help everyone live better, happier lives, and to do so, she merges different design disciplines to produce radical outcomes for humans – and humanity.

As Head of Studio at Idean, a global experience design company, Chi evangelizes a mantra of endless curiosity, fearless execution, and purposeful impact, and as host of the This is HCD Podcast, she is creating a continuous conversation about the future of design. Chi’s goal is to arm as many people as possible with the skills, tools, and confidence they need to create the change they want to see in the world.


Dan Kraemer

HOW TO PROTOTYPE A NEW DIGITAL BUSINESS

“Through the lens of his recent work in guiding leading global companies through digital transformation initiatives, IA Collaborative Founder and Chief Design Officer Dan Kraemer will discuss how to bring a Design Thinking approach to digital innovation – specifically, how to combine user experience, emerging technologies and profit models to prototype entirely new and sustainable digital businesses.”

As Founder and Chief Design Officer at global design and innovation consultancy IA Collaborative, Dan Kraemer is an internationally recognized brand, product, interactive and architectural experience designer who works with some of the world’s most successful companies and brands – including Johnson & Johnson, United Airlines, Samsung, GE, Nike, and Hyatt. Dan leads IA Collaborative’s multidisciplinary design team to identify unseen human needs, frame breakthrough opportunities and drive systemic solutions to commercialization.

His work has been recognized by the International Design Excellence (IDEA) Awards, GOOD DESIGN™ Awards, the Red Dot International Design Awards, the iF World Design Awards and the Design Intelligence Awards. Dan’s work has been featured by Wired, Fast Company, Forbes, the BBC, Branding Magazine and Innovation Magazine, and he is a frequent lecturer on the national and international stage.


cropped-Jose-de-Francisco-LinkedIn-3

THE SOFT & HARD NATURE OF ANYTHING DIGITAL

“Our quest to deliver productivity tools yielding operational excellence for DSPs, Digital Service Providers leads to the design of signature experiences by innovating in the process.”

“The Studio at Nokia Software’s Solutions Engineering is set to work with deceptively simple techniques and elegant sophistication… because neither oversimplification nor self-defeating complexity allow end-to-end systems to efficiently operate at digital speed and global scale.”

“This discussion intersects the soft and hard natures of dynamic systems by modeling Human Machine Systems (HMS) and the design of cybernetics. This practice focuses on critical success factors for the early acceptance and broader adoption of emerging technologies.”

“The work at the Studio embraces a renewed approach to QbD, Quality by Design, which is set to left-shift and unveil instrumental considerations at early design stages. The result is Nokia Studio’s QXbD, Quality Experiences by Design, optimizing for customer delight rather than table-stakes customer satisfaction.”

Jose de Francisco is a Senior Creative Director at Nokia Software Group. His 20+ year experience encompasses global award-winning projects that entail multi-disciplinary leadership responsibilities. Jose is a Distinguished Member of Technical Staff (DMTS) and has worked with Bell Labs on next generation platforms for mobile networks.

He is a Member of the Advisory Board at MIT’s Institute for Data Systems and Society (IDSS) and is the recipient of an MBA in International Marketing and Finance (MBA/IMF) from Chicago’s DePaul University as a Honeywell Europe Be Brilliant Scholar. Jose also holds a postgraduate degree in Human Factors Engineering from BarcelonaTech (UPC) and developed his passion for innovation at Massana Art &Design Center’s Industrial Design program. His thoughts and endeavours can be followed on innovarista.org.


JC Grubbs

DON’T BE DIGITAL

“The motivation to make our businesses more digital often cause us to miss the mark. Business transformation, and “being digital”, isn’t about technology, it’s about becoming more human.”

“In this presentation I’d like to share what I believe should be the driving forces for any company in this era of technology, data, and sometimes noise. Together let’s explore how we translate our humanity into the things we create.”

JC Grubbs is the CEO and founder of DevMynd, a strategy, design, and custom technology firm in Chicago and San Francisco. He has spent his career working to improve the way that technology is designed, built, and delivered. With a focus on human-centered and inclusive design, his company has worked to solve meaningful challenges for organizations like the Department of Defense, Motorola Solutions, Verizon, and AbbVie.


Iana Kouris

EMPLOYEE EXPERIENCE

“Employee experience (EX) is recognized as a key competitive advantage and a prerequisite to deliver outstanding user experience (UX) and customer experience (CX). Companies that invest in EX outperform those that don’t in terms of financial and operational results.”

“Together with Human Resources (HR) colleagues, we have embarked on a journey to make HR our Chief Employee Experience Office. We are applying a Service Design approach, which is revealing human-centered perspectives as we look into our processes and tools from the employee & line manager point of view.”

“User interviews and workshops identified key pain points and game-changing ideas to take EX to new levels: aadequate digital channels and user interfaces are of the essence. The question that we are currently working on and will openly discuss in this session is: “how might we provide the most insightful & engaging digital experience to further personal & career development?”

As a Design Lead at Nokia, I am driving design-led transformation across the company by applying design approach to management and technology challenges. I believe that design is and will remain central for our future, infusing human focus, accelerating learning and fostering collaboration. I have 10+ years of experience in business and creative roles in Telecommunication, High Tech and management consulting, including 5+ years with McKinsey & Company Inc. My education background spans between Business, Mathematics, Philosophy, Fine Arts and Music.


HFE18 Sessions.jpg


HFE18 - Nokia Chicago Tech Center


Nokia HFE Talk Series


NOKIA HFE18 Credits

QbD and Digitalization’s need for Designing Quality into Solutions


“[They] lost their quality leadership to new, aggressive competition. The most obvious consequence was lost of market share (…) [due to] quality features that were perceived as better meeting customer needs [and] they did not fail in service as often.”

“Loss of market share is not the only reason behind [it] (…) a second major force has been the phenomenon of life behind the quality dikes. We have learned that living in a technological society puts us at the mercy of the continuing operation of the goods and services that make a society possible (…) without such quality we have failure of all sorts (…) at the least these failures involve annoyances and minor costs. At their worst they are terrifying.”

“A third major force has been the gathering awareness by companies that they have been enduring excessive costs due to chronic quality-related wastes (…) about a third of what we do consists of redoing work previously done (…) lacking expertise in the quality disciplines, they are amateurs in the best sense of that word.”

J.M. Juran’s assessment on Quality issues in the 1960s-70s.


 

What follows are some of the insights driving the work that I’m doing on reviewing, leveraging and updating QbD (Quality by Design) in the context of today’s fast growing and all-encompassing digitalization.

I am dusting off my research from 2010 on the 3Q Model. Back then I was a senior manager at Alcatel-Lucent’s Solutions & Technology Introduction Department. My current role is Senior Studio Director at Nokia Software’s Solutions Engineering. Note that the scope is End-to-End Solutions. These are holistic system-wide (cross-sectional and longitudinal) undertakings intersecting different domains to deliver the higher value of the whole. I have discussed QbD for Digital Transformation projects at the Design Thinking 2018 event and at the IEEE (Institute of Electrical and Electronics Engineers) conference on CQR (Communications Quality and Reliability) back in April and May of this year. Interestingly enough, both events were held in Austin, Texas.


Juran on QbD book.jpg

QbD was first coined by Juran, a renown pioneer of quality practices, whose work on that specific topic started in the mid 80s. He linked Quality to customer satisfaction and reliability as the two dimensions to focus on:

“Features” were defined as “quality characteristics,” which meant properties intended to satisfy specific customer needs. That would also include “promptness of delivery,” “ease of maintenance,” and “courtesy of service” to name some examples. “The better the features, the higher the quality in the eyes of customers.”

As far as reliability and, therefore, replicability and consistent performance, “freedom from deficiencies” conveyed the fact that “the fewer the deficiencies the better the quality in the eyes of customers.” A “deficiency” is a failure that triggers dissatisfaction, which calls for incurring higher costs to redo prior work.

“Fitness for use” was mentioned as an attempt to capture the above two together. The so-called Juran Trilogy entails Quality Planning, Quality Control, and Quality Improvement.


More than three decades have passed since Juran started to work on “New Steps for Planning Quality into Goods and Services.” Let’s decompose QbD’s acronym at face value and distill its essence.

As a designer, my belief & practice system focuses on “serial innovation” consistently delivering superior value. This is achieved by means of purposeful and elegant solutions equipped with capability models and optimal functionality leading to Quality Experiences.

Customer Delight, rather than just satisfaction, being the sought after outcome. This applies to both small and large undertakings, and as A. Kay, a pioneer in graphical user interfaces, best put it, “simple things should be simple, complex should be possible.”

Following that train of thought, “Designing Quality into Solutions” should become center stage to: (a) collaborative and iterative ideation, (b) agile development, (c) continuous delivery and (d) the dynamic diffusion of (e) new and mass-customizable digital services for consumer and enterprise markets, as well as no-for-profit. Overall, QoB is key to Operational Excellence.


Usability Testing Project


In a world where “Continuous Improvement” leads to incremental and breakthrough innovations, Quality’s critical KPI, Key Performance Indicator, can be expressed in terms of measurable advances in QoUX, the Quality of the Users’ Experiences. These are lagging (outcome) metrics that are far from static because they evolve within and over lifecycles. Therefore, reliability is not just applied to production operations, but also to the solution’s consistent performance and serviceability over time and under changing scenarios and events.


Given Quality’s unequivocal narrative around the “experiential” paradigm and, therefore, human-centric-optics, QbD’s best work should optimize for user “delight,” which is defined as superior “satisfaction,” rather than just aiming for requirements compliance.

It is very tempting to rally around core competencies within comfort zones that exist, and then settling on just aiming for “customer satisfaction” around “must-meet” baseline requirements. Though, that might not suffice given the necessity to innovate and better compete by leveraging unique sources of sustainable differentiation.


Let’s now state the obvious: “designing” Quality Experiences into digital solutions is best addressed by means of Human-Centered methodologies that optimize for (f) users’ “acceptance criteria” and (g) the kind of “adoption levels” that foster user base growth.

The opposite approach would risk the adverse effects (and hidden costs) that can be incurred when technical myopia leads the way. A. Cooper’s “The Inmates are Running the Asylum” captures that very well. His book is referenced below.


Accenture Survey.jpg


Just for the record, the year is 2018 and we are gearing for a pervasive digital world dominated by software defined systems. The 4th Industrial Revolution’s floodgates are set wide-open.

Low and high tech perform best when playing a supporting role. Technology enables “Services” which justify it, otherwise the so-called Chasm and Valley of Death wait around the corner. It pays to emphasize that “Services” are defined by “Use Cases.” So, it shouldn’t take much effort to see that “Use(case)ability” (“usability” being the proper term) is a CSF, Critical Success Factor. “Fitness for use” in other words.

Let’s take that further and couple “usability” with designing for usefulness,” “utility,” “consumability & serviceablity” as well as “affectivity” because perception and human affects orient satisfaction and dissatisfaction levels.

QbD cannot be put to work without adequately addressing Human Dynamics, which entails psychological (e.g. cognitive models, information architecture) physiological (e.g. device form factor, workstation ergonomics) and social dimensions (e.g. network effects increasing value for users.) That happens to be the SoW (Scope of Work) of HFE’s (Human Factors Engineering) interdisciplinary teams in Design Studios… and the topic of my next post on QbD’s Intellectual Capital.


 

A few more thoughts…

In spite of one’s day-to-day work and/or belief system being either closer to or removed from the kinds of jobs and tasks that make tech human, it makes sense to engage in meaningful outcome oriented and goal driven practices by applying HCD, Human-Centered-Design. The purpose is delivering quality and achieving customer acceptance and delight, given that customers are human beings. That is the reason why Design Thinking has outgrown the field of industry design and is applied to a wide variety of domains and disciplines nowadays.

Tech’s roller-coaster industry is packed with well intended technologies that fail. We all know that this is a fiercely competitive environment in constant change. Though, it is also true that, in many of those cases, UX, User Experience, professionals were not engaged at any part of the process, or were purposely involved at the back-end, or were called to come to the rescue in the eleventh hour. That leaves no room for Design to make a difference. Superficial changes just amount to bells-and-whistles and shiny-objects to disguise the underlying quality issues that are likely to re-surface at some point.

QbD’s top objective should be excelling at effectively & efficiently addressing our customers’ acceptance and adoption criteria. That remains true even in the context of full automation. Humans still get promoted and demoted (or fired) based on those system’s performance. D. Newman’s recent article on Forbes magazine rightly states that “you cannot run your business without people (…) you cannot operate technology without people (…) research have shown that people are a critical component for digital transformation.”

Today’s best practice calls for “reverse engineering” solutions by working from that human-centered understanding around Human Machine Systems (HMS.) That is substantially different from only relying on a far riskier “if you build it, they will come” model and its costlier brute-force mindset. 

When dealing with challenging, intractable and complex projects, overlooking that fact typically results in exponential project risk and plenty of the, otherwise, avoidable zig-zagging course corrections ahead (e.g. opportunity costs in financial analysis and hidden and latency costs in systems engineering.)

Agile’s iterative development and ability to pivot shouldn’t be a refuge for either subpar or no design effort, but a vehicle to best implement QbD and augment development capacity while minimizing technical debt. This is why this revision of QbD for today’s tech industry calls for Design Sprints to lead the way.

Last but not least, before dismissing this QbD revision as a philanthropic and humanistic only endeavor, I suggest deep thinking around its (1) business criticality and (2) contribution to risk mitigation.

 

J. de Francisco

Bell Labs, Distinguished Member of Technical Staff

Nokia Software, Senior Studio Director @ Solutions Engineering


Disclaimer:

The above comments are my own and I welcome your feedback on LinkedIn ‘s Messaging and Nokia’s Yammer, which can double as input for further revisions as well as collaboration opportunities.


References:

A. Cooper. The Inmates are Running the Asylum. Why High-Tech Products Drive Us Crazy and How to Restore the Sanity, Sams Publishing, 2004.

D. Newman. 3 Reasons People are Critical for Digital Transformation Success. Forbes, June 2018.

J. de Francisco. IEEE ETR 2018, Emerging Technologies Reliablity Roundtable – Human Factors Session (2). Innovarista: Innovation at Work, July 2018 innovarista.org

J. de Francisco. IEEE ETR 2018, Emerging Technologies – Human Factors Session. Innovarista: Innovation at Work. May 2018 innovarista.org

J.M. Juran. Juran on Quality by Design: the New Steps for Planning Quality into Goods and Services, The Free Press, 1992.