Why Do So Many IT Projects Fail – and What Can We Do About It?

IT projects have an astonishingly high failure rate. According to the CHAOS Report by the Standish Group, only 31% of IT projects are considered successful, while 50% face significant challenges, and 19% fail completely. The common reasons cited include poor communication, unclear requirements, and unrealistic timelines. However, proposed solutions are often generic and fail to address deeper issues.

To better understand the problem, it helps to look at other industries. Large-scale construction projects frequently experience delays and cost overruns—examples include Berlin Brandenburg Airport (BER) or Stuttgart 21. Why does this happen? Because such projects are incredibly difficult to plan, filled with unexpected obstacles, and involve decision-making by too many stakeholders, often without a clear overarching perspective.

But there are also success stories. What do they do differently? When applied to IT, several key challenges emerge.

Software Development Is Too Abstract

Software development is difficult to grasp. In many industries, specialists make most of the critical decisions themselves. In IT, however, the people working on the technology every day rarely have decision-making power.

Instead, key decisions are often made by individuals without deep technical expertise. A common scenario is when major technological choices are made before a project’s actual requirements are fully understood.

Real-World Example

A company decides that its new project will use an Oracle database, a microservices architecture, and a hybrid app. The reasoning?

• The company has been an Oracle partner for years.

• Microservices are currently seen as an industry best practice.

• A single hybrid app seems more efficient than separate mobile and web applications.

At first glance, this may seem logical. However, from a technical perspective, potential problems quickly arise:

• A NoSQL database might be a better fit.

• Microservices could add unnecessary complexity.

• If 95% of users will use the mobile app, a streamlined web version might suffice.

As a result, the project starts with significant disadvantages even before development begins. A similar issue occurs when a project deadline is set before a single requirement has been clearly defined.

Lack of Expertise in Key Roles

IT is an incredibly broad field with countless specializations. However, many job postings list unrealistic and overly long requirements.

Example Job Posting

• Java, JEE, Spring, Spring Boot

• Angular, ULC/Swing

• JNDI, JCA, JPA, SOAP, REST, JSON, JMS, BPMN, DMN

• SQL/PLSQL, Junit, GIT, Bitbucket, SonarQube, Jenkins, Ant, Maven

• Self-motivation, innovative thinking, strong communication skills

This list combines multiple different domains—Java backend, frontend (Angular), cryptography, DevOps, databases, and business process modeling. A single person is unlikely to have deep expertise in all these areas. Yet, companies often seek such “all-rounders” instead of assembling a team of specialists with different strengths.

Other industries handle this more effectively. A surgeon doesn’t perform an operation alone—they have an anesthesiologist, assistants, and nurses. So why is IT expected to function differently?

Unclear Requirements and Insufficient Preparation

While developers are expected to meet high technical standards, many projects lack a clear and well-defined set of requirements.

• Key project stakeholders may not have deep technical or domain expertise.

• Requirements are often vague or incomplete.

• Critical questions remain unanswered.

Why Is This a Problem?

When developers receive unclear or incomplete requirements, they are forced to make assumptions and resolve ambiguities during implementation. This can lead to misunderstandings, delays, and additional costs. In a well-structured project, it is essential to ensure that requirements are clearly documented so that implementation can proceed efficiently.

Lack of Feedback from Business Experts

Business departments are often only minimally involved in the software development process. Many domain experts already have a full-time workload and cannot dedicate enough time to IT projects.

As a result, software is often developed without sufficient real-world feedback. Many business teams test the product only at the very end—by then, changes are expensive and difficult to implement.

The Politics Behind IT Projects

IT projects are expensive. Millions of euros can be spent on a single development initiative, and no one wants to give up control over such budgets.

Budget allocation is often driven more by internal politics and company regulations than by actual technical or business needs.

Some Common Mistakes Include:

• Budgets are distributed based on internal power structures, not actual project needs.

• The cheapest provider is chosen, even if the total cost ends up being much higher.

• Unrealistic budgets lead to artificially low project bids, only for costs to increase through change requests later.

In many cases, contracts are awarded not to the best experts but to those with the right connections.

Successful IT Projects – What Can We Learn?

Not all IT projects fail. Some companies have found ways to navigate complexity and deliver successful solutions.

1. The Mars Rover Software (NASA)

NASA’s Mars Rover software is an outstanding example of a successful IT project. It was developed with an agile approach, where continuous testing, incremental updates, and strict risk management were key success factors. Unlike typical IT projects with rigid deadlines, NASA built flexibility into its development cycles, allowing for rapid adaptation to new challenges.

2. Netflix’s Migration to the Cloud

When Netflix moved its entire infrastructure to the cloud, it faced an enormous technical challenge. Instead of following a fixed roadmap, Netflix adopted a gradual, iterative approach, testing and refining its architecture over several years. This allowed for continuous improvement and risk minimization, proving that large-scale IT projects can be successful when they focus on adaptability rather than rigid planning.

Actionable Steps for More Successful IT Projects

1. Involve developers early – but strategically. Developers should not be treated as mere “code monkeys.” Instead of staffing an entire team from the start, companies should bring in technical experts early to optimize processes before software development begins.

2. Focus on small, testable feature sets. Instead of planning everything in advance, prioritize small releases that deliver real user value and collect feedback early.

3. Allocate budgets in shorter cycles. Instead of locking in funding for years, allocate budgets for a few months at a time, reassess progress, and adjust as needed.

4. Stop planning the unplannable. Software development is more like research than manufacturing. Instead of rigid blueprints, teams should iteratively refine solutions in small steps based on real-world testing and feedback.

Conclusion: Agile, But With Common Sense

IT projects should not be treated as rigid, blueprint-driven processes. Frameworks like Scrum or SAFe are tools, not strict rules. Companies that embrace flexible, user-centric, and technically informed development have a far better chance of delivering valuable software.

The key question every company should ask: “Do we have a real learning process in place, or are we just following a predetermined plan?”

Letting go of the illusion that software can be fully planned from the start is the first step toward successful IT projects.