Why is software engineering so difficult? Is this even engineering?
How to build good software
Things you should never do
No Silver Bullet
The Architecture of Open Source Applications
Real software engineering
essential difficulty: invisibility
We speak human language to communicate ideas. Machines understand code. There is a gap between the two.
The reality of software though is not inherently embedded in space, and thus has no ready geometric representation.
Sometimes pictures can help in presenting ideas. But software visualizations will never be fully accurate.
essential difficulty: conformity and scope
It is import to understand the problem before writing code. But how well do we understand the requirement?
We use prototyping to discover potential solutions. But there is more: prototyping also helps us understanding the problem and identifying new requirements. Testing, at the same time, validates the solution. Development cycles should alternate between those. With other words, consider exploration and consolidation phases.
Assuming that we completely understand all the requirements, software products still exist in a constantly changing environment: users, stakeholders and infrastructure changes over time. (software engineering is a service industry?)
essential difficulty: changeability
How do we know that a software product is complete?
Software is constantly subjected to pressures for change
Software is constantly changing? We need to prepare for that.
But people tend to value known risks over unknowns. It makes us feel smart to think about risks others may not. As a result, we underestimate the unexpected. We rarely hear the following during roadmap planning: we don't need to prepare for that, let's just save those resources for the unexpected.
100:10:1 method: Brainstorm 100 things that could go wrong. Pick 10 on that list that feel like the most likely and investigate them. Find the 1 critical problem you’re going to focus on.
essential difficulty: complexity
What is software complexity? How can we measure that?
Software is limited not by the amount of resources put into building it but complexity.
Solution: alternate between extending and reducing complexity.
Code is not an asset, it is a liability. The more code you have, the more you have to optimize, debug, modernize, refactor, understand. In short, the more code you have — the harder it is to change. A smaller and simpler codebase is easier to understand, improve and extend. - Paul Copplestone
Programmers are most effective when they avoid writing code.
How to measure productivity?
Productivity varies tremendously across the profession, but it may not vary so much within a given company.
Maximising developer productivity is not the goal. Build things that will be used. It's not a problem if a team builds less. Estimate feature value and focus on delivering that. How to calculate feature value? It depends, but it's sure that it needs to be delivered first.
How to improve productivity? Reuse.
The best code is the code that you don't write.
Cut features that don't add value.
“The idea is to start with the desired customer experience when designing new products, going so far as to draft “a press release that literally announces the product as if it were ready to launch and an FAQ anticipating the tough questions.” - Amazon working backwards
Software is largely a service industry operating under the persistent but unfounded delusion that it is a manufacturing industry. — The Cathedral and the Bazaar by Eric S. Raymond
How to find a direct competitor or alternative if it is not clear what problem the software solves?
See frameworks and obscurantism for more.
The main reason why software engineers earn more than other engineers because software is easy to sell.
My own experience, validated by Cockburn’s thesis and Frederick Brooks in No Silver Bullet, is that software development projects succeed when the key people on the team share a common vision, what Brooks calls “conceptual integrity.” This doesn’t arise from any particular methodology, and can happen in the absence of anything resembling a process. – Why don't software development methodologies work
Use the feedback engine effectively. Cost of change is effected by the length of the feedback loop.
- avoid mutable states
- stop using unsafe languages – [Glenn Vanderburg - Real Software Engineering]
Development cycles alternate between:
What is the goal? Feedback loop on different scales.
- A software system can best be designed if the testing is interlaced with the designing instead of being used after the design.
- A simulation which matches the requirements contains the control which organizes the design of the system.
- Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself. – Alan Perlis
How big an increment should be?
Ship as soon as possible. Because shipping means more feedback.
Shipping early, frequently, and on a cadence allows you to shorten the time to feedback and accelerate getting you to where you really need to be.
– Brendan O'Leary - What I learned at GitLab that I don't want to forget
Why are we doing these iterations? To get feedback on our decisions.
Architecture is a buzzword? Equals software design? It is team responsibility, everyone can do that on a certain level.
Remember that the builder has the inventive and creative responsibility for the implementation; so the architect suggests, not dictates. – The Mythical Man-Month
But communicating ideas during software design is hard. Separated architects perform worse, it is best done by a team.
Minimalism improves quality:
Size is the best predictor of code quality
A long paper trail of software engineering studies has shown that many internal code metrics (such as methods per class, depth of inheritance tree, coupling among classes etc.) are correlated with external attributes, the most important of which is bugs. What the authors of this paper show is that when they introduce a second variable, namely, the total size of the program, into the statistical analysis and control for it, the correlation between all these code metrics and bugs disappears. However, this paper uncovers a strong association between most code metrics and the size of the code, both of which together lead to the externally observable attributes like bugs - Source
AI agents fail miserably when trained with misspecified the reward functions. This also applies to software engineering: it doesn't matter if we have 100% test coverage, use proper semantic versioning for all software components.
Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while. You might get a few more after you achieve a certain level of stability and maturity, but the general tendency is to overestimate the contents of your wallet
The Not-Invented-Here Syndrome is considered a classic management pathology, in which a team refuses to use a technology that they didn’t create themselves.
But at the same time:
Find the dependencies — and eliminate them. - In defense of not invented here syndrome
From We are not special
A software component rarely specifies tolerancy and it does not wear out (unlike most physical components). Connection to the real physical world can be explicitly declared and controlled.
Commonly heard: Why don't we fix this hardware issues from software? Software can be changed fast. Also changes can be reversed easily.
This leads to ever chainging requirements? If it is possible, then why not? How much does it cost to build a prototype?
Less hard constraints. If it is a hard constraint, software engineers turn to standards and real-time.
The Rational design process is a caricature for all engineering disciplines, real world software engineering is just one adaptation.
Ultimately it's your application/system/product that needs to work.