8 Techniques
8.1 Patterns for success
Author: Peter Hintjens, key engineer of ZMQ
Patterns for success in software engineering aim to capture the essence of what divides glorious success from tragic failure. They were described as “religious maniacal dogma” by a manager, and “anything else would be effing insane” by a colleague, in a single day. For me, they are science. But treat the Lazy Perfectionist and others as tools to use, sharpen, and throw away if something better comes along.
– Peter Hintjens
8.1.1 The Lazy Perfectionist
Never design anything that’s not a precise minimal answer to a problem we can identify and have to solve.
The Lazy Perfectionist spends his idle time observing others and identifying problems that are worth solving. He looks for agreement on those problems, always asking, “What is the real problem”. Then he moves, precisely and minimally, to build, or get others to build, a usable answer to one problem. He uses, or gets others to use those solutions. And he repeats this until there are no problems left to solve, or time or money runs out.
8.1.2 The Benevolent Tyrant
The control of a large force is the same principle as the control of a few men: it is merely a question of dividing up their numbers.
– Sun Tzu
The Benevolent Tyrant divides large problems into smaller ones and throws them at groups to focus on. She brokers contracts between these groups, in the form of APIs and the “unprotocols” we’ll read about in the next chapter. The Benevolent Tyrant constructs a supply chain that starts with problems, and results in usable solutions. She is ruthless about how the supply chain works, but does not tell people what to work on, nor how to do their work.
8.1.3 The Earth and Sky
The ideal team consists of two sides: one writing code, and one providing feedback.
The Earth and Sky work together as a whole, in close proximity, but they communicate formally through issue tracking. Sky seeks out problems from others and from their own use of the product and feeds these to Earth. Earth rapidly answers with testable solutions. Earth and Sky can work through dozens of issues in a day. Sky talks to other users, and Earth talks to other developers. Earth and Sky may be two people, or two small groups.
8.1.4 The Open Door
The accuracy of knowledge comes from diversity.
The Open Door accepts contributions from almost anyone. She does not argue quality or direction, instead allowing others to argue that and get more engaged. She calculates that even a troll will bring more diverse opinion to the group. She lets the group form its opinion about what goes into stable code, and she enforces this opinion with help of a Benevolent Tyrant.
8.1.5 The Laughing Clown
Perfection precludes participation.
The Laughing Clown, often acting as the Happy Failure, makes no claim to high competence. Instead his antics and bumbling attempts provoke others into rescuing him from his own tragedy. Somehow however, he always identifies the right problems to solve. People are so busy proving him wrong they don’t realize they’re doing valuable work.
8.1.6 The Mindful General
Make no plans. Set goals, develop strategies and tactics.
The Mindful General operates in unknown territory, solving problems that are hidden until they are nearby. Thus she makes no plans, but seeks opportunities, then exploits them rapidly and accurately. She develops tactics and strategies in the field, and teaches these to her soldiers so they can move independently, and together.
8.1.8 The Constant Gardener
He will win whose army is animated by the same spirit throughout all its ranks.
– Sun Tzu
The Constant Gardener grows a process from a small seed, step-by-step as more people come into the project. She makes every change for a precise reason, with agreement from everyone. She never imposes a process from above but lets others come to consensus, and then he enforces that consensus. In this way, everyone owns the process together and by owning it, they are attached to it.
8.1.9 The Rolling Stone
After crossing a river, you should get far away from it.
– Sun Tzu
The Rolling Stone accepts his own mortality and transience. He has no attachment to his past work. He accepts that all that we make is destined for the trash can, it is just a matter of time. With precise, minimal investments, he can move rapidly away from the past and stay focused on the present and near future. Above all, he has no ego and no pride to be hurt by the actions of others.
8.1.10 The Pirate Gang
Code, like all knowledge, works best as collective–not private–property.
The Pirate Gang organizes freely around problems. It accepts authority insofar as authority provides goals and resources. The Pirate Gang owns and shares all it makes: every work is fully remixable by others in the Pirate Gang. The gang moves rapidly as new problems emerge, and is quick to abandon old solutions if those stop being relevant. No persons or groups can monopolize any part of the supply chain.
8.1.11 The Flash Mob
Water shapes its course according to the nature of the ground over which it flows.
– Sun Tzu
The Flash Mob comes together in space and time as needed, then disperses as soon as they can. Physical closeness is essential for high-bandwidth communications. But over time it creates technical ghettos, where Earth gets separated from Sky. The Flash Mob tends to collect a lot of frequent flier miles.
8.1.12 The Canary Watcher
Pain is not, generally, a Good Sign.
The Canary Watcher measures the quality of an organization by their own pain level, and the observed pain levels of those with whom he works. He brings new participants into existing organizations so they can express the raw pain of the innocent. He may use alcohol to get others to verbalize their pain points. He asks others, and himself, “Are you happy in this process, and if not, why not?” When an organization causes pain in himself or others, he treats that as a problem to be fixed. People should feel joy in their work.
8.1.13 The Hangman
Never interrupt others when they are making mistakes.
The Hangman knows that we learn only by making mistakes, and she gives others copious rope with which to learn. She only pulls the rope gently, when it’s time. A little tug to remind the other of their precarious position. Allowing others to learn by failure gives the good reason to stay, and the bad excuse to leave. The Hangman is endlessly patient, because there is no shortcut to the learning process.
8.1.14 The Historian
Keeping the public record may be tedious, but it’s the only way to prevent collusion.
The Historian forces discussion into the public view, to prevent collusion to own areas of work. The Pirate Gang depends on full and equal communications that do not depend on momentary presence. No one really reads the archives, but the simply possibility stops most abuses. The Historian encourages the right tool for the job: email for transient discussions, IRC for chatter, wikis for knowledge, issue tracking for recording opportunities.
8.1.15 The Provocateur
When a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.
– Samuel Johnson
The Provocateur creates deadlines, enemies, and the occasional impossibility. Teams work best when they don’t have time for the crap. Deadlines bring people together and focus the collective mind. An external enemy can move a passive team into action. The Provocateur never takes the deadline too seriously. The product is always ready to ship. But she gently reminds the team of the stakes: fail, and we all look for other jobs.
8.1.16 The Mystic
When people argue or complain, just write them a Sun Tzu quotation.
– Mikko Koppanen
The Mystic never argues directly. He knows that to argue with an emotional person only creates more emotion. Instead he side-steps the discussion. It’s hard to be angry at a Chinese general, especially when he has been dead for 2,400 years. The Mystic plays Hangman when people insist on the right to get it wrong.
8.2 Object-Oriented Patterns
8.2.1 Command-Query Separation
Data structures are state. Therefore, passing around data structures means sharing state, and shared state is the root of all evil. The reason OOP objects were invented was to provide a paradigm where shared state could be minimized and controlled (that’s why we should Package Wisely).
– https://hackernoon.com/objects-vs-data-structures-e380b962c1d2
The two strategies that have so far proven somewhat effective at avoiding bugs induced by state sharing are:
- Statically forbid mutable data (à la Haskell)
- Statically forbid aliasing of mutable references (à la Rust)
Functions that change state should not return values and functions that return values should not change state.
– hackernoon.com, more: Martin Fowler
The really valuable idea in this principle is that it’s extremely handy if you can clearly separate methods that change state from those that don’t. This is because you can use queries in many situations with much more confidence, introducing them anywhere, changing their order. You have to be more careful with modifiers.
What are the two basic assumptions of Haskell as a “pure” language?
Haskell is a pure language, which means that the result of any function call is fully determined by its arguments. Pseudo-functions like rand() or getchar() in C, which return different results on each call, are simply impossible to write in Haskell. Moreover, Haskell functions can’t have side effects, which means that they can’t effect any changes to the “real world”, like changing files, writing to the screen, printing, sending data over the network, and so on. These two restrictions together mean that any function call can be replaced by the result of a previous call with the same parameters, and the language guarantees that all these rearrangements will not change the program result!
8.3 TDD: Producing reliable software
Sometimes we wonder what the value of TDD is.
Test-driven development (TDD) is a way of managing fear during programming. I don’t mean fear in a bad way, pow widdle prwogwammew needs a pacifiew, but fear in the legitimate, this-is-a-hard-problem-and-I-can’t-see-the-end-from-the-beginning sense. If pain is nature’s way of saying “Stop!”, fear is nature’s way of saying “Be careful.” The problem is that fear has a host of other effects:
- Makes you tentative,
- Makes you grumpy,
- Makes you want to communicate less,
- Makes you shy from feedback.
None of these effects are helpful when programming, especially when programming something hard. So, how can you face a difficult situation and
- Instead of being tentative, begin learning concretely as quickly as possible.
- Instead of clamming up, communicate more clearly.
- Instead of avoiding feedback, search out helpful, concrete feedback.
- (You’ll have to work on grumpiness on your own.).
Imagine programming as turning a crank to pull a bucket of water from a well. When the bucket is small, a free-spinning crank is fine. When the bucket is big and full of water, you’re going to get tired before the bucket is all the way up. You need a ratchet mechanism to enable you to rest between bouts of cranking. The heavier the bucket, the closer the teeth need to be on the ratchet.
The tests in test-driven development are the teeth of the ratchet. Once you get one test working, you know it is working, now and forever. You are one step closer to having everything working than you were when the test was broken. Now get the next one working, and the next, and the next. By analogy, the tougher the programming problem, the less ground should be covered by each test.
Source: “TDD By Example”, URL
- How many test do I have to write?
- The minimum amount that lets you write all the production code. The minimum amount, because every test slows down refactoring (when you change production code, you have to fix all the failing tests). On the other hand, refactoring is much simpler and safer on code under tests.
Main idea and discussion on software reliability testing is here.
In testing of web applications, TDD seems the most straightforward way to test is just to check whether objects that should be visible, eg. waypoints in a map, are visible.
One way to test this is by using Luminoth for testing by machine object detection: Luminoth.
8.4 Functional programming
const I = x => x;
const K = x => y => x;
const A = f => x => f(x);
const T = x => f => f(x);
const W = f => x => f(x)(x);
const C = f => y => x => f(x)(y);
const B = f => g => x => f(g(x));
const S = f => g => x => f(x)(g(x));
const P = f => g => x => y => f(g(x))(g(y));
const Y = f => (g => g(g))(g => f(x => g(g)(x)));Source: Common combinators in JavaScript, https://gist.github.com/Avaq/1f0636ec5c8d6aed2e45
8.5 Troubleshooting
Individual Contributors often get sidetracked by:
- Brainstorming/architecture: “I must have thought through all edge cases of all parts of everything before I can begin this project”
- Researching possible solutions forever (often accompanied by desire to do a “bakeoff” where they build prototypes in different platforms/languages/etc)
- Refactoring: “this code could be cleaner and everything would be just so much easier if we cleaned this up… and this up… and…”
- Helping other people instead of doing their assigned tasks
- Jumping on fires even when not on-call
- Working on side projects instead of the main project
- Excessive testing (rare)
- Excessive automation (rare)
Individual Contributors often get stuck when they need to:
- Finish the last 10–20% of a project
- Start a project completely from scratch
- Do project planning (You need me to write what now? A roadmap?)
- Work with unfamiliar code/libraries/systems
- Work with other teams (please don’t make me go sit with data engineering!!)
- Talk to other people (in engineering, or more commonly, outside of engineering)
- Ask for help (far beyond the point they realized they were stuck and needed help)
- Deal with surprises or unexpected setbacks
- Navigate bureaucracy
- Pull the trigger and going into prod
- Deal with vendors/external partners
- Say no, because they can’t seem to just say no (instead of saying no they just go into avoidance mode, or worse, always say yes).
Source: Elided Branches 24
8.6 System modeling and analysis
I have been having good luck with a somewhat different approach, first used in 1960 to debug anALGOLcompiler. The idea is to construct a test file that is about as different from a typical user application as couldbe imagined. Instead of testing things that people normally want to do, the file tests complicated things thatpeople would never dare to think of, and it embeds these complexities in still more arcane constructions.Instead of trying to make the compiler do the right thing, the goal is to make it fail (until the bugs have all been found).
(…)
This method of debugging, combined with the methodology of structured programming and informalproofs (otherwise known as careful desk checking), leads to greater reliability of production software thanany other method I know.
– Donald Knuth, A torture test for TEX, Stanford University, January 1990 25.
Petri nets were originally invented for use in describing chemical reactions. At one point, their usefulness for modeling and analysis of computer systems was determined. This model reaches as far as forming learning Petri nets. This was described by InTechOpen 26.
8.7 Competitive Programming
Irrespective of the problem category, the process of solving a problem can be divided into two broad steps: constructing an efficient algorithm, and implementing the algorithm in a suitable programming language (the set of programming languages allowed varies from contest to contest). These are the two most commonly tested skills in programming competitions.
– https://en.wikipedia.org/wiki/Competitive_programming
8.7.3 Levels
- ICPC
- Regional ICPC,
- Topcoder,
- Codechef,
- SPOJ
Topics:
- Graph algorithms,
- Dynamic programming,
- Searching and sorting,
- Number theory and other mathematical concepts,
- Geometrical and network flow algorithms,
- Data structures.
– Top 10 Algorithms and Data Structures for Competitive Programming
Baseline: ACM ICPC
The ACM ICPC is considered as the “Olympics of Programming Competitions”. It is quite simply, the oldest, largest, and most prestigious programming contest in the world.
– CodeChef, ICPC
Mission:
The ICPC, the “International Collegiate Programming Contest”, is an extra-curricular, competitive programming sport of the universities of the world. ICPC competitions provide gifted students with opportunities to interact, demonstrate, and improve their teamwork, programming, and problem-solving prowess. The ICPC is a global platform for academia, industry, and community to shine the spotlight on and raise the aspirations of the next generation of computing professionals as they pursue excellence.
Registration:
If a team wishes to be able to qualify for the ICPC World Finals, then before competing in ANY regional qualifying event, the coach and all team members must be fully registered in the ICPC Registration System. Registration may not be done retrospectively. Incomplete registration or circumvention that leads to incomplete or false data is grounds for immediate disqualification.