Use whatever feedback loop is most efficient (speed / accuracy).

Sometimes that's building a full app and getting it to market. Sometimes that's isolating a bug in a legacy system to a unit test. Sometimes its the compiler failing to build your code. Sometimes it's a data science problem, and it means training a model in a notebook, to measure its accuracy.

Don't get hung up on using one particular feedback loop.

Getting stuck in the wrong feedback loop is a big anti pattern of software development. When it takes forever to submit a job to some system, then takes 20 minutes to give you a type error in a scripting language, that could be caught by a linter or type checker sooner. Those 20 minutes easily turn into 60 minutes of distraction. Imagine instead you could catch this in 5 seconds with a test / linter / whatever. Then you'd resolve the issue in minutes. OTOH spending hours decomposing simple code into unit tests can be a waste of time, when the most value is submitting that simple code to a batch processing system, to see if it produces output...

100% agree. The faster you can make your feedback loop, the more productive you will be. That’s why automated tests can be a super power. But only (as with everything else) when done right. So how do you know you are doing it right? When your automated tests save more of your time than the time it takes to maintain them. It’s really that simple.

In my experience I get those benefits when testing at the system/use case level but not when doing unit tests. At that higher level automated tests works wonderfully. I haven’t had a bug in production for 5+ years because of test automation. Which means that I spend 95% or more of my time adding new features or optimisations instead of bug hunting and/or getting shouted at because of customer problems. Love it!

You can't "drive the design" with TDD. To write a failing test, you have to have a requirement which is encoded by the test. That requirement doesn't come from TDD, and TDD can't tell you how to procure it.

To go from a high level requirement like "command line application to track transactions and show expenses by category", you need to go through several levels of detailed design before you have anything that is implementable by a test-driven approach, which will necessarily be bottom-up.

Remember that tests can only confirm the presence of a defect, not its absence, and that not everything can be tested due to the intractability of exhaustive testing.

By blindly following test-driven design, you could end up with a useless, inflexible program that correctly categorizes expenses over only a small, fixed vocabulary of categories because it never occurred to anyone that a category must be something user-defined.

A user-defined category feature cannot be implemented with strict TDD because the space of user-defined categories is infinite. Say you have a test which confirms operation for every valid category string up to three characters long. That doesn't prove the system will not fail for a four-character-long category.

The only people who regard TDD as some kind of panacea are those with no formal background of any kind, who don't understand the role of testing in the overall software engineering context, and what its limitations are.

That's very false in my experience, and I did 100% TDD for years (have a more nuanced approach currently in TypeScript/React)

As a programmer with a problem, you first instinct is to start at the solution.

TDD pulls you back, and you first have to write the api and decide how you verify it.

If you're doing it well, you make both of those things as simple as possible, first. Reduce dependencies and inputs, etc.

That's actually the most important part, an easily consumable api that's easy to verify, not the implementation, and TDD forces you to do it first.

Also, a pet peeve of mine is seeing a line or more of code that doesn't do anything. With TDD, such detritus is impossible as you can't write production code unless it's making a red test turn green.

Even a string length api like len(s) is not exhaustively testable; you can't prove it correct with blackbox tests.

TDD has refactoring steps which only have to preserve passing tests; refactoring can easly be the vector that introduces dead code as well as changes behavior for untested input combinations.

I suspect that a lot of code developed TDD is actually deployed on input combinations that are not covered in the TDD test suite.

A string length function developed by TDD will still work on len("supercalifragilisticexpealidocious") even though that exact string never appeared as a test case, and the consumers of that function will use it on all sorts of untested inputs all the time.

> Even a string length api like len(s) is not exhaustively testable;

But this has nothing do with what GP said. That isn't what gp is using TDD for. GP is using TDD for,

1. starting with api of a function vs code.

2. avoiding unused lines of code

3. reducing dependencies, its hard to tdd something that has a lots of deps so it drives your refactor it something more testable and thus more readable/maintainable also.

4. simplest set of input/output.

A very good explanation. Yet what I think is confusing a lot of (very?) good programmers (not me) is that they already do that intuitively by not letting methods become longer than 3 or 4 lines, and whenever implementing a new method, by writing the return statement first (i.e. returning a mock object or similar) and working backwards, i.e. only writing enough code to be able to return the desired value.

If one then goes and tries to write tests for these kind of methods, it feels like superfluous busywork.

That's a good thing, TDD is a prompt to say you don't really understand this requirement. Requirements aren't a design.

TDD makes you define what your input is, and what your output is. What's in the middle your free to choose.

Not necessarily. There are two schools of thought, Outside In TDD and Inside Out.

Don't think in terms of tests -- think in terms of examples.

Write examples of how you would like to be able to solve common cases and handle edge cases if you had the ideal interface based on what you know at the time, then use unit tests with stubbed out interfaces as a tool while trying to develop the implementation.

Refine and even rewrite your examples as you learn about what works and what doesn't work.

Recursively break down complex pieces into smaller modules as you learn what can be handled independently and composed with the other pieces. Break off independent examples depicting how you expect those modules to work and be used as new unit test suites for those new modules.

Don't sweat testing the implementation details, especially when prototyping. What you want is to use tests as a way to speed up your development process and improve the quality of your design and architecture.

Try to write tests that show off how nice your code is to use and how few edge cases you need to handle when calling it. It'll raise the bar for how your code interacts with the rest of the application or system and give you smaller pieces to work and focus on.

Finally, for really green field stuff with a lot of unknown unknowns, you often have to write a lot of standalone, bottom-up throw-away prototypes for things. That doesn't invalidate the usefulness of tests, so don't be afraid to do that pretty regularly.

This is very interesting. I need to try your ideas. Thanks!

That’s a great pragmatic way to think about it.

A year into a novel project, a teammate expressed their concern about a lack of unit tests. They'd even spent time creating a bunch early on. I asked if they still ran or were even relevant any longer and the answer was no.

Particularly with exploratory projects, I'd recommend focusing on rapidly iterating to reach the simplest solution (businesswise and implementation-wise).

If in the end non-trivial logic must be specified, write some tests. Your definition of 'non-trivial' will change with experience.

There's this stupid dogma: tests are good, codebase without tests is bad. It is incredibly hard to fight the idea of "we need more tests" because of that. Anytime engineering team gets some freedom, tests are usually the first thing they decide to work on.

But tests aren't inherently good. Good tests will speed you up, bad ones will slow you down. And it is incredibly hard to write good tests. Might be even harder than writing software that tests supposed to test:

If software works in 80% of cases, it creates 80% of value.

If tests work in 80% of cases, they cause harm in the remaining 20%.

In my experience tests also slow refactoring down, despite the common mantra claiming the opposite.

Want to extract some logic into a new interface or introduce a new parameter to a method?

Well, now instead of just doing that, you will also have to update all tests dealing with said logic.

At my previous job I spent much more time updating tests testing trivial shit than actually writing useful code.

Exactly my experience. Numerous times I found ways to reduce accidental complexity in software. But guess what, my change breaks 50 tests, because accidental complexity happened to be under test. What I anticipated to be an enjoyable 2-hour refactoring turns into 3-day chore. I guess I'll just leave the complexity in there.

Sure, those were "bad" tests. Accidental complexity is not supposed to be tested. And yet I had this experience in every single company I worked at. Maybe I'm just incredibly unlucky. Or maybe there's something wrong with "tests are good" mantra.

"Write tests. Not too many. Mostly integration"

Tests make refactoring slower, because you might have to also update the tests. On the other hand, in a complex enough code base, refactoring is impossible without sufficient test coverage. You can rely on the tests to catch edge cases that you weren't aware of in areas of products built on top of architecture you are maintaining.

Sometimes, all 50 of those stupidly over-complex tests that people have copy-pasted around are garbage that get in the way of your refactor, but sometimes one of them is protecting some important functional requirement that you didn't think about like "Test that this code called from a tight loop doesn't query the db hundreds of time which would lead to the startup time taking minutes rather than seconds".

> Tests make refactoring slower, because you might have to also update the tests

No, tests can make refactoring slower, because they often test implementation detail instead of interface. Being able to distinguish the two is not as easy as it sounds, typically it requires deep understanding of the domain, which people writing tests might not have yet.

> in a complex enough code base, refactoring is impossible without sufficient test coverage

Sure, and in my experience complex code bases are typically complex due to the amount of accidental complexity, not because they actually solve complex problems. Like layers of overengineered classes intertwined through dependency injection for the sake of being more... testable. See the irony?

And indeed, refactoring becomes quite hard. You have two options to make it easier:

1. Identify and remove accidental complexity.

2. Cover it with tests and set the complexity in stone.

It's like fighting cancer with painkillers. It might look like it's working, but you going to die pretty soon.

> Test that this code called from a tight loop doesn't query the db hundreds of time which would lead to the startup time taking minutes rather than seconds

Oh, this is a great example. I saw the exact same thing implemented couple years ago. The whole team hated it because it caused tests to break constantly. And yet nobody on the team had the balls to remove it, and they were even following the pattern for new tests, because past tests had it.

If startup time matters for your app, there will be plenty of signals apart from tests that it's broken. If startup time doesn't matter, why are you testing it?

By the way, you haven't even noticed it, but you conflated startup time with number of queries. You're testing the wrong thing. What if hundreds of queries are still blazingly fast (due to being simple and/or cacheable) and have negligible effect on startup time? What if existing startup time is already extremely high (e.g. you're loading ML models in memory)?

If startup time truly matters for you, you shouldn't be testing it, you should be monitoring it. And I'm not even talking about some complex monitoring systems. When someone on your team says their developer experience got worse, that's the type of alert you need to listen to.

Not my experience at all. I can only safely refactor major parts of the large complex C++ software I am maintaining because of the tests.

If you can’t refactor your code without breaking tests then your tests are testing implementation details and not higher level specification details. Your tests should test your use cases not your implementation details.

In other words, if you can’t refactor your code without breaking your tests then you are doing automatic testing wrong. It’s that simple.

> lack of tests will inevitably result in a slowdown

This is exactly the mindset I'm talking about. Lack of which tests? Manual? Unit? Functional? Integration? End-to-end? Doesn't matter, right? All tests have zero cost, the more the merrier.

And when exactly does the slowdown happen, on day one? Or maybe some time in the future? Do we have a way to tell when exactly should we write tests, how many, and which ones? Most of the time the mindset is "we don't have tests, therefore we need tests". Cargo cult at it's finest.

> Most tests written by people working on the product will of some baseline level of quality

Not in my experience. Tests are often introduced before you have a good understanding of the domain (hello TDD). Epitome of that is hiring junior engineers, realizing you don't want to assign any real work to them, and forcing them to write tests.

> It is much easier to fix bad test

Typically bad tests should not be fixed, they should be thrown away. But pretty much every company sucks at deleting bad tests, because there's the illusion of value they provide.

These old test are the fundament. Test_application_window_rendering may sound simple and useless now, but if this test fails something is really wrong. How else would you test something like dependency upgrades?

A lot of people giving their opinion about TDD, but not many actually answering your question.

The answer depends on your approach. You can go outside-in (top-down) or inside-out (bottom-up). The one you use is a matter of taste. A lot of people find outside-in easier, but inside-out can be a bit faster if you have a good sense of how your system will fit together.

To take an outside-in approach, start at the top level of your application. Possible just under the UI, because UIs can be hard to test. (If the UI isn’t tested, make it do as little work as possible, and have it just forward to the tested code to do real work.)

Test-drive a module, class, or method in that top layer. The smaller the better. If there’s something complicated that should be handled by another module/class/function/method, make the module/function/etc., but just hardcode one answer. Make a note of the things you hardcode.

When you’ve finished TDDing your first thing, look at your notes and choose something that you hardcoded. TDD that in the same way. Repeat recursively until the application works. Focus on getting to a working but incomplete application as soon as possible, by working depth-first instead of breadth-first.

Once you have a working but incomplete application—a “walking skeleton”—think of a feature you’re missing and add it in the same way. Continue until the application is done. Remember to look for opportunities to improve the design and refactor as you go.

Inside-out is very similar, except that you do the depth-first recursion in your head, mentally, and don’t write any code until you get to the leaves of the recursion tree. You test-drive the leaf, then test-drive its caller, etc., until you get to the top. It takes more experience, more thinking about design during TDD, and more willingness to refactor.

I have videos on my site if you’d like more. Start with this series:

https://www.jamesshore.com/v2/projects/lunch-and-learn

Thank you, this is a very useful answer. I will check your videos out for sure.

Don't let TDD destroy your architecture.

TDD encourages a form of programming where you build from the outside-in, building layer after layer of abstraction, putting off solving the actual problem until you get to the messy gooey centre where it ends up being a kludge. It's also very easy to make flawed assumptions in your original TDD spec which you don't realize until you get to that gooey centre, having wasted N days in the process.

I honestly don't tend to start anything until I have a basic idea of how it's going to work (not a fashionable idea these days), and find that projects end up with a much saner structure when they're built from the inside out.

So my preference is to start TDD once I've got a very basic version of something working, at which point you can start diving in to all the corner cases.

Seconded. Only ever add tests after you have a working version. Starting a greenfield project with TDD is a waste of time. Just get something out then start testing your assumptions with unit tests, even then, don't test the whole codebase, test your fundamental assumptions and core features (like user auth, roles, permissions and their invocations, esp if multi-tenanted).

The way to do it using TDD is to do your greenfield development to the point of a working solution to elicit your fine-grained requirements, then throw that away and rewrite it using TDD. The first version is considered a "spike solution" and is not supposed to touch production.

In TDD, all production code must be written against some pre-existing test. Any code which does not meet this criterion is broken.

I have never worked with that sort of time luxury.

It's frequently faster. Half-assed designs will cost you time forever.

Let's not start throwing pejoratives around, there is nothing half-assed about having an idea how your design will work before you blindly start coding probably-wrong stuff.

I've also found having apparent time-luxury to be a rather paradoxical thing, the more of it you have, the more people start spending it by creating bureaucracy and busy-work. It never gets spent how you expect it to be spent.

In all my years of coding, I only knew one developer who did TDD properly. He taught me, as I was struggling trying to track down an intermittent bug and nothing was working. Not going to lie -learning to do TDD properly is difficult. Since learning it, I've not met another dev who does it, and most devs barely write tests. Stick with it, practice. You'll eventually hit that ahah! moment.

A story: I used to work with some devs, most of them would finish coding something in 2 weeks and then spend a week debugging it. I'd take 4 weeks to TDD something. My boss was frustrated at my speed until one day a production system which I'd written, stopped working. Several million dollars was stuck in limbo, the backend team responsible for the paperwork unable to do anything. We had a new dev and the bandwagon was "TDD is crap!" was fully rolling that morning. I asked the backend processing person what had changed? The said we had a new client, they had a weird name. Looked into the client log, sure enough. Added a test with the client's name, and ran the tests. It spat out "1 failing test in module X at line Y because 'this'". Modified the code to handle that test, re-ran the tests, 100% pass. Emailed the results output to the team for a manual upload, processing done, committed and pushed to the production branch. The server would update after business hours. Diagnosed and fixed in 5 minutes. I turned to my colleague who I knew was still stepping through his code for 2 days and said (within earshot of the new dev) "How's that bug you've been hunting? Still not found it? This is the power of TDD. I might be slow at the start, but I fix bugs quickly". The dev manager had everyone take a day of TDD training after that, but nobody really understood. The trainer said my approach to TDD is too strict, with my team doing 110 test runs against the next team of 63 test runs. Our solution had only a few lines of code verses the paragraphs of the others though.

"The said we had a new client, they had a weird name."

This is the material piece of information which lead to the fix. Your TDD simply made it faster to locate the offending line.

Seems like a really poor justification for TDD imo.

I agree, this sounds like a completely normal process of finding and fixing bugs.

The stack trace tells you where the error is occurring, your intuition leads you to the relevant caller function that you should test, stack trace or logs should lead you to the offending parameters (i.e. the weird name).

With these three pieces of information you can write a failing test and implement a patch.

I don't consider this approach to be "TDD", just "software development with good test coverage".

> Not going to lie -learning to do TDD properly is difficult

> The dev manager had everyone take a day of TDD training after that, but nobody really understood

That is exactly the problem with TDD or any other methodology that is hard to get right. Methodology will either push to a pit of success, or a pit of despair. If it's hard to get right, pit of despair it is.

Force 100 people to do TDD. 95 of them will be harmed, 5 will succeed. Same if you do waterfall. Same if you do Visual Basic. Same if you don't do code reviews.

TDD is indistinguishable from any objectively malicious practice. Somehow the fact that few folks were able survive TDD, is twisted to present TDD as some kind of holy grail.

> my first test was along the lines of “it can categorise a single transaction correctly”... I was attempting a more “top down” design strategy here.

Sounds right to me! I usually start projects/features with a test that covers the end-to-end behavior I want to see. When you have no other tests, a smoke test is the highest value test you can add because it measures the thing you care about.

Only the most religious TDD evangelists will force you to "drive the design" using tests. That advice is not relevant for most developers except for perhaps absolute beginners who don't have the experience or mental models to determine what a good design is and TDD might force them to come up with a decent design by the sheer fact of making their design testable, which alone goes a long way (but not necessarily all the way) in making a "good design".

Most experience developers will come up with the architecture/design of what they are working on through a combination of intuition and experience, write tests after and then use TDD to fill the gaps once they have nailed down the architecture.

I do TDD some of the time and my workflow often look like this:

1. Start implementing an object/class, defining APIs of its key methods while having a higher level picture in my head of how it will fit with other components

2. Implement a good chunk if not all of that object (not more than a few 100 lines typically)

3. Write a test to verify that what I did worked.

4. Start commenting out parts of the code I wrote to make sure the applicable test cases fail. Uncomment after verification is done.

5. Use TDD to cover and implement edge case handling and anything leftover

6. Repeat 1-5 for each class/component/module until done

I only get into TDD once the architecture and high level design are finalized, and I have some code written for them and I have some reasonable level of confidence that it is not gonna drastically change. I also generally go "bottom-up" starting from the leafs of the object graph up and use TDD maybe ~50% of the time (just a rough guess).

Trying to force yourself to let "TDD drive the design" will likely result in frustration as you have experienced and slowdown.

If you do ‘red, green, refactor’ TDD and write clean ‘arrange, act, assert’ tests then you’ll likely create nice single responsibility classes that are a pleasure to test and extend.

The tests usually run lightning quick because the units will be small.

When bugs occur in your codebase it will be at the right level of abstraction to write tests for the bug.

I almost never do strict TDD, I hack away to get something going and use the tests to refactor the working code into clean srp code. The end result is the same. However where I would use strict TDD is when there is massive complexity (like creating a video encoder) or if you are following a published specification.

So ‘just barely working, green, refactor, else red, refactor, green’ is probably closer to my actual day-to-day. The working code is clean, the tests are clean, and the coverage is good. The order doesn't matter.

I am not trolling today, TDD doesn't make any sense to me, testing to fail and correcting it until it passes doesn't work with my flow.

In a DAW(Digital Audio Workstation), this is like adjusting the knobs, adding effects when you haven't even added any samples, it just doesn't make any sense, stop forcing yourself to do things that doesn't align with your flow, you'll likely get annoyed for no reason.

The way I work is I do the architecture design, I write the code, then I write my test by following the design graph.

If I don't know what the design would look like, I do what I call "code freestyling" to get a basic idea of what the overall design would look like then I repeat the above step.

Is there a specific tool you use to figure out your design graph, or is it more a 'mind' thing? I know there's some vscode plugins to kinda visualize how code fits together, etc.

It's more of like doing system designing (pre-planning), the system design stage is way of looking at the bigger picture by connecting the dots, and yh, it could be a mind thing but I mostly use my pen and paper or if it's something that's really difficult to nail, I use the C4 modeling (you can use whiteboard, paper or any C4 modeling tool).

C4 modeling would give you the abstract design or system architecture graph, you don't have to do all 4 levels diagram, just do the one that makes sense to you.

If you are doing something simple, you might not need to do any of that but as soon as you are building something of scale with optimal security not doing system designing is like building a skyscraper with no design, well, you know what might go wrong :)

I used to work with a lot of ex-Pivotal folks. Their workflow was something like:

1. Talk through the requirements and write down some key bullets

2. Whiteboard some design options. Make sure you have at least 2 interesting ones.

3. Prototype them out. Decide on your favorite.

4. Delete the prototype.

5. Start writing tests now that you know where you want to be, and TDD from there.

The key point is that TDD has to start from a relatively defined sense of what you want. If you don’t have that it’s totally fine explore. Just don’t ship that code.

What is wrong with your first test? Find some way to persist the result if you like it and make a test that checks that you can still get that result. Now you can sanity check by running the test to make sure it didn't break. That is one benefit of having the test, did something new break something old. Generally that is as far as I take TDD. While writing new routines, it is nice to have a test calling those routines rather than maintain whole executables because many IDEs will run those tests and you can lay down breakpoints to figure out why the routine is bombing or giving an incorrect result. When/if something breaks and you find yourself debugging back into any particular nitty areas, that might indicate a good place to think about putting a test so you don't have to exert yourself getting to that breakpoint again, next time you'd run the test and get a pass fail. I wouldn't worry at all about the purity of it all, think about how having a test can save you time in the debugging process thats it.

I get what you mean with the tip of the iceberg analogy. The problem is that you can't write any insightful tests before you have written the structure of the application and you can write the application before you have written the tests cause of TDD.

So what I do is that I first hack out the application before I write any tests. E.g if I code a matrix multiplication routine I'm not going to waste time writing unit tests for it. Cause perhaps a few hours later I might realize that I don't need matrix multiplication after all. An added benefit is that I don't have to constantly "context switch" between test writing mode and implementation writing mode.

I write tests when I have something interesting to test. So if the application unexpectedly crashes when I run it I write a test that triggers the crash. Then I fix the bug which causes the test to pass. I find these types of tests to be much more valuable than the "cookie-cutter" unit tests TDD expects you to write.

I've found that strict write-the-tests-first TDD doesn't work for me for most problems. The exception is things like "parse specific components out of this string" - really straight-forward this input should result in this output code, where sometimes I'll write the tests first.

What DOES work for me is having a rule that any implementation should be accompanied by tests that prove the implementation works, bundled in the same commit.

I've been doing this habitually for a few years now and it has had a dramatic positive impact on my productivity.

I wrote a bit about how I do this here: https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests...

If you take TDD literally you’ll end up writing the inner domain logic layers first purely because they’re easier to test using objects and APIs. For greenfield projects that’s a risk because you don’t know how that fits with the user/usage side of things.

I’ve found it safer to start with (manual) integration tests where I verify intended behavior from outside in. That avoids building inner layers which turn out to be slightly but annoyingly off compared to what is desired.

Unfortunately, as soon as you jump off the pure OO bits and domain logic, the TDD-ness of your end-to-end program could be all over the place. You’re entirely at the mercy of available tools and your proficiency with them, and the time you can invest in learning those tools.

When I was newer I tried it as a hard rule and it sucked. So much wasted time and silly tests that didn’t test anything interesting.

Now I have a sixth sense for what I should write tests for and when I should write them before vs after the implementation.

Rules are misleading and teach the wrong thing. Instead, here’s one loose question I ask myself: “does the interface inform the implementation or the other way around?”

When the interface is super important and known, I can write a test first. When the implementation will reveal what the interface’s shape might be like, the test might come after.

I don’t know if this is pure or orthodox but: Start with one or two high-level interfaces that should meet the functional requirements.

It defines the information you need to send in and/or receive.

Non functional. Then capture some basic behavior of one of the methods/properties in the tests, then implement the solution. Then start capturing edge-cases the same way.

If you’re asking yourself “how” or “should” I test this, then you’re doing great.

Apply SOLID principles to separate meaningful concerns into separate classes; and iterate there.

Take my viewpoint with a large grain of salt.

TDD is hard when you don't know what your goal is. But it's easy to "just start executing code". I leverage it to get into an exploratory mode in order to think through my designs.

How I TDD 1. create a file with a test. 2. Get it running continuously (dotnet watch test or mocha --watch) 3. Write the code in the same file (just to start the design) as the test so I don't have to bounce around 4. Iterate at step 3 for a while to explore the core domain/model/data (in your example, a Transaction has an amount, a payee, and time for instance), just to reach "escape velocity" 5. Ruthlessly push out dependencies that cross boundaries (file system, network, etc), protecting the core 6. Practice Red-Green-Refactor technique (write a failing/not compiling test, make it pass, refactor) 7. Red-Green phase, I just make things work 8. Refactor phase, I design, reflect on object/function communication patterns

Designing architecture is not easy. And keeping it simple, even less. So I've spent time learning about design patterns, boundaries, and communication patterns and implications to improve my design skills.

I think you are doing it correctly, you just need to break that iceberg into smaller chunks. You’re getting caught up in the ideological nonsense that surrounds TDD and Scrum. “It can categorize a single transaction correctly” should be translated to a problem statement, “I want to be able to categorize a transaction.” You must then brainstorm feasible solutions for this or different aspects of the problem, develop the spec for implementing that, in the form of tests, “able to do this, able to do that,” and build the solution. Then afterwards you can test it against some examples and ask yourself, does this solve the problem? How to improve? Come up with a change or completely different solution, rinse, repeat. I would say a general statement like categorize a transaction is difficult because you don’t have any specific examples. Come up with a list of a dozen transactions that need categorizing and your TDD experience will become much less frustrating.

Don’t. Address the problem correctly. Write good integration tests. Spot the code with corner cases. unit tests these. avoid too much mocks. Docker is your friend

There is a lot of really good advices here about when to do TDD and how to do it, so i would only reiterate earlier comments.

My personal experience is that i start TDD once my project is "mature" enough that the next upgrade/addition can be put at two different place easily enough and i don't know where is the best place to do it. Basically, on my latest project, i asked myself something like: "is this part of this class method, or should the main loop take care of it?". That's where i started completing tests to 90% coverage (main loop + pagination mocks were skipped, ideally i could've reach 95% but laziness caught up) and started TDD. In the end, i still made the wrong choice and added my change to the class method, only to change it back two weeks ago (or roughly six month later). Here, TDD didn't help me to make the best software architecture choice, but having to write test first for this part of code made me decouple it from the method i originally put it in, and allowed me to made a architecture change in less than an hour.

I still recommend to stick with outside-in TDD. It will help you discover interfaces and not over-engineer them. However, don't get too hung up on fulfilling that outermost feedback loop and test immediately. It's OK to spend a lot of time on one of the inner components and commit that and its tests before moving on to the next piece you need to get the outermost test to pass. In fact, it's a great way to make iterative, committable progress towards the larger goal. I know it can feel weird to not get back to the outermost test for a long while, but that just means that your user intercase is simple, but a lot of work is behind the scenes. Nothing wrong with that.

Another piece of general advise on TDD: Adjust what tests and how many you write based on what your language already gives you. E.g. when I write Ruby or JS, I'll write a ton of unit tests for almost Avery conceivable scenario. When writing Rust or Swift I write mostly unit tests for interesting business logs, but not much else.

You don't.

If it doesn't fit your mental model of programming, just don't use it.

> I know I need to, at least, read a CSV, parse a CSV row into some data structure and then categorise it. So I TDDed my way through to complete these “sub problems”. However, when I was about to apply the categorisation rules I realised that the data structure I created didn’t help, was just bad and didn’t work at all

TDD is kind of a forcing function for modular design (whether OOP or functional programming).

In your scenario, you'd want:

1) a module to read the CSV -> test that it can load a file correctly given a path string, correctly handles invalid paths (format, file does not exist, etc)

2) a module to parse a row -> given a string, test that it can parse a 0 row file, a 1 row file, a 2 row file, and correctly handles an unparseable file (e.g. binary file); test that it correctly handles incorrectly formatted rows

3) a module to categorize -> given a set of rows, test that it can categorize a sample set correctly with expected output

If you try to make 2 depend on 1 or 3 depend on 2 and 1, it's no longer a unit test and becomes more of an integration test.

It is indeed a forcing function for modular design but what you get with that if you aren’t careful is heavily over engineered code.

Take that first module, the CSV loader… if you build it in a truly modular way, it can be hard to stop with the features that just support your end user application.

> it can be hard to stop with the features that just support your end user application.

I mean, it's not hard, you just choose to stop.

All it really takes is an awareness that you're programming to solve a specific limited defined task, not programming just for fun.

Of course, if you're doing it for fun, then fine, don't stop.

But nothing about stopping is hard. It's just a choice you have.

This is how I started going about managing complex design processes, BEFORE agile and whatever buzzwords they use to describe the same problem discovery, identification, positioning ad solution proposals. You go as deep as you need to THINK through the details involving all people who know the details. When you do it right all of the correct people fully inform the process and any TDD that is derived from it, all while keeping everyone in consensus about the goal and the decisions being made to get there. But it can all be worked on paper first if you don't cut corners and have the right people involved (like GUI designers and not marketing or the client should be designing interfaces).

http://www.dubberly.com/articles/managing-complex-design-pro...

For me, I usually have an idea for an overall simple design or approach that will likely work. If I'm unsure, I might post an issue that describes the theory of operation for feedback or iteration. TDD is the rest of what follows.

Now what does work is if you can decompose the whole idea into parts, each of which is written using TDD. You might still want to outline some key interfaces so you get the shapes of things to fit. With practice, all this extra stuff often happens in your head as you're doing TDD and you make adjustments along the way.

Also, the way you described making bottom-up things that didn't quite work, is a valid and possibly effective way of doing TDD. If you're able to determine that a lower implementation is unsuitable it can be rewritten and if the tests are not too brittle will give you confidence to do the rewrite safely and faster.

> Before that one, I tried the opposite: a more “bottom up” approach. I know I need to, at least, read a CSV, parse a CSV row into some data structure and then categorise it. So I TDDed my way through to complete these “sub problems”. However, when I was about to apply the categorisation rules I realised that the data structure I created didn’t help, was just bad and didn’t work at all :).

I think what would have worked for you is to write a test for the categorization, but feed it pre-parsed data structures in the tests. If the data structure is wrong, you will find out now. After that works, you have decided on a correct data structure, and you now can write code to read and parse CSV into those same data structures.

It’s all a matter of practice. If you do things in the incorrect order, these things can happen.

First you need to be experienced enough to know what not to do. Like that you shouldn't write mocks, shouldn't test private functions, shouldn't use a ton of global or static vars etc

Then you should find some sort of coverage tool so you can know if you covered most lines or not. I tend to ignore error handling lines but if its easy to get into that case I'll writ a test for it. Then I'll write test that take <1ms to execute. If something needs a network I'll try to separate the logic so it can work on a file OR I'll it connect to something on my local machine.

Generally I try to get 95%+ coverage (including error cases). I have many test and the entire suite run in <100ms. Maybe my project is easier to do TDD on but that's the way I do it

You are allowed to "spike" out a messy implementation before your actual implementation to see what the actual shape of the code will look like. From there you re-implement it using proper TDD.

Don't test private functions. Only test the public interface.

Use dependency injection everywhere.

I think part of it, by reading your question, is that:

a) TDD primarily refers to unit tests (it does not address other types of testing that you need to take into account when designing a program)

b) you have the wrong idea of what constitutes a 'unit' test versus other kinds of testing. Specifically, as per M. Feathers, anything that touches the filesystem, e.g. to read a file or database (as you do here), is not a unit test.

I would highly recommend Chapter 2 of M. Feathers' "Working effectively with Legacy Code" on this topic.

Read about AMDD. The idea is: you not only write tests to design and iterate on that, but also sketch a model of your system and iterate that. Depending on the size you may want to sketch subsystems, etc.

Now the only problem I have with TDD is time. It takes a lot of time to do it right and usually I don't have that time (in my organisation). So maybe you can tdd some parts of your application, maybe the most critical parts. Even for personal projects you may not have the time: demotivation usually kicks in for personal projects. TDD defers results and the positive feedback needed to keep going. At least in my experience

I do a lot of rest api's, and generally, I just manage to test endpoints and data/json structures adhere to design specs or DTO's or typescript types, etc. I'm usually dealing w/ laravel but there's packages to translate backend DTOs to typescript types, for frontend devs to mess w/.

As long as all the restful resources work properly and get an expected outcome, there's a good chance everything else should work fine too.

yes if you have tables -> DTOs -> REST services without much logic the need for TDD beyond endpoints (I don't think it can be called TDD at that point) is limited. If you have services with lots of logic though... it could be useful to test that logic, if it's critical to business. Or if you deal with a framework used by different projects with different requirements, and/or if your software relies a lot on external configuration. Then features can (and will) clash so having some tests that detects those at different layers (unit, service, e2e, etc.) is good. Why TDD? Because it builds a system designed to be covered by tests. It's way harder to introduce tests later in critical parts of the system. This is textbook and in my experience also true. Again, different needs make TDD and automated tests more or less useful.

Thanks. I found an article about it and it looks like a interesting read.

Don’t have a solution to learning best practices other than watching good practitioners at work.

However, a big inflection point for me was getting hold of a good/capable IDE (intellij). It removes a lot of the drudgery.

You write test with yet-to-be defined functions. IDE let’s you run the single test (amongst many) with one shortcut. Fails ofc!. Then you use a shortcut to let IDE implement the function (takes you where it needs to be implemented, easy with python modules eg. YMMV). Rinse and repeat.

I use TDD only when fixing reported bugs. First I write the test for the expected result, see that it fails, fix the code and check that the new test passes.

TDD as it’s supposed to be done according to evangelists doesn’t work. I have the same problem as you every time I try TDD for implementation, and the answer I get is always that I’m doing it wrong somehow. I have better results without TDD in these cases though.

In short: You don't. At least, not in quite the way that the orthodox expansion of TDD implies.

I've found that TDD with a reading of "Testability-Driven Design" was useful for driving adoption of certain architectural patterns that made my software more easily testable. That, in turn, informed how I authored my APIs. And so on.

Most software development these days consists of applying glue code to myriad of frameworks and libraries as opposed to writing algorithmic code. This kind of work does not lend itself well to TDD.

The opposite is probably true. Because you're gluing pieces together, the natural injection point for mocks is precisely where you would apply the glue.

A practical example is that my team is building a product that has a workflow which scrapes websites and then converts the output to internal models (before more processing occurs). The way we make this TDD is by simply loading static "fixtures" during testing instead of actually reaching out to the website to grab the HTML.

You might make the case "what if the HTML DOM changes at the origin"?

That's not the point of the unit test; the unit test is only testing "given this structure, this is the output I would extract from the HTML". If the HTML structure at the source changes, that's an integration or E2E test.

> This kind of work does not lend itself well to TDD.

I couldn't disagree more. Indeed, it's precisely because framework and library black box behavior and upgrades tend to introduce random unforeseeable bugs that TDD is essential.

It's super important to take some known inputs, and after a series of framework/library operations, to check that the outputs are what they're supposed to be, for normal operation and for all the wacko edge cases (zero-length strings, etc.).

I find it hard to implement at the beginning of a project but I find it very easy to implement once I find a bug. Fixing bugs by starting with a failed test make it easy imo.

TDD doesn’t work for me. However having a large number of system/use case tests works wonders. I haven’t had a bug in production for 5+ years.

For me, TDD has been much more about making sure the code works the first time and to the spec and much less about the structure of it.

Tdd only works when you have extremely clear biz requirements like in many big companies.

There are tons of books about the subject. Can tell us which you already read?

Only Kent Beck’s one, but honestly I might need to revisit it (I only vaguely remember how he approached the two examples on the book). If you have other suggestions of books you found insightful please let me know. Thanks!

You can also check out Growing OO Software Guided by Tests.

My guess is that TDD probably works well enough for external consultants or outsourcers, where you have to come up with really good requirements in advance. That said, if you have really good requirements, any methodology will work.

A funny pattern I noticed is most TDD debates is that whoever challenges TDD always comes up with concrete examples, and TDD advocates always stay incredibly vague.

There's a good discussion between DHH and Martin Fowler/Kent Beck on TDD: https://martinfowler.com/articles/is-tdd-dead/

You can also look up Ron Jeffries and Peter Norvig attempts at sudoku solvers. Guess who actually solved the problem, and who wrote 5 blogs posts without getting anywhere?

TDD is snake oil.

Quite frankly very few people can do TDD well. Hence why most developers hate it. They've never been taught by a master.

The majority of developers have less than 5 years experience, and they're teaching new devs. So the skills that we're developed by people who embraced TDD in the early 2000s have been swamped out by new people.

I think the best way learn if you don't have access to a master is probably videos. See if can find kent beck doing some examples. There's subtle knowledge & flows that hard to teach through just text alone.

> The problem with this is that this is testing the “tip of the iceberg” that will be almost the whole, final app.

This is also what I've run into in practice, across numerous systems.

Actually the only cases where I've been able to do TDD effectively (and get >95% test coverage) was when I was in control over the entire codebase and its design from day one, thus being able to enforce whatever modularization was necessary by the tests, otherwise the systems quickly tended to get untestable (at least in regards to being able to start with a test). In addition, I was generally more successful in cases where I was dealing with writing libraries (that are called by other code), as opposed to web applications which need to deal with shuffling around bunches of data.

Let me give you a few examples of what didn't work. Suppose that I'm asked to help out with this pre-existing codebase that's been around for 5 years or so. I look how some request to purchase a product is processed (just a made up example, though along the lines of what I've seen): first you have some server side rendering with the full complexity of PrimeFaces and around 10-20 variables that affect whether certain buttons are visible, possibly there being a modal dialog or redirection in the middle of the process.

Then, you have the view code, which once again contains different methods that interact with services, passing data around and choosing whether to display additional messages or something else. Then, in the service layer, you might have 5-10 services that are involved in retrieving the data necessary for everything, from auditing and e-mail notifications, getting various pieces of information about users/billing/discounts and then checking all of that against what the user actually wants to purchase, perhaps some validation code thrown in along the way.

Validation code which also might need to interact with the database due to incrementally grown business requirements over the years. And from there on out, additional code for returning early in the case of any errors being present, which might need to be further processed for displaying them to the user. Oh, and if the process is successful without any errors, then you might also have some scheduled processes that would be created and would need to be handled. Almost any of these steps could have transaction rollbacks along the way and there might also be a large amount of PrimeFaces/Spring related code and workarounds in there.

And I've seen similar patterns across many projects, almost regardless of technologies used (Java is just a good example): if you give developers the chance to create tightly coupled code, they will jump at it horrifyingly often, especially if they're under time pressure or have a "good enough" mentality. Over time, any sorts of boundaries will dissolve, unless you go for a microservices based approach, but even then you'll still be dealing with "magic" code which will be hard to test (especially if you need some kind of application context to be running during the tests), be it IoC/DI, annotations/decorators, proxies for your code, or even something like ORM interactions.

Speaking of databases, good luck if your code has like 10-20 database calls along the hot path, which you'll need to mock or actually let it hit some database instance, though hopefully one with state that's either reproducible or can be rolled back (both of which will make your CI slow and more error prone). You might go for an in-memory database, but good luck trying to transpose or make Oracle-specific complex queries portable to something like that, given that for many learning JPQL is too much.

In short: there are situations where TDD is simply unsuitable because of how certain systems/domains are architected (I'm yet to see the PrimeFaces layer as something easily testable) and you're better off simply using unit tests where you can, as well as a healthy helping of integration tests (which you may or may not be able to do, depending on how your environments are laid out).

Of course, things might not look so dire for projects that are a bit newer and where you can still alter their structure. For library code, however, personally I'd recommend TDD, especially if you're ever in the need of doing anything security adjacent (e.g. wrapper code around proven security libraries). And for basically all of the cases where you can write mostly functional code with little to no side effects.

Also, here's a not so fun memory: trying to write tests that dealt with loading data from the file system. In Java, you can have a common piece of code for doing that (e.g. Paths.get), though what paths are valid will depend on the actual system and file system that JVM will be running on. In other words, when locally you have Windows and the CI server has a Linux distro or something, your tests will not run consistently. But you don't want to test that bit of code? Better be prepared to add it as an exception to any code coverage quality gate that you've set up. So annoying.

I'm not quite getting your categorization of the first approach as top-down and second as bottom-up, they are the opposite to me.

Your program is going to have a flow like this:

  $ rcd2-awesome-finance-app september2022.csv
  Category    Amount
  Food        $100.00
  Clothes     $200.00
  ...

Internally it'll be like this:

  main(string filename)
    csv = open(filename)
    cat->$ = new dict()
    foreach line in csv
      transaction = parse(line)
      cat = categorize(transaction)
      amount = amount(transaction) // may just be t[2] not a function
      cat->$[cat] = cat->$[cat].or_default(0) + amount // or other logic for first time seeing it
    foreach (cat, amount) in cat->$
      print("{cat}\t{amount}\n")

So your second example is the top-down one, it requires all this logic to exist. That makes TDD harder because going from a one-line CSV file and hardcoding the amount and category (first code is dumb code), to a second one-line CSV and actually categorizing is a massive jump.

Categorizing is the core, so categorizing is the thing that really needs testing. The rest doesn't even need (substantial) testing anyways, it's a bog standard CSV-data driven tool. Which leads to a point I want to make:

You don't need to do TDD on everything to do TDD. TDD is the "show your steps" of programming (analogous to the high school algebra requirement that you show and name each step in a solution). Many people despise that in their math homework, but it's important for the teacher for the same reason TDD can be important to you, the programmer.

Math teachers (ok, some may, they're poor teachers) don't ask students to show their steps just for grins. They ask it so they can observe the thought process of the student. A student tasked with solving for x with `2x = 4` may come up with `x=2`, but how? Did they divide by 2 or subtract 2 (I've done my share of math tutoring, I have seen students subtract what should've been divided). They got the answer by coincidence. So when they are given another problem where that coincidence doesn't work, they get the wrong answer and in more complex problems how they went wrong is non-obvious ( in 3x=9 => x = 6 it would be glaringly obvious, but with more terms and unobserved steps it's not; it's early the coffee hasn't kicked in yet so I can't think of good example problems to illustrate this).

TDD does the same thing, but it's there for you, not an instructor, to observe what you are thinking along the way. Each test is supposed to be small so that you can easily see that the logic is correct. However, if you can see a larger step to take (like that structure I threw up at the top or parts of it) then go for it. I've written I don't know how many CLI apps and I would never use TDD for anything but that inner categorization logic (or its equivalent for them). It would be silly for me to write a test to see that I was reading the file correctly. I can think of only one reason to do it: You don't know the language and API yet so you want to test your own knowledge, more than the program itself, to ensure you're calling it correctly. Even then, unless it has a weird file reading API I wouldn't bother, 99% of them will be the same or similar enough.

Interesting point of view, thanks.